← Back

Ilya Sutskever's 30u30 Reading List & 8 In-Depth Interviews

by 秒秒Guo 4月2日 2026

The real breakthrough lies in recognizing the undiscovered, ideal properties of things that have long existed. The Transformer is considered a breakthrough because it was not obvious to most people.

To truly excel at predicting the next token, the model must understand the underlying reality that produces those tokens. This is not simple statistical pattern matching — it's about understanding "how the world created these statistics." The model needs to infer the thoughts, emotions, beliefs, and behaviors behind human actions.

Deep Understanding of Existing Things — this is the prerequisite. You need a very solid understanding of existing tools, algorithms, and theories. Not just knowing what they are, but knowing why they work the way they do, their strengths, weaknesses, and how they behave under different conditions.

Predicting the next token well means that you understand the underlying reality that led to the creation of that token.

— Ilya Sutskever

Over the past three years, I've rewatched Ilya's interviews again and again. To me, he is a monk-like AI scientist and engineer — every word he says strikes at my most fundamental curiosity about LLMs. Here I've compiled 8 in-depth podcasts and public interviews Ilya has participated in over the past decade, along with Ilya's 30u30. I'll be updating this with notes on Ilya and related papers — hope it helps you too.

① Talking Machines — Machine Learning and Magical Thinking
  • Date January 2015
  • Show Talking Machines
  • Hosts Katherine Gorman & Ryan Adams
  • Links Robohub · Spotify

Ilya recounts his journey from mathematics to machine learning. He found that "learning," from a rigorous mathematical standpoint, seems almost impossible — inductive reasoning cannot be formally proven, yet humans do it all the time. The optimization of deep neural networks likewise comes with no theoretical guarantees, yet empirically it works. He also discussed how the scale of weight initialization held back deep network training for years.

Doing good machine learning research requires a kind of "magical thinking" — if you're used to rigorously proving results, then induction seems almost like magic. I was fascinated by learning precisely because I knew humans could do it, yet from a naive mathematical viewpoint, learning seemed impossible.

② Lex Fridman Podcast #94 — Deep Learning

The evolution of deep learning; the eureka moment of 2010–2011 connecting large-scale data with end-to-end training; the birth of AlexNet; similarities and differences between neural networks and the human brain; aligning AGI with human values; ethical considerations around staged model releases (using GPT-2 as an example); the potential of self-play; and a democratic governance model for AGI — "AI as CEO, humans as the board."

When you know nothing, you only notice very coarse, surface-level patterns — there are spaces between characters, sometimes a comma is followed by a capital letter. Then you might notice certain words appear frequently, notice spelling regularities, notice grammar. And once you become very good at all of that, you start to notice semantics, start to notice facts. But to do that, the language model needs to be bigger.

Neural networks have the ability to reason, but if you train them on a task that doesn't require reasoning, they won't reason. Neural networks will solve the problem you put in front of them in the simplest possible way.

③ Clearer Thinking Podcast — What, If Anything, Do AIs Understand?

Can machines truly be intelligent? GPT-3 was trained on a single task — predicting the next word — so why does it seem to understand so many things? What is the essential connection between prediction and comprehension? What key breakthroughs made GPT-3 possible? Can academia remain at the forefront of AI research? How can AI simultaneously memorize training data and generalize? Is there a conceptual ceiling to scaling data and compute indefinitely? The major categories of AI risk.

A neural network is essentially a kind of parallel computer that can program itself.

Suppose you've read a mystery novel and you're on the last page — if a system can truly predict the next word on that page, it must possess genuine understanding.

Our intuitions about intelligence are not exactly perfect. Many tasks are narrower than people realize — a computer doing them doesn't mean it can do everything.

④ Eye on A.I. — The Mind Behind GPT-4

On the eve of GPT-4's release, Ilya laid out his understanding of what large language models truly are. He argued that next-word prediction is not a shallow statistical game, but a path to deep understanding of the world; that pre-trained models learn compressed representations of real-world processes; that RLHF doesn't teach knowledge but teaches behavior; and that multimodality is useful but not essential — pure text alone can learn color relationships. He also traced his original motivation for entering AI — a philosophical unease about consciousness — and the core intuition he formed when he started working with Hinton at age 17: the human brain is just a neural network with slow neurons.

Learning the statistical regularities is a far bigger deal than meets the eye. To predict, you need to understand the underlying process that produced the data.

What large generative models learn from data are compressed representations of the real-world processes that produced that data.

Our pre-trained models already know everything they need to know about the underlying reality. — This means RLHF isn't teaching knowledge, it's teaching behavior: the model already "knows," it just hasn't learned how to act.

I was very disturbed by consciousness. — His original drive into AI was neither commercial nor engineering, but a philosophical unease about the mystery of consciousness.

⑤ NVIDIA GTC Fireside Chat — AI Today and the Vision of the Future, with Jensen Huang
  • Date March 2023 (recorded the day after GPT-4's launch)
  • Event NVIDIA GTC Spring 2023
  • With Jensen Huang (Founder & CEO, NVIDIA)
  • Links NVIDIA official page · NVIDIA blog

The day after GPT-4's release, Jensen Huang and Ilya sat down for an hour-long conversation. They revisited the decade-long journey from AlexNet to GPT-4; discussed the "discontinuity" of the ImageNet breakthrough; the natural fit between GPUs and neural networks; the fundamental difference between ChatGPT and GPT-4 (GPT-4 predicts the next word better); the importance of multimodality (text, images, video) for AI's understanding of the world; and the boundaries of AI reasoning.

When we train a large neural network to accurately predict the next word across a wide variety of text on the internet, what we're actually doing is learning a world model. On the surface, it looks like we're just learning statistical correlations in text. But it turns out that to really learn these statistical correlations well — to really compress them well — the neural network learns some representation of the process that produced the text.

(The solution to unsupervised learning) Intuitively you can see why it should work. If you compress the data well enough, you must extract all the hidden secrets within it. So that's the key.

⑥ Dwarkesh Podcast (First Interview) — Building AGI

AGI timelines; leaks and espionage; what comes after generative models; the post-AGI economic landscape; working with Microsoft and competing with Google (TPU vs GPU); the difficulty of aligning superhuman AI; the inevitability of technological progress; and whether "new ideas are overrated."

Data exists because computers became better and cheaper… once everyone had a personal computer, they wanted to connect to a network, and you got the internet. Once you had the internet, data suddenly appeared in great quantities. — A deep chain of technological inevitability behind the entire AI revolution.

I just don't want to bet against deep learning. I want to make the biggest possible bet on deep learning. I don't know how, but it will figure it out.

⑦ NeurIPS 2024 Talk

Ilya's first major public appearance after leaving OpenAI. He looked back on a decade of deep learning; reaffirmed connectionism as the enduring core idea that has stood the test of time; discussed biological inspiration and its relationship to AI; raised the "Peak Data" problem — data is running out; explored the possibility of models self-correcting reasoning errors; and addressed AI ethics and rights.

Pre-training as we know it will unquestionably end.

Data is the fossil fuel of AI — it was created somehow, and now we're using it up. We have but one internet.

The more it reasons, the more unpredictable it becomes.

Our selves are parts of our own world models.

⑧ Dwarkesh Podcast (Second Interview) — From the Age of Scaling to the Age of Research

SSI (Safe Superintelligence) strategy; the limits of pre-training and data exhaustion; the paradigm shift from the "age of scaling" to the "age of research"; the puzzling disconnect between models acing benchmarks yet lagging in real-world economic impact; the root causes of poor generalization; value functions and the analogy to emotions; continual learning; a 5–20 year timeline for superintelligence; and a candid account of the alignment challenge.

One thing that guides me personally is an aesthetic of how AI should be by thinking about how people are.

There's no room for ugliness. It's just beauty, simplicity, elegance, with correct inspiration from the brain.

The more they are present, the more confident you can be in a top-down belief. The top-down belief is the thing that sustains you when the experiments contradict you.

Because if you just trust the data all the time, sometimes you can be doing a correct thing, but there's a bug. How do you know if you should keep debugging or you conclude it's the wrong direction?

You must say that things have to be this way, therefore we've got to keep going. That's the top-down belief, and it's based on this multifaceted beauty and inspiration by the brain.

Ilya Sutskever's Deep Learning Reading List

A list of 30 key works (rumored to be) recommended by Ilya for systematically learning deep learning. I read the parts I could understand, and personally find the list highly credible.

Full list: Ilya 30u30 Reading List

mmguo (Lingbo Guo) is a Tsinghua University graduate, now building AI products. She writes interaction experiments, meta prompts, and observations on the AI industry at mmguo.dev. Follow her on X @mmguo_lb · Xiaohongshu 秒秒Guo