What Andrej
Karpathy Learnt

A short, brutal checklist Karpathy has repeated for years:

• You forgot to .zero_grad() (gradients accumulate across batches).
• You passed logits where the loss expected probabilities, or vice versa.
• You left the network in train() mode at eval time (or eval() mode during training).
• Your learning rate is off by 10× or 100×.
• You shuffled the labels but not the data, silently breaking the correspondence.
• You used a view/reshape that scrambled dimensions without throwing an error.

The theme: the framework will not warn you. Correctness is your job.

Karpathy reframed deep learning as a new kind of software, not a new application of software.

Software 1.0 is explicit instructions written by a human in Python or C++.
Software 2.0 is weights found by optimization against a dataset. The programmer specifies a goal and a search space; gradient descent writes the actual program.

Consequences he draws out:

• The dataset is the source code. Curating data is programming.
• Neural nets are more homogeneous, more portable to silicon, and degrade more gracefully than handwritten code.
• Version control, debugging, and IDEs for 2.0 barely exist yet — the tooling is the opportunity.
• Increasingly, 1.0 code is the thin glue around large 2.0 blocks.

Leading Autopilot taught Karpathy how AI works at production scale, where the model is the easy part.

Most of the work is the data flywheel, not the architecture.
A repeatable loop: ship the model, mine the failures, label the hard cases, retrain, ship again. The competitive advantage is the loop, not any single net.

Chase the long tail deliberately.
Trigger the fleet to send back exactly the rare scenarios you fail on — stop signs on a billboard, a truck carrying traffic cones. The interesting examples are rare by definition, so you must hunt them.

The "operation vacation" goal.
Build the engine so well that improving the product becomes mostly automated data work, freeing engineers from hand-tuning.

Labeling is engineering, not grunt work.
Label quality, label consistency, and the ontology you label against decide your ceiling.

In his "Intro to LLMs" and "LLM OS" talks, Karpathy positions large models as the kernel of an emerging operating system, not a chatbot feature.

The LLM is the CPU; the context window is the RAM.
The model orchestrates tools — a browser, a Python interpreter, other models, a file system — the way an OS orchestrates peripherals. Prompting is the new system call.

Pretraining is lossy compression of the internet.
The base model is a "zip file" of human text, recalled probabilistically. It dreams documents. Alignment (SFT + RLHF) turns the dreamer into an assistant.

Think in two systems.
Current models are strong at fast, intuitive "System 1" recall and weak at deliberate "System 2" reasoning. Much of the frontier is buying the model time to think — scratchpads, chains of thought, tool use, search over reasoning.

Jagged intelligence.
LLM capability is spiky, not uniform. A model can prove a theorem and then fail to tell which of 9.11 and 9.9 is larger, or miscount the letters in a word. Competence in one cell says little about the neighboring cell.

Tokenization is the root of a surprising amount of weirdness.
Spelling errors, arithmetic failures, trouble with whitespace and non-English text, and odd prompt sensitivities often trace back to how text is chopped into tokens. "Much of the suffering is tokenization."

Karpathy coined vibe coding: building software by describing intent in natural language and letting the model write the code, accepting diffs without reading every line.

What he actually advocates underneath the meme:

Keep the human in the loop, and keep the loop tight.
The model proposes; you steer. Short feedback cycles beat long autonomous runs that drift.

Treat the LLM as a fast, fallible junior.
It is eager, knowledgeable, and confidently wrong. Verify outputs; never deploy what you have not checked. Build the verification step into the workflow, not as an afterthought.

Context is the scarce resource.
What you put in the window — and what you leave out — determines the answer. Curate it like RAM.

Autonomy has a slider, not a switch.
Choose how much rope to give the model per task. High autonomy for throwaway scripts; tight control for anything that ships or touches real systems.

Build it from scratch at least once.
micrograd is ~150 lines and contains the entire intellectual core of backprop. nanoGPT is a few hundred lines and is a real GPT. Understanding comes from re-deriving, not re-reading.

"The best way to understand something is to build it."
Karpathy's "Zero to Hero" series is the method made explicit: implement the thing end to end, smallest version first, no hidden libraries.

Write the dumb baseline.
A simple model you fully understand beats a complex one you don't. The baseline tells you whether the problem is even hard.

Read the data, not just the papers.
Repeated everywhere in his work: time spent staring at raw examples is never wasted.

Stupidity is mostly impatience.
Most mistakes come from skipping steps — not overfitting a batch, not visualizing inputs, not checking the loss at init. Slow down and the errors disappear.

After OpenAI and Tesla, Karpathy turned to teaching, founding Eureka Labs.

The bottleneck is great explanation, not content volume.
A genuinely good teacher who can hold a learner's hand through hard material is rare and scalable through AI.

AI as the teaching assistant, the human as the curriculum designer.
The model handles patience, repetition, and per-student adaptation; the expert designs the path.

Smallest working example, then scale.
The same pedagogy as his code: start with the minimal thing that runs and is fully understood, then grow it.

The whole guide, one line per chapter.

1. Training recipe — training fails silently; become one with the data, build an end-to-end skeleton, overfit before you regularize, change one thing at a time.
2. Common mistakes — the framework will not warn you: .zero_grad(), train/eval mode, logits vs probabilities, a learning rate off by 10×. Correctness is your job.
3. Software 2.0 — weights found by optimization are a new kind of code; the dataset is the source, and the tooling is still being invented.
4. Tesla data engine — the flywheel is the product: ship, mine failures, label the long tail, retrain, repeat. The model is the easy part.
5. LLMs as a paradigm — the LLM is the kernel, the context window is RAM; intelligence is jagged, and much of the weirdness is tokenization.
6. Working with LLMs — keep the loop tight, treat the model as a fast fallible junior, curate context, and pick your autonomy per task.
7. Learning and building — build it from scratch once; understanding comes from re-deriving, not re-reading.
8. Education — the bottleneck is great explanation; AI is the teaching assistant, the human designs the path.

For the ten aphorisms, see the 10-Line Distillation above.