The Idea File: How Karpathy's LLM Wiki Models a Better Way to Think in Public

Andrej Karpathy recently published a gist called the LLM Wiki, framing it explicitly as an example of an “idea file.” The framing is what makes it interesting. This wasn’t a polished tutorial or a finished explainer. It was a living document — a place where Karpathy accumulates observations, tentative conclusions, open questions, and rules of thumb about large language models as his understanding evolves.

The idea file as a concept is not new. Writers, engineers, and scientists have kept running notebooks for centuries. What Karpathy’s example illustrates is how that practice adapts to a domain that is moving fast enough that your mental model is wrong by default, and staying useful requires constant revision.

What an Idea File Actually Is

An idea file is not a reference document. It is not documentation, not a blog post, not a polished knowledge base. The key property is that it captures your current understanding, including the parts you’re not confident about. Entries in an idea file can be contradictory over time. Old observations get struck through rather than deleted. Questions stay open until the evidence settles.

This is different from the more structured personal knowledge management (PKM) approaches that have become popular in the last few years. Systems like Zettelkasten emphasize atomic notes with explicit links between them. Tools like Obsidian encourage you to build a graph of connected concepts. The second-brain methodology, popularized by Tiago Forte, focuses on capturing and organizing material for eventual retrieval and reuse.

All of those are legitimate practices. But they share an implicit assumption: that knowledge is something you accumulate and retrieve. The idea file assumes something slightly different, which is that your understanding is a process, not a state. The file is less a library and more a scratchpad that happens to persist.

For a domain like LLMs, that distinction matters enormously.

Why LLMs Break Standard Knowledge Management

Most knowledge management systems work well when the underlying domain is stable. If you’re building a reference for SQL query optimization or TCP/IP networking, the facts don’t change much. You write them down once, maybe update them when a new database version ships, and the reference stays useful.

LLMs don’t behave this way. Model behavior changes between versions. Context window sizes, instruction-following fidelity, reasoning depth, tool use reliability — all of these shift with each new release from each provider. Prompt techniques that worked reliably six months ago may behave differently on current models. Observations about GPT-4’s behavior may not transfer cleanly to Claude, or Gemini, or Llama.

More fundamentally, the field’s collective understanding is still being built. Questions that seemed settled, like whether chain-of-thought prompting reliably improves reasoning on arbitrary tasks, have been revisited repeatedly as researchers probe edge cases and distribution shifts. What practitioners know about how to build reliable systems with these models is a moving target.

A Zettelkasten where you captured “LLMs cannot reliably count tokens” in 2023 is now partially obsolete. A static reference doc that says “use temperature 0 for factual tasks” papers over a lot of nuance that depends on the specific model, task, and failure mode you care about. The idea file handles this better because the format doesn’t encourage false confidence. Notes can be dated. Observations can be provisional. The document can contain “I think X, but I haven’t tested this on recent models.”

What the LLM Wiki Actually Contains

Karpathy’s gist covers practical LLM knowledge that is the kind of thing that accumulates through sustained use rather than reading papers. Notes on tokenization behavior, observations about how models handle long context, rules of thumb for prompting, patterns in failure modes, and half-formed hypotheses about why certain approaches work.

The content is organized loosely, more like a FAQ built for one person than a structured curriculum. Some entries are crisp and confident. Others are explicitly speculative. There are no citations and no formal claims, which is appropriate for the format. This is what someone who has spent years working with these systems has noticed, written down in a way that is useful to their own future self.

The meta-point Karpathy was making by sharing it, and the reason the HN discussion lit up, is that the format itself is worth imitating. Not the specific content, but the practice of keeping a document like this at all.

The Compounding Advantage of Writing Things Down

There is a practical reason to maintain an idea file beyond just personal reference. Writing forces a kind of compression that passive reading or watching does not. When you observe something about how an LLM behaves and you write it down in your own words, you’re forced to decide what the actual claim is. Vague intuitions become falsifiable assertions. “This model seems worse at reasoning” becomes “in my testing, when the task requires more than three inferential steps, Claude 3 Haiku reliably fails on the fourth step.”

That compression does two things. It makes the observation more useful to your future self, and it makes the gaps in your understanding visible. When you try to write down why you think something happens, you discover whether you actually have a causal model or just a correlation you’ve observed.

This is the mechanism behind why writing is considered a thinking tool rather than just a communication tool. Richard Feynman’s notebooks, Darwin’s journals, the engineering notebooks that Bell Labs required — these were not record-keeping systems. They were thinking systems that happened to produce records.

For LLM practitioners specifically, this matters because the domain rewards systematic experimentation and careful observation. People who build intuition about model behavior through deliberate practice and note-taking end up with mental models that are more accurate and more transferable than people who rely on general impressions from use.

Public Idea Files as a Knowledge Transfer Mechanism

The unusual choice Karpathy made was to publish this. Most idea files stay private because they’re half-baked by design. Sharing a document full of tentative observations and open questions feels like exposing incomplete work.

But there’s something useful about expert idea files specifically. When Karpathy shares his notes on LLM behavior, the observations carry different weight than if they appeared in a random blog post, not because he’s always right, but because the format tells you what kind of claim each item is. A polished blog post often hides whether the author is confident or guessing. An idea file, by its nature, signals uncertainty where uncertainty exists.

This is similar in spirit to technical memos that circulated inside Bell Labs or Xerox PARC, or the research notes that Leslie Lamport has published alongside his formal work. These are not finished products. They are thinking in progress, shared because the process of thinking is itself instructive.

Publishing an idea file also creates a useful kind of accountability. If you’ve written down “I believe X” and X turns out to be wrong, the document is correctable. Mistakes in public idea files tend to get caught and corrected. Mistakes in private notes compound silently.

Starting Your Own

The practical implication is straightforward. If you’re working seriously with LLMs, or any fast-moving technical area, keeping an idea file is more valuable than keeping a polished set of notes. The mechanics are simple:

• One file per domain, not one file per topic
• Date entries when precision about timing matters
• Distinguish between observed facts, generalizations, and hypotheses explicitly
• Revisit and annotate old entries rather than deleting them
• Write for your future self, not for a hypothetical reader

The tool doesn’t matter much. A Markdown file in a Git repo, a Notion page, a text file on your desktop — the format is less important than the habit. Karpathy chose a GitHub gist, which has the useful property of version history built in.

What makes the LLM Wiki valuable is not the specific observations in it, which will age and require updating. It’s the demonstration that even someone with deep expertise in the area keeps a running document to track their evolving understanding. The alternative is relying on memory, which is lossy, non-persistent, and invisible to anyone else who might benefit from what you’ve figured out.

The idea file is a low-ceremony practice with compounding returns. The right time to start one is whenever you notice yourself forming an opinion about how something works and then not writing it down.