AI Coding Has an Understanding Problem

The problem of pushing code without understanding it is not new.

Even ten years ago, people would copy a snippet from Stack Overflow, push it, and move on without understanding what template specialization actually is.

But today, the problem is happening at a completely different scale. And everyone seems to be fine with it.

A few weeks ago, Elon Musk quote-tweeted a post that equated “a compiler converting source code to assembly” with an “LLM converting natural language into code”.

That tweet prompted me to write the post that follows.

Why the Compiler Analogy Is Absurd

A compiler is a deterministic translator. If you give it the same source code twice, you get the same binary twice, every time, forever.

It also works against a formal specification. If you write something the language doesn’t allow, it doesn’t guess what you meant. It stops and tells you exactly which line is wrong.

None of that is true of a language model.

An LLM’s output is probabilistic. Even if you give it a detailed specification, there is no guarantee that the code it produces actually follows it. It can misunderstand a requirement, make an incorrect assumption, or produce code that looks perfectly reasonable while being subtly wrong.

That is why trusting a compiler and trusting an LLM are very different things.

The Rise of Understanding Debt

Call it understanding debt: the gap between how much software you own and how much of it you actually understand.

This kind of debt has always existed in some form, but it’s bigger today than ever before, for one simple reason: generating code has never been this easy.

But you may ask, why does it matter? If the code works, why should it be a problem that a developer pushed code in a language they’ve never actually studied? Here’s why.

You Won’t Recognize Code That’s Valid but Wrong-Feeling

Code can compile, pass its tests, and still be deeply unidiomatic in ways you wouldn’t notice unless you knew the language well.

For example, say you are building a C++ application where performance matters, so you tell the model to keep overhead to a minimum.

It may take that instruction too literally and use raw pointers and manual memory management across the codebase instead of safer abstractions like std::unique_ptr.

// "Keep overhead minimal" taken literally
class Connection {
  Socket* socket;   // raw pointer
  Buffer* buffer;   
  ~Connection() { delete socket; delete buffer; }
};

The code may compile and run fine, but an experienced C++ developer would immediately question whether that trade-off is actually necessary for the use case.

Code Review Can’t Scale at the Same Speed

You can now ask a tool to rewrite a 20,000-line TypeScript server in Rust in an afternoon.

But you cannot review 20,000 lines of unfamiliar Rust in an afternoon. Not at all.

Code generation scaled by several orders of magnitude the moment these tools got good.

Human review didn’t scale at all, because it’s still one person, reading one screen, at roughly the same speed people have always read code.

Small Misunderstandings Compound

The problem gets worse when you start building new AI-generated code on top of code you never fully understood in the first place.

Each new layer inherits the assumptions, shortcuts, and design decisions of the one below it.

Once that’s done often enough, you end up with a system where nobody really knows why certain decisions were made, or what might break when you change something.

You May Need AI to Explain Your Own Code to You Mid-Incident

If you never understood the code in the first place, the moment it breaks in production, you have to explain your own codebase and the symptoms to a model.

From scratch. And then wait for it to build up enough context to be useful.

That takes time you don’t have during an outage.

And what if the model is unusually slow that day, or just outright down? There is no AI sitting there with a persistent mental model of your codebase.

Every time you come back, you have to give it the context again and hope it reconstructs the system correctly.

More Comments Don’t Mean More Understanding

One of the worst things about AI-generated code is the multi-line, situation-specific comments that seem to come with virtually every new change.

Sometimes there are so many comments that there is almost more commentary than actual code.

// the date field can have arbitrary data at times.
// this was observed, not predicted, during a load test run on a dev laptop,
// so this fallback belongs here. DON'T remove it

The problem is not that comments are bad. Good comments can explain why something is done a certain way when the reason isn’t obvious from the code itself.

The problem is when an AI-generated comment tries to explain every detail, especially when it is based on assumptions that may not be correct.

You end up with code that has plenty of explanation around it, but you still don’t know whether the underlying decision was the right one.

Wrapping Up

Andrej Karpathy said it better than I can::

"You can outsource your thinking, but you can't outsource your understanding."

That line sums up basically everything above.

To be clear, none of this is an argument to stop using LLMs for software development. They’re the single best tool we have for writing code right now, and every developer should be using them.

But there’s a real difference between letting an LLM write the code and letting it do all of the understanding on your behalf.

Let it write the boilerplate. Let it rewrite the TypeScript server in Rust. Let it explain an unfamiliar library, find the bug, write the tests, and suggest the architecture.

Just don’t merge anything until you have fully understood it.