There’s a concept in safety engineering called drift into failure, coined by Sidney Dekker in his work on complex systems. It describes how catastrophic failures rarely arrive suddenly. Instead, organizations slowly normalize small deviations from intended practice until a boundary they didn’t know existed gets crossed. The system looks healthy until it isn’t. Dekker’s framing was about aviation and healthcare, but the pattern generalizes. It applies to individual cognition under automation just as well as it applies to organizational safety culture.
An article circulating this week makes a version of this argument for software development. The machines are fine, it argues. The threat is the comfortable drift toward not understanding what you’re doing. The author is right, and the mechanism deserves unpacking beyond the headline.
What Comfortable Drift Actually Looks Like
The failure mode isn’t obvious when it starts. You use an AI assistant to scaffold a new service. The code works. You ship. Next week you use it to debug an unfamiliar regex. The explanation is correct. You move on. After a month you notice you’re reaching for the tool in situations where, six months ago, you’d have just read the source. You’re faster. Output is up. Nothing is broken.
The problem surfaces later, and not in a clean way. You’re on-call at 2am when a service starts returning unexpected results. The code you’re looking at was partially generated, partially edited, and you haven’t read it end-to-end since it passed review. You understand it at the level you understood it when you accepted the AI’s output, which is approximately the level of “this looks right.” That’s not enough to debug at speed.
This is the shape of comfortable drift. The cost doesn’t appear until a moment when shallow understanding fails, and by then the habit is established.
This Isn’t a New Problem, But the Scale Is Different
Every abstraction layer in computing history has come with a version of this concern. Assembly programmers worried about C programmers not understanding the hardware. C programmers worried about Java developers not understanding memory. The objections were often overwrought, and the productivity gains from higher abstraction were real. Most of the time you don’t need to know what the CPU is doing.
But there’s a meaningful difference between abstracting over a well-specified layer and delegating reasoning to a system that produces plausible-looking output without guarantees. When you use a garbage collector, you’re trading control for safety within a defined contract. When you accept generated code you don’t fully read, you’re delegating the reasoning itself. The abstraction isn’t just below you, it has replaced a step in your thinking process.
Nick Carr made a related argument in The Glass Cage about automation in aviation. Studies of pilots on heavily automated flight decks showed that manual flying skills degraded faster than expected, and that the degradation was invisible during normal operations. The automation handled everything correctly, so there was no feedback signal indicating that the pilot’s independent capability was eroding. The signal only appeared in emergencies, which are rare by design. Software development isn’t as safety-critical, but the feedback loop problem is identical.
The Feedback Signal Is Broken
In most skill domains, doing something badly produces immediate, legible feedback. You write a slow SQL query, the page loads slowly, you profile it. The signal is clear. But when you accept AI-generated code you don’t fully understand, the immediate feedback is often positive. Tests pass. The PR merges. The feature ships. The cost, when it comes, is deferred, indirect, and attributed to other causes.
This is what makes drift comfortable. Comfortable doesn’t mean pleasant, it means low-friction. There’s no friction in the moment that would prompt you to stop and read more carefully. The discomfort arrives much later, in a different context, detached from the original decision.
Researchers studying GPS-dependent navigation have documented spatial memory degradation in people who consistently outsource wayfinding. The effect isn’t dramatic. You don’t forget how to read a map. You just become worse at forming and retaining mental models of unfamiliar spaces, which is the skill that matters when the GPS fails. The same pattern applies to reasoning about code. You don’t lose the ability to read code. You lose the habit of doing it routinely, which atrophies the speed and depth at which you can do it under pressure.
Understanding Is Load-Bearing
The argument against worrying about this usually goes: understanding is overrated, outcomes are what matter, and if the code works, who cares how it was produced. This argument has some validity at the margins. You don’t need to understand every dependency you use. Specialization is efficient.
But understanding has load-bearing functions that only become visible when things go wrong. It enables debugging. It enables safe modification. It enables knowing when a proposed change is wrong. And at a meta level, it enables recognizing when you’ve drifted to a depth where you’re no longer safe to operate independently.
The danger of comfortable drift isn’t just that you don’t understand the code. It’s that you don’t know what you don’t understand. Metacognition about your own competence depends on having engaged with the material enough to map its contours. If you’ve been accepting outputs without reading them, you have no map. You think you understand the system because nothing has failed, not because you’ve verified your understanding.
This is Dekker’s point, applied to individuals. You’ve normalized a deviation from your intended practice so gradually that you no longer recognize it as a deviation.
What the Counter-Habit Looks Like
The corrective isn’t to refuse AI tools. The corrective is to treat generated output the way you’d treat a code review from a junior contributor you’re mentoring: read it, understand it, own it before it merges. That’s not a rejection of the tool, it’s a clarification of your relationship to it. The tool produces candidate code. You produce the final understanding.
More concretely, there are a few behaviors that maintain the feedback signal:
First, maintain a class of problems you work through without assistance. Not as self-punishment, but as calibration. You need a baseline for your own thinking speed and depth so you can notice when it changes.
Second, treat unfamiliar generated code as a reading exercise rather than an acceptance decision. If you can’t explain how it works to someone else, you haven’t understood it yet. Merging it at that point is borrowing against future confusion.
Third, stay inside the debugger. When something breaks, resist the impulse to ask an AI what’s wrong before you’ve traced the problem yourself. The debugging process is precisely where understanding gets built and tested. Outsourcing it consistently means outsourcing the learning.
None of this is about purity or about proving you can code without help. It’s about maintaining the cognitive infrastructure that lets you stay effective when the comfortable path isn’t available. The machines are fine. The question is whether you still are when they’re not enough.