A story making the rounds in TLDR DevOps this week cited a study of 6,080 AI-generated patches written for six recent, real vulnerabilities. Only 26% fully fixed the issue without materially changing how the application behaved. 53.9% either failed to fix the vulnerability, introduced a new one, or both.
That is not a story about a bad model. It is a story about what happens when "the code runs" quietly replaces "the code is right" as the bar for done.
The tension
Every AI product demo makes the same promise: the gap between having an idea and shipping it is closing to zero. Type a sentence, get a patch. Describe a feature, get a PR. For the first time, execution is not the bottleneck.
But a coin flip with a slightly better-than-even chance is what execution being "not the bottleneck" actually looks like in practice, per that same study. Speed without judgment does not remove risk. It just moves the risk downstream, to whoever approves the change without checking it closely enough.
The insight: building got cheap, discernment did not
A separate post in this week's TLDR Dev put a name on the skill that is left over once generation is nearly free: taste. Not aesthetic taste — the ability to tell, quickly and reliably, whether a plausible-looking output is actually the right one for this system, under these constraints, given what could go wrong.
That is a different skill than writing correct code, and it always has been. It just used to be optional, because a human author's first draft came with a baseline of intent and context a reviewer could partially trust. An AI-generated patch comes with no such assumption. It can be syntactically clean, pass the obvious test, and still be the wrong fix — which is exactly what happened in more than half of the patches in that study.
The failure was not "the code did not compile." These patches ran. Many looked like reasonable fixes. The failure was that they did not solve the actual security problem, or quietly broke something else while appearing to succeed — the exact failure mode that only shows up under close, skeptical review, not a quick glance.
That is precisely the muscle a technical interview can test in twenty minutes: hand a candidate a plausible-looking solution and see if they can tell it apart from a correct one.
Evidence: taste is a checklist, not a vibe
The engineering leaders who are burning out right now — a third story in this week's TLDR Product described a wave of them stepping back into fractional work or career breaks — are disproportionately the ones absorbing this gap personally. When a team lacks a shared, repeatable way to evaluate AI output, the burden of catching what is wrong lands on whoever cares enough to keep checking. That does not scale, and it is not sustainable for one person to carry alone.
The candidates who read as senior in an interview are not the ones with the best gut instinct. They are the ones who can turn "does this look right" into a small, repeatable sequence of checks — the same sequence, every time, regardless of how confident the AI-generated output looks.
| Reads as junior | Reads as senior |
|---|---|
| Approves once the code runs and tests pass | Asks what the tests actually cover before trusting them |
| Treats a clean diff as evidence of a clean fix | Checks whether the fix addresses the root cause or just the symptom |
| Scrutiny fades after the first few minutes look fine | Applies the same checklist at minute twenty as at minute two |
What to do about it
Taste is trainable precisely because it is a checklist, not an instinct. Building it before your next interview means practicing the sequence deliberately, out loud, on code you did not write.
Rubduck's AI-assisted interview sessions hand you real code — some of it AI-written, some of it not — and a live voice interviewer that presses on your reasoning the way a real panel would. Building the checklist habit before it matters beats discovering the gap during a live loop. Start your free sessions →