Twenty Years of Cleaning Bugs
The phone goes at ten past nine on a Sunday. It is always a Sunday, and it is always a founder, and the first sentence is always some variant of the same one: something is wrong and nobody knows why.
I have been taking that call, on and off, for about twenty years. The first one I remember properly was in 2005, two years into OM Conseil. A payroll export had produced numbers nobody could explain and the run was Monday. The one last month was a founder whose team had sent a customer a quotation with a price that did not exist anywhere in the company.
Same call. Same Sunday. Completely different bug.
The craft was never about fixing
People outside the work think debugging is about repair. It is not. Repair is the last ten minutes, and it is usually trivial.
Debugging is about narrowing. You have a space of possible causes that is far too large to inspect, and your entire skill consists of cutting it in half, then in half again, without lying to yourself about which half you just eliminated. Reproduce first. Change one thing. Trust the measurement over the story you have already started telling yourself about what happened.
The hardest part has always been the last one. Everybody investigating a failure carries a favourite hypothesis, and the favourite hypothesis is almost always a place where your own competence would be flattered. In 2005 mine was that the accounting package was wrong. It was not. I had wrongly assumed a date format.
The moth was never the point
On 9 September 1947, a team working on the Harvard Mark II found a moth caught in relay 70, panel F. They taped it into the logbook with the note first actual case of bug being found. The logbook, moth included, sits in the Smithsonian, and the story gets retold as the origin of the word.
It is not. Engineers had been calling faults bugs since Edison’s day, and the joke in that logbook only lands because everyone in the room already used the word. Which is precisely what makes the entry worth keeping: what they found funny was that, for once, the fault was actually a physical creature in the machine.
Every other time, it was them. That is the entire history of the profession in one taped-in insect.
Phase one: the bug leaves the code
Somewhere in the second decade, the bugs I was called about stopped being in code.
They were in a firewall rule that had been correct for four years and became wrong the week a supplier changed their egress IP. In a group policy that applied to everyone except the sixteen people who had been migrated in a batch that failed silently. In an ERP data migration where two systems disagreed, politely and consistently, about what a unit of measure was.
None of that is programming. All of it is debugging. The craft transferred intact, because the craft was never about the language. Reproduce, halve, verify, suspect yourself. A malformed field in a CSV and a null pointer are the same animal wearing different clothes.
This is roughly the point at which I stopped describing myself as someone who writes software and started describing myself as someone who finds out what is actually happening. It pays better and it travels further.
Phase two: the bug moves into the answer
Then the failures changed shape again, and this time the change was harder.
A modern failure does not crash. It answers. Fluently, in well-formed sentences, with the tone of something that has checked. The quotation with the impossible price was generated by a perfectly healthy system in which nothing threw an exception and no log line was coloured red. Every component reported success. The output was wrong.
Three things break in the old method at once.
There is no stack trace. The system does not tell you where it went wrong, because in its own terms it did not go wrong. You have to manufacture the failure signal yourself, and that turns out to be most of the job.
Reproduction is no longer free. Ask twice, get two answers. The first reflex is to pin what can be pinned: fix the temperature, log the exact context that was sent rather than the template you think was sent, keep the full request so you can replay it later. Half the AI incidents I have investigated were resolved the moment someone could see what had actually been in the context window, which was never what the team believed.
The search space includes the data. In the quotation case, the model had done nothing unusual. It had faithfully summarised a two-year-old PDF that was still sitting in an indexed folder. The bug was in a shared drive, and it had been there since well before anyone installed any AI. The system did not create the error. It gave it a voice and a confident tone.
What actually carried over
More than I expected, honestly.
Reproduce first still holds, it just costs more. Halving the search space still holds: swap the retrieved documents for known-good ones and see if the failure survives, and you have cut the problem in two exactly as you would with a git bisect. Suspect yourself still holds, and matters more than ever, because a fluent answer is engineered to be believed and the person it fools most easily is the one who wanted it to be right.
What is genuinely new is that the verification step no longer comes free with the failure. Software used to tell you it had broken. Now you have to go and check, every time, and checking requires knowing the answer well enough to grade it. That is the paradox I have written about before: the expertise you needed in order to skip the expertise. In debugging terms it has a name, and the name is a cost. It shows up in every AI project I audit, usually in the column nobody budgeted.
The teams that handle this well are not the ones with the best models. They are the ones who kept a test suite. They wrote down thirty questions whose answers they know, they run them against the system on every change, and when something drifts they find out on Tuesday afternoon rather than from a customer on Sunday morning. That is not an AI practice. That is 1990s engineering hygiene, wearing a new hat.
The Sunday call, 2026 edition
The founder with the impossible price wanted to know whether to switch the whole thing off.
We did not switch it off. We spent Monday deleting four hundred obsolete documents from a folder nobody had opened since 2024, wrote down which document is authoritative for pricing, and added a check that refuses to produce a quotation containing a figure absent from the current price list. Two of those three fixes had nothing to do with artificial intelligence. The third was ten lines.
By Wednesday the system was better than before the incident, because the incident had forced a cleanup that had been owed for two years.
That is the part of this craft that has not moved a millimetre in twenty years. The bug is rarely the interesting thing. The interesting thing is the disorder the bug just made visible, which was always there, patiently waiting for something loud enough to reveal it.
Sometimes it is a moth in relay 70. Usually it is us.
Everyone knows that debugging is twice as hard as writing a program in the first place. So if you’re as clever as you can be when you write it, how will you ever debug it? — Brian Kernighan and P. J. Plauger, The Elements of Programming Style, 2nd edition, 1978