跳转到主要内容

WRITING

I Keep 35 Alarms Watching Me. I Silenced the Most Important One Myself.

July 28, 20269 min readTianli Zeng
engineeringpostmortemClaude Codeautomation
I Keep 35 Alarms Watching Me. I Silenced the Most Important One Myself.

To make one alarm less noisy, I added a line that skipped repeats. From that moment it went permanently silent — for every conversation on this machine.


1. I told it to be quieter, and it never spoke again

While I work, about thirty-odd small alarms sit around me. They don't do any work. They only watch me, waiting for the moment I'm about to do something that's hard to undo.

One of them watches Word documents.

There's a lazy way to edit a document: instead of touching the two paragraphs you actually want changed, regenerate the whole file. Easy — except the fonts, the tables, the numbering, the comments, everything somebody spent an afternoon getting right, all of it is gone. So one alarm shouts at me right at that step.

It shouted far too often. I'd touch a single paragraph and it would shout.

So two days ago I added one line: within the same conversation, only shout the first time.

To do "only the first time," it first has to know which conversation this is. So it asked the system.

The system has no such thing.

And the code already had a fallback written in: if you can't get a name, call it "unnamed."

How one fallback name makes every alarm disappear
It leaves a note on disk: "conversation unnamed — already shouted." The next conversation is also unnamed, sees the note, goes quiet. So is the one after that.

Every conversation on this machine, from then on. Forever.

That note was written at 1:01 in the morning — by my own test run.

From the outside it looked healthy: started on time, exited cleanly, just never said anything.

Trying to make it quieter, I killed it.

The fix isn't to revert that line. Ask a place that actually has a name; then shrink the note from "this conversation" down to "this one document" — so a second, different Word file in the same conversation still gets a shout. That's precisely the case the old design would have swallowed. Nine scenarios, each run for real; that one is the one that matters.


2. The same mistake in three places, so I stopped fixing them one at a time

It wasn't alone. Two other alarms had the identical fallback, and both were calling themselves "unnamed" too.

The embarrassing part comes next: I had already fixed this same bug three days ago.

While fixing it I even wrote a note in the code, naming a leftover file as proof. And in that same file, 340 lines below that note, the part that actually does the work was untouched.

The diagnosis was right. The place doing the work never caught up.

I have a rule of my own: when a third different problem shows up in the same patch of code, what's missing is a concept, not a patch.

So I didn't point-fix the third one. I deleted the fallback name entirely — it had never once worked, and leaving it in would only convince the next person it was a legitimate backup — and added a check that sweeps all 60 scripts and fails the moment any of them asks for that name directly.

From "remember not to write it that way" to "you can't write it that way"
Left: fix it three times. Right: make that shape impossible. It now sweeps 60 scripts, 0 violations.

I verified that check backwards: put the bad shape back on purpose, confirmed it really got blocked.


3. The run that reported "all clear" — all ten sites were dead

I have a health check for my websites. Point it at a site, it tells you whether the site is alive.

I ran it across all ten. All ten failed. The overall verdict: pass.

Two layers. Inside, it counted the failures but never turned that count into a failing verdict. Outside, the script calling it threw the error into a black hole, with a comment right next to it reading "this is not a fatal error."

One failure, swallowed twice
Ten failures walk out the other end as a single green light. On a real deploy, that green light is the thing that lets it through.

A bonus find: the list of ten sites was hardcoded into the script, and that whole group of sites was retired three weeks ago. The routing table it was supposed to read has zero entries left that can be health-checked. The code is still here. The sites are long gone.

The fix splits the verdict into three: alive / retired on purpose / actually broken. The list now comes from the routing table, and when the table yields nothing at all, that's a failure, not a pass.


4. The audit built to catch fake passes was itself faking a pass

Among those thirty-odd, two exist for exactly one purpose: catching a specific mistake — a check that scanned nothing and then reported "all clear."

Directory it was supposed to scan got moved? Folder empty? — "0 problems found," green.

Both of them were doing precisely that.

One is especially expensive. It watches the 124 promises I've made to myself over the past months ("this rule will be enforced by that check"). If its ledger file goes missing, its old response was: report normal.

124 promises unwatched, quietly.

Scanning nothing can still report green
Passing on an empty set is contagious: the one built to stop others doing it was doing it itself.

And then the opposite problem

This morning one of them reported 3 problems. All 3 were false.

The silliest one: the way that audit decides whether a script is "reporting all clear" is to look for a handful of specific words in its output, and one of those words is clean. The file it flagged happens to contain a small function named clean.

It was hunting for "reports all clear" and hit a function name
The other two were even weaker: scratch files I'd left behind an hour earlier.

A false alarm doesn't just waste time. It trains you to skip that whole column — including the one real item sitting next to it.

The same day I dug out a cry-wolf of my own making. One check compares a documentation file against reality, character by character — and that file contains a table counting how many times I've used each command. Every time I use one, it changes. So it reported a problem every single day, because I was using my tools normally. That column is gone now.


5. Before touching anything, let every conclusion take a beating

I had 12 items queued up to fix. Instead of fixing them, I first sent each one out to be attacked, with explicit instructions: assume it's wrong, and go prove it's wrong.

Results:

VerdictCount
Claim and remedy both hold up1
The fact is simply wrong4
Half right; needs reworking before use7
How the 12 conclusions landed
Done as written, 9 of the 12 would have been wasted effort or an outright mistake.

Last round that number was 62.5%. This round the strictly-wrong rate is only 33%. Looks like progress. It isn't. The way things go wrong changed.

Last round the numbers themselves were wrong, and one check exposed them. This round, 5 of those 9 failed in exactly the same way: the numbers were real, they had just expired — they were reading a snapshot from before a batch of fixes two days earlier.

Truths past their date
This kind is far harder to catch than a wrong number, because it looks like homework was done — those numbers really were true once, and you can still find them in the record.

So one rule came out of this round: before asserting anything about a piece of code, first check when that file last changed. If it changed after your claim was written, throw the whole claim out and look again.


6. The ledger for this round

First, a correction to a number I reported earlier.

I've accumulated 34 small tools, and the statistics said 33 of them had never been used. That sounds like time for a spring cleaning.

But the statistics only counted one way of using them: me typing the name. There's a second way to start a tool — midway through a job, the model decides it's the right one and calls it directly. The statistics never looked down that path.

A ruler that only measured half
Of the 33 "never used," 8 were false zeros, covering 18 calls in total.

I added the second ruler but deliberately did not add the two numbers together. Typed count means "how you intend to use me." Auto-invoked count means "how I actually work." Merge them and that distinction disappears.

Checked35 alarms, 2 audits, 124 promises
Silent3, all from the same cause; 1 of them I muted myself two days ago
Crying wolf3 findings this morning, all 3 false
Thrown out9 of the 12 queued items

Installing alarms is not the hard part. The hard part is going back now and then to confirm they still ring — and there is only one way to confirm that: put the accident back in once, and see whether it goes off.

If you take one line away:

A silent alarm and the accident it was supposed to prevent are the same category of thing.