Build the repro first

I shipped four fixes to a bug I never reproduced. All four were wrong. Here's the war story, because the lesson cost me a weekend and it's the most transferable thing I learned all month.

A crash with no error

The symptom: during an email sync, a handful of messages froze on "reading…" forever. No exception. No stack trace. Nothing in the logs. Just a worker that seemed to wander off mid-read on one particular email — a big PDF attachment — and never come back.

So I theorized. The model must be stalling on the big attachment, I figured, so I added a timeout. Still stuck. Then it must need a retry, so I built a retry loop. Still stuck. Then it must be re-hammering an unreadable email forever, so I added a give-up-after-N-attempts. Then a pass to free the attachment from memory partway through. Four theories, four fixes, each one plausible, each one built on top of the last — and not one of them touched the actual problem, because I had never once watched the problem happen. I was debugging a story I'd told myself, and the story was getting more elaborate with every fix.

The forty-line harness

The thing that finally cracked it was giving up on theories entirely and building a headless repro: a small script that ran the real production circuit — the actual sync path, not a mocked-out unit test — on the real problem email, on the actual server, with a memory heartbeat printing every few seconds and a catch-all for any crash the logs might be swallowing.

It took an afternoon to write. It told the truth in a single run.

The truth was boring

The email read fine. Thirty-three seconds, clean, exit zero. The code path I'd been "fixing" for two days was never broken.

What the harness showed instead was memory. Reading that one email — specifically, rendering its three-page PDF into a little thumbnail image — spiked the process past a hard memory ceiling, and the process manager did exactly what it's configured to do at that ceiling: it killed the process. Not a catchable error. A kill signal, from outside the program, invisible to every try/catch I owned. Then the manager restarted the process, the restart re-triggered the sync, the sync marched back to the same fat PDF, and it got killed again. A restart loop that, from the outside, looked exactly like "the model wandered off mid-read."

The real fix was one line: skip the thumbnail render for a big PDF. I'd have found it in an hour if I'd built the harness first instead of fourth.

Why I hadn't

Because my dev machine had been lying to me. The PDF-rendering step depends on a native graphics library that's broken on my laptop's architecture — so locally, that exact operation failed cheap and quiet, and never got anywhere near the memory spike. The one operation that blew up in production literally could not run where I was looking for it.

And that's the trap, the whole trap, in one sentence: "I can't reproduce it here" quietly becomes "I'll reason about it instead." The moment I couldn't make it happen on my machine, I stopped trying to make it happen at all and started reasoning from the symptom — and reasoning about a bug you can't see is just writing fiction with a compiler. Every one of my four fixes was internally sensible. They were answers to a question I'd made up.

The rule

So here's what I banked, and what I'll do every time now: for any "there's no error, it just breaks" bug, build the repro on the real circuit before you touch a line of fix. Reproduce it where it actually happens — not where it's convenient — even if that means running your harness on the production server because your laptop can't run the thing that's failing. Then keep the harness. It's the cheapest insurance you'll ever write; mine is still sitting in the repo, ready for the next ghost.

This is the same instinct as keeping the whole stack boring: the goal is to be able to see what your system is doing, plainly, at the moment it does it. A boring stack you can watch beats a clever one you have to guess at — and a forty-line script that reproduces the truth beats four elegant fixes aimed at a story.

The one-sentence version

The bug wasn't hard. Finding it was — and only because I spent a weekend fixing a story instead of an hour reproducing a fact.