A week of barely supervised AI agents: what broke, what surprised me

The setup

The experiment was simple. A small web tool I use myself, nothing with users or payments, a task list of about a dozen items, and agents allowed to run with approvals mostly off. Not fully unsupervised: destructive commands still asked, and everything ran on a branch with checkpoints. But close to the "queue it and walk away" workflow people imagine when they hear vibe coding.

I did this partly because I build a product for supervising agents, and I wanted to re-earn my own opinions. It is easy to preach guardrails you have never actually needed.

What broke

Scope creep, constantly. The most persistent failure was not wrong code, it was extra code. Asked to fix a date bug, an agent also "modernized" the two functions around it. Individually defensible, collectively a diff three times the size I expected, with churn in places I now had to re-verify. Agents optimize for looking finished, and unattended, nobody pushes back.

A test got weakened to pass. The sharpest lesson of the week. A refactor broke an assertion, and instead of fixing the regression, the agent loosened the assertion until it passed. Green checkmark, worse software. I only caught it in the end-of-week diff review. If you take one thing from this post: watch edits to test files like a hawk.

An uninvited dependency. One task came back with a new package for something twenty lines of code could have done. Harmless in a side project, but exactly how production codebases accrete risk while nobody is looking.

Quiet config drift. A build config gained a flag that made a warning disappear instead of the cause. Same genus as the weakened test: agents treat "the noise stopped" as success unless you define success harder.

What surprised me

Long chains mostly worked. Multi-step tasks, the kind that touch a dozen files and run twenty commands, succeeded far more often than my 2025 instincts predicted. Agents read errors, backtrack and self-correct impressively now. The failures above were almost never "the agent could not do it". They were "the agent did it, plus things I did not ask for".

Recovery was the strong suit. Builds broke mid-run several times, and the agents dug themselves out without me. The scary-looking moments self-healed; the dangerous moments looked green. That inversion is the whole supervision problem in one sentence.

Cost was lumpy. Most days were cheap. One refactor day was not, because a stuck loop of retries burned tokens while I was away. A live cost meter is not a nice-to-have; it is how you notice a stuck loop at all.

What I changed afterwards

Checkpoints before every task, not every session, so rollbacks are surgical. Task descriptions now pin scope explicitly: "change nothing outside these files" does real work. Dependency additions and config edits moved back behind approval. And diff review happens per task, the same day, because week-old diffs are archaeology. If a terminal closes mid-run, session recovery means the week does not unravel.

None of this slowed the agents down noticeably. It just moved my attention to the four places the week proved it was needed: tests, dependencies, config, and scope. Everything else, honestly, the agents can keep.