I skipped testing an agent update and lost a week fixing the fallout. Here's what went wrong and the testing process I use now.
I changed one line in my orchestrator agent’s system prompt on a Tuesday afternoon. By Friday, I had a backlog of 47 failed tasks, three confused client-facing outputs, and a morning briefing that confidently reported numbers from two days ago as current.
One line. Four days of compounding errors before I noticed.
This is a story about why I test agent changes now and why I did not before.
The change was small. I adjusted how the orchestrator prioritized incoming tasks. Previously, client-facing tasks always took priority over internal tasks. I wanted to add a time-sensitivity factor: an internal task due in 30 minutes should outrank a client task due next week.
The logic was correct. The implementation was not. The way I phrased the priority rule in the system prompt created an ambiguity. The orchestrator interpreted “time-sensitive internal tasks” more broadly than I intended. Tasks like “run the daily memory consolidation” suddenly outranked “draft the response for Client X.” The memory consolidation is time-sensitive in that it runs on a schedule. But it is not urgent in the way a client response is urgent.
The result: internal maintenance tasks jumped the queue. Client-facing work piled up. The agents that depended on the orchestrator’s routing received tasks late, processed them with stale context, and produced outputs that were technically complete but based on outdated information.
I did not catch it because the system did not fail. Every task eventually completed. The error was in the ordering, and ordering errors are invisible unless you are looking for them.
The exact architecture, memory layers, and delegation patterns I use to run 50 agents across two businesses.
Get the AI Agent Blueprint →The honest answer: I did not think agent changes needed testing. Not in the traditional software sense.
My reasoning was that agents are probabilistic. The same prompt with the same input can produce different outputs. Traditional unit tests (input X should produce output Y) do not map cleanly to a system where the relationship between input and output is fuzzy.
That reasoning was wrong. Not because traditional testing applies directly. Because I confused “hard to test precisely” with “not worth testing at all.”
After losing that week, I built a testing process. It is not perfect. It does not catch everything. But it catches the kind of error that cost me a week.
Scenario bank. I maintain a set of 30 representative tasks that cover the main categories of work the orchestrator handles. Client requests, internal maintenance, research tasks, content generation, financial operations, and cross-agent coordination.
When I change the orchestrator’s configuration, I run the scenario bank. Each scenario is a task description plus an expected routing decision (which agent should handle it) and an expected priority level.
Before/after comparison. I run the scenario bank before the change and after the change. Then I diff the results. Expected differences confirm the change does what I intended. Unexpected differences reveal side effects.
The priority rule change would have been caught by this method. The scenario bank includes “run daily memory consolidation” as an internal maintenance task with a known priority level. If that priority suddenly jumped above client work, the diff would have flagged it.
Canary deployment. For changes that affect routing or prioritization, I run the new configuration alongside the old one for 24 hours. Both versions process the same incoming tasks. I compare the decisions. If the new version makes decisions I disagree with on real tasks (not just scenarios), I revert.
This catches errors that the scenario bank misses because real task distributions are more varied than any test suite can capture.
Error tracking with context. Every task that requires manual intervention gets tagged with the reason. “Stale context” is different from “wrong agent” is different from “incomplete output.” Over time, the distribution of error types tells me where the system is weakest and where testing efforts should focus.
It is not unit testing in the traditional sense. I am not asserting that specific inputs produce specific outputs. The outputs are generated text, and asserting on generated text is brittle.
It is not integration testing in the traditional sense either. The “integration” between agents is conversational, not API-based. There are no contracts to test against.
What it is closer to is acceptance testing. Given this scenario, does the system make a reasonable decision? “Reasonable” is defined by my judgment when I review the output, not by an automated assertion.
This means the testing process is partially manual. I review the scenario bank diffs. I review the canary comparison. I make judgment calls. This takes about an hour for a significant configuration change.
An hour of testing to prevent a week of debugging. The math is clear.
Beyond the formal testing process, I have developed habits that catch problems before they compound.
Read the morning briefing carefully. The briefing is generated by the system. If the system is malfunctioning, the briefing often shows symptoms. Numbers that seem wrong. Tasks that should have been completed but are listed as pending. Summaries that do not match what I remember from yesterday.
I used to skim the briefing. Now I read it as a diagnostic tool.
Spot-check agent output weekly. Every week, I randomly select ten completed tasks across different agents and review the output. Not for quality (though I check that too) but for signs of systemic issues. If three out of ten outputs reference stale data, something is wrong with the data pipeline.
Track task completion time. A sudden increase in average task completion time indicates something is wrong, even if every task eventually completes. The orchestrator priority bug would have shown up as increased completion time for client tasks within 24 hours if I had been monitoring that metric.
The mistake was not the bad prompt change. Mistakes happen. The mistake was the assumption that agent systems do not need testing because they are “just prompts.”
Agents in production are software. They take inputs, make decisions, produce outputs, and affect downstream systems. The fact that the decision-making mechanism is a language model instead of a function does not exempt them from the same rigor we apply to any other production system.
Testing agents is harder than testing traditional software. The assertions are fuzzier. The test cases are more subjective. The coverage is never complete. But “harder to test” does not mean “should not test.”
It means you need a testing approach that matches the nature of the system. Scenario banks instead of unit tests. Canary deployments instead of staging environments. Judgment-based review instead of automated assertions.
The newsletter covers the full testing and monitoring setup for multi-agent systems.
I lost a week because I treated a production system like a playground. The agents are good enough that everything seemed to work. That is exactly what makes untested agent changes dangerous. They fail quietly.
Subscribe to The Signal
One issue per week. The biggest AI build of the week, the top five videos worth your time, and an operator's read on what actually matters. For builders who ship, not dreamers who scroll.
No spam. Unsubscribe anytime. Just The Signal.