Multi-Agent Systems: When AI Talks to AI and Things Get Weird
Orchestration patterns, failure modes, and lessons from 6 months in production. The submit-verify contradiction, circuit breakers, and why observability is the real product.
Last Tuesday, two parts of my submit-verify automation pipeline disagreed with each other. My submission agent reported a successful form post — record REF-2847, confirmation number APP-2847, status COMPLETE. My verification agent, which I added after the third time the submission agent claimed success on a record that never actually persisted, queried the external portal API and found zero records. No submission. Status: NOT_SUBMITTED.
Both agents had evidence. The submission agent had a screenshot of a confirmation page. The verification agent had a JSON response from the portal API with an empty results array. Both were describing the same event — just 60 seconds apart.
The external portal had a 60-second processing delay between form submission and API visibility. The submission agent was correct at T+0. The verification agent was correct at T+5 seconds. They were both describing ground truth — just from different moments. This is not a hallucination problem. It is a distributed state problem with a confidence display bug layered on top.
Welcome to multi-agent systems in production.
What "Production-Grade" Actually Means
I want to say this upfront, because everything else in this post flows from it: a multi-agent system is a distributed system. Not metaphorically. Literally. You have processes passing messages, shared state that can be observed at different times, external APIs that can fail or lag, and retry logic that can amplify partial failures into full disasters. Every failure mode from microservices applies. The frameworks don't protect you from any of it.
My definition of production-grade for an agent system: failures are contained, visible, and recoverable without human intervention for 95% of cases. The 5% that need human attention are surfaced immediately with enough context to resolve in under 10 minutes. That's it. Not "agents that succeed" — agents that fail gracefully.
| Requirement | Implementation |
|---|---|
| Circuit breakers on external calls | Max 2 retries, then halt + alert |
| Structured data passing | IDs, timestamps, status codes — no natural language |
| Verification after irreversible actions | Agent's claim ≠ source of truth |
| Full execution logs | Queryable inputs, outputs, timestamps for every call |
Three Orchestration Patterns, and When Each One Bit Me
Most tutorials describe these patterns theoretically. Here's how they played out in my actual pipeline.
Sequential. Agent A runs, passes output to Agent B, which passes output to Agent C. Each step waits for the previous one. This is the only pattern I use for anything that involves an irreversible action — form submissions, emails, database writes. The execution log in n8n is a clean, ordered record of every step. When something breaks, I open the execution history, find the failed node, read the exact error. It's boring to describe and invaluable at 11pm when a submission silently failed.
The cost is speed. My sequential submit pipeline takes about 90 seconds per record. I don't care. I process fewer records, not faster ones.
Parallel. I use parallel execution for the ingestion phase — hitting five external sources simultaneously instead of one at a time. That part works well because the tasks are genuinely independent and the merge contract is trivial: union the results, deduplicate by record ID, done. The mistake I made early on was trying to parallelize the submit-and-verify step to speed things up. That's what produced the incident above. The agents were racing against portal processing time and the orchestrator had no way to know whose snapshot was newer.
The rule I now follow: parallel is safe when sub-tasks are independent AND the merge contract is defined before writing a line of code. "We'll figure out merging later" is how you build the submit/verify contradiction I described.
Hierarchical. I don't use this yet at scale. My orchestrator is n8n, which sees the entire workflow as a flat graph. What I've learned from hitting the limits: hierarchical orchestration makes sense when you have genuine ambiguity that requires judgment, not just coordination. If you can express the logic as a decision tree, use sequential with branches. Save hierarchical for things that can't be expressed as a decision tree. And don't add it until you have strong observability, because debugging a conversation between a supervisor agent and three sub-agents where one of them had context drift halfway through is not something you want to do at midnight.
| Pattern | Use When | Avoid When |
|---|---|---|
| Sequential | Irreversible actions, audit trail needed | High-throughput independent tasks |
| Parallel | Independent tasks, trivial merge contract | State-dependent steps, complex merges |
| Hierarchical | Genuine ambiguity requiring judgment | Coordination-only workflows |
The Circuit Breaker That Saved My Pipeline
After the contradicting-agents incident, I added a circuit breaker to every workflow step that touches an external system. The implementation is a single Postgres table:
| Column | Type | Purpose |
|---|---|---|
| workflow_id | TEXT | Workflow identifier |
| step_id | TEXT | Step identifier |
| fail_count | INT DEFAULT 0 | Consecutive failures |
| last_fail | TIMESTAMPTZ | Last failure timestamp |
| cleared_by | TEXT | Who reset it |
| cleared_at | TIMESTAMPTZ | When it was cleared |
Before each n8n node runs: SELECT fail_count FROM circuit_breaker WHERE workflow_id=$1 AND step_id=$2. If fail_count >= 2, skip execution, fire ntfy alert, stop. On success: reset fail_count to 0. On failure: increment fail_count. A human reviews the alert, confirms the step is safe, runs UPDATE circuit_breaker SET fail_count=0, cleared_by='oncall', cleared_at=now() to re-enable it.
| Property | Value |
|---|---|
| State | CLOSED |
| Failure Count | 0/2 |
| Last Trigger | [none] |
| Auto-Reset | manual — human must clear |
Two consecutive failures on the same step = halt. The n8n execution history gives me the exact inputs and outputs for both failed runs. The ntfy alert fires to my phone within seconds. I can look at both failure records, understand what happened, and decide whether to clear the breaker or fix the underlying issue first.
This has triggered four times since I built it. Each time it caught something that would have become a worse problem with another retry: once a rate-limit that needed a 10-minute backoff, once a malformed target URL that was crashing the browser automation, once the portal processing delay issue described above, and once a network timeout that was masking a VPN connectivity drop on the home server.
Idempotency: The Thing Frameworks Skip
If an agent submits a form and the orchestrator doesn't get the confirmation back (network error, timeout), it will retry — and you'll submit twice. Every action with a side effect needs an idempotency key checked before execution. My submit agent writes a fingerprint (record_id + entity_id + date) to Postgres before attempting. If the fingerprint exists, it skips the submission and returns the cached result. This is standard practice in payment systems. It's almost never discussed in AI agent architecture posts.
What Breaks First (In Order)
- Context drift arrives first. Agent A passes a summary to Agent B. Agent B summarizes further. By Agent C, the nuance from A's original observation is gone. The mitigation: pass structured data — Postgres IDs, timestamps, status codes — not natural language. The database is the source of truth. The agent's output is a candidate fact.
- Retry amplification arrives second. An agent fails. The orchestrator retries. The agent fails again, differently, because the world changed between attempts. By attempt three the error is about something completely different. The circuit breaker stops at two, not three, and the ntfy alert includes the first failure's context.
- The confident wrong answer arrives third. A local model states an incorrect conclusion with the same formatting and tone as a correct one. No built-in uncertainty signal. I handle this with a verify step after any consequential action, and structured output with an explicit
confidencefield. Low confidence routes to human review. The agent is never the last gate.
Agents are not sources of truth. They are producers of candidate facts. The database is the source of truth.
The One Thing Tutorials Never Cover: Observability
Every multi-agent tutorial shows you how to build the agents. Almost none show you how to watch them run. I use two layers: LangFuse for LLM call tracing (every call to Ollama, every prompt, every response, latency, token count) and n8n's built-in execution history for workflow-level visibility (every node, every input/output, every error). When the submit/verify contradiction happened, I opened the n8n execution for that run and saw both agent responses side-by-side with their timestamps. The 58-second gap between SUBMIT_AGENT's submission and VERIFY_AGENT's query was right there in the logs. Without that, I would have spent an hour hypothesizing. With it, I had root cause in four minutes.
If you don't have full execution logs with timestamps for every agent call, you are not running a production system. You are running a demo that hasn't failed yet.
Six Months In
The portal incident resolved itself. The submission was there all along — the verify agent had just queried too fast. The orchestrator now waits 60 seconds before running the verification step for any action that involves an external portal submission. That is a hack. It works. Most production systems are full of hacks like this, each one encoding a specific failure mode that someone hit at a bad time.
The pipeline running today is not elegant. It is auditable. Every decision is in a log. Every failure has a circuit breaker. Every irreversible action has an idempotency check and a verification step. It does not cascade failures when one part breaks, and it doesn't silently succeed when it actually failed. Those properties are harder to build than the agents themselves.
Build the observability first. Add the circuit breakers before you go to production. Define the merge contract before you write the parallel step. The agent prompts are the easy part. The distributed systems plumbing is where multi-agent pipelines actually live or die — and almost nobody is writing about it.
Stay with us · dig next
How do you manage discrepancies between multiple agents in your multi-agent systems?
Share your strategies for ensuring consistency and reliability across different agents.
No account needed — pick a take, then keep reading. We rotate these prompts so each piece feels like a conversation, not a clone.
