How to evaluate an agent system for bottlenecks and accuracy

One agent system, one bad week. Every fix uncovers the next problem, and each problem needs a different way to measure.

By Sanyam Wadhwa · About 12 minutes

Most guides to agent evals hand you a list of metrics. That list is useful once you know what's broken. The hard part is the week where every dashboard looks fine and users still say the answers are slow and wrong.

This guide follows one system through a week like that. Each fix uncovers the next problem, and by Friday you'll have the set of evals you'd want on day one.

The system

It answers questions about a company's internal docs and data, things like "why did churn go up in Q3?"

An orchestrator reads the question, makes a plan and hands work to four agents. The Researcher finds documents through a RAG pipeline: it embeds the question, searches a vector index and reranks what comes back. The Analyst pulls numbers with a SQL tool. The Writer drafts the answer, and the Critic, an LLM judge, checks each claim against the sources and can send the draft back.

It's the orchestrator and worker shape Anthropic describes for its own research system, where a lead agent plans and parallel subagents do the work.¹¹ Step through one question below. Watch which branch the rest of the run ends up waiting on.

That last bar is the whole story of Monday.

MondayIt got slow

It starts with a message in the team channel: answers feel slow. The dashboard says median latency is flat, but p95 has roughly doubled since another quarter of documents went into the index.

The instinct is to speed up whatever looks slowest. Hold off. In the run above, the Researcher and the Analyst work in parallel and the Writer waits for both. Total time is set by the longest path through the run, so a faster Analyst changes nothing while the Researcher is the long pole.

So break the Researcher's RAG call into separate spans: embed the question, search the index, rerank, call the model. One slow trace usually tells you the rest.

Here it's the reranker. Retrieve wide, cut the list down before reranking, cache embeddings for questions that repeat, and retune the vector index for its new size instead of keeping settings picked for a smaller corpus.

Why did only p95 move? Averages hide the slow calls people actually feel, which is why SRE teams track percentiles.⁵ And when a query waits on several index shards, the slowest shard sets the pace. Dean and Barroso show that with 100 servers each slow on 1% of requests, about 63% of requests hit at least one slow call.⁴

By Tuesday p95 is back where it was. On Wednesday, support starts forwarding answers that cite documents which don't say what the answer claims.

WednesdayIt got wrong

The tempting fix is a bigger model or a stricter prompt. Before touching either, find out which half failed. A RAG answer can go wrong in retrieval, where the right document never reaches the model, or in generation, where the model has it and ignores it. The fixes for those two have nothing in common.

This is where a golden set earns its keep. If every test question records which chunk holds its answer, one check splits the problem in two.

Run that check across the whole set and you get recall@k, the share of questions where the right chunk made the top k. Monday's fix shows up right away. Cutting the candidate list made reranking fast, and it also pushed the right chunk out for questions with unusual wording. Speed and accuracy turned out to be the same knob.

None of this is unusual. Barnett et al. studied RAG systems in three domains, and missing content and missed top ranked documents both made their list of seven failure points.¹⁷ They also found a RAG system can only really be validated once it's in use, which is the argument for building the golden set from real questions.

The answer is a middle setting that keeps recall at its old level, plus a latency budget so it can't drift back up. For the generation half, keep in mind that position matters. Liu et al. found models use information at the start or end of a long context much better than information in the middle.¹⁸ Then a faithfulness check catches the rest. A judge quotes the source sentence behind each claim, and a plain substring match confirms the quote exists. A quote you can't find is a claim the model made up.

Most of the bad answers were retrieval. A few had nothing to do with documents at all. They had wrong numbers in them.

Thursday morningAn agent broke quietly

Revenue answers now include numbers that look plausible and are wrong, and nothing has errored. Someone renamed a column in the warehouse, the Analyst's query started returning zero rows, and the Analyst still returned something shaped like a number. The Writer built a confident paragraph on it. The Critic checked that paragraph against the Analyst's output, which of course agreed with itself.

Two things catch this. The first is a contract at every boundary. If the Analyst has to return its number together with the SQL that produced it and the row count, a zero row result fails validation before the Writer ever sees it.

The second is replay, for whatever slips past the contracts. Record the failing run, then rerun it with upstream outputs frozen to their golden versions, one part at a time. The first part that disagrees with golden owns the bug.

You might hope a model could read the traces and find the culprit for you. Not yet. On the Who&When dataset of multi-agent failures, the best automated method named the responsible agent 53.5% of the time and the exact failing step only 14.2% of the time.¹³ Replay against golden outputs avoids the guesswork by making the comparison mechanical.

It also helps to know this failure is common. Cemri et al. built a taxonomy of 14 failure modes from traces of popular multi-agent frameworks, and many of them trace back to system design and missing verification rather than the model itself.¹² That's what happened here. No agent checked that the number had a real query behind it.

The fix comes in two halves. Give each agent a timeout and a step limit so a confused agent can't loop. And when an output fails validation, retry once with the error in the prompt. If it fails again, say so in the answer. "I couldn't get the revenue numbers" beats a confident wrong one.

That single retry looked harmless. By Thursday evening it wasn't.

Thursday eveningRetries met a rate limit

At peak hours everything slows down and the bill climbs faster than traffic. The traces explain it once you split time spent waiting from time spent working. The Researcher fires several search calls in parallel, the search API starts rate limiting them, and the extras sit in a queue. In a single timer, that looks exactly like a slow model.

Retries make it worse, right when you're already being throttled. If a step fails with probability p and you allow up to k retries, the expected number of calls for that step is:

With p = 0.3 and k = 2 that's about 1.39 calls. The step really costs around 39% more than a single attempt in a trace suggests.

Cap concurrency per tool below its limit, back off between retries, and add a circuit breaker so a failing tool gets skipped for a while instead of hammered. Then break it on purpose. Switch the search API off in staging and run the golden set. Chaos engineering calls this testing failure in a controlled way before it finds you.¹⁰

Counting retries by where they came from turned up something else. Most of them weren't from the Analyst. They came from the Critic.

FridayThe Critic was the problem

The Critic was sending back drafts that a person reading them would have approved. A judge is a model with its own error rate, and every quality number on the dashboard leans on it. So before trusting it, label a small sample yourself and measure how often the Critic agrees with you.

The research says what to look for. Zheng et al. found that LLM judges favour whichever answer comes first, favour longer answers, and can favour their own outputs.⁷ In this system the Critic and the Writer ran on the same model. G-Eval showed that explicit criteria and step by step reasoning before the score track human ratings better than a bare number.⁸ And Shankar et al. found that people can't fully pin down their criteria before grading real outputs, so expect your rubric to change after the first labelling pass.⁹

The fix was a different model for the Critic, a rubric with described levels instead of a 1 to 10 score, and a quote requirement for every verdict. Anything a plain check can decide, like whether a source is attached at all, moved out of the Critic and into code.

If you want to go further, the judge literature has moved quickly. Gu et al.'s survey collects the known ways to make judges more consistent and less biased, along with a benchmark for testing them.¹⁹ Newer work gives the judge tools of its own, so it can open a source and check a claim instead of reasoning about it from memory.²⁰

Reading Critic traces all afternoon, one more pattern stood out. Easy questions were taking as many steps as hard ones.

Still FridayThe orchestrator over-plans

"Who owns the billing service?" should be a single lookup. The trace shows the Analyst running SQL anyway, then the full Writer and Critic loop. Every agent did its job well. The plan just didn't need most of them.

Log the plan for every run and compare across the golden set: how many steps, which agents, and whether each call fed into the final answer. Be careful about what you grade, though. Anthropic's eval guide suggests grading the outcome rather than forcing one exact sequence of steps, since agents often find valid paths you didn't plan for.² So treat steps and cost as a budget, and correctness as the grade.

That budget is bigger than it looks. In Anthropic's research system, token usage alone explained 80% of the performance variance on their browsing eval, and multi-agent runs used about 15 times the tokens of a chat.¹¹ Kapoor et al. argue that agent evaluations which ignore cost end up rewarding needlessly complex systems, and that simple baselines often match them for far less.¹⁶

The cheap fix is a routing step that classifies the question first, so lookups skip the heavy path. The broader advice from Anthropic applies here too: start with the simplest workflow that works, and add agents only when they clearly help.¹

What I'd set up on day one

Every problem this week could have been caught earlier by something cheap. The checklist at the end collects all of it. Two pieces deserve a word here, because they're the ones people skip.

Traces you can read. Log one span per call, tagged with the part that made it and the attempt number, with queue time kept apart from run time. OpenTelemetry's generative AI conventions already name most of the fields.³ Don't count on a model to read them for you either. On the TRAIL benchmark, built from OpenTelemetry traces, the best model located and correctly classified only about 11% of the errors.¹⁴

Run each question more than once. Agents are inconsistent, so a single pass hides flaky behavior. τ-bench measures this with pass^k, the chance an agent succeeds on all k tries of the same task. A strong model that solved about 61% of retail tasks on one try solved only about 25% of them eight times in a row.¹⁵ Track that number for your golden set, and put all of it in CI so a regression fails the build.² Hamel Husain's evals guide is a good hands-on walkthrough of the setup.⁶

{
  "run_id":     "r_0142",
  "part":       "agent.researcher",
  "step":       "rag.rerank",
  "parent":     "orchestrator.plan",
  "attempt":    1,
  "queued_ms":  ...,
  "run_ms":     ...,
  "tokens_in":  ...,
  "tokens_out": ...,
  "status":     "ok"
}

If you're in one of these right now

The checklist

Tick these off for your own system. Your progress stays in this browser only.

References

1. Anthropic, "Building effective agents" (2024). anthropic.com/engineering/building-effective-agents
2. Anthropic, "Demystifying evals for AI agents" (2026). anthropic.com/engineering/demystifying-evals-for-ai-agents
3. OpenTelemetry, "Semantic conventions for generative AI client spans". opentelemetry.io/docs/specs/semconv/gen-ai
4. J. Dean and L. A. Barroso, "The Tail at Scale", Communications of the ACM 56 (2013). research.google/pubs/pub40801
5. Beyer, Jones, Petoff and Murphy (eds.), Site Reliability Engineering, chapter 6, "Monitoring Distributed Systems". sre.google/sre-book/monitoring-distributed-systems
6. Hamel Husain, "Your AI Product Needs Evals". hamel.dev/blog/posts/evals
7. L. Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (2023). arxiv.org/abs/2306.05685
8. Y. Liu et al., "G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment" (2023). arxiv.org/abs/2303.16634
9. S. Shankar et al., "Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences", UIST 2024. arxiv.org/abs/2404.12272
10. Principles of Chaos Engineering. principlesofchaos.org
11. J. Hadfield, B. Zhang et al. (Anthropic), "How we built our multi-agent research system" (2025). anthropic.com/engineering/multi-agent-research-system
12. M. Cemri et al., "Why Do Multi-Agent LLM Systems Fail?", NeurIPS 2025. arxiv.org/abs/2503.13657
13. S. Zhang et al., "Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems", ICML 2025. arxiv.org/abs/2505.00212
14. D. Deshpande et al. (Patronus AI), "TRAIL: Trace Reasoning and Agentic Issue Localization" (2025). arxiv.org/abs/2505.08638
15. S. Yao, N. Shinn, P. Razavi and K. Narasimhan, "τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains" (2024). arxiv.org/abs/2406.12045
16. S. Kapoor, B. Stroebl, Z. Siegel, N. Nadgir and A. Narayanan, "AI Agents That Matter", TMLR 2025. arxiv.org/abs/2407.01502
17. S. Barnett et al., "Seven Failure Points When Engineering a Retrieval Augmented Generation System", CAIN 2024. arxiv.org/abs/2401.05856
18. N. F. Liu et al., "Lost in the Middle: How Language Models Use Long Contexts", TACL. arxiv.org/abs/2307.03172
19. J. Gu et al., "A Survey on LLM-as-a-Judge" (2024, revised 2025). arxiv.org/abs/2411.15594
20. R. You et al., "Agent-as-a-Judge" (2026). arxiv.org/abs/2601.05111