Your Agent Isn't Slow, It's Queued

A research task takes ninety minutes and nothing about it is slow. The model is fast. The tools are fast. The searches return in twenty seconds each. And yet four of those searches ran one after another, each waiting on a result it never read.

I got tired of arguing about this from intuition, so I wrote the arithmetic down. On a seven-node research pipeline with real timings, half the wall clock was queueing — not work, not model latency, just steps waiting on steps they had no relationship with.

The interesting part is that this is computable. You do not need to profile an agent to find out how much of its runtime is structural. You need its dependency graph and a critical-path calculation, and that fits in fifty lines.

Sequence is not dependency

An edge between two steps should mean exactly one thing: the second step reads what the first produced. If it does not read it, the edge is imaginary, and you are paying for it in wall clock.

Default agent pipelines are written as lines because instructions are written as lines. Do this, then that, then the other thing. But "I wrote it underneath" and "it depends on the output" are different claims, and only the second one costs you anything real.

Take a research task: company filings, academic papers, competitor pricing, expert commentary, then synthesis, then a verification pass. None of the four lookups consume each other. All four feed the synthesis. Written as a chain that is four round trips in a row; written as a fan it is one.

The measurement

Here is the calculator. Give it durations and edges, and it returns the critical path — the longest chain of genuine dependencies — alongside total work.

from collections import defaultdict


def critical_path(tasks, edges):
    """tasks: {name: seconds}. edges: [(before, after)] - after READS before."""
    successors = defaultdict(list)
    indegree = {name: 0 for name in tasks}
    for before, after in edges:
        successors[before].append(after)
        indegree[after] += 1

    finish = {}
    queue = [n for n, d in indegree.items() if d == 0]
    start = {n: 0.0 for n in queue}
    order = []

    while queue:
        node = queue.pop()
        order.append(node)
        finish[node] = start.get(node, 0.0) + tasks[node]
        for nxt in successors[node]:
            start[nxt] = max(start.get(nxt, 0.0), finish[node])
            indegree[nxt] -= 1
            if indegree[nxt] == 0:
                queue.append(nxt)

    if len(order) != len(tasks):
        raise ValueError('graph has a cycle')

    return max(finish.values()), sum(tasks.values())


TASKS = {
    'plan': 4.0, 'filings': 22.0, 'papers': 31.0, 'pricing': 18.0,
    'commentary': 25.0, 'synthesise': 19.0, 'verify': 11.0,
}

CHAIN = [
    ('plan', 'filings'), ('filings', 'papers'), ('papers', 'pricing'),
    ('pricing', 'commentary'), ('commentary', 'synthesise'), ('synthesise', 'verify'),
]

FAN = [
    ('plan', 'filings'), ('plan', 'papers'), ('plan', 'pricing'), ('plan', 'commentary'),
    ('filings', 'synthesise'), ('papers', 'synthesise'),
    ('pricing', 'synthesise'), ('commentary', 'synthesise'),
    ('synthesise', 'verify'),
]

for label, edges in (('chain', CHAIN), ('fan + join', FAN)):
    span, work = critical_path(TASKS, edges)
    print(f'{label:<12} critical path {span:5.1f}s  total work {work:5.1f}s  '
          f'speedup {work / span:4.2f}x')

chain        critical path 130.0s  total work 130.0s  speedup 1.00x
fan + join   critical path  65.0s  total work 130.0s  speedup 2.00x

Same nodes, same durations, same total work. 65 seconds of the chain's 130 was queueing — exactly half the wall clock.

That is not a modelling artefact. Running it for real with the agent call stubbed as a sleep:

chain                       136.2s (scaled)
fan + join                   66.6s (scaled)

The ~5% over the predicted figures is scheduler overhead and sleep granularity, since I scaled the durations down 100× to make the harness runnable. The ratio holds.

A join is a decision, not a formality

Parallelism is easy to add and easy to spoil. The common way to spoil it is a barrier after every stage — wait for all branches, then start the next stage — which quietly turns a fan back into a chain.

I expected this to be expensive and it is, but not always. Consider three documents, each fetched, then extracted, then checked:

barrier after each stage   critical path 39.0s   total work 59.0s   speedup 1.51x
streamed per document      critical path 21.0s   total work 59.0s   speedup 2.81x

The barrier costs 46% extra wall clock for identical work. But the reason is specific and worth knowing: the bottleneck moves. doc0 is slow to fetch, doc2 is slow to extract, doc1 is slow to check. Each barrier makes everyone wait for a different straggler.

When I first wrote this example I made every stage slowest on the same branch — and the barrier cost exactly nothing, 36s either way. That is the real rule, and it is more useful than "barriers are bad": a barrier is free when one branch dominates throughout, and expensive when the bottleneck moves between stages. Since you rarely know in advance which case you are in, stream by default and add a join only where the next node genuinely needs the complete set — deduplicating across sources, ranking candidates against each other, deciding whether coverage is sufficient.

Failure should be a value

A join that aborts when one branch throws is a join that hands you nothing after ninety seconds of successful work from the other three.

async def branch(name):
    if unavailable(name):
        raise RuntimeError(f'{name} unavailable')
    return await step(name)

results = await asyncio.gather(*(branch(n) for n in LOOKUPS), return_exceptions=True)
settled = [r for r in results if not isinstance(r, Exception)]
missing = [str(r) for r in results if isinstance(r, Exception)]

fan, one branch failed       60.0s (scaled)  kept 3/4: papers unavailable

return_exceptions=True is the whole trick in Python — collect what settled, record what did not, and let the next node decide whether three sources out of four is enough to continue. A node that can return "not found" as data is a node the graph can route around. A node that throws is a node that stops the graph.

This is also why node outputs want a shape. If a branch returns prose, the next step has to interpret it, and interpretation is another model call — another sample, another chance to be wrong about something that was already known. Structured output means the graph can branch on a result without asking a model what the result meant.

The cost side, honestly

Parallel breadth is not free, and the case for it is weaker than enthusiasts suggest.

Anthropic published figures from their own multi-agent research system that are worth quoting exactly, because both halves matter. Their multi-agent setup outperformed a single agent by 90.2% on their internal research evaluation. It also used about 15× more tokens than a chat interaction — and, in the detail most summaries drop, plain single agents already use about 4×. So roughly a 4× step for being agentic at all, and a further ~4× on top for fanning out.

Their own conclusion is the honest one: multi-agent systems need tasks whose value justifies the spend, and they suit work with heavy parallelisation and information exceeding one context window. They explicitly do not suit domains where all agents need the same context, or where there are many dependencies between agents.

That second exclusion is the same point as this whole article, read from the other end. If your steps genuinely depend on each other, a graph buys you nothing and costs you a multiplier. The topology is not a performance trick you apply to any pipeline; it pays only where the dependencies were fake to begin with.

Token spend also compounds with what every node carries. Each branch re-sends its instructions on every call, so a bloated preamble is multiplied by fan-out rather than paid once — the token tax of oversized custom instructions is a per-node cost in a graph, which is a good reason to keep node prompts lean. Starting from something tested rather than a blank file helps.

When a graph is the wrong tool

Keep one agent in one loop when the task is short, when a single context holds everything relevant, when there are no independent branches, when failure is cheap, and when a person can check the result in a minute.

Reach for a graph when work genuinely runs in parallel, when different nodes need different tools or permissions, when outputs need independent verification, when a run has to survive interruption, or when cost and authority need controlling by route.

Start with the loop. Draw the graph when the dependencies force you to — and the point of the calculator above is that "force you to" is a measurable condition, not a matter of taste. It is also worth knowing which of your edges will age badly: an edge encoding a belief about how the model should think has roughly the shelf life of a prompt engineering trick, while one encoding a failure policy outlives every model on the price list.

The document-pipeline shape in the barrier example is not hypothetical, incidentally: it is exactly what an indexing run looks like when you build a local RAG assistant, where each document is fetched, chunked, embedded and verified independently. If you are delegating that kind of work to subagents, the habits that actually cut token spend apply here too — brief each branch precisely the first time, because re-briefing costs a round trip on the critical path.

Run it on your own pipeline

Write down every step, its measured duration, and — for each edge — the one question that matters: does this step read the previous step's output, or does it merely appear after it?

Delete every edge that fails. Run the calculator. The gap between total work and critical path is what your current topology costs you, in seconds, before you change a single prompt.

Most pipelines I have written this out for had three real dependencies inside a chain of twelve.