LLM routing: race four models instead of guessing one

Every team using LLMs for coding eventually builds an LLM router. Cheap model for easy tasks, frontier model for hard ones. The problem is that "easy" and "hard" have to be decided before you've seen the answer, usually by a heuristic or, worse, by another model call.

In claude-dev-router we tried the opposite: don't predict. Race.

The idea: LLM routing by racing

Send the same task to several candidates in parallel: a small fast model, a couple of frontier models at low reasoning effort, and a baseline. Each run reports latency, cost, and a confidence signal. Take the fastest run that clears a confidence floor.

src/policy.ts

export function pickBeat(runs: RouterRun[], policy: Policy): RouterRun | null {
  const enabled = runs.filter(
    (r) => policy.enabled.includes(r.routerId) && !r.error,
  );
  if (policy.race === "all") return null;
  const cleared = enabled
    .filter((r) => r.cleared)
    .sort((a, b) => a.latencyMs - b.latencyMs || a.routerCostUsd - b.routerCostUsd);
  return cleared[0] ?? null;
}

A few things fall out of this shape:

Errors are excluded, not ranked. A failed run never wins because it was fast.
Cost is the tiebreaker, not the goal. When two runs clear the bar at the same latency, the cheaper one wins.
Returning null is a valid answer. If nothing clears, the caller knows it has to escalate or wait for the full set. That's more honest than returning the least-bad option.

Why not just use the best model?

Because for a large share of everyday coding tasks, the small model is right, and it's right first. A slow, shaky frontier answer should never beat a fast answer that passed. Racing makes that automatic.

What it costs

You pay for parallel calls you'll throw away. That's real money, and the benchmark harness in the repo exists precisely to measure whether the latency win justifies it for a given workload. For interactive developer tooling, where a human is waiting, it often does. For batch jobs, it usually doesn't.