AI agent testing: catch the failures that look like success

We're building an AI-native product for M&A diligence. Instead of one general chatbot, each stage of the work has its own named agent, built around the document a professional would produce: a quality-of-earnings review, a letter of intent, an investment thesis, a document-review workpaper. Each has its own prompt, budget, and deadline, and our AI agent testing starts with one invariant per agent.

That structure made one kind of testing obvious. The comment at the top of the file says it better than I can:

agents/invariants.ts

// What each agent must be true of its own output, regardless of the deal.
//
// This is the cheapest useful layer of evaluation and most of the value in it.
// "Is the analysis good" needs a partner; "did the QoE lead with the seller's
// number instead of the standard-only one" is a property, and a property can be
// checked for free on every run forever.

Fluent and wrong

The dangerous failure for an agent that writes professional documents isn't gibberish. It's a fluent, well-formatted document that is wrong in the one way the agent was built to prevent.

A quality-of-earnings review that opens with the seller's adjusted earnings figure, rather than the independently normalized one, has quietly adopted the seller's case and dressed it up as diligence. Same sections, same citations, same tone. The only difference is which number a reader anchors on, and that's the whole point of the document.

A reviewer skimming it won't notice. An invariant will.

export interface Violation {
  rule: string
  detail: string
  severity: 'fail' | 'warn'
}
 
export function checkQoe(output: string): Violation[] {
  const headline = headlineRegion(output)
  if (!mentionsStandardFigureFirst(headline)) {
    return [{
      rule: 'qoe-leads-with-standard-figure',
      detail: 'Headline anchors on the seller-adjusted number',
      severity: 'fail',
    }]
  }
  return []
}

Each agent gets a handful of these, and each one exists because it describes a specific way that agent can fail while looking fine. A violation isn't "low quality." It's the agent doing the exact thing it was written not to do, and it blocks a release.

Invariants have bugs too

Two mistakes from our first version are worth passing on.

Refusals are successes. Some agents are designed to decline when the data room can't support an answer. A refusal is short and emits no structured payload, which is correct. Our first invariants scored those as empty outputs and flagged three agents that had behaved exactly as designed. Now there's an explicit isRefusal() check, and a refusal counts as a terminal success.

Check the right region. The "leads with the standard figure" rule originally looked only at the text before the first heading. But the document opens with a title and some progress notes, and the numbers live in the first section. The rule failed on every reference deal while every one of them did lead with the right figure. The check was reading the breadcrumbs.

Where invariants sit among LLM evals

Invariants don't replace expert review or golden-set evals. They sit underneath them. They're deterministic, they cost nothing, and they catch the category of failure that erodes trust fastest: the confident, polished answer that's wrong in the one place it mattered.