LLM guardrails: assert before you render

One of the products we build at Pineapples generates long financial plans for advisors. A deterministic engine does the math (projections, Monte Carlo runs, tax and insurance estimates), and a model writes the narrative around it. The result is a polished, multi-section document that an advisor may put in front of a client, so the last step is a set of LLM guardrails that validate the output before anyone reads it.

The failure mode that scared us most was not a crash. It was a plan that looks finished and is quietly wrong: a NaN in a chart axis, a leftover TODO, a placeholder number, or a narrative that quotes a different figure than the engine computed. Nobody skimming a 40-page PDF catches those.

So in 2026 we added a pre-render sanity layer. Nothing reaches the renderer until it passes.

What the output validation checks

The checks fall into a few buckets, and all of them are cheap:

Text artifacts. Scan the rendered text for things that should never appear in a client document: lorem, TODO, NaN, undefined, null, Infinity, internal case IDs.
Bounds. Definitional numbers have to be finite and plausible. A success probability is between 0 and 100. A net worth is not 10¹⁵.
Reconciliation. When two engines produce the same figure (the plan and the proposal both state an allocation, say), they have to agree.
PII gating. Accounts shown in a plan must belong to the household the plan is for.

In shape, it's just a list of rules that each return violations:

plan-sanity.ts

const HARD_TEXT_PATTERNS: [name: string, re: RegExp][] = [
  ['lorem', /lorem ipsum/i],
  ['todo', /\bTODO\b/],
  ['non-finite', /\b(NaN|Infinity|undefined)\b/],
]
 
export function planSanityCheck(plan: RenderedPlan): Violation[] {
  return [
    ...scanText(plan.text, HARD_TEXT_PATTERNS),
    ...runBoundsChecks(plan.figures),
    ...reconcile(plan.figures, plan.proposalFigures),
    ...gatePii(plan.accounts, plan.householdId),
  ]
}

If anything comes back, generation fails loudly: the document gets a terminal failed status with the reason attached, instead of sitting as a draft that looks fine.

The day the guard was wrong

A few days in, one plan on our demo environment failed on every attempt:

Placeholder/artifact leak detected in rendered content (sentinel-number).

The guard treated 123123 as a developer stub, the kind of number you type while building a form. We scanned the rendered text for it and also checked key figures against it.

It wasn't a stub. Someone had entered 123123 as the amount for every asset, liability, and income line on a demo client. Worse, the household's net worth legitimately computed to exactly -123,123 (six assets minus seven liabilities of the same amount). The number was real data, produced by real arithmetic.

The lesson was sharper than "fix the regex":

We removed numeric sentinels from the text scan entirely and dropped any sentinel value that legitimate arithmetic could reach. The bounds, non-finite, reconciliation, and PII checks stayed, and the regression cases became tests.

And the one hiding behind it

Regenerating that plan exposed a second bug. The loud-failure path set failed plus a reason, but a later successful run never cleared it. A complete, correct plan sat in the UI marked as failed.

The fix is one CASE in the write: a successful content write resets failed back to draft and clears the reason, while leaving review states like approved alone. The general rule: any status you set on failure needs an explicit path back.

Why this layer is worth it

Evaluating whether a financial plan is good needs an expert. Checking whether it contains NaN, disagrees with itself, or shows someone else's account is free, runs on every generation, and catches exactly the mistakes that make people stop trusting AI-written documents.