Cadence
Learn

How the AI is evaluated

A language model is non-deterministic, so the same prompt can return different answers. In a regulated lending context that carries risk. The 2 things the model does, categorising transactions and explaining the decision, are each measured against explicit standards, run by hand before a deploy.

Why

What the evaluations are for

The model can change

The prompt, the model and the retrieved context can all change. Evaluations measure what each change does, so a swap to a cheaper model or a tightened prompt is decided on a number instead of an impression.

The domain is regulated

In a lending decision a wrong figure or an unprompted recommendation carries regulatory weight. The checks below produce a record of what passed. Running them is a person's job here, as is acting on what they report.
Scope

Every place a model writes text is held to a written standard

The credit decision is deterministic code over the categorised totals, so it falls outside these evaluations. The places a model produces text are held to the standards below.

The categoriser

Reads each raw transaction and labels it with a category plus income, recurrence and obligation flags. It converts free-text statement lines into fields the engine can total.

The rationale

Puts the computed decision into plain language for a loan officer. Every sentence must follow from figures the engine already produced. The decision is taken before this runs.
Method

Each check runs in the cheapest way that fits it

Definitive things are code, subjective things are a model acting as judge, and the hardest calls are human. Production systems normally use these together.

Code

Deterministic and cheap. Used when the answer is definitive: a figure that must trace to a source, a banned phrase, a length band. Runs with no model and no API key.

LLM-as-judge

A separate model scores the subjective dimensions that code cannot reach: relevance, balance and tone. It returns a pass or fail, a score from 1 to 5, and a written reason.

Human

A subject-matter reviewer for new features, calibration, and the cases the automated layers flag. Costly and slow, so it is reserved for the calls that genuinely need judgement.
Live today

The categoriser evaluation

The labelled set is produced by a generator that assigns each transaction its true label at creation time, before the model sees anything. The score measures agreement with labels the model had no hand in writing.

What is measured

The categoriser runs over a synthetic set drawn from 8 personas plus 14 hand-authored hard cases, the deliberately ambiguous lines that a naïve rule gets wrong. It is scored on overall accuracy, per-category precision and recall, and a full confusion matrix, with a deterministic offline harness that fails the build if intended outcomes regress.

Open the live model eval
By design

The rationale evaluation

These dimensions define the standard the rationale must meet. Compliance, groundedness and outcome-faithfulness are hard: a breach fails the run. The judged dimensions are advisory. This is the design the checks implement, and applying the result is a decision a person takes.

GroundednessCodeHard gateEvery figure in the text must trace back to the decision package, allowing for rounding and derived values such as debt-to-income. A fabricated number fails.
Outcome faithfulnessCodeHard gateThe outcome the text states must match the outcome the engine computed. A rationale cannot narrate an approval over a decline.
ComplianceCode + judgeHard gateNo advice, no recommendation to lend or decline, no reasoning from protected attributes. A curated deny-list gates it; the judge adds a second pass for paraphrased advice.
CompletenessCodeGateThe rule that drove the outcome has to appear in the explanation. A decline on affordability must say so.
StructureCodeGateA readable length band, and the core affordability figures. Income and what remains after costs must both be present.
Relevance · balance · toneLLM-as-judgeInformationalIs it clear, does it weigh both sides fairly, is it professional? Scored 1 to 5 with a reason. Advisory, and the outcome stays with the deterministic checks.
Cadence

Offline before a deploy, online after

Evaluations are not run once. In a mature setup they gate a deploy and then keep watching production. Neither layer is automated here.

Offline, the pre-deploy check

Runs over the full labelled set before a deploy. Hard checks are meant to block a deploy on a breach, and the remaining dimensions are tracked over time. Here they produce a pass or a fail that a person acts on: scripts/verify-rationale.ts is run by hand, and the categoriser threshold is read by its owner, who decides whether the deploy goes ahead. The checks also serve as the written specification for how the feature should behave.

Online, the standing watch

In production a system samples real traffic, tracks latency and feedback, and watches the input distribution for drift. Failures found here are added to the labelled set, which tightens the offline check over time. This layer is not built here.

The numbers described here can be regenerated. The model evaluation page runs the categoriser over the labelled set on demand and shows accuracy, per-category precision and recall, the confusion matrix and the hard cases it fails.

The dataset

The labelled evaluation set

A fixed set of inputs paired with known-correct outputs, held stable so a system can be measured against it repeatedly and compared across versions. The usual name for one is a golden dataset, which this set is not called here because it meets some of the properties below and not others. Cadence holds 1,067 labelled cases: 1,053 generated statement lines and 14 hand-authored hard cases.

Authoritative labels

An authority other than the model under test decided the correct answer. The Demo Bank generator stamps a label on every line it writes, and 14 hard cases carry a considered human label instead.

Independent of the system under test

Labels must not come from the thing being measured. The generator is written without reference to the language model, so on that path the score measures agreement. It is not independent of the deterministic rules categoriser, which was written to the vocabulary the generator emits, so the rules score 99.3% here and the comparison flatters them. Recorded as finding F7.

Frozen

A seeded generator regenerates identical transactions on every run, and the set lives in version control. A score from March is comparable to a score from September because the inputs cannot have changed.

Representative

It should cover the real distribution including the awkward cases. This is the property the set meets least well, and the 14 hard cases exist to push against it.

Held out

The few-shot prompt and the rules should stay clear of the set, or leakage inflates the score. Measured rather than assumed: 5 of the 6 prompt exemplars use a description that also appears in the set, covering 74 of 1,067 cases, and on all 74 the label the exemplar shows is the correct one. No exemplar matches a case on description and amount together, so the answers are not copied in, and 23.1% of the set opens with a description stem the prompt displays.

Where a labelled set is possible it beats a judge, because scoring against known answers is deterministic and a judge's opinion is not. The judge exists for the rationale, which has no label to mark against.

The limit worth stating plainly: the generator writes both the transaction and its label, so any bias in the generator is invisible to the evaluation. If it writes German salary lines in a shape real banks never produce, the categoriser scores well on data that does not exist. The 14 hard cases are the only genuinely human-labelled part, and there is one rater, so there is no inter-rater agreement.

Limits of the judge

Known failure modes of a model used as a judge

A judge score is an estimate with error attached. The same questions apply to it as to any model in production: which version ran, has it been validated, and against what.

Self-preference

Models score their own family's output higher. With a failover chain the same model can write a rationale and then grade it, which is self-evaluation. Cadence does not record which model judged, so the pairing cannot currently be checked.

The instrument drifts

A change in the judge model makes a score from March incomparable to one from September. The control is to pin the judge version and record it against every score.

An unvalidated judge is decoration

A judge score means something once its agreement with human labels has been measured on a sample. That measurement has not been done here, so the judge dimensions stay advisory.

Position and verbosity bias

Judges tend to prefer whichever answer came first in a comparison, and longer answers over shorter ones. Both push scores around for reasons unrelated to quality.
Findings

What instrumenting this system exposed

Adding tracing to the 3 model call sites surfaced 7 control weaknesses, each recorded before being fixed. They live with the rest of the audit material.

Several exist only because the tracing is there. The application reports success on a run in which the primary provider failed 24 times, so the code's own account of what happened was not enough to find them. The full register, with condition, evidence and disposition for each, is in Audit › Findings, alongside the control tests.
Honesty

What these numbers do and do not prove

Synthetic personas measure consistency against a known generator rather than absolute truth. That is an easier target than the ambiguity of real statements. A deny-list catches overt advisor language and misses paraphrase, which is why a judge runs behind it. The offline gate proves the deterministic path and the checks themselves are sound. It does not, on its own, certify every answer the live model gives.

A production system would add human-reviewed labels from real data, measure inter-rater agreement, and run the online layer described above. None of those are in place here.

Next
Observability