The categoriser & how it is evaluated
A language model reads the statement and labels what each transaction is. It produces structured categories and grounded rationale. It never produces the lending decision. Its quality is then measured against an independent labelled set.
The model labels transactions and deterministic code makes the decision
It categorises and explains
It never decides
The fixed taxonomy every transaction resolves to
The same taxonomy is shared by the data generator's ground-truth labels and the model's output, which makes them directly comparable.
A validated output schema
The model is constrained to a typed schema, validated on the way out. A response that does not parse is rejected rather than trusted, so malformed or hallucinated structure never reaches the engine.
Measured against an independent ground truth
The categoriser is run over a labelled synthetic set and scored on accuracy, per-category precision and recall, and a confusion matrix.
Why the measurement is non-circular
The labelled set is produced by a generator that assigns each transaction its true category at creation time, before the model sees anything. Ground truth therefore comes from a source entirely independent of the categoriser. The model is graded against labels it had no hand in writing, so a high score reflects agreement with the generator rather than the model marking its own work.
The limitation, owned plainly
Synthetic labels measure consistency against a known generator rather than absolute truth. A model that perfectly matches the generator has learned to agree with the generator's conventions. That is a far easier target than the genuine ambiguity of real statements. An ambiguous transfer or an unlabelled standing order has no single objectively correct answer.
A production system would be evaluated on human-reviewed labels drawn from real data, with inter-rater agreement reported so the irreducible disagreement between expert annotators is visible alongside the model's score. This demo is explicit that it measures agreement with the generator.