How good is the categoriser?
The categoriser is run over a labelled synthetic set. Ground truth comes from the Demo Bank generator and a hand-authored hard-case set, written without reference to the language model, so on that path the score measures agreement. The deterministic rules categoriser shares the generator's vocabulary, which flatters it here, and 74 of the labelled cases carry a description the prompt displays. Both are recorded, as F7 and F9. The page opens on the rules baseline over every labelled line; a live run scores a 40-line sample.
Categoriser accuracy
Rules baseline1059 / 1067 labelled transactions correct across 16 categories.
Each click runs the live model (Groq GPT-OSS 120B, with Gemini 2.5 Flash failover) over a 40-line sample. The free-tier providers are rate-limited, so if a fresh call is not available this falls back to the model's cached labels from an earlier run, which are still the model's own output. It falls back to the deterministic rules baseline when the cache is empty too. The model labels transactions. The credit decision is deterministic. Synthetic data throughout.
Read the sample figure and the 95.0% threshold as 2 different measurements. The sample takes every hard case first and fills the rest by striding the seeded lines, so hard cases make up 35% of 40 lines against 1.31% of the full set. That mix is chosen to expose failures and it scores roughly 11 points under the full run for the same categoriser. The threshold that governs a deploy is defined over all 1,067 cases, and the run behind it is reported below. Recorded as F11.
Per-category precision & recall
| Category | Support | Precision | Recall | F1 |
|---|---|---|---|---|
| Salary | 42 | 100% | 98% | 99% |
| Rent | 45 | 100% | 98% | 99% |
| Utilities | 119 | 100% | 100% | 100% |
| Insurance | 37 | 100% | 97% | 99% |
| Groceries | 412 | 100% | 100% | 100% |
| Transport | 102 | 100% | 100% | 100% |
| Subscriptions | 44 | 96% | 100% | 98% |
| Loan repayment | 25 | 100% | 96% | 98% |
| BNPL | 20 | 100% | 95% | 97% |
| Other credit | 12 | 100% | 100% | 100% |
| Discretionary | 135 | 100% | 99% | 100% |
| Transfer | 42 | 95% | 98% | 96% |
| Fees | 19 | 100% | 100% | 100% |
| Gambling | 13 | 100% | 92% | 96% |
Hard cases: 8 of 14 wrong
14 transactions written by hand so that the description alone does not settle the category. The current path gets 8 of them wrong. Each one below carries the label it should have had and the label it was given.
Irregular salary paid as a personal transfer, with no 'Gehalt' keyword to anchor on.
A BNPL instalment that calls itself an 'Abo', so it reads like a subscription.
An informal loan repaid to a family member, which looks like an ordinary transfer.
Gambling top-up behind a generic 'entertainment' merchant name.
Legal-protection insurance billed as a 'membership', which overlaps with subscriptions.
Rent to a private landlord via standing order, without the word 'Miete'.
A café inside a supermarket brand, so the true label is eating out.
A returned rental deposit, which is a credit and not income.
What this score does not measure
Ground-truth labels here come from the Demo Bank generator and a hand-authored hard-case set. They are independent of the categoriser, which keeps the score from being circular. What they measure is consistency against a known synthetic standard rather than absolute truth. A real evaluation needs human-reviewed labels with measured inter-rater agreement, a representative sample of live data, and per-category cost weighting, since mislabelling a loan repayment matters more than mislabelling a café. Treat the number on this page as a development signal.
Accuracy is lowest for the applicants a wrong label hurts most
The categoriser carries a declared threshold of 95.0% accuracy on the labelled set, with a stated consequence: fall below it and the deterministic categoriser carries the traffic. It reads 96.3%, so the threshold holds. Accuracy is not uniform underneath that average. The 2 groups it is lowest for are the ones a wrong label hurts most, and they are 32.8% and 12.7% below everyone else.
| Group | Cases | Accuracy | Below the rest | Can still fall to |
|---|---|---|---|---|
Irregular income and a thin file Income arrives as project transfers rather than a monthly salary line, over 2 months of history rather than 6. Both make the income harder to recognise and the pattern harder to confirm. | 32 3.00% of the set | 84.4% | 12.7% | 54.5% before the gate reacts, which is 14 of its 32 labelled wrong. |
Unusual statement vocabulary Salary paid as a personal transfer, rent to a private landlord with no recognisable word for it, credit dressed as a subscription. The label depends on reading intent rather than matching a term. | 14 1.31% of the set | 64.3% | 32.8% | 0.9% before the gate reacts, which is 13 of its 14 labelled wrong. |
| Everyone else | 1021 | 97.1% | — | This is the population the 95.0% gate is really watching. |
A group of 13 cases or fewer can drop to zero accuracy without this gate reacting at all. At 96.3% against 95.0%, that is what an average over 1,067 cases can absorb. The hard-case group is small enough to be inside that limit today.
The disparity above is a measurement rather than a prediction. It is what the model does now, and the threshold that governs it stays quiet. Recorded as F8.
How this was measured
One run of the whole 1,067-case set on 5 September 2026, answered by GPT-OSS 120B (Groq), scored group by group rather than in total. Group membership comes from the generator that wrote the statements, so it is a property of the data rather than a judgement about it.
It is one run. Repeated runs of this categoriser do not return the same answer, which is finding F4, so read a difference of a point or 2 as noise and a gap of 32.8% as real.
What the model is getting wrong
In the irregular-income group it is one failure repeated: project fees arriving as transfers are labelled recurring income or other, rather than salary. Income the affordability calculation should count is either miscounted or dropped.
The remedy for the blind spot is separate from the remedy for the gap. Set the threshold on the group that carries the risk, and the gap becomes something a deploy can react to.