Layer 3 · Observability

Recording what the model did

Every model call emits a span carrying the provider, the model id, the prompt version, tokens, latency and the session. The application's own logs record that a call succeeded. The traces record which model answered and how many attempts it took.

The stack

Langfuse, OpenTelemetry and the AI SDK

Why each component is here.

Langfuse

The tracing backend, on its EU region. Open source, so a bank can run it inside its own network. Traces carry prompts, model output and customer transaction descriptions, which makes the hosting location a third-party risk question.

OpenTelemetry

The wire format. Spans are standard telemetry, exported through a span processor, so the backend can be swapped without changing application code.

Vercel AI SDK

Where the spans originate. Each call carries a function id, so categorisation, rationale drafting and the judge stay in separate groups.

With no credentials present the span processor is inert and the application runs unchanged. An unreachable backend cannot fail a request.

Cost of being public

Visitor traffic is traced up to 300 model calls a day, then not at all

A public demo that traces everything spends an observability quota the audit evidence also draws on, and a percentage sample sets no ceiling, because it scales with whatever traffic arrives.

The budget is an absolute count held in one row of the database, claimed with a single statement so 2 requests arriving together cannot both decide there is room. Once the day is spent, tracing switches off and the application carries on. It is claimed once per model call rather than once per failover attempt, so a retry does not spend the budget several times over.

Evidence runs are never counted against it. They set CADENCE_TRACE_ENV=audit and are the reason the budget exists.

This deployment reports environment demo and tracing is connected.

Evidence

Real traces, readable without an account

The tool is behind a login and the deployed site carries no keys, so this work could be described here without anyone being able to see it. These rows come from actual runs, and each links to the trace itself.

Exported 2026-09-07 from runs of this system. The statements inside are synthetic, written by the Demo Bank generator. A trace is readable without a Langfuse account where it was declared public as it was written. The rows that open carry a link, and the rest are marked as not shared.

Attachment	Answered by	Prompt	Tokens in / out	Latency	Cost recorded	Trace
decide-application 2026-09-07	groq, gcp.gemini openai/gpt-oss-120b 1 failover	rat-2026-09-05.1	5,409 / 14,683 6 model calls	67.1s	$0.00000	not shared
decide-application 2026-09-07	groq, gcp.gemini openai/gpt-oss-120b 1 failover	cat-2026-09-05.1	5,376 / 17,168 6 model calls	82.4s	$0.00000	not shared
judge-rationale 2026-09-07	groq openai/gpt-oss-120b	judge-2026-09-05.1	839 / 401 1 model call	8.2s	$0.00000	Open
build-rationale 2026-09-07	groq openai/gpt-oss-120b	rat-2026-09-05.1	398 / 349 1 model call	10.5s	$0.00000	Open
categorise-transactions 2026-09-07	groq openai/gpt-oss-120b	cat-2026-09-05.1	881 / 1,132 1 model call	10.0s	$0.00000	Open
judge-rationale 2026-09-07	groq openai/gpt-oss-120b	judge-2026-09-05.1	782 / 231 1 model call	0.8s	$0.00000	not shared
build-rationale 2026-09-07	groq openai/gpt-oss-120b	rat-2026-09-05.1	398 / 324 1 model call	1.0s	$0.00000	not shared
categorise-transactions 2026-09-07	groq openai/gpt-oss-120b	cat-2026-09-05.1	881 / 1,448 1 model call	3.3s	$0.00000	not shared
regenerate-rationale 2026-09-06	groq openai/gpt-oss-120b	not versioned yet	6,921 / 13,449 6 model calls	65.0s	no price registered	not shared
live-eval 2026-09-05	groq openai/gpt-oss-120b	not versioned yet	1,442 / 3,072 1 model call	6.8s	no price registered	not shared

The cost column reports a price setting. Both providers run on free tiers, so no run in this table was billed. The column reports 3 different things: 2 rows where no price is registered for the model, 8 where one is and it is zero, and 0 where a published list price was applied to a request that was never charged.

Costs are computed when a trace is written, so a row keeps whatever price was registered at that moment and no later edit reaches it. Both models now carry an explicit zero, which is why the recent rows read $0.00000 and the older ones do not. The underlying point stands: cost is the wrong axis for ranking these 2 providers while both tiers are free, and what actually binds each one is a quota. Recorded as F5.

What a trace carries

What each field records

All 3 model call sites emit the same 5 fields.

Which provider answered

gen_ai.provider.name, with the model id beneath it. The chain fails over, so the answering provider varies per call and cannot be read off the configuration.

Which of the 3 attachments ran

A function id per call site, so categorisation, rationale drafting and the judge stay in separate groups.

Prompt and model version

The change record. Without it a stored label cannot be tied to the prompt that produced it.

Tokens, latency, temperature

Cost and behaviour. Temperature is recorded because it explains why repeated runs return different answers.

Session

The same identifier the application scopes its writes by, so a trace joins back to the application, decision and audit rows.

Decisions worth explaining

What was settled before collection started

Evidence is separated from demo traffic

Anyone can click through a public demo. A visitor pressing override changes the override rate a reviewer would cite. Every trace carries an environment field, set to audit on evidence runs and demo otherwise, so the 2 populations filter apart. The field had to exist before collection started, since adding it later would invalidate everything gathered up to that point.

The override is recorded as a score

A loan officer changing the outcome makes no model call, so it produces no span. The override is written as a score against the session, with the engine outcome and the officer outcome both recorded. An override rate can be computed from those.

What it caught

What the traces caught that nothing else did

A run the application called a success had failed 24 times

One evaluation over the labelled set produced 24 error observations out of 88, all the same cause: the primary provider hit a rate limit of 8,000 tokens per minute and the work passed to the paid fallback. The run completed, reported an accuracy, and said nothing. A single evaluation had therefore been answered by 2 different models, so the headline figure was a blend of both. F6

Every row claimed a provider that had not answered

The trace recorded one provider while the application stamped another on every row it returned, in the same response. The field had been a hardcoded literal rather than the provider that actually replied. F1

Limits

Where this stops short

Tracing is off in this deployment. No credentials are set here. Public traffic would consume the 50,000 unit monthly quota and mix demo activity into the evidence set. Everything described above runs locally.

A trace records that a model returned a category. Whether the category was correct is measured by the evaluation harness.

Cost attribution reports a price setting. Both models now carry an explicit price of zero, so a new generation records $0.00000, and the older rows keep the price that was registered when they were written. Neither tier has ever charged, and what binds each provider is a quota. Recorded as F5.

AI Transparency