Local explainer

Evaluators & Scores — Langfuse vs CtrlB

How automatic checkers grade chatbot answers, how scores are stored, and where Langfuse and CtrlB match or differ — told through Acme’s support bot.

Rounds 1–8 and phase 3 (whole-trace evaluators) are built and were tested end to end on local Seeker + Orion with ~2,300 real DeepSeek traces (2026-11-18/19). Not on staging yet. What we built. Final additions the same day (D24–D34): a cost estimate from real runs, read-only MCP tools, safer 10% rule defaults, a Scores tab in the AI trace drawer and small UI fixes — final changes. Judge runs now live in their own traces (separate mode, verified locally).

38behaviours compared
14identical
3missing
~59%weighted parity

The big picture in one line

From chat to grade to “does the checker agree with people?”

A customer asks Acme’s support chatbot: “Can I pay with crypto?”

Acme’s docs say nothing about crypto. The bot wrongly answers: “Yes, Acme Cloud accepts crypto payments.” The judge should mark it false: “made up, the docs don’t mention crypto.”

1

App sends a copy of every chat

2

An AI or code checker grades each answer

3

The grade is saved next to the chat

4

People check a few by hand

5

Analytics shows if the checker agrees with people

Who does what

The cast of Acme’s crypto question.

Role What it is In the Acme story
Customer Asks the question. “Can I pay with crypto?”
The bot The company’s own app / code. Answers the customer. Acme’s support chatbot app
AI model as the bot’s brain Writes the answer (e.g. DeepSeek). Produces “Yes, Acme Cloud accepts crypto…”
SDK inside the app Langfuse/CtrlB SDK inside the app. Records each step and sends it. Sends the question, docs lookup, and answer
Langfuse / CtrlB server Stores the data and shows it in a UI. You open the crypto chat later
Evaluator Grades the answer later (automatic checker). Marks the crypto answer false
AI model as the judge A separate AI call, made by the server using the LLM connection. Says: made up — docs don’t mention crypto
Same model, two jobs

DeepSeek can play two roles: the bot’s brain and the judge. Those are two separate calls.

Trace & span

One customer question becomes one tree of steps.

  • Trace = everything for one question.
  • Span / observation = one step inside that question.

Example: Customer asks about crypto. That is one trace with three spans:

Span What it is Why it matters
support-chat Root span — the whole request. Holds the full chat as one unit.
retrieve-docs Looked up help articles. Found none about crypto. Shows the bot had no docs to lean on.
answer-llm The AI call: input = system prompt + docs + question; output = answer; plus model and tokens. This is usually what the judge grades.
Trace tree: support-chat with retrieve-docs and answer-llm support-chat (root) Trace: crypto question retrieve-docs Looked up help articles Found none about crypto answer-llm Prompt + docs + question Output: “Yes… crypto” Judge usually grades answer-llm — if docs aren’t in the span, it can’t check them

Animated dashed lines show the tree. Respects reduced-motion settings.

How the data arrives

  • The app uses the Langfuse SDK, built on OpenTelemetry.
  • It creates spans with input/output attributes.
  • It exports them over OTLP/HTTP to the server.
The tool only sees what the app sends
  • If docs are not in the span, the judge cannot check against them.
  • Same for CtrlB: CtrlB also only receives what the customer’s app sends.
  • Example: Acme’s app must put the (empty) docs into answer-llm input, or the judge has nothing to compare.
  • OpenTelemetry = common standard for recording steps in software.
  • OTLP/HTTP = the wire format used to send those steps to the server.

How the bot knows the answer

Find articles → send them with the question → AI writes a reply.

  1. What it is: The bot finds help articles, then sends question + articles + instructions to the AI. The AI writes the reply.
  2. Example: For “Can I pay with crypto?”, retrieve-docs finds nothing. The AI still answers “Yes… crypto” — it made that up.
  3. Why it matters: No matching docs + sloppy instructions → the bot may invent facts.
Hallucination in one line

Empty docs + weak “don’t invent” rules → wrong confident answers like Acme’s crypto reply.

Evaluator & Score

Set up a checker once. Stick a grade on every matching answer.

Evaluator

What: An automatic checker you set up once.

Example: “Is the answer grounded in Acme docs?” — runs on every answer-llm.

Why: Thousands of chats. Nobody can read them all.

Score

What: The result stuck on the answer — a value plus a reason.

Example: false — “made up, the docs don’t mention crypto”.

Why: You can filter “show all false” and fix those first.

Three kinds of checker

LLM-as-a-judge, decision model, or your own code.

LLM-as-a-judge Flexible

What: You write instructions with {{input}} and {{output}}. The tool fills them with saved span data and sends them to an AI. The AI replies {score, reason}.

Example: Instructions say “mark false if the answer invents payment methods.” Crypto answer → false + reason.
  • Good for fuzzy questions.
  • Costs money.
  • Can be wrong.

How the grade is decided: 0.1 vs 1. The form: evaluator form.

Decision model Structured

What: Fixed short questions (choice / scale / yes-no). Answers with how sure it is (e.g. yes, 92%). One call for all questions.

Example: “Is crypto mentioned in the docs? yes/no” → no, 95%.

How to set it up, and why only some models work: Decision model — how to set it up.

Code Deterministic

What: Your own function evaluate(ctx) returns scores. No AI.

Example: “Is the answer longer than 300 characters?”
  • Free.
  • Instant.
  • Same result every time.

How a rule runs: Code evaluator — how it works.

How the judge decides 0.1 vs 1

You define “correct.” The AI compares the answer to your rules and the docs.

  • What: YOU write the instructions and choose the score type. The AI follows those rules.
  • Example: Scale below. Crypto answer with no docs → 0.0 (made up).
  • Why: The AI’s score is an opinion, not a fact. Bad rules → bad grades.
You are grading Acme’s support answers.

Rules:
- Only facts from the provided docs are allowed.
- If the docs are empty or silent, the answer must say so — not invent.

Scale (numeric 0–1):
  1.0 = all claims come from the docs
  0.5 = half guessed
  0.0 = made up

INPUT (docs + question):
{{input}}

OUTPUT (bot answer):
{{output}}

Return {score, reason}.
Opinion, not truth

Two judges (or two prompts) can disagree on the same crypto answer. That is why humans still score a sample.

Rules: which spans get checked

Filter + sampling + which evaluators. New matches run automatically.

  • Filter — e.g. only answer-llm spans (skip retrieve-docs).
  • Sampling % — e.g. 100% while testing, 10% to save money.
  • Which evaluators — attach one or more checkers to the rule.
  • Optional run on past data — backfill old chats (e.g. last week’s crypto questions).
  • Automatic: new matching spans get checked without you clicking each time.

The screens we filled in: Evaluator vs rule and the rule form.

  • Langfuse sampling: sha256 of the span id, per observation.
  • CtrlB sampling: fnv32 of the trace id, per trace. Different method — not a bug.
  • Both: rules use the latest evaluator version; several evaluators per rule (CtrlB max 10).

Score types

Pick a type — and for numbers, set a range.

Numeric

A number. Set a range like 0–1.

Crypto groundedness: 0.0

Boolean

True or false.

answer_correct = false

Categorical

Your own labels.

Pass / Fail, or “hallucination / ok / unsure”

Text

Free-form notes.

“Docs silent on crypto; bot invented payment method.”
New numbers start at 0–1

A new numeric judge starts at 0–1, same idea as Langfuse. An older judge that still has no range shows a warning — the judge can invent a scale. That was bug B1 (fixed in round 4, not on staging yet).

Example: before the fix, the naming judge returned 0.9, 9, and 60 for the same kind of answer.

Who can give a score

AI, a person, or the app. All three can sit side by side.

Who Source What Example
AI checker EVAL Automatic evaluator false — “made up, docs don’t mention crypto”
A person in the UI ANNOTATION Needs a score config first Human marks the same chat false
The app via API API Your product sends a score Customer clicks 👎 on the crypto reply
  • A person’s score does not replace the AI’s score. Both are kept side by side.
  • Score config = allowed name + type + range/labels. Example: answer_correct = Boolean.
  • Both products add 2 configs: open coding (Text) and pass or fail (Pass=1, Fail=0). CtrlB adds them the first time a workspace opens Score configs, and does not add them again if they were archived.
  • In Langfuse, Annotate scores the whole trace even if a span is open. What the screen shows: Trace scores vs span scores.
  • A score belongs to exactly one thing (trace, span, session, or dataset run). It does not spread.
  • Tags are different: tags belong to the trace and the app’s code copies them to every span.
CtrlB now (rounds 5 and 8) — built and tested locally, not on staging yet
  • EVAL — the judge. ANNOTATION — a signed-in person. API — an API key.
  • Scores API: POST / PATCH / DELETE ai/scores. The author or an admin can edit or delete.
  • A score can sit on the whole trace (empty span id) or on one span.
  • They share the lake dataset, with scope ctrlb.scores, so they do not change evaluator health.
  • The Scores page shows the real Source. The Author column is there, hidden until you show it.
  • Analytics can compare “name · EVAL” with “name · ANNOTATION”.
  • Score configs live in Mongo: name, type (NUMERIC, CATEGORICAL, BOOLEAN, TEXT), range or labels, description, archive. The type cannot change later.
  • Evaluators has a 5th tab, Score configs. The drawer can “Copy from evaluator”. Two defaults: “open coding” (TEXT) and “pass or fail” (Pass=1, Fail=0).
  • AI Traces has an Annotate panel: the whole trace or one span, pick configs, one input per type, then “Scores on this trace”.

Example: a reviewer marks the crypto chat false. The judge’s score stays beside it. Queues come later (phase 2). Detail: What we built.

Why humans still score

Trust the AI only after it agrees with people on a sample.

  • What: AI can be wrong. People score a small sample (20–50), usually the whole trace.
  • Example: Score 30 crypto-style chats by hand. Compare to the judge.
  • Why: Agree → trust AI for the rest. Disagree → fix the AI’s instructions.

Scores API use cases (beyond 👍/👎):

  • Business events (ticket opened, purchase).
  • Your own nightly checks.
  • CI tests that block a bad release.
  • Updating a score by sending the same id.
  • Exporting scores.

Analytics — one score

Totals and shape for a single checker.

  • Total — how many scores.
  • Mean — sum ÷ count. Example: 1, 0, 1, 1 → mean 0.75.
  • Std dev — how spread out. 0 = all the same.
  • Mode and Mode % — most common value and its share.
  • Trend over time — is quality rising or falling?
  • Distribution — how values are spread.
Acme example

If groundedness mean is 0.4 this week, many answers are half-guessed or worse — like the crypto reply.

Analytics — comparing two scores

Same data type only. Matched = items that have both scores.

Type must match
  • Numeric vs numeric, boolean vs boolean, categorical vs categorical.
  • CtrlB only lists scores of the same type as the first one; if there is no other score of that type, the second dropdown is empty.

Boolean / categorical

  • Agreement = same answers ÷ matched.
  • Cohen’s κ = (agreement − chance agreement) ÷ (1 − chance agreement). 1 = perfect, 0 = luck.
  • F1 = 2 · precision · recall ÷ (precision + recall).
  • Confusion matrix — diagonal = agree.
Real Langfuse test — qa_review vs user_feedback

Both sent by our script via the API (not live customers).

20
Matched
75%
Agreement
≈51%
Chance agree
≈0.49
κ (moderate)
0.748
F1

κ = (0.75 − 0.51) ÷ 0.49 ≈ 0.49.

Confusion matrix:

QA false
QA true
Customer false
9
both false
2
cust F / QA T
Customer true
3
cust T / QA F
6
both true

Try it — confusion matrix calculator

The dangerous box

Customer true / QA false (3 in the example): the customer is happy with a wrong answer — like liking the crypto reply when docs never said crypto. Silent failure.

Numeric

  • Pearson r — move together in a straight line (−1…1).
  • Spearman ρ — same ranking order.
  • MAE — average of |A − B|.
  • RMSE — √(average of (A − B)²). Big gaps count more.
  • Heatmap — diagonal = agree.
Real example — quality_qa vs quality_user
20
Matched
0.898
Pearson (strong)
0.931
Spearman (very strong)
0.119
MAE
0.141
RMSE

Note: these ratings were random numbers made by our test script to demo the page, not judge output.

Try it — MAE / RMSE playground

#Score AScore B
1
2
3
4
5

Strength labels

Pearson / Spearman: >0.9 very strong 0.7–0.9 strong 0.5–0.7 moderate <0.5 weak
Cohen’s κ: >0.8 almost perfect 0.6–0.8 substantial 0.4–0.6 moderate 0.2–0.4 fair <0.2 slight
  • Judge vs human
  • Judge vs user feedback
  • Human vs human
  • Judge v1 vs v2
  • Judge on model A vs B

Setting up the judge model

We added DeepSeek as the model that grades answers.

  • Place: Settings → LLM Connections.
  • Adapter: openai. Base URL: https://api.deepseek.com.
  • Custom models: deepseek-flash and deepseek-v4-pro.
  • “Enable default models” was OFF.
  • We tried it in the Playground before using it as a judge.
Example

The connection is just “which AI to call.” The grading instructions come later, on the evaluator.

The evaluator form, field by field

What you fill in before a checker can grade anything.

  1. How to judge. Pick LLM-as-a-judge, Code evaluator, or Decision model, then “with <model>”.
  2. The prompt. Write instructions with blanks {{input}} and {{output}}. “+ Add message” adds another message.
  3. Score shape. “Return a [numeric / boolean / categorical] score.” Numeric is already filled in as “between 0 and 1”.
  4. Map the blanks. {{input}} → Input. {{output}} → Output. The pencil changes that to metadata, tool calls, or a JSON path.
  5. Name and description. On the right, “Test with sample observations”: filter, pick a row, then “Run test on this sample.”
Example

Name the checker “grounded”. Map both blanks. Pick the crypto row. “Run test on this sample” shows the grade before you save a rule.

Where {{input}} and {{output}} come from

The form does not invent the chat. It pastes what was saved earlier.

  • When the app called the AI, the SDK saved the exact messages as that span’s Input, and the reply as its Output (for example langfuse.observation.input / output).
  • Langfuse stores those as columns.
  • Picking a row in the form copies that saved Input and Output into the blanks.
  • Nothing is generated or fetched at grading time.

Like mail merge: the prompt is the letter, each span is a spreadsheet row, and the mapping is which column fills which blank.

  • Other OpenTelemetry setups use standard names (for example gen_ai input and output messages). Langfuse maps those to Input and Output.
  • Unknown names land in metadata. Change the mapping so the blank reads metadata instead.
Example

Select the crypto row and {{input}} becomes the messages already stored on that span — not a new lookup. Which span actually contains the docs: which span to grade.

Which span to grade

Grade the step that contains the docs, not the root.

What the steps are called: Trace & span.

  • The root span support-chat only has {"question"}. No docs. The judge cannot check the answer against docs.
  • The answer-llm GENERATION span’s Input has the system prompt, “Docs: …”, and the question. Grade that one.

Input, system: “You are a cheerful assistant… fill gaps with your best guess.”

Input, user: “Docs: - (no matching docs) Question: Can I pay with crypto?”

Output: “Yes! Acme Cloud accepts crypto payments—typically BTC, ETH, and USDC—at checkout.”

“(no matching docs)” is a trap on purpose

There is no crypto article. A good judge marks the “Yes!” answer as made up.

Observation types

Langfuse labels each step. The app chooses the label when it records the step.

TypeWhat it isExample
SPANA general stepsupport-chat
GENERATIONAn AI model callanswer-llm
RETRIEVERA searchretrieve-docs
TOOLA tool, API, or MCP calla pricing API call
AGENTAn agent decisionthe agent picks the next tool
CHAINA fixed sequencestep A then step B
EMBEDDINGText turned into vectorsembedding the question
GUARDRAILA safety checkblock a risky reply
EVALUATORAn eval step recorded in the tracethe judge run itself
EVENTA single moment, no durationa click

Filter quick buttons:

  • Root spans = isRootObservation:true
  • Generations = type:GENERATION
  • Tools = type:TOOL
  • Experiments
Root spans + Generations together = 0 matches

A generation is never the root. type:GENERATION picks only the AI-call span of each trace (40 traces → 40 answer-llm spans), not the whole trace.

CtrlB has no Langfuse “type”

CtrlB uses gen_ai_operation_name and the span name. Rule filters can use any span field.

Evaluator vs rule

The checker and the switch are two different things.

  • Evaluator = how to judge: instructions, model, and score type.
  • Rule = what and when: the filter, the sampling %, which evaluators, a mapping per evaluator, “run on existing data”, and on/off.
  • No rule → it never runs on live traffic. The status we saw: INACTIVE, then HEALTHY.
  • One rule can run several evaluators. One evaluator can sit on several rules.
  • Judge instructions belong to one evaluator. They are not shared. Copy them, or start from a template.
  • One evaluator = one mode + one set of instructions + one score type.
  • Exceptions: a code evaluator can return several scores (how). A decision model can ask several questions (how).
Example

“Grounded” (the instructions) does nothing until a rule says “run it on answer-llm, 100% of the time.”

The rule form

This is the switch that makes an evaluator run.

  • Configure rule scope — the filter, a preview of matching observations, and a sampling slider.
  • Attach evaluators — shows “2/2 variables mapped” and an estimated LLM cost per week.
  • Then a name, and “Save and activate”.
A $0.00 cost can mean “no price”, not “free”

Langfuse shows $0.00 when it has no price for the model (for example deepseek-chat). Add the price under Settings → Model Definitions. CtrlB’s estimate returns “unknown” (known:false, cost null) instead of $0.

DeepSeek as the judge — what we learned

Two connections failed. One model name worked.

Test 1 — OpenAI adapter — failed

Error: “This response_format type is unavailable now.”

  • Langfuse always sends strict structured output (response_format: json_schema).
  • supportsStructuredOutputs: true is hard-coded for OpenAI-compatible base URLs.
  • DeepSeek does not support json_schema on any model. We tried /, /beta, flash, v4-pro, and chat.
Test 2 — Anthropic adapter — failed

Base URL https://api.deepseek.com/anthropic, model deepseek-flash. Error: “Thinking mode does not support this tool_choice.”

  • DeepSeek’s Anthropic endpoint thinks by default unless the body says thinking: disabled.
  • Langfuse never sends “disabled”. Custom headers cannot change it — it is a body field.
Workaround that works

Same Anthropic connection, but model name deepseek-chat (a non-thinking alias). The judge returned false, with correct reasoning.

“V4.1-Flash-1017” is only the new version behind deepseek-flash. The account lists just deepseek-flash and deepseek-v4-pro.

CtrlB asks for JSON mode (round 3)

On OpenAI-compatible providers, CtrlB sends response_format: json_object. If the provider rejects it, CtrlB retries once without it and remembers that model. Anthropic is unchanged. The parser also accepts JSON written inside text.

Example: DeepSeek accepts json_object. It rejects json_schema, which is why Langfuse’s OpenAI adapter failed above. Built, not on staging yet. Wording: parity row 11.

Templates & extras

Starter instructions, plus buttons on the evaluator page.

Langfuse has 24 templates in 6 categories:

  • Conversational — 8
  • Quality — 5
  • Classifier — 3
  • Retrieval — 3
  • Safety/Security — 3
  • Coding agents — 2

The evaluator page also has “Rules”, “Add alert”, and “Version history”.

CtrlB did not ship alerts on purpose (plan decision E13). Gap: G14.

Example

Start from a Quality template, then change the instructions for Acme’s “don’t invent crypto” rule. The template is a copy, not a shared prompt.

Running live: what we saw

An evaluator sits still until a rule is attached. Then it grades new spans by itself.

What a rule is: Evaluator vs rule.

  • Before a rule: the evaluator shows INACTIVE.
  • After the rule: it turned HEALTHY.
  • Example: with no rule the page said INACTIVE. We attached the rule and it said HEALTHY.
What “healthy” means
  • Langfuse: up to the 50 most recent judge runs in the last 7 days. The tooltip said “5” only because 5 had run.
  • Langfuse order: Blocked → Inactive (no rule, or no runs) → Healthy (0 failed) → Failing (failed ÷ total ≥ 0.5) → otherwise Degraded.
  • CtrlB (round 6): the evaluator’s last 50 runs in the past 7 days, stored in Mongo. Healthy if 10% or fewer failed. Degraded if more than 10% and at most 50% failed. Failing if more than 50% failed. Skipped runs and Test runs are not counted. Failed runs do count (round 1).
  • A rule uses the same rule over its own last 50 runs in 7 days (since 2026-11-19; before, it used 24 h), and it is at least as bad as its worst active evaluator. The list tooltip and the evaluator page show the latest error: type, message, time, plus “Open failed run” and “Open observation”.
  • Empty variables (round 4): if every variable is empty, the run is skipped — no judge call, no cost, no score row. If more than half the matches in 24 h were skipped, the status is Degraded, reason missing_variables.
  • Example: 4 failures in the last 50 runs is 8%, so CtrlB says Healthy. Langfuse would not, because Healthy there means 0 failed.
  • Delayed is not a health status. It is a status of one execution (job) in Langfuse: the run is waiting to be retried. Example: the judge model returned 429 rate limit at 10:00, so the job waits (DELAYED) and retries at ~10:05. CtrlB has no Delayed: it retries inside the scan after 1 s and 3 s, then records an error. Hiding the always-0 Delayed count is still open.
  • Built and tested locally, not on staging yet.
  • Speed: 20 new spans were graded within about 20 seconds of arriving.
  • CtrlB: first scores ~50 s after saving in the staging test. The scan design: How CtrlB works today. What we confirmed: staging test.

View executions

This opens the judge’s own traces, not the bot’s chat.

  • Filters: evaluatorId + environment langfuse-llm-as-a-judge + isRootObservation:true.
  • Each row is one judge call.
  • Name: “Execute evaluator: answer_correct” on live runs, or “Test evaluator” when you click test.
  • Input = the filled judge prompt.
  • Output = {"scoreExplanation": …, "score": …}.
  • Metadata target_trace_id / target_observation_id = which bot span was graded.
  • Example: a live row named “Execute evaluator: answer_correct”. Its target_observation_id is the bot span that was graded (in our story, answer-llm).
  • CtrlB (rounds 1 and 7): the judge span stores the filled prompt, the reply, duration, tokens, and the error in the normal error fields. The Error panel shows the reason. The executions list is no longer empty.
  • Those spans move to a hidden dataset _ctrlb_evaluator_runs_<dataset>. Setting EVALUATOR_RUN_SPANS: inline, both (default — writes to both), or separate. “View executions” and “Open failed run” open the new dataset.
  • Example: after ops sets separate (once the new Orion ships), the customer’s AI Traces stop showing judge traffic. Built, not on staging yet.

Trace scores vs span scores

A score sticks to the whole request, or to one step. The screen never mixes the two.

What a trace and a span are: Trace & span. Who can write a score: Who can give a score.

  • Every score has a traceId and an observationId.
  • observationId empty = trace-level (the whole request).
  • observationId set = span-level (one step).

Trace link opens the trace with the root span selected. Its Scores tab shows trace-level scores. Ours: user_feedback = False, sent by our script with score_trace.

Observation link selects that span. Its Scores tab shows only that span’s scores. Ours: answer_correct = True from the judge on answer-llm.

  • Header chips show only trace-level scores.
  • Tree badges show only that row’s own scores. Langfuse UI code (nodeScores.ts): “badges never mix levels”.
  • Annotate scores the whole trace even if you are viewing a span. Example: answer-llm is selected, and Annotate still writes a trace-level score.
CtrlB (round 5) — built, not on staging yet

A person or the API can score the whole trace (empty span id) or one span. The Annotate panel (round 8) does the same: whole trace or one span. AI scores on a whole trace are built too (phase 3, a whole-trace evaluator). No session scores and no dataset-run scores (G4).

Example: a reviewer marks the crypto trace false while the judge’s score stays on answer-llm.

Decision model — how to set it up

Short fixed questions. One model call answers all of them, and says how sure it is.

Short intro: Three kinds of checker.

  • On the form, choose Use Decision model.
  • Until a supported model is connected, the top says “Add an OpenAI, Google AI Studio or TypeSafe connection”.
  • Until then, Run test and Create stay greyed out.

Questions

  • Up to 50 questions. All of them are answered in one call.
  • “Extra questions add only their own tokens.”
  • Each question has question text and a score name.
  • In the question text, refer to state fields in backticks, for example input and output.

Three types

TypeWhat you fill inWhat gets saved
Choice Options. Each option is a label plus a description. A category (for example grounded / partly_grounded / made_up).
Score At least 2 and up to 10 levels, low to high. Each level has a description. Otherwise it shows “Needs attention — add at least 2 levels”. A number from 0 to levels − 1. It is the expected level, so it can fall between levels (for example 1.9).
Yes / no Optional “refine what yes and no mean”. A number from 0 to 1. That number is the probability of yes.

Build the state

  • Named fields: input from Input, output from Output, and + Add field.
  • “State sent to the model” shows exactly what the model receives.
  • Probabilities for each option or level are kept in the score’s metadata. The score value itself is the label or the number.

Choice. “Does output only use facts from the docs in input?” Score name: groundedness.

Score. “How confident does output sound?” Levels: 0 unsure, 1 neutral, 2 very confident. Score name: answer_confidence.

Yes / no. “Does input ask about payment methods?” Score name: asks_payment.

Expected on the crypto row: made_up, ~1.9, ~0.97.

Why only special models

  • Decision models promise calibrated probabilities. “Yes, 92%” should be right about 92 times in 100.
  • Purpose-built models compute that inside the model: gpt-6-luna on OpenAI, gemini-decision-1 on Google AI Studio, and TypeSafe’s Jev via its /systemone API.
  • A normal chat model, such as DeepSeek, can only write a number that sounds right. Langfuse allowlists the purpose-built models.
  • Example: DeepSeek cannot be used. The OpenAI path needs the exact name gpt-6-luna, which DeepSeek rejects. TypeSafe needs a /systemone API, which DeepSeek does not have.
  • A free Gemini key might work only if gemini-decision-1 is available on it (not confirmed).
  • CtrlB lets any chat model answer and estimate the probabilities (stored in ctrlb_evaluator_probabilities). That is flexible, and it is not calibrated.
One line

Langfuse chooses reliability (fewer models, trustworthy output). CtrlB chooses flexibility (more models, less guarantee).

Code evaluator — how it works

A small rule you write. No AI, no API call, no internet. Same result every time, and free.

Short intro: Three kinds of checker.

  • Langfuse gives the rule the saved span: input, output, metadata, and tool calls.
  • The rule checks something and returns a score.
  • It runs in a sandbox: 2 s limit, standard library only.
  • It reads one span at a time — the span the rule picked — the same way an LLM judge does.
  • You can write it in TypeScript or Python (“written in”).

Python shape, from Langfuse’s own template:

def evaluate(ctx: EvaluationContext) -> EvaluationResult:
    …
    return EvaluationResult(scores=[Score(name=…, value=…, data_type="BOOLEAN", comment=…)])

If the input says “no matching docs”, the answer should say “I don’t know”.

Crypto row → false. Linux row → true.

Turn the input into text (json.dumps if it is not a string). Count the characters. True if there are more than 1.

  • Valid JSON, no model: try JSON.parse. If it throws, the text is not valid JSON.
  • Example: a value that makes JSON.parse throw fails the check. No model is called.

LLM judge vs code

LLM judgeCode
Who checksAIYour rule
Understands meaningYesNo
CostMoneyFree
Same result every timeNot alwaysAlways

Annotation queues

A to-do list of traces for people to review, one after another.

A person’s score is source ANNOTATION. Who can score: Who can give a score. CtrlB does not have queues yet: G5.

  • What: people work the list with “Complete + next”.
  • Example: queue every chat the judge marked false, then walk them one by one.

Create

  • Human annotation → New queue.
  • Name, optional description, score configs (for example answer_correct), optional user assignment.
  • The list shows Completed items, Pending items, Score configs, and “Process queue”.

Fill it

  • Tracing → select rows → Actions → Add to queue.
  • Ideas: all judge = false, all 👎, a random weekly sample, risky topics.
  • You can add as many as you like, and you can have several queues.

Review

  • Process queue → one trace at a time, with the score buttons.
  • Number keys work.
  • Complete + next. Progress shows done vs remaining.
  • Queue vs one-by-one Annotate: same result (an ANNOTATION score). Queues are for many traces, or for several reviewers.
Caveat — span vs trace

The judge scored the answer-llm span. Annotations from a root-span queue land on the trace. If Analytics shows Matched = 0, add answer-llm observations to the queue (filter type:GENERATION), or compare at the right object type. Levels: Trace scores vs span scores. Matched: comparing two scores.

The 5 big differences

The short version. The how-it-works sections below are the detail.

1. What gets graded

Both can grade one span or the whole trace. CtrlB’s whole-trace judge (phase 3) also reads non-AI spans (DB, HTTP) and the trace’s logs. Built and tested locally, not on staging yet.

Human and API scores can already sit on the trace. Table: span vs whole-trace. Design: phase 3.

2. Who can give a score

Langfuse: the judge, a person, or the app. CtrlB phase 1 does the same (rounds 5 and 8): Annotate, a scores API, and score configs. Built and tested locally, not on staging yet. Queues are later.

See who scores and the decision.

3. Speed & status

Langfuse: seconds, with Pending / Delayed / Completed / Error, and retries.

CtrlB: a 60 s scan, first scores ~50 s after saving in our staging test (Langfuse ~20 s), no Delayed status, and only short retries. What we saw live: Running live.

4. Judge reliability

Langfuse pre-fills 0–1, forces the reply shape, and test-calls the model when you save.

CtrlB now starts new numeric judges at 0–1 and asks for JSON mode, with one retry if the provider says no (B1 is the old empty range). It still does not test-call the model when you save.

5. Analytics

Langfuse compares judge vs human on the full data.

CtrlB now computes analytics in the lake for the two scores you pick, over the whole range (pairs capped at 10,000). Judge vs human vs API works (picker shows name · source). A manual recount matched to the digit. The old 2,000-row sample is B2, fixed. Old Seeker still falls back to it. Not on staging yet.

Where CtrlB can win

Logs, infra spans, and metrics. The answer text only shows what the bot said. A log can show a timeout, a retry, or a hot database — the reason the answer went wrong. Langfuse never receives logs. Phase 3 (built) is where CtrlB uses non-AI spans and correlated logs. In a local test of 12 silent billing failures, the judge with the whole trace and logs caught 8 of 8 bad answers; with input/output only it caught 5 of 8 (test). Design.

What Seeker has for this feature

The API and the engine behind CtrlB’s Evaluators page.

  • 25 ai/evaluators endpoints: evaluators (create, read, update, delete, versions, test), observations, estimate, rules (create, read, update, delete, enable), executions and executions/search, scores/summary, scores/histogram (GET and POST), scores/values, settings.
  • 9 older ai/evals endpoints are still there. Judges and LLM keys from that old API are still used.
  • Background engine: every 60 s, and only if EVALUATORS_ENGINE_ENABLED is on. It grades, then writes scores into the lake with an internal ingest (POST …/_json_evolving). Timing detail: How CtrlB works.
  • Mongo holds evaluators, versions, rules, settings, and health.
  • Code evaluators run in sandbox workers. Access is limited by RBAC.
  • Scores API (round 5): POST / PATCH / DELETE ai/scores. Analytics (round 2): POST ai/evaluators/scores/analytics.
  • The Scores page shows evaluator scores and human/API scores. Source is real (EVAL / ANNOTATION / API).
  • MCP (D31): toolset evaluators with 5 read-only tools — list_evaluators, get_evaluator, list_evaluator_rules, search_scores, score_analytics. list_evals still covers the old Evals. Closes gap G13; details: MCP tools.
  • Cost estimate (D30, D32, D33): POST ai/evaluators/estimate prices from the evaluator’s recent real runs, else from real workspace token averages, shown as a range. Cost estimate.

How Langfuse works behind the scenes

Ingest → match rules → job → judge → score in ClickHouse.

  1. On ingest, each observation is checked against active rules in memory (filter + sampling using sha256 of the span id).
  2. Match → a JobExecution row (PENDING) in Postgres + a queue job.
  3. Worker loads input/output, fills the prompt, and forces a structured score. How that force works: parity row 11.
  4. Score saved in the ClickHouse scores table (source, author, config, comment).
  5. Job → COMPLETED.
  • Rate limits → DELAYED and retried over minutes.
  • Bad key / no money / missing model → evaluator blocked (8 block reasons).
  • The judge call is saved as its own trace in hidden environment langfuse-llm-as-a-judge. What “View executions” opens: Running live.
  • Code evaluators run in a separate sandbox (AWS Lambda / external dispatcher), 2 s limit, no network. How the rule is written: Code evaluator.
  • ClickHouse — traces + scores.
  • Postgres — configs, evaluators, rules, job executions, queues.

How CtrlB works today

Scan every 60 s. Scores land as lake logs about 2–3 minutes later.

  • UI: Evaluators, Rules, Scores, Analytics, and Score configs (round 8, built). Annotate sits in the AI trace drawer.
  • Engine: scans the traces dataset every 60 s, only spans older than ~2 min → score appears ~1–3 min after the chat. Whole-trace rules also wait for the trace to be quiet (2 min by default).
  • An observation evaluator reads one span’s gen_ai_input_messages / gen_ai_output_messages. A whole-trace evaluator reads {{trace}}, {{errors}}, {{input}}, {{output}} (phase 3).
  • OpenAI-compatible judges are asked for JSON mode (json_object), with one retry without it if the provider says no. The parser also accepts JSON inside text.
  • New numeric judges start at 0–1. A run with every variable empty is skipped.
  • Evaluator scores are lake records in _ctrlb_evaluators_<traces dataset>. Human and API scores use the same dataset, scope ctrlb.scores, via ai/scores.
  • Judge runs are spans (service ctrlb-evaluator) and move to hidden dataset _ctrlb_evaluator_runs_<dataset>. The same idea in Langfuse: Running live.
  • Evaluators, versions, rules, settings, health, and score configs in MongoDB.
  • Health: last 50 runs in 7 days, for evaluators and (since 2026-11-19) rules. Healthy ≤10% failed, Degraded above that up to 50%, Failing above 50%. A rule is at least as bad as its worst evaluator.
  • Retries on 429/5xx after 1 s and 3 s. No Delayed status.
  • 3 block reasons.
  • TypeScript and Python code evaluators are off by default (turned on only locally for testing). Python still runs on staging (open, infra #16).
  • Scores table Source is the real value: EVAL, ANNOTATION, or API. Author is a column, hidden by default.
  • Old /ai/evals page is switched off; its judge key store is still used.

Customer asks about crypto at 10:00. Bot answers. Score about crypto hallucination shows up around 10:02–10:03 — not instantly.

Span evaluator vs whole-trace evaluator

Span evaluatorWhole-trace evaluator
Looks atOne step (e.g. answer-llm)All steps of the request
RunsAs soon as that step arrivesAfter the request finishes (no new steps for a while)
Score goes onThe spanThe trace
Sees “the docs search failed”?❌✅ (Langfuse adds an “execution issues” list: ERROR spans + message)
CostSmallBigger (more text)
What it readsThat span's input/outputLangfuse: the LLM conversation of the trace + ERROR/WARNING spans. CtrlB (built): every span incl. DB/HTTP calls, their errors, and up to 100 logs picked by text, not level (refs [s1]… so the judge can cite spans)
Langfuse✅✅ new “trace transcript” evaluators (idle 1–60 min, LLM judge + decision model). The old trace evaluator (input/output only) is deprecated
CtrlB✅✅ Built (phase 3), tested locally, not on staging yet. Compact waterfall, quiet wait, then one grade; cited spans highlighted in the Traces drawer. Limits

Span evaluator on answer-llm: “made up — no docs”. Whole-trace evaluator: “made up because retrieve-docs failed (timed out)”.

What CtrlB could do better (idea I2 — phase 3 built: span tree + errors + logs picked by text; metrics, root-cause label and “possibly related” logs still later)
  • Logs: catch hidden problems (retries, caught errors, fallbacks). Langfuse has no logs.
  • Infra context: DB / Kubernetes spans, e.g. “vector DB pod restarted at that moment”.
  • Metrics: e.g. “answers got worse while the DB was at 95% CPU”.
  • Root-cause label: prompt problem vs system problem → the right team fixes it.
  • Cheaper: send the answer + a short error summary, not every step’s full text.
  • Works without linked logs: span errors + nearby logs marked “possibly related”.

Side-by-side parity

38 behaviours compared (2026-11-18, updated 2026-11-19 after the end-to-end test).

✅ identical
14 · 37%
🟡 close
17 · 45%
🔴 different
4 · 10%
⚫ missing
3 · 8%
Weighted ≈ 59%

Identical = 1, close = 0.5 → (14 + 8.5) / 38 ≈ 59%. It was 36% at the start of 2026-11-18, and 54% after rounds 1–8. The final changes D24–D33 do not move any row: rows 6 and 15 get better details but stay close; the cost estimate, MCP tools and UI fixes are not parity rows. The walkthrough fixes and D34 move no row either: the drawer’s new Scores tab shows scores on the trace and span themselves (a Langfuse gap now closed), but no row tracks that. Row 29 is already identical, row 32 stays close (no queues), row 36 stays close (no session or dataset-run scores), and D34 changes how row 26 shows errors, not its health rule.

Updated 2026-11-19: rows 20 (delay) and 21 (whole trace) moved from missing to close, 28 (storage) from different to close, 29 (Scores table) from close to identical. Built and tested locally, not on staging yet. Bar percents use the largest remainder so they add to 100 (14/38, 17/38, 4/38, 3/38).

# Behaviour Parity Langfuse CtrlB
13 evaluator types (LLM, decision, code)✅ identicalyesyes
2Save = new version; rules use latest✅ identicalyesyes
3Delete removes versions + rule links✅ identicalcascadeyes (rule with none left is disabled)
4Project default model + per-evaluator model✅ identicalyesyes
5Prompt messages (system/user/assistant) with {{vars}}✅ identicalyesyes
6Score types numeric/boolean/categorical🟡 close+ categorical multi-matchno multi-match. Multi-label categorical is planned for the next release, off by default
7Numeric range default✅ identicalUI pre-fills 0–1new judges start at 0–1 (round 4). Old judges with no range show a warning. History: B1
8Variable mapping with JSON selector🟡 closejsonSelector, fallback to whole valueJSONPath subset
9Template gallery🟡 close24 managed templatesstatic catalogue in Orion
10Save-time test call to model (blocks bad save)⚫ missingyesonly checks connection exists
11Judge output enforcement🟡 closestrict structured output always on (json_schema for OpenAI-compatible, forced tool call for Anthropic); DeepSeek only works via the deepseek-chat alias on the Anthropic endpointJSON mode (json_object) on OpenAI-compatible providers; one retry without it if rejected, and that model is remembered. Anthropic unchanged. Also accepts JSON inside text. DeepSeek accepts json_object (it rejects json_schema). Still no save-time test call
12Reject out-of-range score when range set✅ identicalzod min/maxErrInvalidScore
13Decision model🔴 differentallowlisted decision models, real probabilitiesany chat model, estimated probabilities
14Code evaluator contract (evaluate(ctx) → scores)🟡 closeTS+Python on, source 256 KB, result 256 KB, Lambda/externalTS+Python off by default, source 20k chars, result 64 KB, own workers
15Test on a sample observation🟡 closeobservation filter + quick buttonsobservation filter. The default sample is the newest one with an input or output (D26); whole-trace evaluators test on a sample trace
16Rule filters🟡 closeobservation + trace filtersone traces dataset + alert filter JSON
17Deterministic sampling🟡 closesha256(span_id), per observationfnv32(trace_id), per trace
18Several evaluators per rule, per-rule mapping✅ identicalyesyes (max 10)
19Run on past data🟡 closetimeScope EXISTING / batch evaluatebackfill ≤1,000, range
20Rule delay setting🟡 closedelay per ruletrace rules: quiet period 2 / 5 / 15 min (L5). Observation rules: fixed 120 s watermark. Intentional
21Whole-trace (transcript) evaluators🟡 closetrace transcript rules with idle time; transcript includes “execution issues” (ERROR/WARNING spans + message); the old trace evaluator is deprecatedbuilt (phase 3), more context: every span incl. DB/HTTP, errors, logs picked by text, caps L1–L8, cited spans. Tested locally. Design
22Trigger timing🔴 differenton ingest (seconds)scan every 60 s (~2–3 min)
23Job record + statuses🔴 differentJobExecution: PENDING/DELAYED/COMPLETED/ERROR/CANCELLEDno row; completed/error, pending = backfill, delayed 0
24Rate-limit retries🔴 differentqueue retries over minutes, DELAYEDin-call retries 1 s, 3 s
25Block reasons🟡 close8 reasons3 reasons
26Evaluator health window🟡 closelast 50 runs in 7 days. Healthy only at 0 failed. Failing at 50% or more. Else Degradedlast 50 runs in 7 days, stored in Mongo (round 6). Healthy ≤10% failed, Degraded above 10% and ≤50%, Failing above 50%. Skipped and Test runs are not counted. Rules use the same window (2026-11-19)
27Never evaluate own judge traces✅ identicalyesyes
28Score storage🟡 closeone table with source, author, config, upsertthe customer’s own lake dataset, with source and author (SQL and dashboards work on it). Human/API scores use scope ctrlb.scores. Edit/delete append a new version; reads take the latest. Configs live in Mongo. Intentional
29Scores table page✅ identicalyesyes. Rows not clickable; Trace link opens the trace peek, Observation link opens it with that span selected; each has its own URL
30Score analytics🟡 closeper-score queries up to 100klake SQL over the whole range (POST ai/evaluators/scores/analytics). Pairs capped at 10,000 (“Compared on 10,000 of N”). Old Seeker still falls back to the latest 2,000. History: B2
31Score sources EVAL/ANNOTATION/API✅ identicalyesreal Source. Human/API scores use scope ctrlb.scores so they stay out of evaluator health
32Manual scoring in UI🟡 closeyesAnnotate panel: whole trace or one span (round 8), edit and delete. Tested locally. No queues yet
33Scores API create/update✅ identicalyesPOST / PATCH / DELETE ai/scores. Tested locally
34Score configs✅ identicalyes (+2 defaults)yes, in Mongo, plus the same 2 defaults (“open coding” TEXT, “pass or fail” Pass=1 / Fail=0). Tab under Evaluators
35Annotation queues⚫ missingyesno
36Scores on trace/session/dataset run🟡 closetrace, span, session, dataset runtrace and span, for human, API and evaluator scores (whole-trace evaluators, phase 3). No session or dataset run (later)
37TEXT scores✅ identicalannotation/API/codeannotation, API, and code. Default config “open coding” is TEXT
38Datasets / experiments⚫ missingyesno (planned later)

Intentional differences (decided, not gaps)

These differ on purpose. Do not “fix” them to match Langfuse.

AreaLangfuseCtrlBWhy
Variable mappingjsonSelectorJSONPath subsetEnough for OTel GenAI messages
Templates24 managed, server-sideShipped in OrionNo server catalogue needed
Judge outputStrict JSON schema / forced tool callJSON mode (json_object)Works with DeepSeek; strict schema doesn’t
Decision modelSpecial decision modelsAny chat model, estimated probabilities (labelled)Those models aren’t available to CtrlB
Code evaluatorsOnOff by defaultRuns user code; keep off until the sandbox is hardened
Rule filtersOwn filter UIReuses the alert filter builderOne filter model across CtrlB
SamplingPer observationPer traceSpans of one request are graded together
Past dataBatch evaluateBackfill ≤1,000 with cost estimateCost control
DelayPer ruleQuiet period only for trace rulesObservation rules need no extra wait
Whole traceLLM conversation + execution issuesAll spans, errors, logs, with caps L1–L8More context finds hidden failures (8/8 vs 5/8)
TriggerOn ingest (~20 s)Scan every 60 s (~1–3 min)The lake has no ingest hook
Job statuses, retriesJob rows, DELAYED, queue retriesFollow from scanningSame reason
Block reasons83The 3 that apply to CtrlB connections
Score storageOne ClickHouse table, upsertCustomer’s lake; explicit edit/deleteSQL and dashboards on scores; the lake is append-only

Planned later (not intentional): annotation queues (phase 2), session scores, datasets and experiments.

Not just chatbots

Same checkers. Different apps. Pick the mode that matches the check.

AppExample checkBest mode
ChatbotIs it relevant, the right tone, and correct?LLM judge
RAG knowledge assistantIs the answer backed by the retrieved pages?LLM judge
Document extractionIs it valid JSON, and do the totals add up?Code
Document extractionDo the values match the document?LLM judge
Classification / routingIs the category right?Decision model
SummarizationDid it add anything not in the original?LLM judge
Text-to-SQLDoes the query run and return rows?Code
Tool / MCP agentsDoes the answer match the tool result?Whole-trace judge
Coding agent / Claude CodeTests failed, but the code was committedCode
Web search with citationsWas the cited link really retrieved?Code
Web search with citationsDoes the source actually say that?LLM judge
Example

An invoice extractor: code checks that the JSON parses and the line items sum to the total. An LLM judge checks that “$20” really appears on the page.

The common pattern

The answer can still be wrong after a tool returns the truth.

  1. The user asks.
  2. The AI calls a tool, API, MCP server, or search.
  3. The tool returns data.
  4. The AI writes the answer — and can still make something up.
  5. The evaluator compares the answer with the recorded tool result.
Tool saidBot said
get_price → 20“Pro is $40”
search_logs → 0 errors“found 12 errors”
npm test → 3 failed“all pass”
get_order → shipped“not shipped yet”

What must be recorded

The judge can only use what was saved on that request.

  1. The AI call’s input, with message content turned on.
  2. The output.
  3. Tool, API, MCP, and search steps, with their inputs and results.
  4. The same trace_id on every step.
  5. Logs that carry trace_id (optional — a CtrlB bonus).

What the judge can do as more is recorded:

  • Only the question and answer → relevance, tone, safety, and “might be made up”.
  • Plus the instructions → “did it follow its own rules?”
  • Plus sources or tool results → correctness.
  • Plus logs (CtrlB) → why it failed.
  • Nothing recorded → nothing to grade.
Example

The pricing bot below only works as a real check if the price call is saved on the same trace as the answer. If that step is missing, the judge can only say the answer states a price.

Checks that need no docs

Most checks compare the answer with what was already recorded. They do not look anything up.

  • Needs no source of truth: relevance, follows instructions, made-up-facts risk, tone, safety, refusal, format, user frustration, topic.
  • Only “is this fact correct?” needs a source of truth: docs in the prompt, expected answers in a dataset, or a human.
  • The judge never looks things up itself. It compares the answer with what was recorded in that request.

The price tag says 20. The cashier says 40. A supervisor watching the CCTV catches the mismatch. The supervisor does not walk the shop to re-check the tag. The recording already has it.

Langfuse vs CtrlB — same pricing bot

The bot calls a pricing API (20) but does not put the price in the prompt. The AI says $40.

What was savedLangfuseCtrlB
Only the AI step. The price never entered the prompt. Judge: “states a price, no source, can’t confirm.” Built (phase 3) A whole-trace evaluator sees the API span, the AI span, and the log “WARN price cache stale”, then says “wrong: API returned $20, bot said $40.” Tested locally; not on staging yet, so staging still reads only the AI span. Phase 3.
The price is in the prompt. Catches it: 20 is in the prompt, the bot said $40. Catches it too. This works today, because the price is on the AI span.
No source at all. Can only flag “no source.” Can only flag “no source.”

Claude Code in CtrlB

Claude Code already speaks OpenTelemetry. Point it at CtrlB.

What it sends:

  • Metrics: session count, tokens, cost in USD, lines of code, commits, pull requests, active time.
  • Events / logs: user_prompt, assistant_response, tool_result, tool_decision, api_request, api_error.
  • Traces (beta): claude_code.interaction → llm_request, and tool → tool.execution.

Turn it on in the shell, or in Claude Code settings.json under "env":

  • CLAUDE_CODE_ENABLE_TELEMETRY=1
  • OTEL_METRICS_EXPORTER=otlp
  • OTEL_LOGS_EXPORTER=otlp
  • OTEL_TRACES_EXPORTER=otlp and CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1
  • OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS

Where the env block may live:

  • Your user file: ~/.claude/settings.json
  • This project only: .claude/settings.local.json
  • Managed settings for the whole company. Managed values cannot be overridden.
  • A repo’s shared .claude/settings.json cannot set the exporter variables.

Prompt text is redacted by default. OTEL_LOG_USER_PROMPTS=1 includes it (that is sensitive). OTEL_LOG_RAW_API_BODIES logs the full bodies.

What you can use it for:

  • Cost per team — OTEL_RESOURCE_ATTRIBUTES=team.id=…
  • Adoption and productivity.
  • Reliability: API errors, 429s, latency.
  • Tool behaviour and rejected edits.
  • Security: risky commands, unapproved MCP servers.
  • Quality: a code evaluator for “tool failed” or “committed while tests failed”. An LLM judge only if prompt text is turned on.
Example

A code evaluator flags a session whose tool_result says tests failed and a later step still created a commit.

Customer example: GoComet

One customer. The steps are in the next section.

GoComet

Stamp the commit on every trace, then compare before and after.

Which commit broke it?

Which commit broke it?

Stamp the commit on every trace. Then compare before and after. Example: GoComet.

  • CI/CD is the pipeline that builds and deploys each commit.
  • It sets OTEL_RESOURCE_ATTRIBUTES=service.version=<commit or release>. In GitHub Actions that value is ${{ github.sha }}.
  • The app’s OpenTelemetry SDK stamps every trace, log, and metric.
  • CtrlB compares the window before that version with the window after.
  • You do not need GitHub’s logs. GitHub gives the pipeline the commit id (GITHUB_SHA). The pipeline gives it to the app. The app carries it to CtrlB.
  • Weekly names work too, such as 2026.w44. Narrow to the commit with git log v2026.w43..v2026.w44, or use 2026.w44+abc123.

If nothing is tagged, you can still guess from:

  • A deploy event log from the pipeline.
  • The container image tag, such as quote-service:abc123.
  • An attribute added by an OpenTelemetry Collector.
  • Pod restarts — time only.
  • A GitHub connection (a new CtrlB feature).
  • Or time-only guessing by hand.
What you seeWhich mode
Errors or latency jumpNo evaluator. Use alerts and compare the two windows.
Outputs break a ruleCode
You need the failure typeDecision model
Is the AI answer still correct?LLM judge, scored by version
Why did it break?Whole-trace judge (built — phase 3)
GoComet example

The pipeline deploys commit abc123 and stamps service.version=abc123. Error rate jumps on those traces. Alerts show the jump. A later whole-trace judge would say whether the answer changed because of that deploy.

Rounds 1–8

Built on branch feat/evals-scores-v2 and tested end to end on local Seeker + Orion (2026-11-18/19). Not deployed to staging yet.

RoundWhatStatus
1 Health counts failed runs. A rule uses its own runs (since 2026-11-19 its own last 50 in 7 days, like evaluators), and it is at least as bad as its worst active evaluator. The list tooltip and the evaluator page show the latest error (type, message, time) with “Open failed run” and “Open observation”. The judge span stores the filled prompt, the reply, duration, tokens, and the error, so the Error panel shows the reason. Example: a failed judge no longer looks healthy with an empty error box. Built · tested locally
2 Analytics is SQL in the lake (POST ai/evaluators/scores/analytics): totals, trend, distribution, and the comparison of the two scores you picked, over the whole range. Pairs stop at 10,000, with the note “Compared on 10,000 of N”. Old Seeker still falls back to the latest 2,000 rows. Example: “Last 1 day” is that whole day, not a short slice of 2,000 rows. Built · tested locally
3 OpenAI-compatible judges are asked for JSON mode (response_format: json_object). If the provider rejects it, one retry without it, and that model is remembered. Anthropic is unchanged. JSON inside text still parses. Example: DeepSeek accepts json_object and rejects json_schema. Built · tested locally
4 If every variable is empty, the run is skipped: no judge call, no cost, no score row. If more than half the matches in 24 h were skipped, status is Degraded, reason missing_variables. New numeric judges start at 0–1. An old judge with no range shows a warning. An empty mapping asks “Fix mapping” or “Save anyway”. Escape never saves. Example: a blank {{input}} does not spend a judge call. Built · tested locally
5 Human and API scores. Configs live in Mongo (name, type NUMERIC / CATEGORICAL / BOOLEAN / TEXT, range or labels, description, archive; the type cannot change). POST / PATCH / DELETE ai/scores. Signed-in users are ANNOTATION. API keys are API. The author is stored. The author or an admin can edit or delete. A score can sit on the whole trace (empty span id). Same lake dataset, scope ctrlb.scores, so they do not change evaluator health. The Scores page shows the real Source. Author is hidden by default. Analytics can compare “name · EVAL” and “name · ANNOTATION”. Example: a reviewer marks the crypto chat false, next to the judge. Built · tested locally
6 Health is the evaluator’s last 50 runs in the past 7 days, stored in Mongo. Healthy if 10% or fewer failed. Degraded if more than 10% and at most 50% failed. Failing if more than 50% failed. Langfuse is Healthy only when 0 failed, and Failing at 50% or more. Skipped runs and Test runs are not counted. Example: 4 failures in 50 runs is 8%, so Healthy in CtrlB. Built · tested locally
7 Judge traces move to a hidden dataset _ctrlb_evaluator_runs_<dataset>. Setting EVALUATOR_RUN_SPANS: inline, both (default — writes to both), or separate. “View executions” and “Open failed run” open that dataset. Customer AI Traces stop showing judge traffic after ops sets separate, once the new Orion ships. Example: the crypto judge call is not mixed into the bot’s trace list. Built · tested locally
8 A 5th Evaluators tab, Score configs: a create/edit drawer with “Copy from evaluator”. Two defaults, added the first time a workspace opens the tab, and not added again if archived: “open coding” (TEXT) and “pass or fail” (Pass=1, Fail=0). An Annotate panel on AI Traces: whole trace or one span, pick configs, one input per type, “Scores on this trace” (newest first, updates at once after save, edit or delete). Example: open the crypto trace, pick “pass or fail”, mark Fail. Built · tested locally

Phase 3 — whole-trace evaluators

Built and tested locally (2026-11-19). Not on staging yet. The judge grades the whole request, not one step.

  • Choice on the evaluator: “Grades: One observation | Whole trace”.
  • What the judge reads: a short waterfall — every span, including DB and HTTP, plus errors and the most important logs of that trace. Variables: {{trace}} (waterfall with refs [s1]…), {{errors}}, {{input}}, {{output}}.
  • Clean trace: an empty {{errors}} reads “No errors or problem logs in this trace.” Before this fix, clean traces failed with missing_variables (97 of 129 runs).
  • Logs found by trace id through the same correlation as the Related logs tab.
  • When: once, after the trace is quiet. Default 2 minutes. Also 5 or 15.
  • CtrlB’s edge: non-AI spans and logs tied to the same request. Langfuse’s “trace transcript” evaluators wait 1–60 min and read the LLM conversation plus “execution issues” (ERROR/WARNING observations). Its old trace evaluator (input/output only) is deprecated.
  • Example: the crypto answer is false because retrieve-docs timed out, and a WARN log says the docs cache was stale.
  • The judge may return citedSpans. The Traces drawer shows a “Trace scores” strip and “Highlight spans the judge cited”.

Limits

LimitWhat it means
L1≤300 spansA small trace is sent as it is. Up to 1,000 spans are read, then cut.
L2Over 300Keep the root, errors, LLM spans, and the slowest spans first.
L316 KB textLLM input and output are cut to 1 KB each.
L4≤100 logs / 4 KB, any levelPicked by importance: ERROR, WARN, then text like “failed”, “timeout”, “retry” or an HTTP 5xx, then the rest. Repeats shown once (×N), shown in time order. Levels are normalised (INFO/info/Information → INFO; none → “-”). Levels are often missing or wrong — on staging ~1.4 M trace logs are Envoy access logs with no level, and failures are logged at INFO — so the judge reads the text.
L52 min quietGrade after no new spans. Choices: 2, 5, or 15 min.
L6Never quietGrade after 30 min anyway, and mark “still running”.
L710% sampleDefault for trace rules. Since D29, new observation rules start at 10% too (they were 100%).
L8Weekly costShown before you save.

Staging trace sizes

Dataset k8s_demo_llm, 24 h, about 2 million traces.

StatValue
Median spans3
p95 spansabout 22
p99 spansabout 25
Max spans284
p99 durationabout 5 s
Longest traceabout 24 h
Example

Most chats are 3 spans, so they fit under L1. A 284-span trace is trimmed (L2) and then cut to 16 KB (L3).

End-to-end test (2026-11-18/19)

Local Seeker (:8095) + Orion (:5173) against the staging lake and Mongo, with ~2,300 real DeepSeek traces from a seed script. Everything below is fixed and verified.

Experiment: whole trace + logs vs input/output only

Judge readsBad answers caughtNote
Whole trace + logs8 / 8One catch was visible only in logs: 503 access lines with no level, and an INFO “retry failed”
Input / output only5 / 8—
Example

12 traces where billing failed silently: the answer text looked fine, but a backend step failed. The judge that read the whole trace and logs caught 3 more bad answers than the input/output judge; one of them only showed in no-level 503 access logs and an INFO “retry failed”. Cost was about $0.0005 per trace.

Found and fixed — Seeker

FoundFix
Clean traces failed missing_variables because {{errors}} was empty (97 of 129 runs)Reads “No errors or problem logs in this trace.”
Rule health used 24 h, so a rule said Failing while its only evaluator said DegradedRules use their own last 50 runs in 7 days, and are at least as bad as their worst evaluator
Orion got a 400 on every page load while checking whole-trace supportSettings now report traceEvaluators
Code sandbox test: a deep JSON.parse gave a new error textMapped to “Maximum call stack size exceeded.”; the worker survives

Found and fixed — Orion (UI)

  • Whole-trace evaluators reopened as “One observation” → fixed; clone keeps it.
  • Human and API scores were missing from the Scores table and the Annotate list → fixed.
  • Annotate list updates at once after save, edit or delete, newest first; buttons say “Saving...” / “Deleting...”.
  • Scores table works like Langfuse: rows not clickable; Trace and Observation links open the trace peek, each with its own URL.
  • Prompt Preview fills variables from the selected sample; long values scroll instead of being cut.
  • Last-error links keep the dataset and a valid time range; “Open trace” for whole-trace errors.
  • Score configs tab: one header, “Show archived” switch, locked data type still visible.
  • Whole-trace wording (“Select a trace”, “Matching traces”); the save dialog offers a 10% traces rule.
  • Analytics says “516 scores · Human”, not “runs”; /evaluators/analytics redirects; interval list capped at the range.
  • Scores chart notes when it covers only loaded scores (“Covers the N newest scores, from …”).
  • Smaller fixes: sample traces list shows Spans / Errors / Duration; Pass/Fail buttons not squashed; 1 px overflow; version history shows “Grades” and the range.
Tests

Orion: full suite 14,475 passed (2 failures fixed after), feature dirs 2,109 passed, design-system audit 0 findings in feature files; remaining ESLint errors only in TracesDrawer.tsx (pre-existing on dev). Seeker: all 44 packages pass with Mongo.

How to test locally

  • Seeker on :8095 against the staging lake and Mongo. Orion on :5173. In the browser: localStorage.seekerURL = "http://localhost:8095".
  • Test data: ~/Desktop/skkr/evaluators-langfuse/ctrlb_ai_seed.py (real DeepSeek calls; scenarios good, sloppy, unknown, wrongdoc, outage, hidden, logonly, big). It writes shivam_12_ai_traces and shivam_12_ai_logs through a local-only helper that is never committed.
  • Example (~$0.01): .venv/bin/python ctrlb_ai_seed.py --count 150 --workers 8 --yes

Final changes (2026-11-19, D24–D35)

Built after the end-to-end test, on the same branch. They close several open items. Not on staging yet.

#What changedExample
D24“Scored” column. The Evaluators list column “Observations scored” is now just “Scored” (header and tooltip). Same count: runs that returned a verdict.A whole-trace evaluator that graded 120 traces shows “Scored 120”, not “Observations scored 120”.
D25Log tie-break. When logs are equally important (same rank as in L4), logs on an ERROR span of the trace come first, then logs on its slowest spans, then the earliest. Budgets, “×N” and time order are unchanged.A trace whose only problems are three 404 ERROR spans used to fill the budget with hundreds of earlier access logs. Now those spans’ own logs are kept.
D26Test sample has data. The Test panel starts on the newest matching observation with an input or output, else the newest. Your own pick is never replaced. Whole-trace tests keep the newest trace.The newest row is an empty execute_tool span → the panel opens on the newest chat span instead, so the first test is useful (fixes S10).
D27Model picker: working connections first. Models of providers that are set up come first. A provider with no key says “not set up”; a key that failed its check says “connection not working”. Search covers every provider.Only DeepSeek has a key → DeepSeek models are on top; OpenAI says “not set up”.
D28Judge runs count as LLM calls. In a run-spans dataset (_ctrlb_evaluator_runs_*) an evaluation span counts as an LLM call, and “Spans by operation” calls it “evaluation”, not “Other”. Code-evaluator runs call no model, so they stay at 0. Customer datasets are unchanged.“View executions” on an LLM judge shows “LLM calls 1” per run, not 0.
D29New rules sample 10%. The “Evaluator saved” dialog’s rule and a brand-new rule in the rule drawer start at 10% for observation rules too (was 100%). The dialog has a sampling field, and its cost line names the rate: “About $X a week at 10% sampling, …”.One click on a big dataset was ≈ $550–950/week at 100% with no filter; at 10% it is about a tenth of that (fixes S8).

Cost estimate (D30, D32, D33)

  • From real runs first (D30): a saved evaluator is priced at the average cost of its own runs in the last 24 h on the same dataset, counting only runs after it (and the project default model, if it uses that) was last saved. The response says basis: "recent_runs" and shows one value. Example: the judge cost $0.000164 per call in its recent runs → the weekly estimate uses $0.000164 per call.
  • Fallback with no runs (D32): basis: "estimate". Output tokens come from real completed judge runs in the workspace (last 24 h, else 7 days); only if there are none, 60 per LLM judge and 25 per decision question (was 120 / 40). Input is the real messages, averaged over the 5 newest matching observations.
  • Any model, as a range (D33): history of the same provider + model comes first, then the workspace average. A reasoning model with no same-model history gets +500 output tokens (it bills hidden thinking as output). Orion shows “≈ $low–$high a week (estimate)”: the guessed part at 0.7×–1.5×, high 2× for a reasoning model. Example: a $1.00 guess shows “≈ $0.70–$1.50 a week (estimate)”; with a reasoning model, “≈ $0.70–$2.00”.
Local check

Real cost $0.000164 per call. The new fallback guessed $0.000146 per call (~11% low). The old guess was $0.000263 (~60% high).

MCP read tools (D31)

New toolset evaluators. Read only, no writes. Each tool uses the same service and the same access checks as its HTTP route.

ToolWhat it returnsExample question
list_evaluatorsEvaluators with status and reason, last error, 24 h stats; optional status filter“Which evaluators are failing?”
get_evaluatorOne evaluator by id or name: prompt, scores, mapping, rules“What does answer_correct check?”
list_evaluator_rulesRules with dataset, filters, sampling %, evaluators, status“Which rules run on k8s_demo_llm?”
search_scoresScores (EVAL / ANNOTATION / API), newest first, filtered; at most 200 rows per call“What scored below 0.5 today?”
score_analyticsOne or two scores: totals, distribution, agreement, kappa, correlations, MAE, RMSE“Does the judge agree with humans?”

The search_scores and score_analytics window is capped by MCP_SEARCH_MAX_RANGE. The default stays 24 h (owner decision), the same as search_logs. list_evals still lists the old Evals.

Walkthrough fixes

Found while clicking through the built pages.

  • Scores tab in the AI trace drawer. A “Scores (N)” tab lists evaluator, human and API scores. On the trace root: the trace scores, plus how many scores sit on other spans. On a span: that span’s scores. A “View in Scores” link opens the Scores page for that trace; with none it says “No scores yet.” Annotate and the tab show the evaluator’s name instead of “—”. The old Evals tab is unchanged behind its switch. This closes a Langfuse gap: scores are visible on the trace and observation themselves. Example: a trace with 2 trace scores and 3 span scores shows “Trace scores, and 3 scores on other spans”; the answer-llm span shows its answer_correct score with the evaluator’s name.
  • Last error always shown (D34). A Degraded or Failing evaluator always shows its last error and an “Open trace” link. Failed recent runs keep their error, so it comes from the same 7 days as the status (before, only from the last 24 h). Example: an evaluator whose only failure was 4 days ago is Degraded and now shows that error and “Open trace”, not an empty box.
  • Test panel, no traces in range. It says “No traces in the last 1 day. Pick a longer time range.” instead of “No traces dataset with AI spans found”. Example: the newest AI trace is 3 days old and “Last 1 day” is picked → you are told to widen the range.
  • Lists re-fit on resize. Evaluators, Rules and Score configs re-fit their columns when the window resizes; the Name column is wider. Example: a narrower browser window no longer cuts off columns, and long names fit.
  • Scores chart wording. It says “18,908 scores · 7,068 failed runs” and the legend says “Failed runs”. Failed means evaluator runs that errored, not scores that are false. Example: a judge that answered “false” 500 times adds 500 scores, not 500 failed runs.
  • Score configs header says “Updated at”, like the other lists.

Judge runs live in their own traces (decision #6 / E2b)

Each judge run is one span in its own trace. It points back at what it judged with ctrlb_evaluated_trace_id (and the evaluated span). It is never added to the bot’s trace.

EVALUATOR_RUN_SPANSWhere run spans goWhen
inlineOnly the rule’s own traces dataset (old behaviour)Not used any more
bothThe rule’s dataset and _ctrlb_evaluator_runs_<ds>Default, temporary Old Orion’s “View executions” still reads the rule’s dataset
separateOnly _ctrlb_evaluator_runs_<ds>Target Same idea as Langfuse’s hidden langfuse-llm-as-a-judge environment
  • Datasets: run spans in _ctrlb_evaluator_runs_<ds>, scores in _ctrlb_evaluators_<ds>, both in the same org as the rule’s dataset.
  • Next step: flip staging and prod to separate once the new Orion ships.
  • Example: a rule on shivam_12_ai_traces grades 66 bot traces → the Traces page for that dataset lists 66 bot spans and no evaluation spans; the 66 judge runs sit in _ctrlb_evaluator_runs_shivam_12_ai_traces.
Verified locally with separate

New judge runs go only to _ctrlb_evaluator_runs_<ds>. The bot’s dataset on the Traces page shows only bot spans (66 of 66). The judge dataset is never in the Traces dataset dropdown: Seeker filters _ctrlb_evaluator_runs_* out of dataset lists.

Where judge runs are viewed Done · A + C

  • Judge runs are opened only in AI Traces: Evaluators › “View executions” → the drawer.
  • In that drawer, “Open in traces” is hidden for judge runs and replaced by “Open evaluated trace”, which opens the bot trace that was judged. Verified.
  • Bot traces keep “Open in traces”.
  • Where bot traces come from: in a real workspace, the customer’s own app (OpenTelemetry). Locally, shivam_12_ai_traces is our DeepSeek seed data (ctrlb_ai_seed.py; scenarios good, outage, hidden, logonly, wrongdoc, sloppy, unknown, big).

Citations (cited spans)

  • Only whole-trace evaluators cite spans. A per-span evaluator never does.
  • Visible in the Test panel result and on the Traces page drawer (Waterfall → “Highlight spans the judge cited”).
Worked example

An outage trace: 3× GET billing-api returned 503, plus one chat deepseek-chat call. The judge scored 1, “handled honestly”: the bot told the user billing was down. The three 503 spans are highlighted red (cited, error) and the chat span yellow (cited, no error).

Annotate: humans score the bot, not the judge

Annotate is for people scoring the bot’s traces. Those human scores are the ground truth the judge is checked against.

  1. Find the wrong score (Scores page, or the trace’s Scores tab).
  2. Annotate the bot trace with the right value.
  3. Open the judge run (View executions → drawer) and read why it scored that way.
  4. Edit the evaluator (a new version) and Test it on that same trace.
  5. Check Analytics: judge vs human — agreement, kappa, confusion matrix.
Example

The judge gave 0 to an honest “billing is down, try later” answer. Annotate that bot trace as 1. The judge run says “did not answer the question”. Add “an honest outage message counts as correct” to the prompt and Test on the trace: now 1. Analytics shows agreement going up as new scores come in.

In progress Not done

ItemWhat it will doStatus
D35 Analytics drill-downClick a confusion-matrix or heatmap cell → list of the pairs in it, with both scores, the reasoning and linksIn progress
Test panel datasetOpens on the rule’s dataset when the evaluator has exactly one ruleIn progress
Judge-run drawer headerCounts the evaluation call as an LLM callIn progress
Annotate on judge runsHidden: judge runs are not ground truthIn progress
AI drawer Scores tabShows “Cited:” with links to the cited spansIn progress

Planned: CtrlB vs Langfuse head-to-head Planned

  • The same ~100 seed traces and the same DeepSeek judge. Langfuse’s judge reads input/output; CtrlB’s reads the whole trace + logs.
  • Result: a scorecard by scenario (good, outage, hidden, logonly, wrongdoc, sloppy, unknown).
  • Earlier small test: trace + logs caught 8/8 hidden failures; input/output caught 5/8.

Decisions made

These are chosen. They are not still open. Full table with code pointers (D1–D35; D35 drill-down is in progress): Seeker docs/evaluators-plan.md §0.

#DecisionWhat it meansWhy
D1Default numeric range 0–1New numeric LLM scores start “from 0 to 1”Judges invented scales (1, 9, 60)
D2Analytics in Seeker SQLPer score, capped at 10,000 pairsOld page used 2,000 rows of all scores; recount matched to the digit
D3Health = last 50 runs in 7 daysHealthy ≤10%, Degraded ≤50%, Failing >50%; skipped and Test runs excludedLike Langfuse; recovers quickly after a fix
D4Rules use the same health ruleOwn last 50 runs; at least as bad as the worst evaluator; 24 h fallbackRule said Failing, evaluator said Degraded
D5–D9Human scores, Scores API, score configs, trace-level scores, judge vs human vs APIAnnotate; ai/scores; 2 default configs; empty span id = trace; name · sourceLangfuse G1–G4, G6
D10Judge runs in a separate dataset_ctrlb_evaluator_runs_<dataset>, EVALUATOR_RUN_SPANS=both by default; _ctrlb_ datasets hidden from pickersKeeps customer traces clean
D11–D14Whole-trace evaluators, limits L1–L8, logs by text, empty errors is a valueSee phase 3More context, bounded cost
D16JSON mode for OpenAI-compatible judgesjson_object with a fallback retry; Anthropic unchangedWorks with DeepSeek
D17Skipped runsAll variables empty → no model call; >50% skipped → DegradedHonest health
D18Settings report whole-trace supporttraceEvaluators in settingsRemoved a 400 on every page load
D19Code evaluators off by defaultOn only locally for testing; staging unchangedRuns user code
D20Backward compatible both waysAdditive API changes onlyRollout safety
D22No edits to Orion shared components or design-system filesFeature CSS overrides onlyTeam rule
D23Analytics compares the same level only“Traces” = trace-level scores; a span score never pairs with a trace scoreLike Langfuse
D24“Scored” columnWas “Observations scored”; same countWhole-trace evaluators score traces
D25Log tie-breakEqual-rank logs: error spans first, then slow spansError spans’ own logs were dropped
D26Test sample has dataNewest sample with an input or outputFirst test was often on an empty span (S10)
D27Model picker: working connections firstNo key → “not set up”Unconfigured providers came first
D28Judge runs count as LLM callsLegend label “evaluation”Judge runs showed “LLM calls 0”
D29New rules sample 10%Observation rules too; sampling field in the saved dialogOne click at 100% ≈ $550–950/week
D30, D32, D33Cost estimateRecent real runs first; else real token averages as a rangeOld guess ran ~60% high. Details
D31MCP read tools5 tools; window capped by MCP_SEARCH_MAX_RANGE (24 h)Answer questions without the UI. Details
D34Last error matches the status windowDegraded/Failing evaluators show their newest error from the same 7 days as the status, with “Open trace”A Degraded evaluator showed no error or link when the failure was older than 24 h. Details
  • Sampling (#13, D29): trace rules default to 10% (L7), and since D29 new observation rules do too.
  • Health cutoffs: keep CtrlB’s ≤10% / ≤50% bands. Do not copy Langfuse’s “healthy only at 0 failed”.
  • Score configs (20c): a tab under Evaluators, not under Settings.
  • Annotation: phase 1 is built (rounds 5 and 8). Queues are phase 2.

Still open

Not done. Do not read these as shipped. Owner decisions are in Issues & Plans.

  • Delayed (#7): hide the always-0 Delayed count.
  • Annotation queues: phase 2. Also later: session scores, datasets and experiments.
  • Python on staging (#16): code evaluators should be off until the sandbox is ready. They still run. Infra work.
  • B3: CSV download still stops at 10,000 rows.
  • “Copy from evaluator”: a mismatch warning is not built yet.
  • G14: evaluator alerts — unchanged.
  • In progress: D35 Analytics drill-down and four small UI items — see In progress. (Done since: Test result showing cited spans (citations), “Scored” wording D24, test sample D26, model picker D27, MCP tools G13 / D31.)
  • Next phase — “professional polish”: after end-to-end sign-off. One-row header, a compact last-error line instead of the big banner, one-line help with ⓘ details, full-height split panes, tighter test-panel controls. Must follow the design system and not change design-system files.

Later / next release

  • Multi-label categorical scores (Langfuse’s multi-match), off by default.
  • Professional, Langfuse-like UI polish (above).
  • Older open items: the list above, plus open decisions 4–6.

CtrlB staging test — confirmed (2026-11-18)

Hands-on on CtrlB staging. The fixes were then tested end to end locally (2026-11-18/19 test); staging still runs the old code.

What was tested

  • Test evaluator shivam-101. LLM judge gpt-5-mini, score 0–1.
  • Rule shivam-102 on k8s_demo_llm chat spans at 0.05%, with backfill 20.
  • Unsaved tests of the decision model and a Python code evaluator.
  • Both test items were deleted afterwards. Their 18 score records stay in the lake.

Confirmed findings

IDSeverityStatusWhat we confirmed
S1 High Fixed r4 An empty numeric range lets the judge flip scales. The naming judge (gpt-5.5, generic prompt, no range) gave 1, 9, 9.5, 9.5 on the same answer in 4 tests. gpt-5-mini gave 0.9, 0.9, 1, 1. New judges now start at 0–1. Built, not on staging yet.
S2 High Fixed r1+r4 An evaluator can fail 100% silently. “Detect Out-of-Scope Request” was Failing, 0 scored, 23/23 errors. The reason (missing_variables: {{input}} resolved to nothing) was only inside the judge span output. Its mapping Input › [0] › content does not match OTel GenAI messages ([0] › parts › [0] › content). Now the error is shown (round 1). If every variable is empty, the run is skipped (round 4). Saving an empty mapping asks “Fix mapping” or “Save anyway”. Escape never saves. Built, not on staging yet.
S3 Med Fixed r1 The execution Error panel said “No error message” while the error was in Output. The error is now in the standard error fields, so the panel shows the reason. Built, not on staging yet.
S4 Med Fixed r1 The executions list showed Input —, Output —, 0.00 µs, 0 LLM calls, 0 tokens for every judge run. The span now stores the filled prompt, the reply, duration, tokens, and the error. Built, not on staging yet.
S5 Med Fixed r1+r6 Evaluator health ignored failed runs. shivam-101 said “Healthy: every run completed” while Scores said “18 scores (2 failed)”. Failed runs now count. Health is the last 50 runs in 7 days. Built, not on staging yet.
S6 Med Fixed r1 + 2026-11-19 Rule health did not reflect its evaluators. A rule showed Healthy while its only evaluator failed 100%. A rule now uses its own last 50 runs in 7 days (until 2026-11-19 it still used 24 h), and it is at least as bad as its worst active evaluator. Tested locally, not on staging yet.
S7 Med Fixed r2 Analytics used the latest 2,000 rows: “Showing the latest 2,000 of 54,712 scores”. With “Last 1 day” the trend only spanned ~06:00–08:00. Analytics is now SQL over the whole range. Pairs cap at 10,000. Old Seeker still falls back to 2,000. Built, not on staging yet.
S8 Med Fixed D29 The one-click “Start scoring” after saving defaulted to 100% sampling with no filter (≈ $331.83/week here; ≈ $550–950/week on the demo dataset with 3.4 M spans/week). New observation rules also defaulted to 100%. Now both start at 10%, the dialog has a sampling field, and its cost line names the rate (D29). Trace rules already defaulted to 10% (L7). Tested locally, not on staging yet.
S9 Med (security config) Open (infra #16) Python code evaluators run on staging. CtrlB’s plan says keep Python off until a non-root, no-network sandbox.
S10–S14 Low Open (S10, S14 fixed)
  • S10. The default test sample could be an empty span. Fixed It now starts on the newest sample with an input or output (D26).
  • S11. The decision score name auto-fills as answer_s_request, and typing appends to it.
  • S12. Rules are named after their filter (two “Type any of invoke_agent”).
  • S13. One existing rule is failing lake_query.
  • S14. /evaluators/analytics showed “Evaluator not found”. Fixed It now redirects to /evaluators/scores/analytics.
Worked well
  • First scores ~50 s after saving (Langfuse ~20 s).
  • Backfill of 20 in ~3 min for $0.0158.
  • With a range, scores were 0–1 with good reasons.
  • The decision model labels “Probabilities are model-estimated”.
  • Clear cost estimates and delete dialogs.

What is wrong in CtrlB (bugs)

B1 and B2 are fixed and tested locally. B3 is still open. Not on staging yet, so staging can still show the old behaviour.

B1 High Fixed · round 4 Numeric LLM scores have no range by default

The judge invented its own scale. The naming evaluator returned 0.9, 9 and 60 for the same kind of answer (about 400 on a 0–1 scale, about 1,400 on 1–10, about 12 values of 15–60 in one day). Means, charts and comparisons were meaningless.

With no min/max the instruction was just “score is a number”. New numeric judges now start at 0–1. An existing judge with no range shows a warning. Langfuse UI defaults to “between 0 and 1”.

Seen on staging (S1). Built, not on staging yet.

B2 Medium Fixed · round 2 Analytics compared on the latest 2,000 rows of all evaluators

Not of the two chosen scores. Total/Mean/Min/Max used all data (Seeker summary), but Std dev, Trend, Distribution, Matched, Pearson, Spearman, MAE, RMSE and the heatmap used the small unfiltered sample.

Example: naming vs testing_1234 → Matched 11 although both had 1,400+ runs. One card mixed “all data” and “sample” numbers (mean 2.49 with std 0.598 and max 60 is impossible).

Now the lake computes those panels for the two chosen scores over the whole range. Pairs cap at 10,000 (“Compared on 10,000 of N”). Old Seeker still falls back to the latest 2,000. Langfuse queries each score separately, up to 100k values.

Seen on staging (S7). Built, not on staging yet.

B3 Low Open CSV stops at 10,000 rows

Download CSV silently stops at 10,000 rows and fires about 50 browser requests (200 per page). With 1.1 lakh scores the file is partial (there’s a toast, but it’s easy to miss). Candidate for a server-side export.

What is missing (gaps)

Features Langfuse has that CtrlB does not — in simple words.

High Do these first

G1 Built · round 8 Manual (human) scoring

A person can mark a chat in the Annotate panel (whole trace or one span). Example: mark the crypto answer false. Built, not on staging yet. Queues are still later: decision.

G2 Built · round 5 Scores API

POST / PATCH / DELETE ai/scores. A signed-in user is ANNOTATION. An API key is API. Example: the app sends a thumbs-down on the crypto reply. Built, not on staging yet.

Medium Next

G3 Built · rounds 5 and 8 Score configs

Configs live in Mongo. The Score configs tab adds “open coding” (TEXT) and “pass or fail” (Pass=1, Fail=0) the first time a workspace opens it, and does not add them again if archived. Built, not on staging yet. Rounds.

G4 Built · trace + span Where a score can sit

Human and API scores can sit on a trace (empty span id) or a span (round 5). AI scores on a whole trace are built too (phase 3). Tested locally, not on staging yet. Still no session score and no dataset-run score. The Langfuse screen: Trace scores vs span scores.

G5 Agreed · phase 2 No annotation queues

CtrlB has none, so handing out 20–50 crypto chats is harder. How a Langfuse queue works: Annotation queues.

G6 Built · round 5 Analytics can compare judge vs human

You can compare “name · EVAL” with “name · ANNOTATION”. Example: the judge’s crypto score vs a reviewer’s. Built, not on staging yet. The lake query is round 2.

G6a Fixed · round 5 Source column

The Scores table shows the real source: EVAL, ANNOTATION, or API. Author is a column, hidden by default. Built, not on staging yet.

G7 Intentional Slower and less status tracking

Same speed and status gap as difference 3. The lake has no ingest hook, so CtrlB scans every 60 s (intentional differences). What a live run looked like: Running live.

G12 Built · phase 3 Whole-trace evaluators

Built and tested locally: the judge sees every span, errors and logs, so it can tell that another step failed (8 of 8 vs 5 of 8 in the test). Staging still has only span evaluators until deploy. Design: phase 3. Table: span vs whole-trace.

Low / note Later

G8 Intentional Decision models guess probabilities

How the two products differ: Decision model — how to set it up.

G9 Can’t compare judge v1 vs v2 on the same answers

A new version replaces the old one. Langfuse can compare versions only in experiments. CtrlB has no experiments.

G10 Intentional Code evaluators off by default

TypeScript and Python are off (own sandbox workers). Langfuse runs them in AWS Lambda or an external sandbox. How a Langfuse code rule works: Code evaluator.

G11 Agreed · later No datasets / experiments / CI gating

Langfuse has datasets, experiments, and a RegressionError GitHub Action. CtrlB has none.

G13 Built · D31 MCP tools for Evaluators, Rules, and Scores

Five read-only tools: list_evaluators, get_evaluator, list_evaluator_rules, search_scores, score_analytics. list_evals still lists the old Evals. Example: an assistant answers “which evaluators are failing?” without the UI. Built, not on staging yet. Details.

G14 Open No alerts from an evaluator

Langfuse has “Add alert”. CtrlB skipped alerts on purpose (plan decision E13).

What we plan to change

Status of each item. “Built” means in code and tested end to end locally (2026-11-18/19), not on staging yet.

  • Built · round 5 POST / PATCH / DELETE ai/scores → check auth, RBAC, the trace, and the value → the lake dataset _ctrlb_evaluators_<…>.
  • Built · round 5 Lake fields that shipped: source, author, and scope ctrlb.scores so human and API scores stay out of evaluator health.
  • Built · rounds 5 and 8 Score configs in Mongo, tab under Evaluators, two defaults. Decision: annotation.
  • Built · round 5 Read rules: missing source on old rows reads as EVAL; an edit or delete appends a new version and the latest row per score id wins (the lake is append-only).
  • Built · round 4 Default numeric range 0–1 (fix B1). Old judges with no range show a warning.
  • Built · round 2 Analytics for the two chosen scores over the whole range (fix B2). Pairs cap at 10,000. Old Seeker still falls back to the latest 2,000.
  • Built · phase 3 Idea I1: the whole-trace judge reads other spans and logs picked by text, found through the same correlation as the Related logs tab. Caps are L1–L8. Phase 3.
  • Built · phase 3 Idea I2: whole-trace evaluators (G12). Metrics, a root-cause label and “possibly related” logs are still later. Table: span vs whole-trace.
  • Built · round 7 Idea I3 — separate dataset for judge runs. Hidden dataset _ctrlb_evaluator_runs_<dataset>. Setting EVALUATOR_RUN_SPANS: inline / both (default, writes to both) / separate. Scores stay in _ctrlb_evaluators_<dataset>. Customer AI Traces drop judge traffic after ops sets separate, once the new Orion ships. CtrlB did this once (commit 7be3a545) before PR #516 moved runs back into the app dataset.

Idea I3 — status

#ItemStatusNote
1Engine writes run spans to the new datasetBuilt · round 7EVALUATOR_RUN_SPANS default both writes to the app dataset and to _ctrlb_evaluator_runs_<ds>
2Create + permission the datasetBuilt · round 7Hidden dataset, same org. Not on staging yet
3“View executions” opens the runs datasetBuilt · round 7“Open failed run” opens it too. Customer traces stay quiet after separate, once the new Orion ships
4Link back to the graded app spanBuilt · round 1Latest error links: “Open failed run” and “Open observation”
5Hide _ctrlb_* datasets from pickersBuiltHidden from normal pickers. separate is what stops judge traffic in the app dataset itself

Rough size: Seeker 150–250 LOC, Orion 80–150 LOC. Not chosen: the _ctrlb_meta org — cross-org reads, customers lose direct SQL/dashboards, strict scoping needed.

Backward compatibility — new Seeker must work with old Orion

  • Only add, never break: new optional fields, params and endpoints. Never remove or rename a field or change what it means.
  • Old requests behave the same.
  • Anything old Orion can’t follow ships behind a config flag or a “do both” phase until Orion is updated.
ChangeOld-Orion safe?How
I3 run spans to a new datasetBuilt · round 7EVALUATOR_RUN_SPANS: inline / both (default) / separate. separate waits until the new Orion ships
Hide _ctrlb_* datasetsBuiltOnly new Orion hides them; old lists unchanged
Human / API scores (G1/G2)Built · rounds 5 and 8ai/scores plus the Annotate panel. Not on staging yet
Default numeric range (B1)Built · round 4New judges start at 0–1. Old judges with no range show a warning
Analytics fix (B2)Built · round 2Lake SQL for the whole range. Pairs cap at 10,000. Old Seeker falls back to 2,000
Whole-trace evaluators (G12)Built · phase 3New optional fields; old Orion doesn’t show the option. Not on staging yet
Score configs (G3)Built · rounds 5 and 8Mongo, plus two defaults. Not on staging yet
Whole-trace support flagBuiltSettings report traceEvaluators; new Orion probes trace-preview only on older Seeker (works both ways)

A reviewer marks the crypto chat false (ANNOTATION). A thumbs-down from the app lands as API. Analytics compares judge vs human on the same chats, with real source labels. Queues are still later. Decision.

Decided: yes — phase 1 built (rounds 5 and 8), queues later (phase 2)

The decision is yes. Phase 1 is built and tested locally. It is not on staging yet. Queues wait for phase 2.

Why we said yes

Without human scores, nobody can tell if a judge is right. Example: the staging case “80% should_have_refused”. Range, analytics, and ideas I1–I3 are in Plans.

Where in CtrlB

#PlaceWhen
1AI Traces → Annotate (the whole trace, or one span)Built · round 8
2Scores tab → real Source (EVAL / ANNOTATION / API). Author column, hidden by default. Author or admin can edit or deleteBuilt · round 5
3Score configs — a tab under Evaluators (not Settings)Built · round 8
4Analytics → compare name · EVAL vs name · ANNOTATIONBuilt · round 5
5Annotation queuesOpen · phase 2

Behind it

  • Score configs live in Mongo.
  • POST / PATCH / DELETE ai/scores → the lake dataset, with source, author, and scope ctrlb.scores.
  • Trace-level scores are allowed: empty span_id (G4).
  • Backward compatible: new endpoints and optional fields only.

Phases

PhaseWhatStatus
1Configs + score API + Annotate + Source column + Analytics by sourceBuilt · rounds 5 and 8 — not on staging yet
2QueuesOpen — later

How Langfuse queues work today: Annotation queues.

What was decided

  • Yes. Phase 1 first. Queues later.
  • Score configs sit under Evaluators (decision 20c).
  • A signed-in user writes ANNOTATION. An API key writes API. The author or an admin can edit or delete.
  • Two default configs. Judge traces moved (#6).
Where the code is

Branch feat/evals-scores-v2. Built and tested locally, not on staging yet. The “copy from evaluator” mismatch warning is still open.

Open decisions (for the owner)

Found during the end-to-end test. 1–3 are now decided; the rest are not changed yet.

#QuestionContext
1Lower the per-span default on the “Evaluator saved” dialog?Decided · D29 New rules start at 10%, with a sampling field in the dialog. Was 100% with no filter (≈ $550–950/week on the demo dataset). See S8
2Fix the cost estimate?Decided · D30, D32, D33 Priced from recent real runs; else real token averages, shown as a range. Old guess ran ~60% high ($0.000263 vs real $0.000164 per call); the new fallback is ~11% low ($0.000146). Details
3What does Analytics “Traces” mean?Decided · D23 Like Langfuse: only trace-level scores; a span score never pairs with a trace score
4Scores chart over the whole range?Today it covers only loaded rows, with a note. Options: load all, or a server aggregate
5Comparison rows still openCategorical multi-match (next release, off by default); test call to the model on save; hide always-0 Delayed; Python on staging (#16); “Copy from evaluator” warning; evaluator alerts (G14); CSV cap 10k (B3). Done since: cited spans in the Test result and on the Traces drawer, test sample (S10, D26), “Scored” wording (D24), log tie-break (D25), model picker order (D27), MCP tools (G13, D31)
6API timing (reported, not changed)New evaluator sample list ~9 s on large datasets (rows query, then a full-day COUNT; pre-existing); attributes fetched for all traces datasets on that page; AI drawer fetches score configs 3× and the trace 2×; Analytics ~2.4 s
7“Professional polish” phaseAfter sign-off: one-row header, compact last-error line, one-line help with ⓘ, full-height split panes, tighter test-panel controls. Design system only; no design-system file changes

FAQ

Short answers. Acme examples where they help.

  • Neither Langfuse nor CtrlB fetches docs when it grades. The app already saved them on the span.
  • Example: the crypto answer can be checked only if answer-llm’s Input contains “Docs: …” — even “(no matching docs)”.
  • Walk-through: where input and output come from and which span to grade.
  • Observation evaluators: one span’s gen_ai input and output messages.
  • Whole-trace evaluators (phase 3, built and tested locally): a short waterfall of every span, errors, and up to 100 logs picked by text. Not on staging yet. Phase 3.
  • Example: the crypto judge can say the docs search timed out, and point at that span (cited spans).
  • Scores go into the lake dataset _ctrlb_evaluators_<traces dataset>.
  • Same ingest path the product already uses for logs.
  • Scores and Analytics already read that dataset.
  • Score values stay in the lake, not as Mongo score rows.
  • Mongo holds evaluators, versions, rules, settings, health, and score configs (round 5).
  • Example: “pass or fail” is a config in Mongo. The Fail on the crypto chat is a lake row.
  • No. Round 2 computes totals, trend, distribution, and the comparison in the lake, for the whole range.
  • Matched pairs stop at 10,000. The note says “Compared on 10,000 of N”.
  • Old Seeker still falls back to the latest 2,000. Built, not on staging yet. History: B2.
  • No. A new numeric judge starts at 0–1 (round 4).
  • An older judge with no range shows a warning.
  • Example: the naming judge used to return 0.9, 9, and 60. That was B1.
  • No. OpenAI-compatible judges are asked for JSON mode (json_object).
  • If the provider rejects it, CtrlB retries once without it and remembers that model. Anthropic is unchanged. JSON inside text still parses.
  • Example: DeepSeek accepts json_object and rejects json_schema. Built, not on staging yet.
  • No. It is the last 50 runs in the past 7 days (round 6), stored in Mongo.
  • Healthy ≤10% failed. Degraded above 10% and up to 50%. Failing above 50%. Skipped and Test runs are not counted.
  • Rules use the same rule over their own last 50 runs (since 2026-11-19), and are at least as bad as their worst evaluator. A rule or evaluator with no record yet falls back to 24 h.
  • Example: 4 failures in 50 runs is Healthy in CtrlB. Langfuse is Healthy only when none failed.
  • It meant the Source column always printed EVAL, written in the UI, even if the row was not from a judge.
  • Round 5 writes the real source. The column shows EVAL, ANNOTATION, or API.
  • Example: a thumbs-down from the app shows API, not EVAL. Built, not on staging yet.
  • How linking works: CtrlB finds a trace’s logs by looking for log rows that carry the same trace_id, in the same time window.
  • No trace_id in the logs (app didn’t add it) → simply no logs found. No error.
  • Other spans are always there — every span of a request shares the trace_id.
  • Phase 3 (built): the waterfall includes the most important logs that share the trace (any level, picked by text). If none are linked, those lines are simply absent. A trace with no errors and no problem logs gets “No errors or problem logs in this trace.” in {{errors}}. Not on staging yet.
  • Example: a crypto chat with linked logs can mention “docs search timed out”. With no linked logs, phase 3 still has the spans.
  • No logs, but a system error: the waterfall still includes span errors (status ERROR, exceptions). That is part of phase 3, built.

Glossary

Short definitions. Same words used above.

Logs vs spans vs traces
A trace is the whole request: a tree of spans that share one trace_id (one crypto chat). A span is one timed step — start, end, duration, status OK or ERROR, attributes, and events including exceptions (example: answer-llm; Langfuse also calls this an observation). A log is a free-text line at any moment; a step can have many or none. Spans show where and how long. Logs show what happened, including caught errors, retries, and things outside a request.
CI/CD
The pipeline that builds and deploys each commit. Example: GitHub Actions stamps service.version with ${{ github.sha }}.
service.version
A label on every trace, log, and metric naming the commit or release. Example: abc123 or 2026.w44.
Structured output
The model must reply in a fixed JSON shape. Langfuse always sends json_schema (OpenAI-compatible) or a forced tool call (Anthropic). CtrlB asks OpenAI-compatible models for json_object, retries once without it if they refuse, and also accepts JSON inside text.
Thinking mode
The model reasons in a hidden step before the answer. DeepSeek’s Anthropic endpoint does this unless the body says thinking: disabled.
Observation type
The kind of step Langfuse stores (SPAN, GENERATION, TOOL, and others). The app chooses it when recording. Example: answer-llm is GENERATION.
SDK
Library inside the app that records steps and sends them.
OpenTelemetry / OTLP
Standard way to record spans and send them over HTTP.
Evaluator
Automatic checker set up once (LLM, decision, or code).
Judge
The AI call that grades an answer using your instructions.
Rule
Filter + sampling + which evaluators run on matching spans.
Sampling
% of matching spans that get checked (e.g. 10% to save money).
Score
Value + reason stuck on an answer (or trace/span).
Score config
Allowed name + type + range/labels (e.g. answer_correct = Boolean).
Source (EVAL / ANNOTATION / API)
Who created the score: AI checker, person in UI, or app API.
Annotation queue
List of items for humans to score, with “Complete + next”.
Backfill
Run evaluators on past data (old chats).
Execution
One run of an evaluator on one span (job / status).
Lake
CtrlB’s log store; scores sit in _ctrlb_evaluators_… datasets.
Matched
Items that have both scores you are comparing.
Agreement
Same answers ÷ matched (boolean/categorical).
κ (Cohen’s kappa)
(agreement − chance agreement) ÷ (1 − chance agreement).
F1
2 · precision · recall ÷ (precision + recall).
Pearson
Do two numeric scores move together in a straight line (−1…1).
Spearman
Do two numeric scores share the same ranking order.
MAE
Average of |A − B|.
RMSE
√(average of (A − B)²); big gaps count more.