A customer asks Acme’s support chatbot: “Can I pay with crypto?”
Acme’s docs say nothing about crypto. The bot wrongly answers: “Yes, Acme Cloud accepts crypto payments.” The judge should mark it false: “made up, the docs don’t mention crypto.”
How automatic checkers grade chatbot answers, how scores are stored, and where Langfuse and CtrlB match or differ — told through Acme’s support bot.
Rounds 1–8 and phase 3 (whole-trace evaluators) are built and were tested end to end on local Seeker + Orion with ~2,300 real DeepSeek traces (2026-11-18/19). Not on staging yet. What we built. Final additions the same day (D24–D34): a cost estimate from real runs, read-only MCP tools, safer 10% rule defaults, a Scores tab in the AI trace drawer and small UI fixes — final changes. Judge runs now live in their own traces (separate mode, verified locally).
From chat to grade to “does the checker agree with people?”
A customer asks Acme’s support chatbot: “Can I pay with crypto?”
Acme’s docs say nothing about crypto. The bot wrongly answers: “Yes, Acme Cloud accepts crypto payments.” The judge should mark it false: “made up, the docs don’t mention crypto.”
App sends a copy of every chat
An AI or code checker grades each answer
The grade is saved next to the chat
People check a few by hand
Analytics shows if the checker agrees with people
The cast of Acme’s crypto question.
| Role | What it is | In the Acme story |
|---|---|---|
| Customer | Asks the question. | “Can I pay with crypto?” |
| The bot | The company’s own app / code. Answers the customer. | Acme’s support chatbot app |
| AI model as the bot’s brain | Writes the answer (e.g. DeepSeek). | Produces “Yes, Acme Cloud accepts crypto…” |
| SDK inside the app | Langfuse/CtrlB SDK inside the app. Records each step and sends it. | Sends the question, docs lookup, and answer |
| Langfuse / CtrlB server | Stores the data and shows it in a UI. | You open the crypto chat later |
| Evaluator | Grades the answer later (automatic checker). | Marks the crypto answer false |
| AI model as the judge | A separate AI call, made by the server using the LLM connection. | Says: made up — docs don’t mention crypto |
DeepSeek can play two roles: the bot’s brain and the judge. Those are two separate calls.
One customer question becomes one tree of steps.
Example: Customer asks about crypto. That is one trace with three spans:
| Span | What it is | Why it matters |
|---|---|---|
support-chat |
Root span — the whole request. | Holds the full chat as one unit. |
retrieve-docs |
Looked up help articles. Found none about crypto. | Shows the bot had no docs to lean on. |
answer-llm |
The AI call: input = system prompt + docs + question; output = answer; plus model and tokens. | This is usually what the judge grades. |
Animated dashed lines show the tree. Respects reduced-motion settings.
answer-llm input, or the judge has nothing to compare.Find articles → send them with the question → AI writes a reply.
Empty docs + weak “don’t invent” rules → wrong confident answers like Acme’s crypto reply.
Set up a checker once. Stick a grade on every matching answer.
What: An automatic checker you set up once.
answer-llm.Why: Thousands of chats. Nobody can read them all.
What: The result stuck on the answer — a value plus a reason.
false — “made up, the docs don’t mention crypto”.Why: You can filter “show all false” and fix those first.
LLM-as-a-judge, decision model, or your own code.
What: You write instructions with {{input}} and {{output}}. The tool fills them with saved span data and sends them to an AI. The AI replies {score, reason}.
How the grade is decided: 0.1 vs 1. The form: evaluator form.
What: Fixed short questions (choice / scale / yes-no). Answers with how sure it is (e.g. yes, 92%). One call for all questions.
How to set it up, and why only some models work: Decision model — how to set it up.
What: Your own function evaluate(ctx) returns scores. No AI.
How a rule runs: Code evaluator — how it works.
You define “correct.” The AI compares the answer to your rules and the docs.
You are grading Acme’s support answers.
Rules:
- Only facts from the provided docs are allowed.
- If the docs are empty or silent, the answer must say so — not invent.
Scale (numeric 0–1):
1.0 = all claims come from the docs
0.5 = half guessed
0.0 = made up
INPUT (docs + question):
{{input}}
OUTPUT (bot answer):
{{output}}
Return {score, reason}.
Two judges (or two prompts) can disagree on the same crypto answer. That is why humans still score a sample.
Filter + sampling + which evaluators. New matches run automatically.
answer-llm spans (skip retrieve-docs).The screens we filled in: Evaluator vs rule and the rule form.
sha256 of the span id, per observation.fnv32 of the trace id, per trace. Different method — not a bug.Pick a type — and for numbers, set a range.
A number. Set a range like 0–1.
True or false.
Your own labels.
Free-form notes.
A new numeric judge starts at 0–1, same idea as Langfuse. An older judge that still has no range shows a warning — the judge can invent a scale. That was bug B1 (fixed in round 4, not on staging yet).
Example: before the fix, the naming judge returned 0.9, 9, and 60 for the same kind of answer.
AI, a person, or the app. All three can sit side by side.
| Who | Source | What | Example |
|---|---|---|---|
| AI checker | EVAL | Automatic evaluator | false — “made up, docs don’t mention crypto” |
| A person in the UI | ANNOTATION | Needs a score config first | Human marks the same chat false |
| The app via API | API | Your product sends a score | Customer clicks 👎 on the crypto reply |
answer_correct = Boolean.open coding (Text) and pass or fail (Pass=1, Fail=0). CtrlB adds them the first time a workspace opens Score configs, and does not add them again if they were archived.POST / PATCH / DELETE ai/scores. The author or an admin can edit or delete.ctrlb.scores, so they do not change evaluator health.Example: a reviewer marks the crypto chat false. The judge’s score stays beside it. Queues come later (phase 2). Detail: What we built.
Trust the AI only after it agrees with people on a sample.
Scores API use cases (beyond 👍/👎):
Totals and shape for a single checker.
If groundedness mean is 0.4 this week, many answers are half-guessed or worse — like the crypto reply.
Same data type only. Matched = items that have both scores.
Both sent by our script via the API (not live customers).
κ = (0.75 − 0.51) ÷ 0.49 ≈ 0.49.
Confusion matrix:
Customer true / QA false (3 in the example): the customer is happy with a wrong answer — like liking the crypto reply when docs never said crypto. Silent failure.
Note: these ratings were random numbers made by our test script to demo the page, not judge output.
| # | Score A | Score B |
|---|---|---|
| 1 | ||
| 2 | ||
| 3 | ||
| 4 | ||
| 5 |
Strength labels
We added DeepSeek as the model that grades answers.
openai. Base URL: https://api.deepseek.com.deepseek-flash and deepseek-v4-pro.The connection is just “which AI to call.” The grading instructions come later, on the evaluator.
What you fill in before a checker can grade anything.
{{input}} and {{output}}. “+ Add message” adds another message.{{input}} → Input. {{output}} → Output. The pencil changes that to metadata, tool calls, or a JSON path.Name the checker “grounded”. Map both blanks. Pick the crypto row. “Run test on this sample” shows the grade before you save a rule.
The form does not invent the chat. It pastes what was saved earlier.
langfuse.observation.input / output).Like mail merge: the prompt is the letter, each span is a spreadsheet row, and the mapping is which column fills which blank.
Select the crypto row and {{input}} becomes the messages already stored on that span — not a new lookup. Which span actually contains the docs: which span to grade.
Grade the step that contains the docs, not the root.
What the steps are called: Trace & span.
support-chat only has {"question"}. No docs. The judge cannot check the answer against docs.answer-llm GENERATION span’s Input has the system prompt, “Docs: …”, and the question. Grade that one.Input, system: “You are a cheerful assistant… fill gaps with your best guess.”
Input, user: “Docs: - (no matching docs) Question: Can I pay with crypto?”
Output: “Yes! Acme Cloud accepts crypto payments—typically BTC, ETH, and USDC—at checkout.”
There is no crypto article. A good judge marks the “Yes!” answer as made up.
Langfuse labels each step. The app chooses the label when it records the step.
| Type | What it is | Example |
|---|---|---|
| SPAN | A general step | support-chat |
| GENERATION | An AI model call | answer-llm |
| RETRIEVER | A search | retrieve-docs |
| TOOL | A tool, API, or MCP call | a pricing API call |
| AGENT | An agent decision | the agent picks the next tool |
| CHAIN | A fixed sequence | step A then step B |
| EMBEDDING | Text turned into vectors | embedding the question |
| GUARDRAIL | A safety check | block a risky reply |
| EVALUATOR | An eval step recorded in the trace | the judge run itself |
| EVENT | A single moment, no duration | a click |
Filter quick buttons:
isRootObservation:truetype:GENERATIONtype:TOOLA generation is never the root. type:GENERATION picks only the AI-call span of each trace (40 traces → 40 answer-llm spans), not the whole trace.
CtrlB uses gen_ai_operation_name and the span name. Rule filters can use any span field.
The checker and the switch are two different things.
“Grounded” (the instructions) does nothing until a rule says “run it on answer-llm, 100% of the time.”
This is the switch that makes an evaluator run.
Langfuse shows $0.00 when it has no price for the model (for example deepseek-chat). Add the price under Settings → Model Definitions. CtrlB’s estimate returns “unknown” (known:false, cost null) instead of $0.
Two connections failed. One model name worked.
Error: “This response_format type is unavailable now.”
response_format: json_schema).supportsStructuredOutputs: true is hard-coded for OpenAI-compatible base URLs.json_schema on any model. We tried /, /beta, flash, v4-pro, and chat.Base URL https://api.deepseek.com/anthropic, model deepseek-flash. Error: “Thinking mode does not support this tool_choice.”
thinking: disabled.Same Anthropic connection, but model name deepseek-chat (a non-thinking alias). The judge returned false, with correct reasoning.
“V4.1-Flash-1017” is only the new version behind deepseek-flash. The account lists just deepseek-flash and deepseek-v4-pro.
On OpenAI-compatible providers, CtrlB sends response_format: json_object. If the provider rejects it, CtrlB retries once without it and remembers that model. Anthropic is unchanged. The parser also accepts JSON written inside text.
Example: DeepSeek accepts json_object. It rejects json_schema, which is why Langfuse’s OpenAI adapter failed above. Built, not on staging yet. Wording: parity row 11.
Starter instructions, plus buttons on the evaluator page.
Langfuse has 24 templates in 6 categories:
The evaluator page also has “Rules”, “Add alert”, and “Version history”.
CtrlB did not ship alerts on purpose (plan decision E13). Gap: G14.
Start from a Quality template, then change the instructions for Acme’s “don’t invent crypto” rule. The template is a copy, not a shared prompt.
An evaluator sits still until a rule is attached. Then it grades new spans by itself.
What a rule is: Evaluator vs rule.
missing_variables.This opens the judge’s own traces, not the bot’s chat.
evaluatorId + environment langfuse-llm-as-a-judge + isRootObservation:true.{"scoreExplanation": …, "score": …}.target_trace_id / target_observation_id = which bot span was graded.target_observation_id is the bot span that was graded (in our story, answer-llm)._ctrlb_evaluator_runs_<dataset>. Setting EVALUATOR_RUN_SPANS: inline, both (default — writes to both), or separate. “View executions” and “Open failed run” open the new dataset.separate (once the new Orion ships), the customer’s AI Traces stop showing judge traffic. Built, not on staging yet.A score sticks to the whole request, or to one step. The screen never mixes the two.
What a trace and a span are: Trace & span. Who can write a score: Who can give a score.
traceId and an observationId.observationId empty = trace-level (the whole request).observationId set = span-level (one step).Trace link opens the trace with the root span selected. Its Scores tab shows trace-level scores. Ours: user_feedback = False, sent by our script with score_trace.
Observation link selects that span. Its Scores tab shows only that span’s scores. Ours: answer_correct = True from the judge on answer-llm.
nodeScores.ts): “badges never mix levels”.answer-llm is selected, and Annotate still writes a trace-level score.A person or the API can score the whole trace (empty span id) or one span. The Annotate panel (round 8) does the same: whole trace or one span. AI scores on a whole trace are built too (phase 3, a whole-trace evaluator). No session scores and no dataset-run scores (G4).
Example: a reviewer marks the crypto trace false while the judge’s score stays on answer-llm.
Short fixed questions. One model call answers all of them, and says how sure it is.
Short intro: Three kinds of checker.
input and output.| Type | What you fill in | What gets saved |
|---|---|---|
| Choice | Options. Each option is a label plus a description. | A category (for example grounded / partly_grounded / made_up). |
| Score | At least 2 and up to 10 levels, low to high. Each level has a description. Otherwise it shows “Needs attention — add at least 2 levels”. | A number from 0 to levels − 1. It is the expected level, so it can fall between levels (for example 1.9). |
| Yes / no | Optional “refine what yes and no mean”. | A number from 0 to 1. That number is the probability of yes. |
input from Input, output from Output, and + Add field.Choice. “Does output only use facts from the docs in input?” Score name: groundedness.
Score. “How confident does output sound?” Levels: 0 unsure, 1 neutral, 2 very confident. Score name: answer_confidence.
Yes / no. “Does input ask about payment methods?” Score name: asks_payment.
Expected on the crypto row: made_up, ~1.9, ~0.97.
gpt-6-luna on OpenAI, gemini-decision-1 on Google AI Studio, and TypeSafe’s Jev via its /systemone API.gpt-6-luna, which DeepSeek rejects. TypeSafe needs a /systemone API, which DeepSeek does not have.gemini-decision-1 is available on it (not confirmed).ctrlb_evaluator_probabilities). That is flexible, and it is not calibrated.Langfuse chooses reliability (fewer models, trustworthy output). CtrlB chooses flexibility (more models, less guarantee).
A small rule you write. No AI, no API call, no internet. Same result every time, and free.
Short intro: Three kinds of checker.
Python shape, from Langfuse’s own template:
def evaluate(ctx: EvaluationContext) -> EvaluationResult:
…
return EvaluationResult(scores=[Score(name=…, value=…, data_type="BOOLEAN", comment=…)])
If the input says “no matching docs”, the answer should say “I don’t know”.
Crypto row → false. Linux row → true.
Turn the input into text (json.dumps if it is not a string). Count the characters. True if there are more than 1.
JSON.parse. If it throws, the text is not valid JSON.JSON.parse throw fails the check. No model is called.| LLM judge | Code | |
|---|---|---|
| Who checks | AI | Your rule |
| Understands meaning | Yes | No |
| Cost | Money | Free |
| Same result every time | Not always | Always |
A to-do list of traces for people to review, one after another.
A person’s score is source ANNOTATION. Who can score: Who can give a score. CtrlB does not have queues yet: G5.
answer_correct), optional user assignment.The judge scored the answer-llm span. Annotations from a root-span queue land on the trace. If Analytics shows Matched = 0, add answer-llm observations to the queue (filter type:GENERATION), or compare at the right object type. Levels: Trace scores vs span scores. Matched: comparing two scores.
The short version. The how-it-works sections below are the detail.
Both can grade one span or the whole trace. CtrlB’s whole-trace judge (phase 3) also reads non-AI spans (DB, HTTP) and the trace’s logs. Built and tested locally, not on staging yet.
Human and API scores can already sit on the trace. Table: span vs whole-trace. Design: phase 3.
Langfuse: the judge, a person, or the app. CtrlB phase 1 does the same (rounds 5 and 8): Annotate, a scores API, and score configs. Built and tested locally, not on staging yet. Queues are later.
See who scores and the decision.
Langfuse: seconds, with Pending / Delayed / Completed / Error, and retries.
CtrlB: a 60 s scan, first scores ~50 s after saving in our staging test (Langfuse ~20 s), no Delayed status, and only short retries. What we saw live: Running live.
Langfuse pre-fills 0–1, forces the reply shape, and test-calls the model when you save.
CtrlB now starts new numeric judges at 0–1 and asks for JSON mode, with one retry if the provider says no (B1 is the old empty range). It still does not test-call the model when you save.
Langfuse compares judge vs human on the full data.
CtrlB now computes analytics in the lake for the two scores you pick, over the whole range (pairs capped at 10,000). Judge vs human vs API works (picker shows name · source). A manual recount matched to the digit. The old 2,000-row sample is B2, fixed. Old Seeker still falls back to it. Not on staging yet.
Logs, infra spans, and metrics. The answer text only shows what the bot said. A log can show a timeout, a retry, or a hot database — the reason the answer went wrong. Langfuse never receives logs. Phase 3 (built) is where CtrlB uses non-AI spans and correlated logs. In a local test of 12 silent billing failures, the judge with the whole trace and logs caught 8 of 8 bad answers; with input/output only it caught 5 of 8 (test). Design.
The API and the engine behind CtrlB’s Evaluators page.
ai/evaluators endpoints: evaluators (create, read, update, delete, versions, test), observations, estimate, rules (create, read, update, delete, enable), executions and executions/search, scores/summary, scores/histogram (GET and POST), scores/values, settings.ai/evals endpoints are still there. Judges and LLM keys from that old API are still used.EVALUATORS_ENGINE_ENABLED is on. It grades, then writes scores into the lake with an internal ingest (POST …/_json_evolving). Timing detail: How CtrlB works.POST / PATCH / DELETE ai/scores. Analytics (round 2): POST ai/evaluators/scores/analytics.evaluators with 5 read-only tools — list_evaluators, get_evaluator, list_evaluator_rules, search_scores, score_analytics. list_evals still covers the old Evals. Closes gap G13; details: MCP tools.POST ai/evaluators/estimate prices from the evaluator’s recent real runs, else from real workspace token averages, shown as a range. Cost estimate.Ingest → match rules → job → judge → score in ClickHouse.
JobExecution row (PENDING) in Postgres + a queue job.scores table (source, author, config, comment).COMPLETED.DELAYED and retried over minutes.langfuse-llm-as-a-judge. What “View executions” opens: Running live.Scan every 60 s. Scores land as lake logs about 2–3 minutes later.
gen_ai_input_messages / gen_ai_output_messages. A whole-trace evaluator reads {{trace}}, {{errors}}, {{input}}, {{output}} (phase 3).json_object), with one retry without it if the provider says no. The parser also accepts JSON inside text._ctrlb_evaluators_<traces dataset>. Human and API scores use the same dataset, scope ctrlb.scores, via ai/scores.ctrlb-evaluator) and move to hidden dataset _ctrlb_evaluator_runs_<dataset>. The same idea in Langfuse: Running live./ai/evals page is switched off; its judge key store is still used.Customer asks about crypto at 10:00. Bot answers. Score about crypto hallucination shows up around 10:02–10:03 — not instantly.
| Span evaluator | Whole-trace evaluator | |
|---|---|---|
| Looks at | One step (e.g. answer-llm) | All steps of the request |
| Runs | As soon as that step arrives | After the request finishes (no new steps for a while) |
| Score goes on | The span | The trace |
| Sees “the docs search failed”? | ❌ | ✅ (Langfuse adds an “execution issues” list: ERROR spans + message) |
| Cost | Small | Bigger (more text) |
| What it reads | That span's input/output | Langfuse: the LLM conversation of the trace + ERROR/WARNING spans. CtrlB (built): every span incl. DB/HTTP calls, their errors, and up to 100 logs picked by text, not level (refs [s1]… so the judge can cite spans) |
| Langfuse | ✅ | ✅ new “trace transcript” evaluators (idle 1–60 min, LLM judge + decision model). The old trace evaluator (input/output only) is deprecated |
| CtrlB | ✅ | ✅ Built (phase 3), tested locally, not on staging yet. Compact waterfall, quiet wait, then one grade; cited spans highlighted in the Traces drawer. Limits |
Span evaluator on answer-llm: “made up — no docs”. Whole-trace evaluator: “made up because retrieve-docs failed (timed out)”.
prompt problem vs system problem → the right team fixes it.38 behaviours compared (2026-11-18, updated 2026-11-19 after the end-to-end test).
Identical = 1, close = 0.5 → (14 + 8.5) / 38 ≈ 59%. It was 36% at the start of 2026-11-18, and 54% after rounds 1–8. The final changes D24–D33 do not move any row: rows 6 and 15 get better details but stay close; the cost estimate, MCP tools and UI fixes are not parity rows. The walkthrough fixes and D34 move no row either: the drawer’s new Scores tab shows scores on the trace and span themselves (a Langfuse gap now closed), but no row tracks that. Row 29 is already identical, row 32 stays close (no queues), row 36 stays close (no session or dataset-run scores), and D34 changes how row 26 shows errors, not its health rule.
Updated 2026-11-19: rows 20 (delay) and 21 (whole trace) moved from missing to close, 28 (storage) from different to close, 29 (Scores table) from close to identical. Built and tested locally, not on staging yet. Bar percents use the largest remainder so they add to 100 (14/38, 17/38, 4/38, 3/38).
| # | Behaviour | Parity | Langfuse | CtrlB |
|---|---|---|---|---|
| 1 | 3 evaluator types (LLM, decision, code) | ✅ identical | yes | yes |
| 2 | Save = new version; rules use latest | ✅ identical | yes | yes |
| 3 | Delete removes versions + rule links | ✅ identical | cascade | yes (rule with none left is disabled) |
| 4 | Project default model + per-evaluator model | ✅ identical | yes | yes |
| 5 | Prompt messages (system/user/assistant) with {{vars}} | ✅ identical | yes | yes |
| 6 | Score types numeric/boolean/categorical | 🟡 close | + categorical multi-match | no multi-match. Multi-label categorical is planned for the next release, off by default |
| 7 | Numeric range default | ✅ identical | UI pre-fills 0–1 | new judges start at 0–1 (round 4). Old judges with no range show a warning. History: B1 |
| 8 | Variable mapping with JSON selector | 🟡 close | jsonSelector, fallback to whole value | JSONPath subset |
| 9 | Template gallery | 🟡 close | 24 managed templates | static catalogue in Orion |
| 10 | Save-time test call to model (blocks bad save) | ⚫ missing | yes | only checks connection exists |
| 11 | Judge output enforcement | 🟡 close | strict structured output always on (json_schema for OpenAI-compatible, forced tool call for Anthropic); DeepSeek only works via the deepseek-chat alias on the Anthropic endpoint | JSON mode (json_object) on OpenAI-compatible providers; one retry without it if rejected, and that model is remembered. Anthropic unchanged. Also accepts JSON inside text. DeepSeek accepts json_object (it rejects json_schema). Still no save-time test call |
| 12 | Reject out-of-range score when range set | ✅ identical | zod min/max | ErrInvalidScore |
| 13 | Decision model | 🔴 different | allowlisted decision models, real probabilities | any chat model, estimated probabilities |
| 14 | Code evaluator contract (evaluate(ctx) → scores) | 🟡 close | TS+Python on, source 256 KB, result 256 KB, Lambda/external | TS+Python off by default, source 20k chars, result 64 KB, own workers |
| 15 | Test on a sample observation | 🟡 close | observation filter + quick buttons | observation filter. The default sample is the newest one with an input or output (D26); whole-trace evaluators test on a sample trace |
| 16 | Rule filters | 🟡 close | observation + trace filters | one traces dataset + alert filter JSON |
| 17 | Deterministic sampling | 🟡 close | sha256(span_id), per observation | fnv32(trace_id), per trace |
| 18 | Several evaluators per rule, per-rule mapping | ✅ identical | yes | yes (max 10) |
| 19 | Run on past data | 🟡 close | timeScope EXISTING / batch evaluate | backfill ≤1,000, range |
| 20 | Rule delay setting | 🟡 close | delay per rule | trace rules: quiet period 2 / 5 / 15 min (L5). Observation rules: fixed 120 s watermark. Intentional |
| 21 | Whole-trace (transcript) evaluators | 🟡 close | trace transcript rules with idle time; transcript includes “execution issues” (ERROR/WARNING spans + message); the old trace evaluator is deprecated | built (phase 3), more context: every span incl. DB/HTTP, errors, logs picked by text, caps L1–L8, cited spans. Tested locally. Design |
| 22 | Trigger timing | 🔴 different | on ingest (seconds) | scan every 60 s (~2–3 min) |
| 23 | Job record + statuses | 🔴 different | JobExecution: PENDING/DELAYED/COMPLETED/ERROR/CANCELLED | no row; completed/error, pending = backfill, delayed 0 |
| 24 | Rate-limit retries | 🔴 different | queue retries over minutes, DELAYED | in-call retries 1 s, 3 s |
| 25 | Block reasons | 🟡 close | 8 reasons | 3 reasons |
| 26 | Evaluator health window | 🟡 close | last 50 runs in 7 days. Healthy only at 0 failed. Failing at 50% or more. Else Degraded | last 50 runs in 7 days, stored in Mongo (round 6). Healthy ≤10% failed, Degraded above 10% and ≤50%, Failing above 50%. Skipped and Test runs are not counted. Rules use the same window (2026-11-19) |
| 27 | Never evaluate own judge traces | ✅ identical | yes | yes |
| 28 | Score storage | 🟡 close | one table with source, author, config, upsert | the customer’s own lake dataset, with source and author (SQL and dashboards work on it). Human/API scores use scope ctrlb.scores. Edit/delete append a new version; reads take the latest. Configs live in Mongo. Intentional |
| 29 | Scores table page | ✅ identical | yes | yes. Rows not clickable; Trace link opens the trace peek, Observation link opens it with that span selected; each has its own URL |
| 30 | Score analytics | 🟡 close | per-score queries up to 100k | lake SQL over the whole range (POST ai/evaluators/scores/analytics). Pairs capped at 10,000 (“Compared on 10,000 of N”). Old Seeker still falls back to the latest 2,000. History: B2 |
| 31 | Score sources EVAL/ANNOTATION/API | ✅ identical | yes | real Source. Human/API scores use scope ctrlb.scores so they stay out of evaluator health |
| 32 | Manual scoring in UI | 🟡 close | yes | Annotate panel: whole trace or one span (round 8), edit and delete. Tested locally. No queues yet |
| 33 | Scores API create/update | ✅ identical | yes | POST / PATCH / DELETE ai/scores. Tested locally |
| 34 | Score configs | ✅ identical | yes (+2 defaults) | yes, in Mongo, plus the same 2 defaults (“open coding” TEXT, “pass or fail” Pass=1 / Fail=0). Tab under Evaluators |
| 35 | Annotation queues | ⚫ missing | yes | no |
| 36 | Scores on trace/session/dataset run | 🟡 close | trace, span, session, dataset run | trace and span, for human, API and evaluator scores (whole-trace evaluators, phase 3). No session or dataset run (later) |
| 37 | TEXT scores | ✅ identical | annotation/API/code | annotation, API, and code. Default config “open coding” is TEXT |
| 38 | Datasets / experiments | ⚫ missing | yes | no (planned later) |
These differ on purpose. Do not “fix” them to match Langfuse.
| Area | Langfuse | CtrlB | Why |
|---|---|---|---|
| Variable mapping | jsonSelector | JSONPath subset | Enough for OTel GenAI messages |
| Templates | 24 managed, server-side | Shipped in Orion | No server catalogue needed |
| Judge output | Strict JSON schema / forced tool call | JSON mode (json_object) | Works with DeepSeek; strict schema doesn’t |
| Decision model | Special decision models | Any chat model, estimated probabilities (labelled) | Those models aren’t available to CtrlB |
| Code evaluators | On | Off by default | Runs user code; keep off until the sandbox is hardened |
| Rule filters | Own filter UI | Reuses the alert filter builder | One filter model across CtrlB |
| Sampling | Per observation | Per trace | Spans of one request are graded together |
| Past data | Batch evaluate | Backfill ≤1,000 with cost estimate | Cost control |
| Delay | Per rule | Quiet period only for trace rules | Observation rules need no extra wait |
| Whole trace | LLM conversation + execution issues | All spans, errors, logs, with caps L1–L8 | More context finds hidden failures (8/8 vs 5/8) |
| Trigger | On ingest (~20 s) | Scan every 60 s (~1–3 min) | The lake has no ingest hook |
| Job statuses, retries | Job rows, DELAYED, queue retries | Follow from scanning | Same reason |
| Block reasons | 8 | 3 | The 3 that apply to CtrlB connections |
| Score storage | One ClickHouse table, upsert | Customer’s lake; explicit edit/delete | SQL and dashboards on scores; the lake is append-only |
Planned later (not intentional): annotation queues (phase 2), session scores, datasets and experiments.
Same checkers. Different apps. Pick the mode that matches the check.
| App | Example check | Best mode |
|---|---|---|
| Chatbot | Is it relevant, the right tone, and correct? | LLM judge |
| RAG knowledge assistant | Is the answer backed by the retrieved pages? | LLM judge |
| Document extraction | Is it valid JSON, and do the totals add up? | Code |
| Document extraction | Do the values match the document? | LLM judge |
| Classification / routing | Is the category right? | Decision model |
| Summarization | Did it add anything not in the original? | LLM judge |
| Text-to-SQL | Does the query run and return rows? | Code |
| Tool / MCP agents | Does the answer match the tool result? | Whole-trace judge |
| Coding agent / Claude Code | Tests failed, but the code was committed | Code |
| Web search with citations | Was the cited link really retrieved? | Code |
| Web search with citations | Does the source actually say that? | LLM judge |
An invoice extractor: code checks that the JSON parses and the line items sum to the total. An LLM judge checks that “$20” really appears on the page.
The answer can still be wrong after a tool returns the truth.
| Tool said | Bot said |
|---|---|
get_price → 20 | “Pro is $40” |
search_logs → 0 errors | “found 12 errors” |
npm test → 3 failed | “all pass” |
get_order → shipped | “not shipped yet” |
The judge can only use what was saved on that request.
trace_id on every step.trace_id (optional — a CtrlB bonus).What the judge can do as more is recorded:
The pricing bot below only works as a real check if the price call is saved on the same trace as the answer. If that step is missing, the judge can only say the answer states a price.
Most checks compare the answer with what was already recorded. They do not look anything up.
The price tag says 20. The cashier says 40. A supervisor watching the CCTV catches the mismatch. The supervisor does not walk the shop to re-check the tag. The recording already has it.
The bot calls a pricing API (20) but does not put the price in the prompt. The AI says $40.
| What was saved | Langfuse | CtrlB |
|---|---|---|
| Only the AI step. The price never entered the prompt. | Judge: “states a price, no source, can’t confirm.” | Built (phase 3) A whole-trace evaluator sees the API span, the AI span, and the log “WARN price cache stale”, then says “wrong: API returned $20, bot said $40.” Tested locally; not on staging yet, so staging still reads only the AI span. Phase 3. |
| The price is in the prompt. | Catches it: 20 is in the prompt, the bot said $40. | Catches it too. This works today, because the price is on the AI span. |
| No source at all. | Can only flag “no source.” | Can only flag “no source.” |
Grade what was saved, not the live web.
cited_text).The bot cites a help URL and quotes “Pro is $20.” Code checks that sentence is in the saved search result. The judge then checks the answer didn’t turn $20 into $40.
Claude Code already speaks OpenTelemetry. Point it at CtrlB.
What it sends:
user_prompt, assistant_response, tool_result, tool_decision, api_request, api_error.claude_code.interaction → llm_request, and tool → tool.execution.Turn it on in the shell, or in Claude Code settings.json under "env":
CLAUDE_CODE_ENABLE_TELEMETRY=1OTEL_METRICS_EXPORTER=otlpOTEL_LOGS_EXPORTER=otlpOTEL_TRACES_EXPORTER=otlp and CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERSWhere the env block may live:
~/.claude/settings.json.claude/settings.local.json.claude/settings.json cannot set the exporter variables.Prompt text is redacted by default. OTEL_LOG_USER_PROMPTS=1 includes it (that is sensitive). OTEL_LOG_RAW_API_BODIES logs the full bodies.
What you can use it for:
OTEL_RESOURCE_ATTRIBUTES=team.id=…A code evaluator flags a session whose tool_result says tests failed and a later step still created a commit.
One customer. The steps are in the next section.
Stamp the commit on every trace. Then compare before and after. Example: GoComet.
OTEL_RESOURCE_ATTRIBUTES=service.version=<commit or release>. In GitHub Actions that value is ${{ github.sha }}.GITHUB_SHA). The pipeline gives it to the app. The app carries it to CtrlB.2026.w44. Narrow to the commit with git log v2026.w43..v2026.w44, or use 2026.w44+abc123.If nothing is tagged, you can still guess from:
quote-service:abc123.| What you see | Which mode |
|---|---|
| Errors or latency jump | No evaluator. Use alerts and compare the two windows. |
| Outputs break a rule | Code |
| You need the failure type | Decision model |
| Is the AI answer still correct? | LLM judge, scored by version |
| Why did it break? | Whole-trace judge (built — phase 3) |
The pipeline deploys commit abc123 and stamps service.version=abc123. Error rate jumps on those traces. Alerts show the jump. A later whole-trace judge would say whether the answer changed because of that deploy.
Built on branch feat/evals-scores-v2 and tested end to end on local Seeker + Orion (2026-11-18/19). Not deployed to staging yet.
| Round | What | Status |
|---|---|---|
| 1 | Health counts failed runs. A rule uses its own runs (since 2026-11-19 its own last 50 in 7 days, like evaluators), and it is at least as bad as its worst active evaluator. The list tooltip and the evaluator page show the latest error (type, message, time) with “Open failed run” and “Open observation”. The judge span stores the filled prompt, the reply, duration, tokens, and the error, so the Error panel shows the reason. Example: a failed judge no longer looks healthy with an empty error box. | Built · tested locally |
| 2 | Analytics is SQL in the lake (POST ai/evaluators/scores/analytics): totals, trend, distribution, and the comparison of the two scores you picked, over the whole range. Pairs stop at 10,000, with the note “Compared on 10,000 of N”. Old Seeker still falls back to the latest 2,000 rows. Example: “Last 1 day” is that whole day, not a short slice of 2,000 rows. |
Built · tested locally |
| 3 | OpenAI-compatible judges are asked for JSON mode (response_format: json_object). If the provider rejects it, one retry without it, and that model is remembered. Anthropic is unchanged. JSON inside text still parses. Example: DeepSeek accepts json_object and rejects json_schema. |
Built · tested locally |
| 4 | If every variable is empty, the run is skipped: no judge call, no cost, no score row. If more than half the matches in 24 h were skipped, status is Degraded, reason missing_variables. New numeric judges start at 0–1. An old judge with no range shows a warning. An empty mapping asks “Fix mapping” or “Save anyway”. Escape never saves. Example: a blank {{input}} does not spend a judge call. |
Built · tested locally |
| 5 | Human and API scores. Configs live in Mongo (name, type NUMERIC / CATEGORICAL / BOOLEAN / TEXT, range or labels, description, archive; the type cannot change). POST / PATCH / DELETE ai/scores. Signed-in users are ANNOTATION. API keys are API. The author is stored. The author or an admin can edit or delete. A score can sit on the whole trace (empty span id). Same lake dataset, scope ctrlb.scores, so they do not change evaluator health. The Scores page shows the real Source. Author is hidden by default. Analytics can compare “name · EVAL” and “name · ANNOTATION”. Example: a reviewer marks the crypto chat false, next to the judge. |
Built · tested locally |
| 6 | Health is the evaluator’s last 50 runs in the past 7 days, stored in Mongo. Healthy if 10% or fewer failed. Degraded if more than 10% and at most 50% failed. Failing if more than 50% failed. Langfuse is Healthy only when 0 failed, and Failing at 50% or more. Skipped runs and Test runs are not counted. Example: 4 failures in 50 runs is 8%, so Healthy in CtrlB. | Built · tested locally |
| 7 | Judge traces move to a hidden dataset _ctrlb_evaluator_runs_<dataset>. Setting EVALUATOR_RUN_SPANS: inline, both (default — writes to both), or separate. “View executions” and “Open failed run” open that dataset. Customer AI Traces stop showing judge traffic after ops sets separate, once the new Orion ships. Example: the crypto judge call is not mixed into the bot’s trace list. |
Built · tested locally |
| 8 | A 5th Evaluators tab, Score configs: a create/edit drawer with “Copy from evaluator”. Two defaults, added the first time a workspace opens the tab, and not added again if archived: “open coding” (TEXT) and “pass or fail” (Pass=1, Fail=0). An Annotate panel on AI Traces: whole trace or one span, pick configs, one input per type, “Scores on this trace” (newest first, updates at once after save, edit or delete). Example: open the crypto trace, pick “pass or fail”, mark Fail. | Built · tested locally |
Built and tested locally (2026-11-19). Not on staging yet. The judge grades the whole request, not one step.
{{trace}} (waterfall with refs [s1]…), {{errors}}, {{input}}, {{output}}.{{errors}} reads “No errors or problem logs in this trace.” Before this fix, clean traces failed with missing_variables (97 of 129 runs).trace evaluator (input/output only) is deprecated.retrieve-docs timed out, and a WARN log says the docs cache was stale.citedSpans. The Traces drawer shows a “Trace scores” strip and “Highlight spans the judge cited”.| Limit | What it means | |
|---|---|---|
| L1 | ≤300 spans | A small trace is sent as it is. Up to 1,000 spans are read, then cut. |
| L2 | Over 300 | Keep the root, errors, LLM spans, and the slowest spans first. |
| L3 | 16 KB text | LLM input and output are cut to 1 KB each. |
| L4 | ≤100 logs / 4 KB, any level | Picked by importance: ERROR, WARN, then text like “failed”, “timeout”, “retry” or an HTTP 5xx, then the rest. Repeats shown once (×N), shown in time order. Levels are normalised (INFO/info/Information → INFO; none → “-”). Levels are often missing or wrong — on staging ~1.4 M trace logs are Envoy access logs with no level, and failures are logged at INFO — so the judge reads the text. |
| L5 | 2 min quiet | Grade after no new spans. Choices: 2, 5, or 15 min. |
| L6 | Never quiet | Grade after 30 min anyway, and mark “still running”. |
| L7 | 10% sample | Default for trace rules. Since D29, new observation rules start at 10% too (they were 100%). |
| L8 | Weekly cost | Shown before you save. |
Dataset k8s_demo_llm, 24 h, about 2 million traces.
| Stat | Value |
|---|---|
| Median spans | 3 |
| p95 spans | about 22 |
| p99 spans | about 25 |
| Max spans | 284 |
| p99 duration | about 5 s |
| Longest trace | about 24 h |
Most chats are 3 spans, so they fit under L1. A 284-span trace is trimmed (L2) and then cut to 16 KB (L3).
Local Seeker (:8095) + Orion (:5173) against the staging lake and Mongo, with ~2,300 real DeepSeek traces from a seed script. Everything below is fixed and verified.
| Judge reads | Bad answers caught | Note |
|---|---|---|
| Whole trace + logs | 8 / 8 | One catch was visible only in logs: 503 access lines with no level, and an INFO “retry failed” |
| Input / output only | 5 / 8 | — |
12 traces where billing failed silently: the answer text looked fine, but a backend step failed. The judge that read the whole trace and logs caught 3 more bad answers than the input/output judge; one of them only showed in no-level 503 access logs and an INFO “retry failed”. Cost was about $0.0005 per trace.
| Found | Fix |
|---|---|
Clean traces failed missing_variables because {{errors}} was empty (97 of 129 runs) | Reads “No errors or problem logs in this trace.” |
| Rule health used 24 h, so a rule said Failing while its only evaluator said Degraded | Rules use their own last 50 runs in 7 days, and are at least as bad as their worst evaluator |
| Orion got a 400 on every page load while checking whole-trace support | Settings now report traceEvaluators |
Code sandbox test: a deep JSON.parse gave a new error text | Mapped to “Maximum call stack size exceeded.”; the worker survives |
/evaluators/analytics redirects; interval list capped at the range.Orion: full suite 14,475 passed (2 failures fixed after), feature dirs 2,109 passed, design-system audit 0 findings in feature files; remaining ESLint errors only in TracesDrawer.tsx (pre-existing on dev). Seeker: all 44 packages pass with Mongo.
:8095 against the staging lake and Mongo. Orion on :5173. In the browser: localStorage.seekerURL = "http://localhost:8095".~/Desktop/skkr/evaluators-langfuse/ctrlb_ai_seed.py (real DeepSeek calls; scenarios good, sloppy, unknown, wrongdoc, outage, hidden, logonly, big). It writes shivam_12_ai_traces and shivam_12_ai_logs through a local-only helper that is never committed..venv/bin/python ctrlb_ai_seed.py --count 150 --workers 8 --yesBuilt after the end-to-end test, on the same branch. They close several open items. Not on staging yet.
| # | What changed | Example |
|---|---|---|
| D24 | “Scored” column. The Evaluators list column “Observations scored” is now just “Scored” (header and tooltip). Same count: runs that returned a verdict. | A whole-trace evaluator that graded 120 traces shows “Scored 120”, not “Observations scored 120”. |
| D25 | Log tie-break. When logs are equally important (same rank as in L4), logs on an ERROR span of the trace come first, then logs on its slowest spans, then the earliest. Budgets, “×N” and time order are unchanged. | A trace whose only problems are three 404 ERROR spans used to fill the budget with hundreds of earlier access logs. Now those spans’ own logs are kept. |
| D26 | Test sample has data. The Test panel starts on the newest matching observation with an input or output, else the newest. Your own pick is never replaced. Whole-trace tests keep the newest trace. | The newest row is an empty execute_tool span → the panel opens on the newest chat span instead, so the first test is useful (fixes S10). |
| D27 | Model picker: working connections first. Models of providers that are set up come first. A provider with no key says “not set up”; a key that failed its check says “connection not working”. Search covers every provider. | Only DeepSeek has a key → DeepSeek models are on top; OpenAI says “not set up”. |
| D28 | Judge runs count as LLM calls. In a run-spans dataset (_ctrlb_evaluator_runs_*) an evaluation span counts as an LLM call, and “Spans by operation” calls it “evaluation”, not “Other”. Code-evaluator runs call no model, so they stay at 0. Customer datasets are unchanged. | “View executions” on an LLM judge shows “LLM calls 1” per run, not 0. |
| D29 | New rules sample 10%. The “Evaluator saved” dialog’s rule and a brand-new rule in the rule drawer start at 10% for observation rules too (was 100%). The dialog has a sampling field, and its cost line names the rate: “About $X a week at 10% sampling, …”. | One click on a big dataset was ≈ $550–950/week at 100% with no filter; at 10% it is about a tenth of that (fixes S8). |
basis: "recent_runs" and shows one value. Example: the judge cost $0.000164 per call in its recent runs → the weekly estimate uses $0.000164 per call.basis: "estimate". Output tokens come from real completed judge runs in the workspace (last 24 h, else 7 days); only if there are none, 60 per LLM judge and 25 per decision question (was 120 / 40). Input is the real messages, averaged over the 5 newest matching observations.Real cost $0.000164 per call. The new fallback guessed $0.000146 per call (~11% low). The old guess was $0.000263 (~60% high).
New toolset evaluators. Read only, no writes. Each tool uses the same service and the same access checks as its HTTP route.
| Tool | What it returns | Example question |
|---|---|---|
list_evaluators | Evaluators with status and reason, last error, 24 h stats; optional status filter | “Which evaluators are failing?” |
get_evaluator | One evaluator by id or name: prompt, scores, mapping, rules | “What does answer_correct check?” |
list_evaluator_rules | Rules with dataset, filters, sampling %, evaluators, status | “Which rules run on k8s_demo_llm?” |
search_scores | Scores (EVAL / ANNOTATION / API), newest first, filtered; at most 200 rows per call | “What scored below 0.5 today?” |
score_analytics | One or two scores: totals, distribution, agreement, kappa, correlations, MAE, RMSE | “Does the judge agree with humans?” |
The search_scores and score_analytics window is capped by MCP_SEARCH_MAX_RANGE. The default stays 24 h (owner decision), the same as search_logs. list_evals still lists the old Evals.
Found while clicking through the built pages.
answer-llm span shows its answer_correct score with the evaluator’s name.Each judge run is one span in its own trace. It points back at what it judged with ctrlb_evaluated_trace_id (and the evaluated span). It is never added to the bot’s trace.
EVALUATOR_RUN_SPANS | Where run spans go | When |
|---|---|---|
inline | Only the rule’s own traces dataset (old behaviour) | Not used any more |
both | The rule’s dataset and _ctrlb_evaluator_runs_<ds> | Default, temporary Old Orion’s “View executions” still reads the rule’s dataset |
separate | Only _ctrlb_evaluator_runs_<ds> | Target Same idea as Langfuse’s hidden langfuse-llm-as-a-judge environment |
_ctrlb_evaluator_runs_<ds>, scores in _ctrlb_evaluators_<ds>, both in the same org as the rule’s dataset.separate once the new Orion ships.shivam_12_ai_traces grades 66 bot traces → the Traces page for that dataset lists 66 bot spans and no evaluation spans; the 66 judge runs sit in _ctrlb_evaluator_runs_shivam_12_ai_traces.separate
New judge runs go only to _ctrlb_evaluator_runs_<ds>. The bot’s dataset on the Traces page shows only bot spans (66 of 66). The judge dataset is never in the Traces dataset dropdown: Seeker filters _ctrlb_evaluator_runs_* out of dataset lists.
shivam_12_ai_traces is our DeepSeek seed data (ctrlb_ai_seed.py; scenarios good, outage, hidden, logonly, wrongdoc, sloppy, unknown, big).An outage trace: 3× GET billing-api returned 503, plus one chat deepseek-chat call. The judge scored 1, “handled honestly”: the bot told the user billing was down. The three 503 spans are highlighted red (cited, error) and the chat span yellow (cited, no error).
Annotate is for people scoring the bot’s traces. Those human scores are the ground truth the judge is checked against.
The judge gave 0 to an honest “billing is down, try later” answer. Annotate that bot trace as 1. The judge run says “did not answer the question”. Add “an honest outage message counts as correct” to the prompt and Test on the trace: now 1. Analytics shows agreement going up as new scores come in.
| Item | What it will do | Status |
|---|---|---|
| D35 Analytics drill-down | Click a confusion-matrix or heatmap cell → list of the pairs in it, with both scores, the reasoning and links | In progress |
| Test panel dataset | Opens on the rule’s dataset when the evaluator has exactly one rule | In progress |
| Judge-run drawer header | Counts the evaluation call as an LLM call | In progress |
| Annotate on judge runs | Hidden: judge runs are not ground truth | In progress |
| AI drawer Scores tab | Shows “Cited:” with links to the cited spans | In progress |
These are chosen. They are not still open. Full table with code pointers (D1–D35; D35 drill-down is in progress): Seeker docs/evaluators-plan.md §0.
| # | Decision | What it means | Why |
|---|---|---|---|
| D1 | Default numeric range 0–1 | New numeric LLM scores start “from 0 to 1” | Judges invented scales (1, 9, 60) |
| D2 | Analytics in Seeker SQL | Per score, capped at 10,000 pairs | Old page used 2,000 rows of all scores; recount matched to the digit |
| D3 | Health = last 50 runs in 7 days | Healthy ≤10%, Degraded ≤50%, Failing >50%; skipped and Test runs excluded | Like Langfuse; recovers quickly after a fix |
| D4 | Rules use the same health rule | Own last 50 runs; at least as bad as the worst evaluator; 24 h fallback | Rule said Failing, evaluator said Degraded |
| D5–D9 | Human scores, Scores API, score configs, trace-level scores, judge vs human vs API | Annotate; ai/scores; 2 default configs; empty span id = trace; name · source | Langfuse G1–G4, G6 |
| D10 | Judge runs in a separate dataset | _ctrlb_evaluator_runs_<dataset>, EVALUATOR_RUN_SPANS=both by default; _ctrlb_ datasets hidden from pickers | Keeps customer traces clean |
| D11–D14 | Whole-trace evaluators, limits L1–L8, logs by text, empty errors is a value | See phase 3 | More context, bounded cost |
| D16 | JSON mode for OpenAI-compatible judges | json_object with a fallback retry; Anthropic unchanged | Works with DeepSeek |
| D17 | Skipped runs | All variables empty → no model call; >50% skipped → Degraded | Honest health |
| D18 | Settings report whole-trace support | traceEvaluators in settings | Removed a 400 on every page load |
| D19 | Code evaluators off by default | On only locally for testing; staging unchanged | Runs user code |
| D20 | Backward compatible both ways | Additive API changes only | Rollout safety |
| D22 | No edits to Orion shared components or design-system files | Feature CSS overrides only | Team rule |
| D23 | Analytics compares the same level only | “Traces” = trace-level scores; a span score never pairs with a trace score | Like Langfuse |
| D24 | “Scored” column | Was “Observations scored”; same count | Whole-trace evaluators score traces |
| D25 | Log tie-break | Equal-rank logs: error spans first, then slow spans | Error spans’ own logs were dropped |
| D26 | Test sample has data | Newest sample with an input or output | First test was often on an empty span (S10) |
| D27 | Model picker: working connections first | No key → “not set up” | Unconfigured providers came first |
| D28 | Judge runs count as LLM calls | Legend label “evaluation” | Judge runs showed “LLM calls 0” |
| D29 | New rules sample 10% | Observation rules too; sampling field in the saved dialog | One click at 100% ≈ $550–950/week |
| D30, D32, D33 | Cost estimate | Recent real runs first; else real token averages as a range | Old guess ran ~60% high. Details |
| D31 | MCP read tools | 5 tools; window capped by MCP_SEARCH_MAX_RANGE (24 h) | Answer questions without the UI. Details |
| D34 | Last error matches the status window | Degraded/Failing evaluators show their newest error from the same 7 days as the status, with “Open trace” | A Degraded evaluator showed no error or link when the failure was older than 24 h. Details |
Not done. Do not read these as shipped. Owner decisions are in Issues & Plans.
Hands-on on CtrlB staging. The fixes were then tested end to end locally (2026-11-18/19 test); staging still runs the old code.
shivam-101. LLM judge gpt-5-mini, score 0–1.shivam-102 on k8s_demo_llm chat spans at 0.05%, with backfill 20.| ID | Severity | Status | What we confirmed |
|---|---|---|---|
| S1 | High | Fixed r4 | An empty numeric range lets the judge flip scales. The naming judge (gpt-5.5, generic prompt, no range) gave 1, 9, 9.5, 9.5 on the same answer in 4 tests. gpt-5-mini gave 0.9, 0.9, 1, 1. New judges now start at 0–1. Built, not on staging yet. |
| S2 | High | Fixed r1+r4 | An evaluator can fail 100% silently. “Detect Out-of-Scope Request” was Failing, 0 scored, 23/23 errors. The reason (missing_variables: {{input}} resolved to nothing) was only inside the judge span output. Its mapping Input › [0] › content does not match OTel GenAI messages ([0] › parts › [0] › content). Now the error is shown (round 1). If every variable is empty, the run is skipped (round 4). Saving an empty mapping asks “Fix mapping” or “Save anyway”. Escape never saves. Built, not on staging yet. |
| S3 | Med | Fixed r1 | The execution Error panel said “No error message” while the error was in Output. The error is now in the standard error fields, so the panel shows the reason. Built, not on staging yet. |
| S4 | Med | Fixed r1 | The executions list showed Input —, Output —, 0.00 µs, 0 LLM calls, 0 tokens for every judge run. The span now stores the filled prompt, the reply, duration, tokens, and the error. Built, not on staging yet. |
| S5 | Med | Fixed r1+r6 | Evaluator health ignored failed runs. shivam-101 said “Healthy: every run completed” while Scores said “18 scores (2 failed)”. Failed runs now count. Health is the last 50 runs in 7 days. Built, not on staging yet. |
| S6 | Med | Fixed r1 + 2026-11-19 | Rule health did not reflect its evaluators. A rule showed Healthy while its only evaluator failed 100%. A rule now uses its own last 50 runs in 7 days (until 2026-11-19 it still used 24 h), and it is at least as bad as its worst active evaluator. Tested locally, not on staging yet. |
| S7 | Med | Fixed r2 | Analytics used the latest 2,000 rows: “Showing the latest 2,000 of 54,712 scores”. With “Last 1 day” the trend only spanned ~06:00–08:00. Analytics is now SQL over the whole range. Pairs cap at 10,000. Old Seeker still falls back to 2,000. Built, not on staging yet. |
| S8 | Med | Fixed D29 | The one-click “Start scoring” after saving defaulted to 100% sampling with no filter (≈ $331.83/week here; ≈ $550–950/week on the demo dataset with 3.4 M spans/week). New observation rules also defaulted to 100%. Now both start at 10%, the dialog has a sampling field, and its cost line names the rate (D29). Trace rules already defaulted to 10% (L7). Tested locally, not on staging yet. |
| S9 | Med (security config) | Open (infra #16) | Python code evaluators run on staging. CtrlB’s plan says keep Python off until a non-root, no-network sandbox. |
| S10–S14 | Low | Open (S10, S14 fixed) |
|
B1 and B2 are fixed and tested locally. B3 is still open. Not on staging yet, so staging can still show the old behaviour.
The judge invented its own scale. The naming evaluator returned 0.9, 9 and 60 for the same kind of answer (about 400 on a 0–1 scale, about 1,400 on 1–10, about 12 values of 15–60 in one day). Means, charts and comparisons were meaningless.
With no min/max the instruction was just “score is a number”. New numeric judges now start at 0–1. An existing judge with no range shows a warning. Langfuse UI defaults to “between 0 and 1”.
Seen on staging (S1). Built, not on staging yet.
Not of the two chosen scores. Total/Mean/Min/Max used all data (Seeker summary), but Std dev, Trend, Distribution, Matched, Pearson, Spearman, MAE, RMSE and the heatmap used the small unfiltered sample.
Example: naming vs testing_1234 → Matched 11 although both had 1,400+ runs. One card mixed “all data” and “sample” numbers (mean 2.49 with std 0.598 and max 60 is impossible).
Now the lake computes those panels for the two chosen scores over the whole range. Pairs cap at 10,000 (“Compared on 10,000 of N”). Old Seeker still falls back to the latest 2,000. Langfuse queries each score separately, up to 100k values.
Seen on staging (S7). Built, not on staging yet.
Download CSV silently stops at 10,000 rows and fires about 50 browser requests (200 per page). With 1.1 lakh scores the file is partial (there’s a toast, but it’s easy to miss). Candidate for a server-side export.
Features Langfuse has that CtrlB does not — in simple words.
A person can mark a chat in the Annotate panel (whole trace or one span). Example: mark the crypto answer false. Built, not on staging yet. Queues are still later: decision.
POST / PATCH / DELETE ai/scores. A signed-in user is ANNOTATION. An API key is API. Example: the app sends a thumbs-down on the crypto reply. Built, not on staging yet.
Configs live in Mongo. The Score configs tab adds “open coding” (TEXT) and “pass or fail” (Pass=1, Fail=0) the first time a workspace opens it, and does not add them again if archived. Built, not on staging yet. Rounds.
Human and API scores can sit on a trace (empty span id) or a span (round 5). AI scores on a whole trace are built too (phase 3). Tested locally, not on staging yet. Still no session score and no dataset-run score. The Langfuse screen: Trace scores vs span scores.
CtrlB has none, so handing out 20–50 crypto chats is harder. How a Langfuse queue works: Annotation queues.
You can compare “name · EVAL” with “name · ANNOTATION”. Example: the judge’s crypto score vs a reviewer’s. Built, not on staging yet. The lake query is round 2.
The Scores table shows the real source: EVAL, ANNOTATION, or API. Author is a column, hidden by default. Built, not on staging yet.
Same speed and status gap as difference 3. The lake has no ingest hook, so CtrlB scans every 60 s (intentional differences). What a live run looked like: Running live.
Built and tested locally: the judge sees every span, errors and logs, so it can tell that another step failed (8 of 8 vs 5 of 8 in the test). Staging still has only span evaluators until deploy. Design: phase 3. Table: span vs whole-trace.
How the two products differ: Decision model — how to set it up.
A new version replaces the old one. Langfuse can compare versions only in experiments. CtrlB has no experiments.
TypeScript and Python are off (own sandbox workers). Langfuse runs them in AWS Lambda or an external sandbox. How a Langfuse code rule works: Code evaluator.
Langfuse has datasets, experiments, and a RegressionError GitHub Action. CtrlB has none.
Five read-only tools: list_evaluators, get_evaluator, list_evaluator_rules, search_scores, score_analytics. list_evals still lists the old Evals. Example: an assistant answers “which evaluators are failing?” without the UI. Built, not on staging yet. Details.
Langfuse has “Add alert”. CtrlB skipped alerts on purpose (plan decision E13).
Status of each item. “Built” means in code and tested end to end locally (2026-11-18/19), not on staging yet.
_ctrlb_evaluators_<…>.source, author, and scope ctrlb.scores so human and API scores stay out of evaluator health.source on old rows reads as EVAL; an edit or delete appends a new version and the latest row per score id wins (the lake is append-only)._ctrlb_evaluator_runs_<dataset>. Setting EVALUATOR_RUN_SPANS: inline / both (default, writes to both) / separate. Scores stay in _ctrlb_evaluators_<dataset>. Customer AI Traces drop judge traffic after ops sets separate, once the new Orion ships. CtrlB did this once (commit 7be3a545) before PR #516 moved runs back into the app dataset.| # | Item | Status | Note |
|---|---|---|---|
| 1 | Engine writes run spans to the new dataset | Built · round 7 | EVALUATOR_RUN_SPANS default both writes to the app dataset and to _ctrlb_evaluator_runs_<ds> |
| 2 | Create + permission the dataset | Built · round 7 | Hidden dataset, same org. Not on staging yet |
| 3 | “View executions” opens the runs dataset | Built · round 7 | “Open failed run” opens it too. Customer traces stay quiet after separate, once the new Orion ships |
| 4 | Link back to the graded app span | Built · round 1 | Latest error links: “Open failed run” and “Open observation” |
| 5 | Hide _ctrlb_* datasets from pickers | Built | Hidden from normal pickers. separate is what stops judge traffic in the app dataset itself |
Rough size: Seeker 150–250 LOC, Orion 80–150 LOC. Not chosen: the _ctrlb_meta org — cross-org reads, customers lose direct SQL/dashboards, strict scoping needed.
| Change | Old-Orion safe? | How |
|---|---|---|
| I3 run spans to a new dataset | Built · round 7 | EVALUATOR_RUN_SPANS: inline / both (default) / separate. separate waits until the new Orion ships |
Hide _ctrlb_* datasets | Built | Only new Orion hides them; old lists unchanged |
| Human / API scores (G1/G2) | Built · rounds 5 and 8 | ai/scores plus the Annotate panel. Not on staging yet |
| Default numeric range (B1) | Built · round 4 | New judges start at 0–1. Old judges with no range show a warning |
| Analytics fix (B2) | Built · round 2 | Lake SQL for the whole range. Pairs cap at 10,000. Old Seeker falls back to 2,000 |
| Whole-trace evaluators (G12) | Built · phase 3 | New optional fields; old Orion doesn’t show the option. Not on staging yet |
| Score configs (G3) | Built · rounds 5 and 8 | Mongo, plus two defaults. Not on staging yet |
| Whole-trace support flag | Built | Settings report traceEvaluators; new Orion probes trace-preview only on older Seeker (works both ways) |
A reviewer marks the crypto chat false (ANNOTATION). A thumbs-down from the app lands as API. Analytics compares judge vs human on the same chats, with real source labels. Queues are still later. Decision.
The decision is yes. Phase 1 is built and tested locally. It is not on staging yet. Queues wait for phase 2.
Without human scores, nobody can tell if a judge is right. Example: the staging case “80% should_have_refused”. Range, analytics, and ideas I1–I3 are in Plans.
| # | Place | When |
|---|---|---|
| 1 | AI Traces → Annotate (the whole trace, or one span) | Built · round 8 |
| 2 | Scores tab → real Source (EVAL / ANNOTATION / API). Author column, hidden by default. Author or admin can edit or delete | Built · round 5 |
| 3 | Score configs — a tab under Evaluators (not Settings) | Built · round 8 |
| 4 | Analytics → compare name · EVAL vs name · ANNOTATION | Built · round 5 |
| 5 | Annotation queues | Open · phase 2 |
POST / PATCH / DELETE ai/scores → the lake dataset, with source, author, and scope ctrlb.scores.span_id (G4).| Phase | What | Status |
|---|---|---|
| 1 | Configs + score API + Annotate + Source column + Analytics by source | Built · rounds 5 and 8 — not on staging yet |
| 2 | Queues | Open — later |
How Langfuse queues work today: Annotation queues.
Branch feat/evals-scores-v2. Built and tested locally, not on staging yet. The “copy from evaluator” mismatch warning is still open.
Found during the end-to-end test. 1–3 are now decided; the rest are not changed yet.
| # | Question | Context |
|---|---|---|
| 1 | Lower the per-span default on the “Evaluator saved” dialog? | Decided · D29 New rules start at 10%, with a sampling field in the dialog. Was 100% with no filter (≈ $550–950/week on the demo dataset). See S8 |
| 2 | Fix the cost estimate? | Decided · D30, D32, D33 Priced from recent real runs; else real token averages, shown as a range. Old guess ran ~60% high ($0.000263 vs real $0.000164 per call); the new fallback is ~11% low ($0.000146). Details |
| 3 | What does Analytics “Traces” mean? | Decided · D23 Like Langfuse: only trace-level scores; a span score never pairs with a trace score |
| 4 | Scores chart over the whole range? | Today it covers only loaded rows, with a note. Options: load all, or a server aggregate |
| 5 | Comparison rows still open | Categorical multi-match (next release, off by default); test call to the model on save; hide always-0 Delayed; Python on staging (#16); “Copy from evaluator” warning; evaluator alerts (G14); CSV cap 10k (B3). Done since: cited spans in the Test result and on the Traces drawer, test sample (S10, D26), “Scored” wording (D24), log tie-break (D25), model picker order (D27), MCP tools (G13, D31) |
| 6 | API timing (reported, not changed) | New evaluator sample list ~9 s on large datasets (rows query, then a full-day COUNT; pre-existing); attributes fetched for all traces datasets on that page; AI drawer fetches score configs 3× and the trace 2×; Analytics ~2.4 s |
| 7 | “Professional polish” phase | After sign-off: one-row header, compact last-error line, one-line help with ⓘ, full-height split panes, tighter test-panel controls. Design system only; no design-system file changes |
Short answers. Acme examples where they help.
answer-llm’s Input contains “Docs: …” — even “(no matching docs)”._ctrlb_evaluators_<traces dataset>.json_object).json_object and rejects json_schema. Built, not on staging yet.EVAL, written in the UI, even if the row was not from a judge.trace_id, in the same time window.{{errors}}. Not on staging yet.Already answered above — open that section:
Short definitions. Same words used above.
trace_id (one crypto chat). A span is one timed step — start, end, duration, status OK or ERROR, attributes, and events including exceptions (example: answer-llm; Langfuse also calls this an observation). A log is a free-text line at any moment; a step can have many or none. Spans show where and how long. Logs show what happened, including caught errors, retries, and things outside a request.service.version with ${{ github.sha }}.abc123 or 2026.w44.json_schema (OpenAI-compatible) or a forced tool call (Anthropic). CtrlB asks OpenAI-compatible models for json_object, retries once without it if they refuse, and also accepts JSON inside text.thinking: disabled.answer-llm is GENERATION._ctrlb_evaluators_… datasets.