benchouse Methodology
How the analytics agent benchmark is graded, judged and kept honest
Season 2026-S2 · question bank v1 · universe seed 42 · judge agreement 100.0% · 13 questions on the diagnostic tier.
Season 2026-S2 shows real runs (nao, Snowflake Cortex Analyst, Lightdash AI) against question bank v1. The harness already audits SQL by execution where question goldens permit it; those audit results ship in run artifacts and join the public views as coverage matures.What is Benchouse?
Benchouse is a public benchmark and leaderboard for analytics agents — AI systems that answer business questions against a data warehouse. Agents run against a simulated but realistically messy e-commerce warehouse and are graded against golden answers derived from a withheld ground-truth ledger.
What does the ranking measure?
Answer correctness: the share of questions answered correctly, judged against golden answers derived from the withheld truth ledger, measured identically for every agent. The harness additionally audits SQL by execution where questions permit it (distinguishing genuinely-correct answers from "lucky" ones that arrived via unverified SQL); those audit results are recorded in run artifacts and will join the public views as audit coverage matures across tiers and agents.
What are the divergence quadrants?
Each audited question lands in one of four quadrants: correct (right answer, verified SQL), lucky (right answer, unverified SQL), interpretation failure (verified SQL, wrong answer), and wrong (both wrong). The quadrant breakdown separates agents that are genuinely reliable from agents that happen to guess well; it is measured in the harness today and will be published as audit coverage matures.
Who grades the answers, and can the grader be trusted?
A pinned LLM judge grades answer correctness — currently claude-opus-4-8 (calibrated, n=124). The judge itself is measured: goldens are perturbed into known-wrong answers and the judge's agreement, precision and recall against withheld ground truth are computed on a calibration set. That agreement score is stamped on every run and published on the leaderboard.
How does Benchouse prevent training-data contamination?
The universe is simulated and its truth ledger withheld, so no agent can have seen this data during training. From Season 2 onward, the entire universe — customers, orders, ad spend, support tickets — will regenerate deterministically from a new random seed each season, with golden answers re-derived automatically, and any past season reproducible exactly from its seed. Fresh data alone will not be enough, though: the structural traps will also rotate (see "What will change each season?"), so knowing last season's gotchas will not help.
What will change each season?
Three things will move at different cadences. Every season from Season 2 onward: a new seed will regenerate all data, and the trap set will rotate — which sources deliberately disagree, where the mess is injected — so the diagnostic tier stays a closed-book exam even after past seasons are public. Periodically, the question bank version will bump: new question templates, new business domains and subsystems. Fixed forever: the real-world connector schemas (a Shopify export looks like a Shopify export) — knowing those is legitimate professional knowledge, exactly as it is for a human analyst. Because traps rotate, each season will imply a new bank version, which is why scores will never be compared across seasons and rank will be the only cross-season signal.
What do the tiers mean?
Tiers stack three kinds of questions. Descriptive (42 questions this season) asks "what" — e.g. "What is the number of orders this month?". Diagnostic (13) adds "why": explaining changes and reconciling sources that deliberately disagree — "Why did orders drop in March?". Prescriptive (4) adds "how to fix": recommendations grounded in the data — "How can we avoid orders dropping again?".
What is being measured — the model, the agent, or the semantic layer?
Three components move independently in every result: the LLM (e.g. claude-sonnet-4-6), the agent (e.g. nao, Cortex Analyst, Lightdash AI), and the semantic layer — the curated metric definitions the agent may answer through (dbt MetricFlow, Cube, LookML, Snowflake semantic views, Lightdash models, or none at all, meaning raw SQL against the warehouse). The semantic layer matters because it can contain crystallized answers: a well-authored metric definition can defuse a reconciliation trap before the agent runs. Benchouse therefore discloses the semantic layer on every entry, next to the model. As the field grows, raw-warehouse and with-semantic-layer results will be ranked as separate divisions, with semantic layers authored during a time-boxed onboarding against the schema only, frozen before questions are asked. The coding-world analogue is the agent-computer interface and code-intelligence tooling: same model, different interface, very different scores.
How does a richer semantic layer affect fairness — the reference context layer?
A curated semantic layer can pre-bake business logic, so a richer layer sits closer to encoding the answers; a gap between two agents can be the layer, not the agent. To make that measurable, each season will define a reference context layer — a technology-neutral manifest of semantic atoms (each metric, dimension, join and named business rule) over that season's warehouse, regenerating with the seed. Because no single layer format is universal, Benchouse will author a reference implementation in each layer technology as the fair baseline, and will measure every layer against the manifest — including its own reference implementations, which score below 100% wherever a technology cannot express an atom. Each layer will carry two numbers: coverage (the fraction of reference atoms it defines — below the baseline means thinner) and enrichment (atoms defined beyond the reference — the signal that a layer is fat with extra pre-baked logic). Coverage and enrichment are disclosed for transparency and will feed future divisions and score normalization; they do not enter today's ranking, which stays answer correctness.
Who runs the evaluations?
Leaderboard runs are executed on Benchouse infrastructure in sandboxed environments, with a receipt (run id, universe seed, exact command, timestamp) published per entry. Vendors may also self-report runs; those entries carry a self-reported receipt, and any entry run on an older question-bank version is shown but not ranked — results are only comparable within a bank version.
Can results be reproduced?
Yes. Every leaderboard entry carries a receipt: the run id, the universe seed, and the exact command that produced the run. The data generator is deterministic, so the entire benchmark universe can be rebuilt bit-for-bit from the seed. Raw run artifacts are withheld while the season's question bank is active, because they contain the questions themselves.