BENCHOUSE
The independent benchmark for Analytics Agents.
7 agent configurations. The same 59 analytical tasks against the same deliberately-messy e-commerce warehouse. Scored on correctness against withheld ground truth — not vibes.
The first run is complete. Names, numbers and receipts go on the board at launch — the results go to the list first.
Benchmark updates, not a newsletter: new results when we run them, methodology changes when we make them, nothing else.
a sneak preview
Illustrative — the blurred names and numbers are fake. Every real score ships with a 95% interval, and statistical ties share a rank.
what we can already tell you
- The most expensive agent is not the most accurate.
- The context layer matters: same agent, same model, same facts — a different layer format alone moved the score.
- Every agent got worse the moment we asked why.
how we score
- Every configuration — agent + LLM + context layer — sits the same 59 tasks on a simulated e-commerce warehouse, at three levels: what happened, why, and what should we do about it.
- Answers are graded against a truth ledger computed independently and withheld from every agent. An agent is right when the number is right, not when the SQL looks plausible.
- Statistical ties share a rank; every score carries a 95% interval. No fake precision.
- No vendor pays to be listed. No vendor sees questions in advance. Every number carries a receipt: run id, seed, command.
on the bench
… more
Building an analytics agent? Get benchmarked →