BENCHOUSE

The independent benchmark for Analytics Agents.

Fifteen agent configurations answered the same 300 analytical questions against the same deliberately messy e-commerce warehouse. Every answer was scored against ground truth the agents never had access to.

The first run is finished. Names, numbers and receipts go on the board at launch, and the list sees them before that.

a sneak preview

The table below is illustrative. Every name and number in it is invented and the agents shown do not exist. Real scores publish on 1 September, each with a 95% interval, and configurations whose intervals overlap share a rank.

what we can already tell you

• The most expensive agent is not the most accurate.
• With the same agent, the same model and the same underlying facts, changing only the format of the context layer moved the score.
• Every agent scored worse on the questions that asked why something happened.

how we score

• Every configuration (an agent, an LLM and a context layer) answers the same 300 questions on a simulated e-commerce warehouse. The questions come at three levels: what happened, why it happened, and what to do about it.
• Answers are graded against a truth ledger that is computed separately and withheld from every agent. An agent only scores when its number matches, however plausible its SQL looks.
• Every score carries a 95% interval, and configurations whose intervals overlap share a rank.
• No vendor pays to be listed or sees the questions in advance. Every number has a receipt recording its run id, seed and command.

Read the methodology →

on the bench

Products run in season one. The logos are their owners’ trademarks and show what was tested. They do not indicate partnership, sponsorship or endorsement, and no vendor approved, reviewed or paid for inclusion.
Queued next, not yet run: Hex, Databricks Genie, Looker, ThoughtSpot, Wren AI, Vanna.

Building an analytics agent? Get benchmarked →