A fluent answer is no reason to believe the number.
I want to know which answers an analytics agent gets wrong, why it gets them wrong, and
what to change. That means recording what it did and checking the result.
I kept being asked which analytics agent to buy. The answers available came from vendor
demos. I had been Head of Data at Babylist, where I built the data org to about 20 people
across engineering, analytics and platform. I knew what it meant to be the person a wrong
number lands on. A convincing demo wasn't enough to recommend a tool.
So I built a deliberately messy simulated e-commerce warehouse with ground truth the agents
never see. Each season, I give them the same 300 questions and publish the results. I check whether they get the answer right, whether
they answer at all, and whether they decline questions the data cannot answer.
What the benchmark showed
Running the benchmark showed me where these agents come apart, and what it takes to improve
them.
The easy questions hide the difference
Every agent aces “what happened?”, with accuracy ranging from 94.7% to
98.9%. They separate on “why did it happen?” and come apart on “what should I
do?”, where accuracy ranges from 0.0% to 72.0%.
Accuracy over answered questions, one gray line per agent configuration; the red bar and
its label are the worst-to-best range. Every configuration is in the full table.
Fluency tells you very little
In the chat window, a great answer and a dangerous one can look identical. I only see the
difference when I record what the agent did and check it.
Write down the decisions
In an experiment with nao,
accuracy went from about 50% to 96%. A stronger model helped, but the big moves came from
documenting the warehouse and writing down business decisions. The agent had built its own
holiday calendar instead of using the company's agreed holiday flag.
The promises I make
Nobody can buy a position on the benchmark or removal from it.
I publish the results whether vendors like them or not.
I never share a client's results with their vendor, or publish anything about them without
their approval.
Where it is going
I want every team running an analytics agent to know which answers are wrong and why. The Benchouse product keeps every answer, traces wrong and refused answers to
their causes, and proposes fixes to the semantic layer, dbt models and agent context, ranked
by how many questions each defect broke.
