How we benchmark analytics agents

The benchmark launched Sept 1 2026, and it really resonated with the community. One of the most commonly asked questions was some version of “How are you running these?“. This article answers that.

BTW: No vendor can buy their position, or their removal from the benchmark.

The moving pieces, treated fairly

Every time an agent gets a question it uses its own capabilities and three inputs to answer it:

• The Data
• The Semantic layer
• An LLM

The hard part here is how you measure agent effectiveness fairly when all three inputs can differ between agents.

The Data. This is by far the easiest to keep the same. Tables and columns all behave the same in Snowflake, Databricks and ClickHouse, the three databases we used in the initial benchmark.

The Semantic layer. The semantic layer was by far the hardest and most work to keep equivalent between different agents. We came up with a system that I think is as fair as possible, by transpiling a dbt layer to all the different target semantics deterministically. More on that below.

An LLM. For the LLMs we took the easy but fair road from a user perspective: we used the default preferred LLMs of the agent vendor. We will be running more benchmarks in the future that will run one agent on multiple LLMs.

The generated messy data

We generated a fully simulated e-commerce warehouse for Driftline Aquatics, a fictional Dutch aquascaping/planted-tank D2C store. A pretty standard store: they use Shopify, NetSuite, etc. They track events through Segment and advertise on all the major platforms. For every piece of software they use we built an emitter that emits data. We try to have every emitter behave as human-like as possible, so shopping events happen during simulated visits across the site, and a marketer makes daily decisions based on historical data. Obviously we can only come so close and I would have loved to do a better job, but I’m pretty proud of where we are.

All of this results in hundreds of tables across sources and intermediate layers, after which we consolidate it into 60 mart tables.

Because of the way we generate data we get entries that are duplicated, too late or even missing. Event names that change over time, etc.

The mart has its own gotchas: if you have 60 tables there are enough places to hide a fake join, similar-looking metrics and inconsistencies that I have definitely seen in my 15+ year career.

All of this is to say, this is a deliberately messy dataset, not a 10-table clean dataset, which I have seen vendors benchmark themselves on.

Why we ended up having 300 questions

It all started with 60 questions, I thought that’s way more than the 10-15 questions some vendors use, it should be fine. But when you start looking at how statistically significant a difference between agents needs to be, 60 is definitely not enough, that gives you accuracy within ±12 percentage points! We wanted to be a lot narrower, while keeping costs maintainable. 300 questions gives ±5 percentage points, for ±1 percentage point we would have to have more than 8,000 questions!

So 300 it is for now, we’re bootstrapped after all.

The 300 questions are split up in 3 tiers:

• What happened? (descriptive). “How many orders did I have in June?”
• Why did it happen? (diagnostic). “Why did fish tank sales go up between June and August?”
• What should I do? (prescriptive). “What do I need to do to get more fish tank sales in the next three months?”

These tiers are really where the benchmark starts to show a difference in accuracy between agents: most agents, even basic text-to-SQL ones, score really well in tier 1. Tiers 2 & 3 are where the competition gets real.

The Not Invented Here syndrome reflected in semantic layers

Who loves all the different semantic layers? Nobody, except maybe a vendor reading this.

In order to be fair, we needed to give every agent an equivalent semantic layer. Our semantic source of truth lives in dbt. Tables, columns, metrics, relationships you name it, it’s all documented there. To translate it to every target layer we built semglot. It translates semantic models between formats through a neutral representation, so every agent gets the same semantic content in the format it can actually consume. The translation is not always 1:1, but pretty close!

It’s open source, you are free to use it.

Systematic scoring setup

300 questions times 7 configurations, that’s a lot. Not something I’d ever volunteer to score by hand. So we built an LLM judge that grades each answer against per-question criteria: what the golden answer is, what a correct explanation must identify.

Three metrics roll out of the judge:

• Accuracy. Of the questions an agent answered, how many were correct.
• Completeness. How many of the answerable questions were actually answered.
• Restraint. A seeded subset of questions is unanswerable on purpose. The agent should refuse to answer these.

Refusing something answerable costs completeness; answering something unanswerable costs restraint.

What this doesn’t measure

Consistency

We ask the question once, and the agent one-shots an answer. In an ideal world we ask the same question 3 times and measure how consistent the agent is. This would triple the cost, and is not in scope for now.

Your data

This reflects a synthetic dataset, your data might be more or less messy. The quality of the semantic layer might differ, and that’s OK. We can help you measure the effectiveness of agents on your data.

Multi turn

The agent gets one shot to return the right answer; in real life a user might give clarification or different metric names in subsequent turns.

What’s next

Expect a lot more agents over the coming months.

I love sharing this. So if you want to chat about the results, or know a place where this would be valuable to present, shoot me an email at andre@benchouse.ai.