← The Independent Analytics Agent Benchmark
hex vs nao
Season 2026-S2 · question bank seed-v1 · judge claude-opus-4-8-r4 · k=1
Both configurations answered the same 300 questions against the same deliberately messy simulated warehouse, graded by the same judge against withheld ground truth. hex (gpt-5.6 (Hex) · hex semantic model (dbt)) answers 74.4% of the questions it attempts correctly (195/262); nao (claude-sonnet-5 · nao table docs) answers 70.0% (203/290). Neither vendor paid to be measured, chose the questions, or saw results early.
| Configuration | Accuracy | Completeness | Restraint | $/question |
|---|---|---|---|---|
| hex gpt-5.6 (Hex) · hex semantic model (dbt) | 74.4% | 100.0% | 100.0% | $0.35 |
| nao claude-sonnet-5 · nao table docs | 70.0% | 100.0% | 100.0% | $0.95 |
By question tier
Accuracy over answered questions; "declined" is the share of that tier's answerable questions the configuration chose not to answer.
| Tier | hex | nao |
|---|---|---|
| What happened? | 97.8% | 95.8% |
| Why did it happen? | 68.7% | 67.7% |
| What should I do? | 55.2% | 46.9% |
How to read this
- This page is generated from the same data as the live leaderboard — figures are pulled from the season's run artifacts, never written by hand, and every configuration's receipt (run id, seed, reproduce command) is on its detail page: hex, nao.
- Accuracy counts only answered questions. Declining answerable questions costs completeness; answering seeded unanswerable trap questions costs restraint. Read all three together.
- Each configuration runs its vendor-default LLM and its own semantic-layer format carrying equivalent content — this compares products as shipped, not models under lab control. Part of any gap belongs to the underlying model.
- k=1 (each question asked once) and the judge's agreement with human graders is not yet measured for claude-opus-4-8-r4 — see how we benchmark analytics agents for everything this measurement does and does not support.
- One season, one simulated e-commerce warehouse. Your data, semantic layer and question mix are different.