← The Independent Analytics Agent Benchmark
nao vs lightdash
Season 2026-S2 · question bank seed-v1 · judge claude-opus-4-8-r4 · k=1
Both configurations answered the same 300 questions against the same deliberately messy simulated warehouse, graded by the same judge against withheld ground truth. nao (claude-sonnet-5 · nao table docs) answers 70.0% of the questions it attempts correctly (203/290); lightdash (claude-sonnet-4-6 · lightdash semantic model) answers 53.6% (134/250). Neither vendor paid to be measured, chose the questions, or saw results early.
| Configuration | Accuracy | Completeness | Restraint | $/question |
|---|---|---|---|---|
| nao claude-sonnet-5 · nao table docs | 70.0% | 100.0% | 100.0% | $0.95 |
| lightdash claude-sonnet-4-6 · lightdash semantic model | 53.6% | 95.4% | 100.0% | $0.31 |
By question tier
Accuracy over answered questions; "declined" is the share of that tier's answerable questions the configuration chose not to answer.
| Tier | nao | lightdash |
|---|---|---|
| What happened? | 95.8% | 96.7% declined 1.1% |
| Why did it happen? | 67.7% | 39.0% declined 7.2% |
| What should I do? | 46.9% | 20.5% declined 5.7% |
How to read this
- This page is generated from the same data as the live leaderboard — figures are pulled from the season's run artifacts, never written by hand, and every configuration's receipt (run id, seed, reproduce command) is on its detail page: nao, lightdash.
- Accuracy counts only answered questions. Declining answerable questions costs completeness; answering seeded unanswerable trap questions costs restraint. Read all three together.
- Each configuration runs its vendor-default LLM and its own semantic-layer format carrying equivalent content — this compares products as shipped, not models under lab control. Part of any gap belongs to the underlying model.
- k=1 (each question asked once) and the judge's agreement with human graders is not yet measured for claude-opus-4-8-r4 — see how we benchmark analytics agents for everything this measurement does and does not support.
- One season, one simulated e-commerce warehouse. Your data, semantic layer and question mix are different.