← The Independent Analytics Agent Benchmark
genie vs cortex-analyst
Season 2026-S2 · question bank seed-v1 · judge claude-opus-4-8-r4 · k=1
Both configurations answered the same 300 questions against the same deliberately messy simulated warehouse, graded by the same judge against withheld ground truth. genie (LLM not disclosed · genie space instructions) answers 65.7% of the questions it attempts correctly (119/181); cortex-analyst (claude-sonnet-4-6 · snowflake semantic view) answers 54.8% (131/239). Neither vendor paid to be measured, chose the questions, or saw results early.
| Configuration | Accuracy | Completeness | Restraint | $/question |
|---|---|---|---|---|
| genie LLM not disclosed · genie space instructions | 65.7% | 68.0% | 80.0% | — |
| cortex-analyst claude-sonnet-4-6 · snowflake semantic view | 54.8% | 81.0% | 100.0% | $0.13 |
By question tier
Accuracy over answered questions; "declined" is the share of that tier's answerable questions the configuration chose not to answer.
| Tier | genie | cortex-analyst |
|---|---|---|
| What happened? | 95.6% declined 1.1% | 94.7% declined 1.1% |
| Why did it happen? | 43.4% declined 6.2% | 45.7% declined 8.0% |
| What should I do? | 0.0% declined 84.0% | 0.0% declined 47.0% |
How to read this
- This page is generated from the same data as the live leaderboard — figures are pulled from the season's run artifacts, never written by hand, and every configuration's receipt (run id, seed, reproduce command) is on its detail page: genie, cortex-analyst.
- Accuracy counts only answered questions. Declining answerable questions costs completeness; answering seeded unanswerable trap questions costs restraint. Read all three together.
- Each configuration runs its vendor-default LLM and its own semantic-layer format carrying equivalent content — this compares products as shipped, not models under lab control. Part of any gap belongs to the underlying model.
- k=1 (each question asked once) and the judge's agreement with human graders is not yet measured for claude-opus-4-8-r4 — see how we benchmark analytics agents for everything this measurement does and does not support.
- One season, one simulated e-commerce warehouse. Your data, semantic layer and question mix are different.