Beyond the Right Number: How to Evaluate Analytics Agents

When we asked Genie what percentage of orders were refunded, it answered 7.10%. Our reference calculation was 7.0989%, so we accepted the rounded answer.

A revenue reconciliation is harder to grade. If you ask why marketing’s revenue is lower than finance’s and get an accurate table of both totals, you still need an explanation. The table doesn’t tell you why they differ or which number to use in your meeting.

We saw a similar problem with a safety-stock recommendation. The agent identified the products with the most stockouts and recommended buying more, but left out the cost of carrying that inventory.

Checking the numbers would have missed both problems. We needed to grade the explanation and the recommendation as well.

In our benchmark of analytics agents, we organize that work into three layers: descriptive, diagnostic, and prescriptive. The examples below come from our simulated e-commerce business, Driftline Aquatics. They show what we check and where grading gets difficult.

The cards show recorded answers and grades. You can inspect the expected reasoning, reference numbers, and supporting excerpts together. The grade applies to the full recorded response; omissions in excerpts are marked with […]. Where a recorded grade deserves another look, we explain why.

Descriptive: did it get the number right?

Descriptive questions ask what happened. How much revenue did we book? What percentage of shipments arrived on time? Usually, you can calculate a reference value independently and compare the answer against it.

The refund-rate example is straightforward. The reference is 7.0989%. Genie’s 7.10% is a valid rounding. It gives the reader the requested information at a sensible level of precision.

Before running an evaluation, specify the metric, population, time period, and acceptable tolerance. An accurate calculation over the wrong set of orders still answers the wrong question.

Be explicit about rounding, too. An on-time rate of 93.6% is consistent with the 93.6043% reference. 93.74% differs by 0.1357 percentage points, beyond this question’s 0.01 tolerance. An answer that looks close on the page can still be outside the requirement.

Diagnostic: does the explanation hold up?

Diagnostic questions ask why something happened. We check both the explanation and the numbers used to support it.

Take the gap between marketing and finance revenue. A useful answer needs to establish the size of the gap, explain the different scopes, and tell the reader when each metric is appropriate. Those are three separate requirements you can write down before the agent runs.

In the first card, nao reports the revenue figures, explains that paid-channel attribution covers a narrower scope than finance’s total, and describes what each measure is useful for. It explains why the totals should differ.

Explanations also need numerical scrutiny. In the refund-category example, nao correctly identifies Tanks & Kits as the largest contributor, but gives 465,797 in refunded revenue against a reference of 490,636.33. The ranking sounds convincing; the amount is about 5.1% low.

For your own evaluations, turn “explains the result” into a short list of observable requirements. Which comparison must the answer establish? Which difference must it explain? What conclusion should the reader be able to draw? Then check the facts supporting that reasoning separately.

Prescriptive: can you use the recommendation?

Prescriptive questions ask what to do. They need evidence and a proposed action, plus the consequences or trade-offs relevant to the decision.

The safety-stock answer illustrates the gap. Genie names the leading variants and reports the correct counts: 418, 277, and 191 stockout days. It recommends prioritizing those variants. But it never weighs holding costs against lost-sales risk, which the question’s rubric requires.

You could agree with the direction and still need more work before placing an order.

Several recommendations may be defensible. The rubric should specify what the reasoning must establish: the problem, a concrete action, the expected effect, and the relevant trade-off. Avoid requiring the agent to imitate one preferred paragraph.

For forecasts, we look for the assumptions behind the estimate. In the category-returns example, Hex labels its margin-exposure calculation as a prioritization proxy. It does not claim that the estimate is a measured accounting loss.

Turn those expectations into a judge

We use a model to assess explanations and recommendations, and code to check its output. The model must identify supporting passages for its decisions. Code verifies those quotes, checks direct numeric comparisons, and derives the grade.

The judge receives the question, rubric, agent response, and reference evidence. For a simple metric, the evidence is a stored value. For questions supported by SQL, we execute the reference query and include all its rows. A trend across five platforms cannot be fairly assessed if part of the platform data is missing. If required evidence cannot be prepared, grading stops.

The model then assesses each required criterion as pass, fail, or unverifiable. A pass needs a supporting excerpt from the answer. A fail identifies a wrong claim or missing requirement. Unverifiable means the supplied evidence cannot resolve a material claim.

Code checks that every criterion appears exactly once and that the quoted words occur in the recorded response. For each passing criterion, we can see the passage the judge relied on. A real quote can still be interpreted incorrectly, so that semantic judgment remains something to validate with people.

For direct numeric claims, the model identifies the reported number and maps it to a reference value. Code checks the original number token, applies the declared unit conversion, and calculates the difference. An extractor cannot turn −42 into 42 by dropping the sign, or use 42 as evidence for an answer that actually says 420.

Tolerances are part of this check. We use an explicit absolute tolerance when the question supplies one. Scalar and table comparisons otherwise allow for displayed precision; rubric comparisons without a specified tolerance use a 1% relative fallback. The model still needs to select the correct metric, reference value, and units. Derived calculations and forecasts need their own explanation of the method and assumptions.

A failed required check prevents a pass. An unverifiable assessment produces a review-required error rather than a forced good or wrong grade. We save the assessment, evidence, and computed checks so we can investigate the result later. Reused grades must match the question, evidence, answer, and judge configuration that produced them.

Evaluate the judge, too

Some of the passes in these cards deserve review. The judge accepted the order-total explanation even though it did not explicitly recommend deduplicating orders, as the rubric required. Saving the reasoning helps us spot mistakes like this; it does not prevent them.

To start your own evaluation, choose a small set of questions people actually ask. Write the requirements first, calculate the reference values independently, and have people review the agent responses. Include difficult cases: a correct table with no explanation, sound reasoning with one wrong number, and a sensible recommendation that omits a cost.

Compare the judge’s decisions with those human labels. Inspect false passes, false failures, and cases sent to review separately. Keep a held-out set that you do not use to tune the prompt. When a disagreement reveals an ambiguous question or incomplete evidence, improve those inputs as well.

Also keep unanswered questions, platform failures, and judge errors visible alongside your scores. Removing a response from grading does not establish that the agent answered it correctly.

We have implemented and tested the checks described here. Measuring their agreement with human reviewers remains a separate validation step. Code tests establish that the checks execute as intended; human review helps establish whether the decisions are fair.

If you’re building an evaluation, start with a few real questions and write out what each answer needs to contain. Check the numbers, read the explanations, and look for the costs or assumptions behind recommendations. When you disagree with a grade, keep the answer and the reason. Those cases will tell you what to fix in the judge, the rubric, or the question itself.