Cortex Analyst

• ✓ evals: supported
• ✓ logs: supported
• ✓ cost: supported
• ✓ users: supported

Snowflake logs every Cortex Analyst request in SNOWFLAKE.LOCAL.CORTEX_ANALYST_REQUESTS_V, one row per question. This view turns that log into turns/v1 rows.

• Analyst writes either SQL or prose. A request with SQL restates the question first (“This is our interpretation of your question: …”); that restatement becomes a message, not the answer. A request with no SQL answers in prose, which is what declining looks like, so statements is [] there and the turn counts against completeness.
• Analyst logs '', not NULL, when it wrote no SQL. The view turns that into [], and Benchouse ignores an empty statement as well.
• Keep the WHERE on the semantic models your app sends Analyst, the same ones you give the agent in Benchouse. Without it the view reads every model on the account, and the agent’s page lists the ones its evals don’t ask about.
• Snowflake bills Analyst per message, and only for successful ones. Each successful turn is one message in Benchouse, priced at the rate per 1,000 messages on the connection.

1. Create the view

As a role that holds the SNOWFLAKE.CORTEX_ANALYST_REQUESTS_VIEWER application role, create a view of the request log. Put your semantic models in the WHERE.

CREATE DATABASE IF NOT EXISTS BENCHOUSE;
CREATE OR REPLACE VIEW BENCHOUSE.PUBLIC.TURNS_V1 AS
WITH req AS (
    SELECT
        r.REQUEST_ID,
        r.TIMESTAMP                                   AS ASKED_AT,
        r.LATEST_QUESTION                             AS QUESTION_TEXT,
        r.SEMANTIC_MODEL_NAME                         AS SEMANTIC_MODEL,
        r.USER_ID,
        r.USER_NAME,
        r.PRIMARY_ROLE_NAME                           AS USER_ROLE,
        COALESCE(TO_VARCHAR(r.SOURCE), 'api')         AS CLIENT_INTERFACE,
        r.RESPONSE_METADATA:"model_names"[0]::string  AS MODEL_NAME,
        NULLIF(TRIM(r.GENERATED_SQL), '')             AS GENERATED_SQL,
        r.RESPONSE_STATUS_CODE                        AS STATUS_CODE,
        r.RESPONSE_BODY                               AS BODY,
        TRY_CAST(r.RESPONSE_METADATA:"analyst_latency_ms"::string AS NUMBER) AS LATENCY_MS,
        OBJECT_CONSTRUCT(
            'warnings', r.WARNINGS,
            'feedback', r.FEEDBACK,
            'question_category', r.RESPONSE_METADATA:"question_category",
            'tables_referenced', r.TABLES_REFERENCED) AS EXTRA
      FROM SNOWFLAKE.LOCAL.CORTEX_ANALYST_REQUESTS_V r
     WHERE r.SEMANTIC_MODEL_NAME IN ('SALES.PUBLIC.SV_ORDERS')
),
parts AS (
    SELECT q.REQUEST_ID,
           f.index                                    AS IDX,
           f.value:"text"::string                     AS TXT,
           (q.GENERATED_SQL IS NOT NULL
            OR f.value:"text"::string LIKE 'This is our interpretation of your question:%') AS IS_RESTATEMENT
      FROM req q,
           LATERAL FLATTEN(input => q.BODY:"message":"content") f
     WHERE f.value:"type"::string = 'text'
),
answer AS (
    SELECT REQUEST_ID,
           NULLIF(LISTAGG(CASE WHEN NOT IS_RESTATEMENT THEN TXT END, ' ')
                    WITHIN GROUP (ORDER BY IDX), '')  AS ANSWER_TEXT,
           NULLIF(LISTAGG(CASE WHEN IS_RESTATEMENT THEN TXT END, ' ')
                    WITHIN GROUP (ORDER BY IDX), '')  AS RESTATEMENT
      FROM parts
     GROUP BY REQUEST_ID
)
SELECT
    q.REQUEST_ID                                      AS turn_id,
    q.ASKED_AT                                        AS asked_at,
    q.QUESTION_TEXT                                   AS question_text,
    a.ANSWER_TEXT                                     AS answer_text,
    DATEADD(millisecond, q.LATENCY_MS, q.ASKED_AT)    AS completed_at,
    CASE WHEN q.STATUS_CODE BETWEEN 200 AND 299 THEN 'success' ELSE 'error' END AS outcome,
    CASE WHEN q.STATUS_CODE NOT BETWEEN 200 AND 299 THEN q.BODY:"error_code"::string END AS error_type,
    CASE WHEN q.STATUS_CODE NOT BETWEEN 200 AND 299 THEN q.BODY:"message"::string END AS error_message,
    q.USER_ID                                         AS user_id,
    q.USER_NAME                                       AS user_name,
    q.USER_ROLE                                       AS user_role,
    q.CLIENT_INTERFACE                                AS client_interface,
    q.MODEL_NAME                                      AS model_name,
    q.SEMANTIC_MODEL                                  AS semantic_model_ref,
    CASE WHEN q.GENERATED_SQL IS NULL THEN ARRAY_CONSTRUCT()
         ELSE ARRAY_CONSTRUCT(OBJECT_CONSTRUCT(
                'sql', q.GENERATED_SQL, 'status', 'not_executed', 'is_answer', TRUE))
    END                                               AS statements,
    CASE WHEN a.RESTATEMENT IS NULL THEN ARRAY_CONSTRUCT()
         ELSE ARRAY_CONSTRUCT(OBJECT_CONSTRUCT('kind', 'restatement', 'text', a.RESTATEMENT))
    END                                               AS messages,
    q.EXTRA                                           AS extra
  FROM req q
  LEFT JOIN answer a ON a.REQUEST_ID = q.REQUEST_ID;

2. Create the Benchouse user

The view is all this role can read:

CREATE ROLE benchouse_read;
GRANT USAGE ON WAREHOUSE eval_wh TO ROLE benchouse_read;
GRANT USAGE ON DATABASE BENCHOUSE TO ROLE benchouse_read;
GRANT USAGE ON SCHEMA BENCHOUSE.PUBLIC TO ROLE benchouse_read;
GRANT SELECT ON VIEW BENCHOUSE.PUBLIC.TURNS_V1 TO ROLE benchouse_read;
CREATE USER benchouse_read TYPE = SERVICE_AGENT DEFAULT_ROLE = benchouse_read;
GRANT ROLE benchouse_read TO USER benchouse_read;
ALTER USER benchouse_read ADD PROGRAMMATIC ACCESS TOKEN benchouse_read ROLE_RESTRICTION = 'BENCHOUSE_READ';

For evals as well, Benchouse asks Analyst your eval questions as this role, and Analyst’s SQL runs as it too. Add the semantic view and the tables behind it:

GRANT DATABASE ROLE SNOWFLAKE.CORTEX_USER TO ROLE benchouse_read;
GRANT USAGE ON DATABASE SALES TO ROLE benchouse_read;
GRANT USAGE ON SCHEMA SALES.PUBLIC TO ROLE benchouse_read;
GRANT SELECT ON SEMANTIC VIEW SALES.PUBLIC.SV_ORDERS TO ROLE benchouse_read;
GRANT SELECT ON ALL TABLES IN SCHEMA SALES.PUBLIC TO ROLE benchouse_read;

3. Add the agent

1. In Agents, click Add agent and choose Cortex Analyst.
2. Choose your Snowflake connection, or create one with the account identifier, warehouse, the role BENCHOUSE_READ and the token.
3. Enter the semantic view your app sends Analyst. If your app routes between several, add each one: an eval question can then name the one it asks through. Semantic views on the account are offered.
4. Leave the query as SELECT * FROM BENCHOUSE.PUBLIC.TURNS_V1 and click Test query. It runs the view without reading a row, names any column that is missing, and says which optional columns the view leaves out.
5. Enter your price per 1,000 messages, and click Add and sync.

The agent’s page shows each sync: how many rows were read and stored, any skipped with the reason, and semantic models in the log that the agent’s evals don’t ask.