The Data Agent Stack - Part 7: Evals, Provenance, and Production Feedback Loops

Why a data agent is only useful if you can prove where the answer came from

The agent answered the same question correctly yesterday.

Overnight, a context document, metric definition, schema, model, or tool changed.

Today, the SQL still compiles. The query still returns rows. The answer is different.

Which part of the system regressed?

Part 6 ended with a distinction:

A receipt makes a run explainable.

An eval tells you whether it was right.

Part 7 goes one layer further.

The production data agent is the system under test.

An eval is not a score attached to the final paragraph. It is a structured test of the evidence path that turned a governed question into a returned or published answer.

A receipt can prove exactly what happened and still document the wrong answer.

A receipt is evidence, not a verdict

A run receipt should expose enough of the run to explain what happened to its intended audience.

Which question did the system think it was answering? Which metric and context versions did it use? Which tables did it select? Which tools and SQL ran? What data boundary applied? Which policy decisions allowed, denied, redacted, or redirected the run? Where did the result go?

That evidence makes the run inspectable.

It does not make the run correct.

The agent can use the right identity, obey every permission, execute valid SQL, attach the result link, and still choose the wrong canonical table. The receipt faithfully records the failure. It does not erase it.

The reverse is also possible.

The agent may return the right number, but the system did not record the context version, metric contract, data snapshot, tool configuration, or policy bundle. The answer may be correct, but the team cannot distinguish a repeatable result from a lucky one.

This is why four related concepts need separate names.

Provenance records where the answer came from.

Observability shows how the run behaved operationally.

Auditing lets an authorized reviewer inspect accountability and policy compliance.

Evaluation determines whether the behavior was acceptable under a defined question, evidence boundary, evaluation criteria, and policy path.

It also helps to distinguish the run manifest from the run receipt.

The run manifest is the complete internal record assembled across the execution. The receipt is a scoped projection exposed to a user, operator, auditor, or evaluator. Different audiences may need different views of the same run.

OpenLineage offers a useful analogy. Its object model distinguishes jobs, runs, input and output datasets, runtime events, and design-time metadata. Its facet model adds extensible, versioned metadata to runs, jobs, and datasets.

A data-agent run needs additional fields for resolved intent, model and workflow versions, selected context, tools, SQL, policy outcomes, assumptions, final claims, memory operations, publishing, and grader versions.

My read is that the run manifest should reference sensitive documents and full result sets in their governed systems rather than copy them into a second uncontrolled store.

Keep stable identifiers, versions, hashes, timestamps, policy decisions, bounded result digests, validation outcomes, and evidence references. Preserve enough information for replay and diagnosis without creating a shadow warehouse.

The manifest itself also needs an access policy, retention class, deletion behavior, and tamper-evident storage. Even without raw result rows, it may expose sensitive questions, identities, table names, SQL, business assumptions, and publishing destinations.

The evaluation invariant is:

Every dependency that can change an answer must be identifiable in the run manifest. Every deployable dependency must be versioned and tested against representative cases before broad rollout.

Each receipt should expose the subset needed by its intended audience.

Replay should be defined in levels.

Use exact replay only where deterministic dependencies, model and tool versions, randomness controls, data snapshots, and policy state can be restored.

Use behavioral replay when the same system and fixture can be restored, but probabilistic outputs may differ. Run the case several times and measure the distribution.

Use diagnostic reconstruction when the original state no longer exists, but the manifest still contains enough evidence to explain the likely failure path.

A run that supports none of these is not merely hard to debug. Its observability contract is incomplete.

Evaluate the whole analysis path

A data-agent run has several independently fallible stages.

The user may ask an ambiguous question. The system may resolve the wrong metric. Retrieval may miss the authoritative definition. Table selection may choose a deprecated asset. The tool router may select the wrong engine. The SQL may use the wrong grain. The result may be stale. The final prose may overstate what the evidence supports. The policy path may silently broaden authority.

Evaluating only the final answer compresses all of those failures into one score.

If you remember one picture from this post, make it this one:

Each stage needs its own observable artifact and its own pass-or-fail rule.

Intent resolution can be checked against a canonical metric, cohort, period, comparison, and ambiguity policy.

Context retrieval can be checked for required sources, prohibited sources, authority, freshness, relevance, and permissions.

Table selection can be checked for canonical status, population, grain, lineage, ownership, and deprecation.

Tool calls can be checked for capability, schema, arguments, bounds, side effects, and effective authority.

SQL can be checked for parsing, schema binding, joins, filters, aggregation, grain, cost, and semantic intent.

Results can be checked against exact values, normalized row sets, tolerances, reference queries, or business invariants.

The answer can be checked for claim support, visible assumptions, uncertainty, evidence links, and prohibited conclusions.

Memory and context writes can be checked for truth, scope, sensitivity, ownership, and approval.

Publishing can be checked for destination, audience, freshness, evidence, and authority.

Policy is enforced by the runtime. The eval harness verifies that the recorded allow, deny, redact, transform, and fallback outcomes matched the case.

Policy failures should remain hard failures. A model judge should not average an unauthorized retrieval path into an otherwise high-quality score.

This requires both component evals and end-to-end evals.

Component evals tell you where the failure entered the system. End-to-end evals tell you whether the components worked together to answer the user’s actual question.

Neither replaces the other.

Pinterest’s public analytics-agent writeup makes this distinction concrete. Its benchmarking framework measures table discovery separately from SQL generation, then slices difficult cases across complex analytical logic, ambiguous business terms, cross-domain questions, and schema evolution.

One aggregate answer-quality score would hide which capability failed.

The same distinction applies to existing data tests.

dbt Semantic Layer validations can verify that configuration parses, the semantic graph satisfies defined constraints, physical tables contain the declared fields, and generated SQL executes.

Those are important prerequisites. They do not prove that the agent selected the right metric, used the right context, answered the requested question, or preserved the right policy path.

The broader agent literature reinforces the same design. The ReAct paper models an interleaving of actions and environmental observations. For production evaluation, the useful implication is to inspect observable tool calls, outputs, retries, validation steps, and revisions rather than grading only the final response.

The final answer is only the last artifact in the system under test.

Golden cases need more than golden SQL

A useful golden case starts with a question.

A production case also has to define what the question means, which evidence is acceptable, and what the system should do when the meaning is unresolved.

A complete case should include:

1. The natural-language question and any relevant conversation prefix.

2. The canonical intent, metric, dimensions, cohort, period, comparison, and requested output.

3. The metric contract, including grain, filters, exclusions, ownership, and version.

4. Required, preferred, and prohibited context sources.

5. Required, acceptable, and prohibited tables or query patterns.

6. One or more reference SQL implementations.

7. A fixture, snapshot, as-of timestamp, or current reference-query strategy.

8. The expected result, tolerance, comparison rule, or business invariant.

9. The expected policy result, including allow, deny, redact, transform, or authorized fallback.

10. The required evidence, assumptions, caveats, destination, and answer characteristics.

11. The ambiguity policy: clarification, a disclosed default, multiple interpretations, or abstention.

12. Domain, difficulty, sensitivity, ownership, and failure-slice tags.

Here, the evaluation criteria are the rules, references, invariants, or reviewer judgments used to determine whether a case passes.

The OpenAI public data-agent case study describes curated questions paired with manually authored reference SQL. The system executes both the generated and expected SQL, compares the SQL and returned data, allows for acceptable variation, and uses the combined signals to produce a score and explanation.

The same cases operate as development regression tests and production canaries.

GitHub describes a related pattern in its internal analytics-agent writeup. Its structured cases contain prompts, known correct answers, ground-truth SQL, domain, and difficulty. Each case runs through multiple parallel trials, and the framework aggregates completion, accuracy, and duration before comparing configurations.

A complete case has multiple correctness checks:

Reference SQL is one correctness check, not the only one.

Two queries can use different CTEs, aliases, join order, predicate forms, or date expressions and still implement the same analysis. Exact string matching would reject a valid solution.

Result comparison is stronger, but it is not conclusive by itself.

A wrong query can return the expected number by coincidence on today’s data. A deprecated and canonical table may temporarily agree. A missing filter may remain invisible until a new population appears. A many-to-many join may produce the right aggregate for one fixture and fail on the next.

The comparison strategy should match the case:

• Use exact scalar comparison for stable fixtures and exact arithmetic.

• Normalize types, aliases, nulls, timestamps, and row ordering before comparing tabular results.

• Allow subsets or supersets when additional diagnostic columns are harmless.

• Use numeric tolerances for floating-point values, approximate computations, or bounded late-arriving adjustments.

• Use relational invariants for uniqueness, reconciliation, cohort separation, valid ranges, and grain.

• Execute a current canonical reference query when the underlying value legitimately changes.

• Use SQL semantic analysis to inspect tables, joins, filters, aggregation, and window logic.

• Use human review when metric meaning is contested or multiple analytical conclusions are defensible.

Mutable data makes the data boundary part of the test contract.

A case can use a stable fixture, a versioned warehouse snapshot, a time-travel identifier, an as-of timestamp, a current reference query, or a set of invariants.

Without one of those, a changed result cannot be classified. The agent may have regressed, or the data may have legitimately changed because of late arrivals, a backfill, a metric revision, a schema migration, or a freshness incident.

The grader order matters too.

Start with deterministic policy and structural checks. Then execute the query. Compare results and invariants. Inspect SQL semantics where useful. Use a model judge for questions such as whether claims are supported, whether assumptions are visible, and whether ambiguity was handled appropriately. Escalate high-risk cases and grader disagreements to calibrated human review.

A model judge should not override a deterministic policy failure, a broken invariant, or a clear result mismatch.

OpenAI’s evaluation best practices recommend scoped tests at every stage, task-specific cases that reflect real production distributions, comprehensive logging, continuous evaluation, and human calibration of automated scoring.

The guidance also warns that model judges can exhibit position and verbosity bias.

The public data-agent case study names the Evals API used when the article was published. OpenAI’s published deprecation timeline lists October 31, 2026 for read-only mode and November 30, 2026 for shutdown.

The durable pattern is the dataset, comparators, graders, release gates, and canaries, not one product API.

Golden SQL is a reference implementation, not a monopoly on correctness.

Every answer-changing artifact needs a release gate

In ordinary software, the deployable artifact is usually obvious.

For a data agent, the answer depends on more than application code.

A model change can alter tool selection. A prompt change can alter clarification behavior. A tool-description change can redirect authority. A new context document can change table ranking. A retrieval-index rebuild can surface stale query patterns. A semantic-contract change can legitimately change the expected number. A pipeline migration can invalidate a table context card. A policy change can turn a previously valid path into a denial or fallback. A grader change can move the score without changing the agent.

Every one of those is an answer-changing artifact.

Every answer-changing artifact needs a version, an owner, and a test.

GitHub treats context and agent-configuration changes as release candidates. Contributors submit context through pull requests, and each change goes through offline evaluation before it ships. Multiple trials expose variance, while accuracy, completion, and duration support configuration comparison.

Spotify’s context-layer writeup shows why the release gate has to continue after deployment. Its expert-owned clusters contain datasets, documentation, and vetted question-SQL pairs.

Cluster health tracks whether the underlying data remains healthy, curated pairs survive schema changes, context covers current questions, and generated SQL remains reproducible. Questions, answers, SQL, and feedback return to named cluster owners.

Pinterest reinforces the need for capability-specific gates. It separately measures table discovery and SQL generation, uses human review and EXPLAIN validation, updates indexes, improves judges, and adds SME-verified and production-derived cases as the system expands.

The release path should compare the candidate with the current production baseline:

change manifest → impacted component evals → critical end-to-end suite → repeated trials → baseline and slice comparison → review → shadow or canary → rollout → monitoring → promotion or rollback

The exact suite depends on the artifact.

A model change should run the broad critical suite with repeated trials, tool-choice distributions, latency, cost, and policy checks.

A context change should emphasize retrieval, source authority, required and prohibited sources, table selection, domain coverage, and stale-context cases. Its rollback unit may be a document version, curated-example version, context-card version, or index snapshot.

A semantic-definition change should update the metric version and the affected cases explicitly. A changed answer may be the intended result.

A tool change should test schema compatibility, authority, argument validation, bounds, timeouts, retries, and side effects.

A policy change should run allowed, denied, revoked, redacted, transformed, fallback, memory, export, and publishing cases.

A grader change should be versioned and tested against a frozen human-labeled set so the team can distinguish an agent improvement from a change in the ruler.

Release thresholds should be slice-specific.

Block or roll back on policy failures, critical metric regressions, prohibited table selection, missing required evidence, or cost and latency breaches outside the approved budget.

One average score is not a release gate.

The score should be sliced by domain, metric family, ambiguity, table complexity, join type, schema evolution, permission path, tool path, and failure class.

A common domain can improve enough to hide a complete failure in Finance, experimentation, or a sensitive-data path.

Production feedback is a data pipeline

Offline gates catch known failures.

Production creates the cases you did not know to write.

A user corrects a metric definition. A reviewer rejects an answer. A query retries three times. A tool route changes. A result is technically valid but misleading. A policy fallback answers a narrower question. A user abandons the session after repeated clarification.

Those are signals.

They are not automatically ground truth, and they are not automatically learning.

Grab’s engineering writeup makes the operating loop concrete. Reviewers can approve, reject, refine, reroute, or annotate responses.

Grab then samples annotations into offline cases, analyzes failure patterns, tracks rejection rates, targets prompt, guardrail, and tool improvements, and builds regression suites from actual failures.

Spotify routes questions, answers, generated SQL, and user feedback back to named domain owners. That ownership turns telemetry into maintained context rather than an unreviewed memory dump.

Meta’s analytics-agent writeup adds a useful scope distinction. It describes personal query-history context, visible SQL, domain cookbooks, reusable analytical recipes, custom validations, and tool controls.

Those patterns show that a correction may belong to a user, a team workflow, a shared semantic asset, or a validation rule. It does not automatically belong in global memory.

The production loop should force every confirmed failure through an owner:

The root cause determines the artifact.

A clarification that matters only to the current analysis should remain session-local.

A stable user preference may become personal memory.

A team convention may become team-owned context with an owner, scope, review date, and deletion path.

A canonical business rule should become an approved semantic contract or domain document.

A wrong population may require a table-context or pipeline fix.

A bad tool route may require a tool contract or workflow change.

A confirmed recurring failure should usually become a regression case.

An unsupported, unauthorized, or overly local correction may require no shared change at all.

The feedback invariant is:

No correction becomes shared system knowledge until it is validated, scoped, authorized, assigned to an owner, encoded in the right artifact, and protected by a regression test.

Feedback without triage is telemetry. Feedback with ownership becomes reliability work.

Offline gates are not enough

Offline evals provide controlled data boundaries, repeatability, and fast comparison.

Production keeps moving.

The user-question distribution changes. Schemas evolve. Metrics get redefined. Context documents decay. Retrieval indexes are rebuilt. Models and tools change behavior. Permissions are revoked. Late-arriving data changes yesterday’s result. A canary question that represented production six months ago may no longer represent it today.

Production evaluation therefore needs several controls.

Shadow runs execute a candidate configuration without publishing its answer.

Canary questions exercise critical metrics, domains, data boundaries, and policy paths against live dependencies.

Limited rollouts expose the candidate to a bounded set of users, questions, or domains.

Sampled production replay turns reviewed traffic into comparison cases.

Online monitoring detects shifts in table selection, context health, tool failures, retries, result anomalies, grader disagreement, latency, cost, and policy outcomes.

Keep these signals separate because satisfaction, reliability, analytical correctness, and governance failures require different owners and release decisions.

User satisfaction includes ratings, corrections, answer edits, repeated use, and abandonment.

System reliability includes completion, tool errors, retries, timeouts, latency, and cost.

Analytical correctness includes reference-result agreement, invariant failures, table-selection accuracy, and claim support.

Governance correctness includes unauthorized retrieval, incorrect fallbacks, redaction failures, memory-scope violations, and publishing errors.

Production drift includes changing question distributions, schema and metric changes, context decay, model and tool changes, permission changes, and declining reproducibility.

A thumbs-up rate is useful.

It is not proof of correctness.

The evaluation surface also expands as the agent receives more responsibility.

A recent data-agent autonomy taxonomy frames its highest L4 and L5 levels as a research roadmap, not as normal production deployments.

The useful design implication is narrower: as agents run longer workflows and gain more authority, evaluation has to expand beyond task accuracy into robustness, adaptability, safe recovery, policy preservation, operator intervention, cumulative cost, latency, and rollback behavior.

Offline evals protect known behavior.

Production signals tell you when the world, the system, or the workload has moved.

You need both.

Failure modes

1. Exact SQL comparison rejects a valid query

The generated query uses different CTEs, aliases, predicates, or join order but implements the same metric and returns the correct result.

A string matcher calls it wrong.

Control: Execute both queries against the same data boundary, normalize their results, and use SQL semantic features as diagnostics rather than the sole verdict.

2. Matching results hide the wrong evidence path

The agent queries a deprecated table and gets the expected number because the old and new tables still agree today.

A result-only test passes.

Control: Check required and prohibited tables, metric versions, grain, source authority, and relational invariants in addition to the result.

3. Live data makes the golden value stale

Late events, corrections, backfills, or a legitimate metric revision change the expected result.

A fixed scalar makes the healthy system look broken.

Control: Record a snapshot or as-of boundary, execute a current canonical reference query, or use stable invariants and tolerances.

4. A context change regresses table discovery

A new document or query example outranks the canonical table for one domain.

The common eval cases still pass because they do not cover that table family.

Control: Gate context changes with required and prohibited source checks, table-selection cases, domain slices, health monitoring, and a versioned rollback unit.

5. One average score hides a broken domain

A model upgrade improves common product questions and completely breaks a smaller Finance or permission-denied slice.

The overall score rises.

Control: Set release thresholds per critical slice, not only globally. Treat policy failures and critical metric failures as hard blockers.

6. The grader shares the agent’s blind spot

The agent and model judge apply the same incorrect definition of “active user.” The answer is polished, and the judge rewards it.

Control: Put deterministic metric contracts, reference execution, invariants, independent human labels, and grader-disagreement analysis ahead of stylistic judgment.

7. A production failure is fixed but never becomes a test

An engineer patches a prompt or context document, closes the incident, and moves on.

The same class of failure returns after the next model, index, or schema change.

Control: Do not close the issue until it links the root cause, owning artifact, versioned fix, regression case, rollout decision, and follow-up production signal.

Builder checklist

1. Define a versioned golden-case schema.
   Require canonical intent, semantic contracts, ambiguity behavior, context expectations, a data boundary, an expected result or invariant, policy and fallback outcomes, evidence requirements, slice tags, and a named owner.

2. Test each stage and the complete run.
   Maintain component evals for intent, retrieval, table selection, tools, SQL, results, answer support, provenance, policy, memory, and publishing, plus critical end-to-end cases.

3. Emit a complete run manifest.
   Record the run ID, request and intent, requester and effective policy, model, prompt, workflow and tool versions, selected context, metric contracts, generated and executed SQL, engine and query ID, data boundary, result reference, validations, assumptions, memory reads and writes, destination, grader version, human-review outcome, and retention class.

4. Use multiple correctness checks.
   Compare normalized results, tolerances, invariants, reference execution, and semantic SQL features. Do not depend on SQL strings or a single model judge.

5. Version every answer-changing artifact.
   Models, prompts, workflows, tools, context, indexes, semantic contracts, pipelines, policies, and graders all need owners, change manifests, release gates, and rollback units.

6. Run repeated trials and report slice-level variance.
   Track completion, correctness, tool behavior, latency, cost, and grader disagreement by domain, metric, ambiguity, policy path, and complexity.

7. Calibrate automated graders against humans.
   Freeze the grader model, rubric, reference set, and thresholds with each result. Measure agreement on a human-labeled holdout and investigate disagreements.

8. Close failures through ownership.
   A confirmed production failure should reach the right artifact owner, create a regression case, pass offline evals, survive a canary, and meet explicit promotion or rollback criteria.

Recap

Provenance is necessary.

It is not sufficient.

A receipt tells you which system, context, tools, data, policy, and destination produced the answer. An eval tells you whether that complete path was acceptable.

One final-answer score is too compressed. SQL strings are too brittle. Result matching can be fooled by coincidence. Model judges can share the agent’s blind spot. User corrections can be wrong or scoped too narrowly. Offline suites can age while production moves.

The durable pattern is layered:

Component evals provide diagnosis. End-to-end cases provide release confidence. Run provenance supports replay and reconstruction. Canaries exercise live dependencies. Production signals detect drift. Ownership turns confirmed failures into durable tests.

The complete data-agent stack is the unit that must become testable.

What the seven-part stack adds up to

The seven parts describe one operating system for trustworthy analysis.

Meaning and metric contracts define the question. The data foundation makes trusted assets legible. Context assembly selects the evidence for the turn. Pipeline code explains how that evidence was produced. Tools execute the analyst loop. Governance preserves authority across retrieval, execution, memory, and publishing. Provenance records the run, and evaluation determines whether the path was acceptable.

Production feedback closes the system. A confirmed failure reaches an owner, changes the right artifact, passes the relevant evals, and returns through a controlled rollout.

The cumulative loop is:

meaning → context → execution → authority → evidence → evaluation → improvement

A production data agent becomes trustworthy when meaning is owned, context is curated, actions are bounded, authority is preserved, evidence is recorded, and confirmed failures become tests.

Thanks for reading this seven-part series. Subscribe for future writing on the systems, boundaries, and operating patterns behind production agents.

If you are building this

Which failure is hardest to turn into a durable regression test in your data-agent stack?

Metric ambiguity, context retrieval, table selection, SQL semantics, result drift, policy behavior, grader quality, or production feedback?

The Data Agent Stack

• Part 1: What Is a Data Agent?

• Part 2: The Data Foundation Is the Agent

• Part 3: Context Assembly for Data Agents

• Part 4: Meaning Lives in Code

• Part 5: Tools, Query Execution, and the Analyst Loop

• Part 6: Trust, Permissions, and Governance

References

• OpenAI, Inside OpenAI’s in-house data agent, Evaluation best practices, and Working with evals, covering curated cases, reference SQL, result comparison, graders, continuous evaluation, canaries, human calibration, and the published Evals deprecation timeline.

• Spotify Engineering, Encoding Your Domain Expert: The Context Layer Behind Spotify’s Data Assistant, covering domain ownership, vetted question-SQL pairs, context health, reproducibility, and feedback routing.

• GitHub, How we built an internal data analytics agent, covering pull-request-gated context changes, structured cases, repeated trials, configuration comparison, and aggregate statistics.

• Grab Engineering, From firefighting to building: How AI agents restored our team’s core productivity, covering reviewer actions, production annotations, offline cases, targeted fixes, and regression suites.

• Pinterest Engineering, Unified Context-Intent Embeddings for Scalable Text-to-SQL, and Analytics at Meta, Inside Meta’s Home Grown AI Analytics Agent, covering capability-specific evaluation, difficult slices, visible SQL, scoped context, validations, and tool controls.

• OpenLineage, Object Model and Facets and Extensibility; dbt, Semantic Layer Validations; and the ReAct and Data Agents papers, supporting the distinctions among lineage, run provenance, semantic validation, observable action loops, and broader evaluation as agent authority grows.