Make Claude Code and Codex Data Analysis Trustworthy

To make Claude Code or Codex data analysis trustworthy and reusable, give the agent five things:

1. Maintained business context.
2. Shared definitions.
3. Reusable analytical building blocks.
4. A visible evidence and validation path.
5. A way to escalate ambiguity to a responsible data person.

Most teams begin with access. They connect the warehouse, add MCP servers, put a data dictionary in Google Drive, and assume the AI will figure out the rest.

We tried versions of that ourselves. It worked just often enough to be frustrating.

The agent could write a query and produce a plausible answer. But sometimes it did not retrieve the dictionary. Sometimes it guessed what retention meant. Sometimes a customer used an internal phrase such as “first-stage cycle,” and the AI had no idea what that phrase meant inside the business. The answer still arrived with confidence.

We would discover the failure too late—after somebody used the analysis, or after two people began arguing because their numbers did not match.

The lesson was not that Claude Code, Codex, MCP, or data dictionaries are bad. They are useful. The lesson was that connectivity is not an operating system. I have already written about why access is not the same as understanding. This is the practical system we learned to put around that idea.

The AI needs a connected path from the business question to the evidence, and a responsible person at the end of the path when software should not guess.

Why warehouse access feels better than it is

Coding agents are excellent at data work.

They can inspect files, write SQL, run code, clean exports, compare segments, generate visualizations, and package the result. OpenAI's official KPI root-cause workflow shows how an agent can combine dashboards, definitions, source exports, segment cuts, launch context, and stakeholder discussions into a source-backed analysis.

That list is also the clue.

The agent does not become trustworthy merely because it can query the warehouse. It needs the metric definition, the relevant operating context, the source evidence, and instructions for separating confirmed findings from hypotheses.

Suppose you ask, “What was retention last quarter?”

To answer correctly, the agent may need to know:

• whether retention is measured for users, accounts, subscriptions, or revenue;
• which event qualifies a customer as active;
• whether the cohort begins at signup, activation, or payment;
• how upgrades, pauses, reactivations, and refunds are handled;
• which timezone and calendar rules apply;
• whether the source data was current;
• and which individual customers sit behind the final percentage.

Raw tables do not answer those questions. SQL does not choose the business meaning for you.

A query can run perfectly and still be wrong.

1. Give the agent business context it can actually retrieve

Every company has language that makes sense internally and nowhere else.

“First-stage cycle.” “Healthy account.” “Activated publisher.” “Qualified customer.” “Core order.” “Expansion.” These terms may encode years of operating knowledge. They may also be scattered across a strategy deck, a sales document, a dashboard description, a meeting transcript, and one person's memory.

Putting all those files in a shared drive does not guarantee the agent will use the correct one.

Anthropic describes context engineering as the work of curating and maintaining the information available to an agent across system instructions, tools, MCP, external data, and message history. Its guidance on effective context engineering emphasizes retrieving relevant information just in time rather than assuming that more context automatically produces a better answer.

This matches what we experienced. We created dictionaries. We connected the files. But if the agent did not know that it needed a particular definition, did not retrieve the right document, or ranked an outdated explanation above the current one, the information might as well not have existed.

Business context needs structure:

• What are the important entities, activities, metrics, and dimensions?
• Which definition is authoritative?
• Which decision or workflow uses it?
• What changed, when, and why?
• Who owns the meaning?
• Which source should win when two records disagree?

The goal is not to load the entire company into one enormous prompt. The goal is to give the agent a map and clear retrieval paths.

2. Turn definitions into shared operating logic

A data dictionary is helpful. A definition the system actually uses is better.

If “retained customer” is described in a document but every analysis rewrites the logic from raw tables, the definition is advisory. The agent can ignore it, misunderstand it, or implement a slightly different version each time.

A shared definition should connect three layers:

1. Human meaning: what the business intends the term to mean.
2. Executable logic: how that meaning is represented in the data.
3. Visible examples: which records qualify and which do not.

For example:

A retained account is a paying company that completed at least one qualifying activity in each of the first three full calendar months after activation.

That sentence is not enough by itself. The system also needs to define “paying company,” “qualifying activity,” “activation,” “full calendar month,” and the treatment of paused or reactivated accounts. Then it needs to let a person inspect the accounts included in the result.

This is why we organize data around business Activities in Polyform. The agent should not invent retention directly from a maze of event tables. It should assemble retention from stable building blocks that already represent meaningful business behavior.

The same definition can then support a dashboard, a cohort analysis, a customer list, an investigation, or an automation. Reuse begins when the meaning and the logic stay together.

3. Give the AI steps and building blocks—not a blank warehouse

People often expect an agent to move directly from a broad question to raw data:

Go into the warehouse and tell me why retention fell.

That request hides several jobs. The agent has to interpret retention, find the relevant tables, understand identity, reconstruct customer behavior, choose comparison periods, validate freshness, calculate the metric, investigate segments, and distinguish evidence from a story that merely sounds plausible.

It may succeed. But if the whole process is improvised, the next run can take a different path.

A dependable system gives the agent reusable steps:

business question
→ approved metric and scope
→ underlying Activities and Dimensions
→ cohort or comparison logic
→ source records and freshness
→ visible evidence
→ conclusion, caveats, and next action

The agent can still explore. It can still write code. It can still discover a new segment or challenge an old assumption. But it begins from shared primitives and leaves useful work behind.

This is the difference between an impressive chat and a compounding data capability.

In a chat, the company gets an answer. In a system, the company also gets the definition, dataset, cohort, chart, evidence, and operating history that make the next answer better.

4. Make every important number inspectable

If the AI says the answer is 32, you should be able to see the 32.

That sounds obvious. Many AI-analysis experiences stop at the aggregate, the chart, or the generated SQL. A user asks whether the answer is correct, and the system explains its reasoning in more detail. But another explanation is not validation.

Trust requires an evidence path:

• Show the records behind the number.
• Show the definition and filters used.
• Show the time window and data freshness.
• Show the transformation or reusable logic that produced the dataset.
• Show exclusions, unknowns, and data-quality warnings.
• Save enough state that another person can reproduce the result.

This is one of my biggest concerns with treating an MCP connection to a dashboard tool as a complete analytics architecture. MCP is valuable connectivity. The official MCP architecture defines how hosts, clients, and servers exchange resources, tools, and prompts while preserving clear boundaries. It does not define your retention metric, decide which dashboard is authoritative, verify warehouse freshness, or appoint the person responsible when the answer is disputed.

The connection can give the agent a way to retrieve a number. Your system still needs to establish why that number should be trusted.

5. Give the AI a way out

This is the part most AI data systems miss.

What should happen when the AI cannot find a definition? What if two definitions conflict? What if the source is stale, the identity mapping is incomplete, or the business term is known only to the person who created it? What if the answer will drive a board decision or trigger an automated customer action?

The agent should not keep trying until it manufactures certainty.

It should escalate.

A good escalation includes:

• the original question;
• what the agent tried;
• the sources and definitions it inspected;
• the exact ambiguity or failure;
• the candidate interpretations;
• the records or systems affected;
• and the decision it needs from a person.

Then a named data owner resolves the issue. They may approve one definition, repair a model, clarify a business term, correct a permission, or decide that the analysis is not safe to use yet. The resolution should become part of the maintained system so the next person and the next agent do not hit the same dead end.

Without this loop, errors stay hidden until people notice conflicting numbers downstream. With it, uncertainty becomes visible work.

This is not a generic “human in the loop” slogan. The human has to own the repair.

A practical escalation rubric

The AI should stop and request review when any of these conditions are true:

Ambiguity

More than one plausible business definition would materially change the answer.

Missing evidence

The agent cannot expose the records behind a consequential aggregate.

Conflicting sources

Two systems or approved reports disagree and no precedence rule exists.

Stale or incomplete data

An expected source has not updated, a processing job failed, or coverage is unknown.

High consequence

The result will influence financial reporting, customer treatment, compensation, compliance, or a production action.

Irreversible action

The agent is about to change data, contact customers, alter a live workflow, or trigger an outcome that is difficult to undo.

Novel logic

The analysis depends on a new definition, transformation, identity rule, or causal claim that has not been reviewed.

The reference workflow

A trustworthy and reusable AI-analysis path looks like this:

1. State the business question and intended decision. “Why did activated-account retention fall, and should we change onboarding?” is better than “analyze retention.”
2. Load the relevant context. Retrieve the current definition, decision history, known incidents, and ownership—not every file in the company.
3. Use approved building blocks. Start from stable entities, Activities, Metrics, and Dimensions before querying raw data.
4. Inspect freshness and quality. Confirm that the expected sources and transformations are current enough for the question.
5. Run the analysis. Let the agent write code, compare cohorts, segment behavior, and investigate plausible drivers.
6. Expose the evidence. Save the dataset, examples, filters, logic, and source links with the result.
7. Separate findings from hypotheses. State what the data proves, what it suggests, and what remains unknown.
8. Escalate when necessary. Send ambiguity, conflict, or unsafe actions to the named data owner with the investigation attached.
9. Persist the learning. Update the definition, building block, test, or operating context so the next analysis starts from the resolution.
10. Publish a reusable result. Make the final dataset, cohort, dashboard, decision document, or workflow visible to the people who need it.

This path does not make every answer slow. It makes the easy work fast and the dangerous work visible.

Why we built Polyform as one connected system

We learned this problem by trying the obvious approach.

We connected tools through MCP. We created shared dictionaries. We put definitions in Google Drive. Then we watched the AI occasionally skip the material, misunderstand a customer term, or produce an answer whose flaw appeared only after people had begun using it.

We realized that data AI could not be a chat window floating above disconnected tools. The system had to think the way data work actually happens.

Polyform connects permissions, business context, definitions, Activities, Metrics, models, source data, analysis, dashboards, workflows, and operating history. The outputs stay visible. A cohort can be inspected. A dataset can be reused. A dashboard remains tied to the logic and evidence behind it.

And a dedicated data expert meets with the company every week. That person learns how the business works, maintains the context, reviews consequential results, and becomes the escalation path when something goes haywire.

The software gives the expert leverage. The expert gives the result accountability.

I am biased in recommending Polyform because we built it around this exact failure mode. But even if you build the system yourself, I would insist on the same five requirements: context, definitions, reusable building blocks, inspectable evidence, and escalation.

Without them, the AI can help one person move faster.

With them, the company can trust and reuse what the AI produces.

That is the difference between AI doing analysis and an AI-first company building a data capability.

That is also the purpose of Polyform’s maintained AI context: keep definitions, decisions, evidence, and operating history available to the work instead of rebuilding them in every prompt.

References and further reading

• access is not the same as understanding
• KPI root-cause workflow
• effective context engineering
• official MCP architecture