Anthropic’s 95% Analytics Accuracy Is Not the Finish Line

Anthropic published one of the clearest descriptions I have seen of what it takes to make AI analytics work inside a real company. My honest reaction is still: this is impressive engineering, and 95% is not a good enough trust standard.

At the reported aggregate rate, roughly one answer in every 20 is inaccurate. That does not mean every twentieth question fails on schedule. It means that across enough questions, wrong answers remain normal—not exceptional.

Now put that inside a company with hundreds of employees. People will ask hundreds of questions. Some wrong answers will be caught. The dangerous ones will look reasonable, circulate in a meeting, and influence a decision before anyone knows they were wrong.

Separate what Anthropic disclosed from what I am assuming

A useful critique has to be precise. The article gives us enough evidence to see the difficulty of the problem, but it does not give us every detail of the experiment.

What I am not assuming is a Metric count. Anthropic says the agent may face hundreds of viable choices and millions of fields. It also says each domain skill can route among roughly 30 reference files. Neither fact tells us whether the semantic layer held 20 Metrics, 200, or 2,000.

That unknown matters. A system that performs well over a narrow governed surface may behave very differently as the number of definitions, exceptions, teams, and historical versions grows. The article is evidence that structure helps. It is not evidence of a particular Metric-scale ceiling.

Anthropic's own details show how hard 95% was

This was not Claude pointed at a warehouse. Anthropic describes canonical datasets, human-owned Metric definitions, metadata, lineage, curated reference documents, skills, CI checks, offline evaluations, adversarial review, provenance, monitoring, correction harvesting, and domain ownership.

The team watched offline accuracy fall from about 95% to about 65% in one month before treating skill maintenance as an engineering responsibility. Roughly 90% of its data-model pull requests now include a skill change. That is not a prompt. It is a maintained operating system around the model.

Even then, Anthropic says the silent failure remains: the answer is wrong, looks plausible, and is used without correction. The company mitigates that failure with provenance, human sign-off for leadership-bound work, and daily checks against blessed dashboards. It explicitly says it does not have a robust solution yet.

So I do not read 95% as proof that AI analytics is solved. I read it as a very capable team showing us the amount of work required to reach a result that still leaves one answer in 20 wrong in aggregate.

The missing hierarchy

Do not put your semantic layer directly on top of SQL

A semantic layer is necessary. Anthropic is right that humans should own Metric definitions and that the agent should use the semantic layer first. The mistake is treating a Metric name, formula, and compiled query as the complete explanation.

SQL is an implementation language. It is precise, powerful, and often the right thing for a technical reviewer to inspect. It is not how most people understand the business. Showing a stakeholder a generated query does not help them decide whether “conversion” means the same thing they meant.

The semantic layer should sit on top of an assembly of simple business concepts. At Polyform, that assembly is a Dataset: a reusable population with a grain, related behavior, columns, filters, and views. The Dataset sits on top of Activities: timestamped, human-understandable events such as Started Web Session and Submitted Lead. Those Activities can sit on top of SQL and warehouse transformations.

This distinction is important: SQL still exists. Polyform can generate and inspect it. But the user does not have to understand a long query to understand the answer. The explanation moves from the Metric to the Dataset population to the Activities and examples. SQL is available at the end when implementation-level review is necessary.

Session conversion should be explainable as a customer journey

Suppose the Metric is session conversion rate. The Dataset grain is one web session. A session converts when the first qualifying Submitted Lead Activity happens after Started Web Session and within the agreed session window.

Now a person can inspect both sides of the definition. Here is a customer whose session converted, the exact Activities that made it qualify, and the relationship between them. Here is another customer whose session did not convert, and why. That is far more understandable than “the SQL looked reasonable.”

Activities work well as the base layer because they use simple verbs, are defined from one customer's perspective, and naturally open into customer-journey examples. They give the AI a small vocabulary of business facts it can assemble without hiding the meaning inside a new query every time.

Can this hierarchy answer every business question?

This is the first concern I hear: a Dataset definition sounds too constrained for the long tail of questions. My experience says the opposite.

I have spent eight years building data systems around Activities. In that time, I have not found a business question we could not express through Activities, relationships, Datasets, Metrics, and Dimensions. That is my founder experience, not a mathematical proof or a guarantee that the first model is always right.

The constraint is useful. When a new question appears, we can see whether it needs a new Activity, a different population, a new relationship, or a new measure. The question improves the shared model instead of disappearing into another one-off SQL file.

If you have a question you believe this hierarchy cannot answer, bring it to Polyform. Submit the question and let our Activity Schema team work through it with you. I would genuinely like to be proven wrong, because the failure would teach us where the model needs to grow.

The real benchmark is whether a person can challenge the answer

Anthropic's article moves the industry forward because it does not pretend model intelligence is enough. Its biggest gains came from governed sources, structured context, maintenance, evaluation, and review.

My disagreement is about where the explanation stops. A provenance footer can tell a user that an answer came through the semantic layer. It cannot, by itself, show what population was assembled, which customer behavior qualified, or how a boundary case was treated.

For AI analytics to become dependable self-service, the user needs a path from the number back to business language and individual examples. That path is:

Metric → Dataset → Activities → SQL.

Use the semantic layer. Govern it. Maintain it. But do not make SQL the layer where the business has to go to understand itself.

References and further reading

• Anthropic: How Anthropic enables self-service data analytics with Claude
• Polyform: Why AI Keeps Using the Wrong Business Definition
• Polyform: The Activity Schema approach