The AI Analytics Demo Is Easy. The Maintenance Loop Is the Product
The first impressive answer proves the model can perform. The next month proves whether the product can survive reality.
The AI analytics demo is easy. The maintenance loop is the product.
I do not mean that the demo is technically easy. The team may have done excellent work. They may have connected the model, cleaned the example data, written the right instructions, tested the prompt a thousand times, and built a beautiful interface around the result.
I mean that a demo answers the easiest version of the product question:
Can this system produce one impressive answer under conditions the company prepared?
The answer is often yes. AI can do almost anything if you narrow the use case, give it enough context, and keep iterating until the path looks magical.
But that is not the question a buyer actually needs answered.
The real question is what happens after the first impressive answer—when a source changes, a definition drifts, a permission is removed, a workflow fails, the result looks wrong, and somebody has to find the problem, repair it, rerun the work, and explain what changed.
That loop is the product.
The old demo trickShow me the table underneath it
I used to see this in Looker demos long before generative AI.
The demo would be: look how easy it is to see this metric. Now drill into it. Now filter it. Now find the insight. Every click was fast and every answer was clean.
I would say, “Awesome. Show me the table underneath it.”
And sometimes it was one table with about a thousand rows and a column that might as well have been called insight.
Of course the demo worked. The difficult parts of data work had already been removed from the environment.
That anecdote is not a complete description of Looker. Modern Looker has a real modeling layer. Its own LookML documentation covers joins, derived tables, validation, caching, reuse, and the maintenance work required to turn warehouse data into governed analytics.
That is exactly the point. The real product was never the clean drill-down by itself. It was the system and the people required to make that drill-down keep meaning the right thing.
AI made the gap 100,000 times worse
That is not a measured statistic. It is the only phrase that feels proportional to the change.
A traditional analytics demo could hide complexity by preparing the data. An AI demo can hide complexity and create the feeling that the product will handle anything you ask next.
The model speaks naturally. It generates a chart. It explains the result. It follows the presenter's exact wording. The audience does not see the failed prompts, the extra documents added to the context, the fields renamed for the demo, or the questions the system was trained not to receive.
This is not necessarily dishonest. A demo is supposed to show the possibility.
The mistake is treating possibility as operating proof.
We see the same thing across AI. Apple Intelligence demos showed personal context and actions across apps. Those are meaningful ambitions, but Apple's own developer guidance explains that apps must expose their data, entities, and actions through App Intents and contextual annotations. The keynote moment depends on the real app, the real permission, the real context, and the real action all being available together.
Microsoft's Power BI Copilot documentation is even more explicit. It tells model owners to simplify the schema, create verified answers, add business-specific AI instructions, test common questions, and continuously test as the model and available tools change. Microsoft also warns that the same prompt and grounding data can produce different outputs.
That does not mean Copilot is fake. It means the demo is the visible tip of a maintained analytical system.
The first answer is where the work starts
Suppose you ask, “Why did retention fall last month?” The AI finds a billing change, shows the affected cohort, and produces a convincing explanation.
Great. Now the company starts using the result.
Then the billing platform changes an export. Finance updates the definition of an active subscription. A contractor loses access to customer-level data. The weekly report runs before a late source arrives. A dashboard filter keeps the old plan names. The customer-success workflow fails after an API change.
None of those failures make for a good demo. All of them are normal product conditions.
A source, definition, permission, dependency, or result no longer matches the prior run.
Find the models, reports, workflows, people, and decisions that inherited the problem.
Resolve whether the change is a bug, an approved business decision, or missing context.
Change the permitted data logic, definition, test, permission, or workflow.
Rebuild every affected output instead of stopping after the diagnosis.
Check the final state and keep the decision, evidence, and test for the next run.
A diagnosis is not recovery. The loop closes only when the corrected result is running again.
This is not theoretical. Anthropic recently described how its own analytics-agent documentation fell out of date as the underlying data model changed. In its account of building self-service analytics with Claude, the company says offline accuracy drifted from about 95% to about 65% in one month before it treated skill maintenance as an engineering problem. It now says roughly 90% of its data-model pull requests include a related skill change.
I love that example because Anthropic is not a careless buyer with bad data. It is one of the teams building the frontier models. And its result still decayed when the business environment changed.
Their conclusion is the right one: maintenance and validation keep the system from rotting.
Is the product an engine, or is it a paintbrush?
This is a classic data-product move.
The demo sells a powerful engine. Then something is slow, wrong, or hard to maintain, and the answer becomes: you did not model the data well. You did not name the fields correctly. You asked the wrong question. You did not prepare the semantic layer. Your warehouse is too complicated.
Some of that can be true. A data product cannot invent the company's domain truth. Good data modeling still matters. Permissions still matter. A customer still has to decide what “active,” “revenue,” or “retained” means.
But if the product only works after the customer has already solved the hard parts, be honest about what is being sold.
A paintbrush can be an excellent product. It gives a skilled person leverage. But do not sell me an engine and then explain every failure by reminding me that I am holding the brush.
The evaluation question is not “Can a talented data team make this work?” A talented data team can make almost anything work.
The question is: What work does the product remove, what work does it expose, and who owns the work that remains?
My favorite demo is still a demo account
I am not against demos. I love them. At Polyform, one of my favorite things to show is a synthetic e-commerce account built from a single prompt.
The useful part is not the first chart. It is watching the system create the pieces underneath it: model the Activities, build reusable Datasets and Metrics, add tests, connect reports, keep the work visible, and show what remains for review. I wrote up the exact boundary in the e-commerce workspace demo.
That demonstrates reach. It shows that AI can do substantial setup and construction instead of only describing what a person should build.
It still does not prove what I ultimately want you to know.
For that, I tell people the same thing: try it for a month.
Do not test AI analytics against our clean question. Test it against your messy month.
Your sources. Your history. Your strange internal language. Your conflicting definitions. Your permission boundaries. Your late files. Your new questions. Your users who do not know which table they were supposed to ask about.
A polished demo can show magic. A month shows whether the magic accumulates into a capability or dissolves into support tickets.
How I would evaluate an AI analytics product
- Use real messy data.
Include several sources, imperfect names, duplicate concepts, historical exceptions, and at least one feed that changes during the test.
- Ask who owns the failure.
When an answer is wrong or slow, does the vendor investigate and repair the path, or does every explanation end with your modeling?
- Change something on purpose.
Update a definition, remove a permission, rename a source field, or break a safe test dependency. See whether the system detects the effect and carries the repair through the rerun.
- Test scale through complexity, not row count.
One billion rows in one clean table can be easier than 200 tables with overlapping customers, changing definitions, and ten years of decisions.
- Find customers a year in.
Any system can look amazing with one or two tables. Ask what it feels like after hundreds of tables, new teams, old dashboards, migrations, and staff changes.
- Talk to the people consuming it.
The builders know every happy path. Ask the finance, sales, operations, or product users whether they rely on the result and what happens when they challenge it.
- Inspect what survives the answer.
Does the work leave behind a definition, dataset, test, evidence trail, report, workflow, decision, owner, and repair history—or only a chat transcript?
- Prefer a contract that keeps the pressure on.
If the product claims to deliver recurring value, monthly terms give the customer a clean way to judge that value without being trapped for a year.
Monthly contracts are product discipline
I do not think every annual contract is bad. Large implementations can require real commitments on both sides. Procurement and planning can also make annual terms useful.
But the default annual lock-in across software is strange when the product's value is supposed to recur every month.
We prefer monthly contracts because they force the relationship to remain honest. The customer should not stay because the demo was great eleven months ago. They should stay because the system is still useful now.
That changes what the vendor has to care about. Not only activation. Not only the first dashboard. Not only the first “wow” answer.
It has to care about the source-to-result path staying operable, the business meaning staying current, the permissions staying correct, and the failed workflow becoming repaired work. In Polyform, a dedicated expert owns maintenance and recovery inside the agreed foundation. The boundary is visible in our monthly commercial model; new expert-delivered outcomes remain separately approved rather than turning maintenance into an open-ended promise.
Reality is the benchmark
A demo can always look amazing. That is true in analytics, and it is true across AI.
You can iterate a thousand times on one workflow. You can choose the cleanest example. You can preload the right context. You can record the run that worked. You can make the model look like it understands the entire company.
Delivering the product is different.
The data becomes messier. The business adds complexity. The source schema changes. The definition that was obvious to one team is wrong for another. Permissions evolve. Workflows fail. People leave. A new executive asks the question in a way nobody anticipated.
The product is what happens next.
Does the system notice? Can it show what changed? Can it repair what it is allowed to repair? Does it know when not to guess? Can a responsible person take over with the evidence attached? Can the result be rerun, verified, and trusted again?
Do not buy the answer.
Buy the loop that keeps the answer alive.
References and further reading
- Google Cloud: Write LookML
- Microsoft Learn: Prepare a semantic model for Copilot in Power BI
- Apple Developer: Provide contextual cues to Apple Intelligence and Siri
- Anthropic: How Anthropic enables self-service data analytics with Claude
- Polyform: Self-Service Analytics Is Evolving Again
- Polyform: Make Claude Code and Codex Data Analysis Trustworthy