Where AI stops
AI can answer. Then the work comes back to you.
Claude, ChatGPT and coding agents are genuinely good at answering. There are five points where serious business work stops and a person picks it up again.
One business question
work returns to a person
“Why is this number wrong?”
AI answers
and then?
Five places it stops
Pick the sentence you recognise
Each stop has the same shape.
Where it breaks
→
What teams try
→
What closes it
STOP 01
It doesn’t know what our business means.
Takeaway
Meaning has a structure. Documents don’t carry it.
“When I say total sales, it does not know whether I mean a signed contract, an order, or a payment.”What teams tell us
Where it breaks
Business question
Retrieved paragraph
✕
No Metric, Dataset, Activity or lineage
the AI reconstructs meaning per question
A believable wrong answer
Three definitions of revenue
Every tool has its own “active customer”
The definition changed last month
What teams try today
Total sales means signed contracts, not orders…
Understood. Recalculating.
Also, retention is 90 days here…
Paste the definitions into every chat
✕Gone at the next conversation.
metrics-glossary.pdf
Q3 planning notes.docx
data-dictionary-v4.xlsx
A knowledge base or RAG index
✕More documents is not analytical structure.
metrics:
- name: total_sales
agg: sum
# journeys, history, reports: elsewhere
A semantic layer plus MCP servers
✕Defines the metric, not the system around it.
What closes it
One hierarchy every answer is built from.
METRICTotal sales
DATASETPopulation, grain, filters
ACTIVITYSigned Contract
TRANSFORMOne focused SQL definition
WAREHOUSEYour source data
Metric · Total sales
Validated
Built fromSigned Contract activity
Reused in4 Reports · 2 Workflows
Last change14 Aug · reason recorded
The same Metric powers the dashboard, the investigation and the Workflow. Your dedicated expert keeps it current.
STOP 02
I can’t defend the answer.
Takeaway
Proof should arrive with the answer, not after the challenge.
“The AI says retention is 42 percent, but finance has 37. I cannot tell which one is right.”What teams tell us
Where it breaks
42%the answer
SELECT count(distinct c.id) …LEFT JOIN orders o ON …
✕
SQL is not a human explanation
which customers? which window? how fresh?
Paralysis, or false confidence
A citation shows what was retrieved, not whether it was right
The proof gets built only after someone objects
What teams try today
DATA-2291 · Verify retentionOpen
DATA-2284 · Rebuild logicQueued
Send the query to an analyst
✕The queue is the thing AI was meant to shorten.
42%
AI
37%
Finance
?
Board
Compare against a trusted dashboard
✕Two opaque numbers, no way to reconcile them.
not_null · orders.customer_id
unique · customers.id
nothing tests the business meaning
Tests, lineage and freshness monitors
✕Technical checks a business reader can’t read.
What closes it
The number opens into four things a person can read.
42%Retention · last quarter
1What the Metric means here
2Which customers were counted
3What happened to one of them
4How current the data is
Dataset · Retained customers
1,284 rows
Harbourline Co.Order · 12 Julcounted
Meridian FoodsOrder · 03 Augcounted
Castlegate Ltdno qualifying orderexcluded
FRESHNESS
Orders current to 06:10 today
DIFFERENCE FOUND
Finance used a 6-month window
STOP 03
I keep passing context between tools.
Takeaway
You should supervise the outcome, not transport the state.
“Every tool is helpful, but I am the one keeping the whole job in my head.”What teams tell us
Where it breaks
DASHBOARD AI
Finds the drop
can’t reach the definition
CODING AI
Changes the model
can’t see the dashboard
AUTOMATION TOOL
Schedules the alert
no business context
NOBODY
Checks the whole result
The integration and supervision layer is you.
What teams try today
WarehouseMCP
BI toolMCP
metadata only✕
Connect everything through MCP
✕Actions, without a maintained map of how objects relate.
!
Chain the handoffs in n8n or Zapier
✕Automates the steps, not the judgement between them.
screenshot.png
error.txt · “column not found”
definition (again)
Carry it yourself, tool to tool
✕Your clipboard becomes the memory system.
What closes it
One conversation supervising the whole job.
ONE ASK, IN POLY
“Add time from demo to signed contract to the weekly dashboard, split by lead source, and alert us when it grows.”
Check existing Activitiesdone
Build the Dataset and Metricdone
Add to Report, inspect it visuallyrunning
Create and test the alert Workflowqueued
Deep Work · 3 streamssupervised by Poly
Model the qualifying-demo Activity12m
Extend the pipeline Report4m
Alert threshold needs a business callTask
When the AI is genuinely stuck, the gathered context becomes a Task for your dedicated expert — not a dead end.
STOP 04
It finds the problem. Nobody owns the fix.
Takeaway
A diagnosis is not a repaired result.
“The AI explained the error, but the production fix still came back to us.”What teams tell us
Where it breaks
Dashboard is wrong
AI explains a likely cause
✕
No change rights, no rerun, no owner
the investigation context is dropped here
Another ticket, still wrong
The job says success; the Salesforce field is still empty
Or nothing broke — the definition changed
What teams try today
ERROR: relation "stg_orders_v2" does not exist
> here is a corrected query…
Paste the error into a coding assistant
✕A valid query is not a repaired business result.
BI logsok
Orchestratorsuccess
CRM fieldempty
Read four systems’ logs separately
✕Every handoff is coordinated by the customer.
Assigned · the one person who knows
Status · on leave until Monday
Open a ticket for the expert who knows
✕The bad number sits there until they are free.
What closes it
A loop that ends at the result, with a name on it.
DETECTThe Report looks wrong
TRACEMetric, Dataset, run, definition history
DECIDEFailed process, changed rule, or new request
REPAIRChange the permitted work and rerun
VALIDATECheck the original Report again
Workflow run · Sync to SalesforceFailed node
Read Dataset · 812 rows
Map fields
Write account_tier · 403retry
WHEN JUDGEMENT IS NEEDED
Task → your dedicated expert, with the investigation attached. They own it until the Report is useful again.
STOP 05
The useful work disappears into chat.
Takeaway
A good answer is not always the final product.
“That analysis was useful. Now I need to rerun it next week — and the team needs a dashboard, not my chat.”What teams tell us
Where it breaks
A genuinely good analysis
Pasted into a slide or sheet
✕
Cut off from data, definitions and freshness
next month it has to be rebuilt from scratch
A saved conversation is not a reusable interface
Nobody can filter it, drill it, or rerun it
What teams try today
Copy it into a spreadsheet or deck
✕A snapshot, stale the day after.
Rebuild in the BI tool3 days
Filters and assumptions drift
Ask an analyst to rebuild it as a dashboard
✕The work is rebuilt, and the logic diverges.
at_risk_accounts_final.csv
at_risk_accounts_final_v2.csv
Export the list into another tool
✕An export is not an operating process.
What closes it
One analysis, four surfaces, one foundation.
REPORT
A dashboard the team watches daily
DATASET
A Cohort others can filter and drill
PRESENTATION
A board story that can be refreshed
WORKFLOW
A recurring update, or an action
ONE FOUNDATION
Same Cohort, Metrics, Activities and current data
Report · Feature adoption & retentionlive
Recurring delivery
Monday 08:00Slack · growth team
Reply “why?”journeys behind the change
Five stops, three causes
Dependable AI needs a system around the model.
01
A complete system
Reach the work, keep it, prove it, repair it.
Stop 02
Stop 03
Stop 05
02
Maintained business context
The approved meaning, permissions and history — kept current.
Stop 01
Stop 02
03
Accountable expert judgment
A named person who finishes blocked and consequential work.
Stop 03
Stop 04
Start with one problem
Bring us the stop you keep hitting.
One definition everyone reuses. One number you can defend. One broken path repaired. That is a good first result.