What Data Work Should Never Be Fully Automated?

Data work is not SQL work.

SQL is the part we can point at. There is a table. Someone writes a query. A number comes out. Now AI can write that query in seconds, so it is tempting to believe the rest of the data job is about to disappear.

It is not. The query is an implementation of decisions the business has already made—or decisions the query author is quietly making on its behalf.

The work that should never be fully automated is the work where the company has to decide what something means, who may do it, whether the consequence is acceptable, and who owns recovery when the answer is wrong. AI should help with all of that work. It should not become the unnamed authority behind it.

The false shortcut

SQL can be correct while the data work is wrong

Imagine a table called sales. If you know nothing about how the company works, the request sounds easy: sum the amount column and group it by month.

But when does a sale happen? When a contract is signed? When an order is placed? When the first payment succeeds? Does a renewal count? What about a refund, a failed payment, a cancellation, a backdated contract, or a CRM migration that copied every old opportunity into a new system?

Each answer can produce valid SQL. Each answer can also produce a different number.

The PostgreSQL documentation describes the mechanics of queries through table expressions, filters, grouping, window functions, and result selection. Those are powerful operations. They do not decide which real-world event the company wants the result to represent.

Data work begins by deciding what exists

Before you can measure a business, you have to decide what its building blocks are.

A customer is not necessarily a row in the CRM. A sale is not necessarily a payment. A product user may be a person, an account, a household, a location, or a team. The source systems record operational events. Data work decides how those events become a durable model of the business.

That model also has to survive time. If you changed CRMs, how do you know that the same opportunity did not become two opportunities? If finance changed the treatment of refunds in July, should the old board report be restated or preserved? If an account merged with another account, which history follows it?

Stitching tables is the technical part. Deciding what the stitched history means is the business part.

Taste enters before the query and after the result

There is another part of the job that is easy to miss: deciding what is worth measuring.

A request arrives as “show me sales and users.” A good data person asks what changed, what decision is coming, which behavior could plausibly explain it, and what the next question will be if the first hypothesis is true. They have seen similar businesses. They recognize a pattern. They know when a clean chart is hiding a bad denominator.

That is taste.

Taste is not mystical, and it is not a license to ignore evidence. It is compressed experience: knowing which distinctions matter, which anomalies deserve skepticism, which definitions will break later, and which result is technically interesting but operationally useless.

AI can improve this process. It can surface candidate hypotheses, find comparable patterns, inspect edge cases, and argue against the first explanation. But somebody still has to decide which question deserves the company's attention and which standard the answer has to meet.

Automate execution as far as it can safely go

I am not making an argument for keeping people busy with work a machine can do.

AI should write and refactor SQL. It should inspect schemas, profile tables, suggest joins, generate tests, trace lineage, compare periods, find anomalies, document models, build draft reports, rerun jobs, and monitor recurring work. When the definition and permissions are settled, many of those tasks should run without a person touching every step.

The right boundary is not “AI does nothing important.” The right boundary is that authority does not disappear just because execution becomes cheap.

Five decisions should always have a human owner

“Never fully automated” does not mean the AI must be excluded. It means the AI cannot be the final, unaccountable authority.

1. Financial and operating definitions

AI can compare existing calculations, show examples, identify contradictions, and propose a definition. A person with the relevant business authority must decide what revenue, margin, retention, activation, or customer health means for this use.

2. Permissions

AI can recommend the minimum access needed for a task and explain why. It should not grant itself a wider scope, cross a company or regional boundary, or decide that sensitive data is acceptable because it makes the analysis easier. An authorized person owns the grant.

3. Customer-impacting actions

A model can identify accounts that appear at risk. That does not mean it should silently change prices, issue refunds, deny service, close accounts, or message customers. The more material the consequence, the clearer the approval and appeal path must be.

4. Irreversible or expensive changes

AI can prepare the migration, produce the diff, simulate the downstream impact, write the rollback plan, and run the checks. A named owner should authorize destructive data changes, production writes, or decisions that are difficult to undo.

5. Unresolved ambiguity

When two definitions are plausible, the source history is incomplete, or the evidence cannot distinguish between explanations, the correct behavior is not to choose quietly. It is to stop, show the uncertainty, and route the decision to somebody who can resolve it.

“Human in the loop” is not enough

A review button does not create accountability. A person who cannot see the evidence, change the decision, or repair the outcome is not meaningfully in the loop.

The person has to be named before the consequential step. They need the authority to say no, the context to understand the tradeoff, the evidence to inspect what happened, and the responsibility to recover when the result fails.

This is close to the distinction made in the NIST AI Risk Management Framework. NIST calls for organizations to define roles for human-AI oversight, document how outputs will be used, test systems in context, monitor them in production, and assign responsibility for disengaging systems whose behavior does not match their intended use. The OECD's accountability principle similarly connects accountability to traceability across datasets, processes, and decisions.

Those frameworks are broader than analytics, but the operating idea is the same: oversight must be designed into the work. It cannot be an anonymous person asked to glance at the final number after every important choice has already been hidden.

The quality problem is not solved by better SQL generation

I have seen AI generate SQL that runs and still produces terrible analytics. I have also seen people do the same thing.

I do not need a theory about the model's training data to explain the failure. There is an enormous amount of analytics work that was never reviewed against the business decision, never tested with real edge cases, and never maintained after the source systems changed. Copying that style faster does not improve the standard.

Software engineering has visible review conventions: tests, pull requests, interfaces, performance budgets, and open-source code that many people can inspect. Analytics often has weaker feedback. The query runs. The dashboard loads. The number looks believable. The business moves on.

That is why I would not hire someone merely because they know how to write SQL. SQL is learnable, and increasingly automatable. I would look for someone who understands analytics: someone who can frame the question, design the definition, inspect the evidence, anticipate the next question, and take responsibility for what the company does with the result.

Why Polyform still puts a person on the work

At Polyform, we have spent years making more data execution reusable. Our Activity Schema approach gives the work a consistent grammar. The platform connects data, definitions, reports, investigations, tasks, workflows, and operating history. AI can do a surprising amount across that system.

We still put a dedicated expert on the relationship.

That person meets with the company, learns how the business works, maintains the context, reviews consequential work, and steps in when the software should not guess. The decisions and evidence stay in the system, so accountability does not collapse into one consultant's private memory. The dedicated expert model exists because execution can be automated while ownership cannot.

The customer also remains part of the boundary. The company supplies domain truth, grants access, sets priorities, and approves decisions that belong to it. Accountability is a relationship between people with authority, not a feature we switch on inside the model.

Do not ask only whether the AI can write the query

Ask who decided what the number means.

Ask who may use the underlying data. Ask who approves the action. Ask what happens if the source changes, the definition becomes disputed, or the result hurts a customer. Ask who has both the evidence and the authority to recover.

Then automate everything else you safely can.

SQL can be automated. Data work can be accelerated. Accountability has to belong to someone.

Automate the execution. Keep the accountability.

References and further reading

• PostgreSQL: Queries
• NIST: AI Risk Management Framework Core
• OECD AI Principle: Accountability
• Polyform: Business context for AI