How to Measure Whether an AI Agent Completed the Business Job

Do not measure whether an AI agent completed a business job by asking the agent whether it is done.

Define the job, evidence, acceptance tests, failure conditions, escalation path, and responsible owner before the work starts. Let the agent do the work. Then have a separate supervisor review the resulting files, records, system state, or customer outcome against that definition.

For important work, I would measure six things:

1. Did the required output or action actually exist?
2. Did it pass the business and technical acceptance tests?
3. Can a reviewer inspect the evidence behind it?
4. Did it avoid disqualifying safety, permission, or quality failures?
5. Did the downstream business outcome occur?
6. What did an accepted outcome cost in time, AI usage, retries, and human review?

The first four tell you whether the agent completed the job. The fifth tells you whether the job was worth doing. The sixth tells you whether the system is economical.

Those are related questions, but they are not the same question.

There are also two levels of proof. At the product level, an evaluation suite asks whether the agent can complete this class of job reliably across representative cases. At the user level, the review experience asks whether this specific job was completed correctly today. A production agent needs both. A benchmark cannot accept today's customer refund, and one successful refund does not prove the system is reliable.

A plausible output is not proof of completion

In a normal chat, the user is often the evaluator.

You ask ChatGPT to write an email, summarize a document, or make a table. The response appears in front of you. You read it, decide whether it is good enough, and either use it or ask for a revision.

The review step is built into the experience because the output and the user occupy the same conversation.

Agents change that relationship.

An agent may work for minutes or hours. It may use an MCP server, operate software, query a database, change files, create records, or delegate parts of the job to other agents in parallel. The user no longer sees every important intermediate result. At the end, the agent says the task is complete.

That message is useful, but it is not evidence.

Imagine a flight-booking agent that says, “Your flight has been booked.” The business outcome is not the sentence. The outcome is a valid reservation in the booking system with the correct passenger, route, date, price, and payment state. Anthropic uses essentially this distinction in its guide to evaluating AI agents: the transcript may say the flight was booked, while the outcome must be verified in the environment.

The same principle applies to business work:

• “The deck is finished” is not the same as a correct, reviewable deck built from current data.
• “The customer was refunded” is not the same as a valid refund in the payment system and a correctly updated account.
• “The CRM is clean” is not the same as duplicate records being resolved without losing activity history.
• “The report was repaired” is not the same as the underlying logic being corrected and every affected output being revalidated.

An MCP server can confirm that a tool call was accepted or that a record was returned. That still may not prove the larger job was completed. The record could belong to the wrong customer, the requested change could require a second system, or the successful action could have created a downstream inconsistency. MCP provides connectivity; the outcome contract defines what the connected work must accomplish.

The agent's completion message is a claim. The business needs a verification path.

Use a supervisor, not only a worker

The pattern I recommend is a worker and a supervisor.

The worker agent executes the job. The supervisor holds the original request, the success criteria, and the responsibility for reviewing what comes back. It inspects the actual result, identifies gaps, requests more work when necessary, and decides whether the outcome is ready for the user.

“Supervisor” is the plain-language term I use. Anthropic describes a related multi-agent design as an orchestrator-worker pattern: a lead agent plans the work, delegates bounded tasks to subagents, evaluates what they return, and synthesizes the result. The important point is not the label. It is that the agent doing the work should not be the only authority declaring that the work is correct.

This matters even when there is only one worker. In its work on long-running application-development agents, Anthropic observed that agents tended to praise work they had produced even when human reviewers found the quality mediocre, and found that separating generation from evaluation was a useful lever. A worker may have missed a source, misunderstood an instruction, accepted a partial result, or verified the artifact it intended to create instead of the artifact it actually created.

The supervisor should therefore review independent evidence:

• the files that now exist;
• the records that changed;
• the tests that ran;
• the source data and definitions used;
• the rendered output a person will see;
• the downstream system state;
• and any unresolved ambiguity or failure.

It should not merely summarize the worker's summary.

Write the test before the agent starts

The simplest way to improve agent reliability is to define “done” before the work begins.

For a one-off task in ChatGPT, Claude, Codex, or another agent, ask the agent to write a short plan that answers:

• What business job are we trying to complete?
• What will exist or be different when the job is done?
• What evidence will prove it?
• What tests should pass?
• What would make the result unacceptable even if the output looks good?
• When should the agent stop and ask for help?

Save that agreement in a file the agent cannot conveniently forget as the conversation grows. Then, when the agent finishes, instruct the reviewing agent to reopen the file, inspect the real result, run the tests, and report each condition as passed, failed, or not proven.

This is a lightweight version of the system we use at Polyform. It turns an informal request into a small outcome contract.

It also prevents a subtle failure: redefining success after seeing what the agent happened to produce. If the acceptance criteria are written only at the end, it is easy for both the user and the AI to rationalize a partial result as success.

The AI outcome contract

An outcome contract is the record the worker, supervisor, and human owner use to agree on the job.

It does not need to be complicated. For most business workflows, it should contain these fields:

Field	Question it answers
Business job	What useful job is the agent responsible for completing?
Recipient or decision	Who will use the result, and for what?
Required output or state	What must exist or change?
Objective evidence	Which files, records, queries, or system states can prove completion?
Acceptance tests	What must be true for the result to be accepted?
Failure conditions	What automatically disqualifies the result?
Escalation	When must the agent stop, and who decides what happens next?
Downstream outcome	What business result should follow if the job is useful?
Cost	What time, AI usage, retries, and human review did acceptance require?
Owner	Who is accountable for accepting the result and repairing the process?

Here is a compact machine-readable example for a monthly leadership deck:

{
  "business_job": "Produce the July operating deck for the leadership meeting",
  "recipient_or_decision": "Leadership uses it to review performance and choose August priorities",
  "required_output": {
    "type": "presentation",
    "location": "approved leadership reporting folder",
    "period": "2026-07"
  },
  "objective_evidence": [
    "rendered slide images",
    "source dataset and freshness timestamp",
    "metric definitions and filters",
    "reconciliation report",
    "links to the records behind consequential totals"
  ],
  "acceptance_tests": [
    "all required sections are present",
    "metrics reconcile to approved sources within stated tolerances",
    "every comparison uses the intended period and population",
    "charts and labels are legible in the rendered deck",
    "findings are separated from hypotheses",
    "a named reviewer accepts the final deck"
  ],
  "failure_conditions": [
    "a metric definition is guessed",
    "a required source is stale or incomplete",
    "confidential data is exposed to an unauthorized recipient",
    "a consequential total cannot be traced to underlying records"
  ],
  "escalation": {
    "owner": "reporting owner",
    "trigger": "definition conflict, missing source, failed reconciliation, or unsupported conclusion"
  },
  "downstream_outcome": "leadership can make the intended decisions without a second manual rebuild",
  "cost_measure": "AI and tool cost plus retries, elapsed time, and reviewer minutes",
  "accountable_owner": "head of operations"
}

The exact format matters less than the separation of concerns. A polished deck can pass the output check and fail the metric check. A correct deck can pass the completion check and still fail to influence a useful decision. A technically successful workflow can become uneconomical if every run requires hours of expert correction.

One score hides those differences. The contract makes them visible.

For data work, replay a result you already know

Data work creates a special problem: a result can look completely reasonable while using the wrong definition, grain, population, identity rule, or time window.

Before trusting an agent with next month's analysis, give it a month whose answer you already know.

Suppose your team has an approved June operating deck and needs the agent to produce July's version. Do not begin by handing it July's data and hoping the slides look right. Start with a historical replay:

1. Give the agent the June source data, approved metric definitions, and the final June deck.
2. Ask it to reconstruct the deck from the underlying data without copying the displayed totals.
3. Compare its metrics, populations, charts, annotations, and source records with the approved version.
4. Investigate every meaningful difference. Determine whether the agent is wrong, the old deck is wrong, or the definition is ambiguous.
5. Save the accepted logic, tolerances, exceptions, and review decisions in the outcome contract.
6. Only then ask the agent to apply the tested process to July.

I think of this as a defined experiment. In engineering terms, it is close to a historical replay or backtest: the agent has to reproduce a known result before it earns the right to operate on an unknown one.

This does more than check arithmetic. It tests whether the agent understands what each metric means.

For example, the June deck may say there were 412 active customers. Reproducing 412 requires the agent to understand:

• what counts as a customer;
• what “active” means;
• which date establishes the reporting period;
• how merged, paused, refunded, or test accounts are treated;
• whether late-arriving data changes the total;
• and which 412 records belong in the population.

If the agent produces 409, that gap is valuable. The goal is not to force it to match the reference at any cost. The goal is to explain the three-record difference and resolve the business meaning before the process runs in production.

The replay becomes your first evaluation set. Every meaningful mistake becomes a regression case the system should continue to pass.

Grade the outcome without over-prescribing the path

Agents can reach the same valid outcome through different steps. One may use three queries; another may use ten. One may build a temporary validation file; another may inspect the records directly.

Do not make the evaluation so rigid that it fails a good result merely because the agent took an unexpected path.

Anthropic recommends combining code-based, model-based, and human graders, choosing deterministic checks where possible and using model or human judgment where nuance requires it. Its guidance also recommends grading what the agent produced rather than prescribing every tool call when multiple valid approaches exist.

For a business workflow, that can look like this:

Deterministic checks

• Did the expected file or record exist?
• Did required tests pass?
• Did totals reconcile within the approved tolerance?
• Were required fields populated?
• Was the action performed exactly once?
• Did cost, latency, or retry count stay within the allowed range?

Model-based checks

• Does the narrative accurately reflect the evidence?
• Are findings distinguished from hypotheses?
• Does the artifact follow the requested structure and audience?
• Are material caveats visible rather than buried?

Human checks

• Is the business interpretation correct?
• Is the result good enough to use in the real decision?
• Is the action appropriate given the customer relationship or operating context?
• Should an ambiguity become a new rule, definition, or test?

The higher the consequence and the harder the action is to reverse, the more important the human review boundary becomes.

Measure completion, impact, and efficiency separately

Teams often jump from an agent's technical metrics directly to ROI.

They monitor tokens, latency, tool errors, model scores, or trace quality. Those measurements are useful for operating the agent. They still do not prove that the agent completed the business job.

Other teams jump to the far end and ask whether revenue increased. That matters, but it may be too delayed or influenced by too many other factors to diagnose whether the agent itself worked.

A better measurement chain has three layers.

1. Completion

Did the agent create the required artifact or state, pass the acceptance tests, expose the evidence, and avoid disqualifying failures?

Useful measures include accepted completion rate, failure rate by condition, escalation rate, correction rate, and time to acceptance.

2. Business impact

Did the accepted result create the intended downstream value?

Depending on the workflow, that may mean a case was resolved, a qualified meeting was booked, a report supported a decision, an invoice was collected, a defect was repaired, or a customer received the correct service.

Measure this separately so a perfectly executed but low-value job does not look like a successful AI investment.

3. Efficiency

What was the full cost of an accepted outcome?

Include model and tool usage, failed attempts, elapsed time, human review, corrections, and recovery—not only token price. OpenAI similarly recommends measuring the full cost per accepted outcome rather than treating the cheapest model or lowest token cost as the goal.

This structure lets you diagnose the system. Low completion suggests an agent, context, tool, or evaluation problem. High completion with low impact suggests the company automated the wrong job. Good impact with excessive review cost suggests the workflow works but is not yet economical.

How we use this pattern at Polyform

At Polyform, Poly is the conversation and control layer. It holds the user's question, the relevant context, the progress of the work, and the responsibility for returning an understandable result.

When the job requires substantial execution, Poly coordinates DeepWork. A DeepWork task gives a coding agent a cloud computer where it can use Codex, write and run code, inspect files, query data, work through the Polyform command line, use connected tools, and persist the artifacts required for review. Multiple DeepWork tasks can investigate separate parts of a problem in parallel.

The important part is what happens when the work comes back.

Poly does not need to accept a worker's “done” message as the final answer. It can compare the returned work with the original goal, inspect the supporting evidence, reconcile results from parallel tasks, request additional work, and bring the user the completed outcome or the exact issue that still needs judgment.

The product architecture helps, but software alone is not enough for consequential data work. A dedicated Polyform expert maintains the business context, resolves definitions, reviews important changes, and owns recovery when the AI cannot safely finish the job.

The complete pattern is:

business job
→ outcome contract
→ DeepWork execution
→ independent review against evidence
→ human judgment when required
→ accepted result
→ downstream outcome
→ new tests and maintained context

That is how agent work improves instead of resetting with every conversation.

I am biased because we built Polyform around this pattern. A technically capable team can assemble it with coding agents, MCP servers, an evaluation platform, its own data environment, and internal reviewers. When the entire job lives inside one well-instrumented product, a specialist observability or evaluation tool may be enough. And for low-consequence one-off work, a person reading the output may be the only supervisor required.

Polyform is most useful when proving completion requires connecting agent behavior to company-specific definitions, customer context, operational systems, financial or product outcomes, recurring reporting, permissions, and an accountable recovery path.

The practical workflow

If you are putting an agent into production, start with one real job:

1. Name the business job. Describe the useful result, not the tool activity.
2. Define the resulting state. Say what must exist or change outside the conversation.
3. Write the tests first. Record acceptance criteria, failure conditions, evidence, and escalation.
4. Create a historical replay. Use a prior case with a known, reviewed outcome.
5. Let the worker execute. Permit flexible methods inside the allowed boundaries.
6. Have a supervisor inspect the result. Review the environment and evidence, not only the transcript.
7. Require human judgment where consequence or ambiguity demands it. Make the owner explicit.
8. Measure the downstream outcome. Confirm that accepted work was useful.
9. Measure full cost per accepted outcome. Include retries, correction, and review.
10. Turn every important failure into a test. Let the evaluation system accumulate the business's operating knowledge.

The goal is not to prove that the agent is generally intelligent.

The goal is to prove that, for this business job, in this environment, under these conditions, the system produced an acceptable result—and that someone is accountable when it did not.

That is the difference between an agent that says it finished and a company that knows the job was done.

Polyform’s AI-first operating model keeps substantial execution, review, evidence, and escalation connected to the business result rather than ending at the agent transcript.

References and further reading

• evaluating AI agents
• orchestrator-worker pattern
• long-running application-development agents
• full cost per accepted outcome