How to Track AI Agent Work: The Constraint-First Briefing

Six · 8 min read · May 28, 2026

How to Track AI Agent Work: The Constraint-First Briefing

Say you've handed an AI agent the vendor renewal review ahead of a budget meeting in three weeks. It has to list every contract that renews next quarter, pull what each one costs and how much it's used, and recommend which ones are worth renegotiating.

On Tuesday you check on it. The status says it's working. It has read 12 documents, made 41 tool calls and hit no errors.

All of that can be true, and you still don't know whether you'll walk into the meeting with a recommendation.

What you need to know is that the two largest contracts are PDFs in a finance folder the agent can't open. Without them, the cost comparison covers everything except the two contracts that matter most.

What the technical records show

Engineers track agents with traces, logs and metrics, and they should. OpenTelemetry's conventions for AI agent spans cover creating and invoking an agent, with fields for the error type and the tokens used, and tool calls get their own spans. The conventions are still marked Development, and the fields that would hold the agent's instructions and messages are opt-in, so by default the trace may not show what the agent was asked to do.

Records like these are where the 41 tool calls and the clean error count come from. They tell you whether the agent ran and where it failed, but not whether the renewal review is any closer to done.

The U.S. Government Accountability Office's AI accountability framework, written for federal agencies and other organizations using AI, asks them to "assess performance against defined metrics" so a system "functions as intended." In the renewal review, the defined result is a recommendation on every contract up next quarter. Until that's written down, nothing in the trace counts as progress.

So tracking agent work takes more than the trace. Engineers keep their records, someone checks them against the result, and the person who has to decide gets a short briefing that starts with what's in the way.

The Constraint-First Briefing

The briefing has six lines. The first is what's in the way. The second says what the work is for, because a blocker means nothing until you know what it's blocking. The other four follow from those two.

Here it is with the renewal review filled in.

  1. Constraint: The one thing stopping the next useful step. "The two largest contracts are in a finance folder the agent can't open." If nothing is in the way, write "nothing known." That doesn't mean the work is healthy. It means nobody has found a problem yet.
  2. Outcome: What the work should produce, for whom and by when. "A renewal recommendation for every contract up next quarter, ready for the budget meeting."
  3. Evidence: Where each claim came from and how current it is. "Costs for 18 contracts come from last month's invoice export. Usage for three tools is estimated from seat counts. Terms for the two largest contracts are missing." A link isn't proof. You still have to ask whether the source answers the question and is recent enough for the decision.
  4. Decision or help needed: The smallest thing a person has to do. "Someone with access shares the two contracts, or agrees that the review goes ahead without them."
  5. Owner: Who follows up. In this case, you.
  6. Next commitment and review condition: What happens next and what reopens the review. "When the files arrive, the agent adds both contracts and sends the full comparison. If they aren't shared by Thursday, you ask the CFO whether the review goes to the meeting with both flagged."

Three more lines sit underneath, kept apart from the six on purpose.

The first is who can authorize the decision. In the renewal review, that's the finance controller, not you.

The second is where the records are. Link the traces, the invoice export and the agent's list of sources, so anyone who doubts a cost figure can check it without asking.

The third is what's unknown: "The agent hasn't checked notice periods on any contract." Notice periods can carry deadlines, so ask the agent to check them this week. Everything else waits for one full read before the meeting: every contract's findings, not just the briefing.

Why the constraint goes first

On Tuesday you don't need the 41 tool calls. You need to know two contracts are missing, because that's the part you can still fix before the meeting. A status that leads with activity leaves you to dig that out of the log yourself.

Constraint-first isn't constraint-only, though. A blocker without the rest of the lines can send someone to fix the wrong thing. If the briefing said only "can't open the finance folder," you might ask for access to the whole folder. The evidence line shows that two files would do. The outcome line shows why those two matter more than the other 18.

That's also why the request in line four is the smallest one. "Share these two contracts" gets done in a minute. "Give the agent finance access" turns into a separate conversation that has nothing to do with the budget meeting.

Who follows up, and who can say yes

The GAO framework lists "roles, responsibilities, and delegation of authority" as things to define for an AI system. In the renewal review they belong to three different people.

You own the follow-up. The finance controller decides who sees the finance folder. And if the review has to go to the meeting without the two largest contracts, your CFO decides whether a partial recommendation is worth presenting at all.

Write those down separately. Owning a task doesn't give you permission to grant folder access, and an agent that can open a folder hasn't necessarily been cleared to use what's in it. When nobody knows who can authorize something, put that in the line: "unclear who can approve sharing these files." Then route the question to someone who can answer it. Filling the gap with whoever seems senior enough is how a review gets signed off by someone who couldn't sign off on it.

End every review with a decision

Each review should end with one decision. Most come down to one of these:

In the renewal review, the right call on Tuesday is probably to provide context. Ask the controller for the two files and let the agent carry on. If the controller says no, the call becomes a scope question for the CFO.

Whatever you choose, write down three things with it: the next action, who follows up and what reopens the review. "Ask the controller" isn't a commitment until someone owns it and you know when you'll look again.

How often you look again depends on the work. GAO says each use case needs its own monitoring frequency and, citing the research literature, that it should be "proportional to the impact of an incorrect output." NIST's report on monitoring deployed AI systems records workshop attendees asking "What is the right cadence for monitoring?" and calls best practices still nascent. It says its findings have particular relevance to frontier generative AI.

Nobody has a standard interval to hand you, and the renewal review doesn't need one. It needs a look when the files arrive or Thursday passes without them, when the notice periods come back this week and once more before the meeting.

Where a short briefing goes wrong

The first problem is filtered reporting. A 2009 study in MIS Quarterly surveyed 561 project managers about how they report project status to their executives. It found that optimistic biasing, one kind of selective reporting, "has a degrading effect on reporting quality," and that reporting quality had a positive effect on project performance. Those were people reporting on information systems projects, not agents. But an agent summarizing its own work is also choosing what goes in. The unknowns line says what it didn't check, and the full read before the meeting shows what the briefing left out.

The second is trusting the review step too much. In two experiments published in Cognitive Research: Principles and Implications, participants in a simulated decision process judged defendants, with support from what they were told was an AI system. When that support was wrong, their accuracy dropped, particularly when they saw the AI's answer before giving their own judgment. The study was about judging defendants, not reviewing agent work, and it didn't test any briefing format. It's still a reason to read the evidence line and form your own view of the costs and usage before you read which contracts the agent says to renegotiate.

The third is the line that says everything's fine. Say a contract the agent marked "no action" needs 90 days' notice to cancel, and that window closes next week. Nothing on the constraint line would catch it, because nothing is blocked. The unknowns line is what flags it, and waiting for the full read before the meeting would mean finding the deadline after it had passed.

A blank briefing to copy

Fill in each line from the work in front of you. Leave a line blank until there's something real to put in it, except the constraint, which says "nothing known" when nothing is.

Underneath:

Running the renewal review on Supanova

Say the renewal review ran on Supanova atoms instead of a single agent. Before the work starts, you choose how much your workforce checks in, from approving every step to working more independently. A review that feeds a budget decision is a good one to approve step by step, at least the first time.

Finished work comes back for your review, so you see the comparison before anyone takes it into the meeting. Ask for the six lines in the brief itself, with what's in the way first, where each cost figure came from and what you need to decide.

Next time you check on agent work, ask what's in the way before you ask what it has done. If the answer is nothing, read the evidence anyway.

Start with Supanova

Sources

human reviewAgent MonitoringAutonomous OperationsAI OperationsAI Agents