How to Define AI Quality Before the Work Starts

Six · 9 min read · July 23, 2026

How to Define AI Quality Before the Work Starts

Say you're deciding which competitors deserve a closer look. You ask for a comparison built from their own websites: what each one sells, what it charges and who it says it's for. You'll use it to pick two or three for deeper research.

The table comes back clean. Every cell is filled. There's a customer count for each company and a short recommendation at the bottom. It reads well. Then you start reviewing it, and the questions arrive one at a time. Did that customer count come from the company's own site? What was supposed to happen when a competitor doesn't publish its prices? And did you ask for a recommendation at all?

You can give the table a score now, but the score can only measure it against what you think good looks like after you've seen it. Nobody wrote down beforehand which sources counted, what a blank should look like or who makes the final call. The table got built without those answers.

So write them down first. Before the work starts, say what the comparison is for, what you'll need to see to trust it and what should happen when the evidence runs out. Then review what comes back against that record. Checks and scores still matter once the table is back, because they tell you whether it met what you wrote down.

We call that record an evidence brief. It takes six short lines.

One score hides the problems you need to see

A single quality score for the comparison would blend separate questions together. Did it stick to what the competitors' pages say, and did it cover what you needed? Those can come apart.

Microsoft draws that line in the RAG evaluator documentation for its Foundry platform. Groundedness, in its terms, is about precision: the response doesn't contain content outside the material it was given. Response completeness, which Microsoft lists as a preview evaluator, is about recall: the response doesn't miss critical information compared with the expected answer.

In the comparison, one draft might stick to the competitors' pages and leave out the pricing tier your customers actually buy. Another might cover every tier and slip in a customer count that isn't on any page it cites. Blended into one number, the two could land close together, and they need different fixes.

Microsoft built those evaluators for retrieval systems, but the split works on one table too. Ask whether every claim is on a cited page, and separately whether the table covers the prices and tiers you needed. A summary score can still help you sort several drafts, as long as those two answers stay visible and a failure that should sink the work on its own, like an invented number, can't be averaged away.

What goes in an evidence brief

Writing down what AI is for before it runs isn't a new idea. NIST's AI Risk Management Framework (version 1.0, from 2023) asks organizations to record what an AI system is intended for, which tasks it supports and how people may use and oversee what it produces. Those actions are written for whole systems, and NIST says they aren't a checklist. An evidence brief brings the same habit down to one assignment. For the comparison, that's six things to settle before anyone opens a competitor's website.

What it's for

Name what should come back, who will use it and what decision it feeds. For the comparison, that's a table with source notes that you'll use to choose which competitors get deeper research. The table informs that choice. It doesn't make it.

Ask: What should come back, who uses it, and what will it inform?

What you need to see

This is the evidence a reviewer should be able to check without redoing the work: where each claim came from and when, how a figure was worked out, which claims couldn't be supported. For the comparison, every price and every customer count should point to a dated page on the competitor's own site. A citation next to a claim doesn't make the claim true, and more sources aren't automatically better ones, so name what matters for this use.

Ask: What support has to sit next to each claim that matters?

Where the work stops

These are the boundaries that would make an otherwise good result unusable: which sources are allowed, what dates count, what the work must not claim or do. For the comparison, that's the competitors' own websites and no guesses about roadmaps, private features or how well a company is doing. Only write a line here if it changes the work or the review.

Ask: Which boundary, if crossed, would make a good-looking table unusable?

How to handle what isn't known

Say what can be estimated, what has to be labeled, what should be left open and what must be checked before it goes in. For the comparison, "not publicly listed" is an acceptable answer for a competitor's price. A brief that rewards a complete table and never allows a blank pushes the work toward filling cells it can't support. And a competitor that doesn't list a feature hasn't told you it lacks one, so the brief should keep those two apart.

Ask: Which gaps can be estimated, which labeled, which left open, and which checked first?

When to stop and ask

Name the situations that should pause the work and bring it back to a person. In the comparison, that might be two of a competitor's own pages contradicting each other, a question the allowed sources can't answer, or a point where finishing the table would mean making the choice the brief kept for you. A comparison that comes back with the conflict flagged has done what you asked.

Ask: What specific situation means stop and ask rather than keep going?

Who checks it and who decides

Name who looks at the evidence and who can accept the result. They can be different people. Someone on your team checks the source notes, and you decide which competitors move forward. For low-stakes work, an agreed rule can do the accepting. A name on the brief doesn't mean anyone looked carefully, so the person checking needs the evidence in front of them and the time to read it.

Ask: Who inspects the evidence, and who can accept the result?

What a check can tell you

Once the comparison comes back, part of the brief can be checked mechanically. Does every row have a source link? Is every page dated? Does any cell make a kind of claim the brief ruled out? A program can answer those every time. It can confirm a link is there, but not that the page says what the table claims.

Apple's guidance for developers evaluating AI features divides the checking three ways. Code takes anything with a computable answer, like a word limit, a schema or a numeric range. A model judge takes what code can't, like whether a summary gets the important points. People, Apple says, are too slow and expensive for routine evaluation, so it keeps them for high-stakes work, for calibrating the model judges and for discovering gaps in what the scoring looks at. Apple is writing about features tested across many outputs. Your comparison is one document you'll read yourself, but the split still shows which parts a program can check and which need a person reading the page.

Say you ask a model to pick the better of two drafts of the table. In an exploratory study of 15 LLM judges across 22 tasks, presented at IJCNLP-AACL 2025, the judges' preferences shifted with the order the answers appeared in. The authors found the bias wasn't random chance. It varied significantly across judges and tasks, and the quality gap between the answers strongly affected it. So a judge's pick between your drafts is one input, and the brief should say who weighs it.

The UK government's AI Playbook, written for public-sector teams, is plain about who decides: responsibility for any output or decision made or supported by an AI system always rests with the public organisation. Your company isn't bound by it, but the comparison works the same way. A check can tell you whether a price has a source. It doesn't get to pick which competitors move forward. You do, and the brief says so before the work starts.

When you only find out what you wanted after you see it

The fair objection is that sometimes you can't know what good looks like until you see a draft. You open the comparison and realize you care whether each competitor offers a free tier, which you never mentioned.

That happens to experts too. In one study by Szymanski and colleagues, 10 domain experts and 8 lay users wrote criteria for judging AI answers about nutrition and math teaching. It was a small study in two domains, and the authors say the staged review process they propose has yet to be tested in practical settings. Like you with the free tier, the participants added to or refined their criteria after reviewing model outputs. The experts generally wrote more than the lay users, especially after seeing outputs, but both groups rarely changed what they'd first written and built new criteria within the same themes instead.

Treat the brief the same way. Write it first, and when a draft teaches you something, add it as a visible revision: "free tier, added after the first draft." The new line guides the next pass. It didn't guide the first one, so don't judge the first table as if it had.

The brief, filled in

Here's the blank form. If a line wouldn't change what comes back, how it's checked or whether it gets accepted, cut it.

What it's for: Create [the thing] for [who] to inform [the decision it doesn't make].

What you need to see: Make [claims, sources, calculations or expert checks] easy to inspect.

Where the work stops: Stay within [scope and sources]. Don't [claim, guess, act or disclose].

What isn't known: [Estimate, range, label, leave open or verify] each kind of gap.

When to stop and ask: Stop and ask when [conflict, missing input, permission or a decision outside scope] comes up.

Who checks and who decides: [Role] checks the evidence. [Role or agreed rule] accepts the result.

And here's the brief for the competitor comparison.

What it's for: A comparison table with source notes, so you can choose which competitors get deeper research. The table doesn't make that choice.

What you need to see: Every claim that matters links to a dated page on the competitor's own site. Any figure shows how it was worked out. Claims that couldn't be supported are labeled or left out.

Where the work stops: Competitors' own websites only. No guesses about roadmaps, unpublished customer counts, private features, intentions or market strength. No recommendation.

What isn't known: Write "not publicly listed" when the pages don't support a claim. Keep "we found no public evidence" separate from "it doesn't exist."

When to stop and ask: Stop and ask when a competitor's pages contradict each other, the allowed sources can't answer a question the table needs, or finishing would mean choosing for you.

Who checks and who decides: Someone on your team checks each claim against its source note. You decide which competitors move forward.

Keep the lines that hold up

After a few comparisons, some lines show up every time: own-site sources only, "not publicly listed" allowed. Keep those as a starting point for the next brief, with a note on where each one came from. A line that keeps coming back still isn't right for every job. A source rule for competitor research may be wrong for a summary of customer interviews, and one person's preference isn't automatically company policy.

Running the comparison on Supanova

Say the comparison ran on Supanova atoms. You'd hand them the six lines with the assignment, so the work starts from the same standard you'll use at the end. Finished work comes back for your review. That's when you open the brief next to the table and check it line by line: does each price point to a dated page, does "not publicly listed" show up where the sites are silent, and did the atoms leave the choice of competitors to you?

If the table misses a line, you can say exactly which one in the next assignment. If the first draft taught you something about free tiers, that goes into the next brief as a new line, marked as added.

Before the next piece of work you hand off, try writing the six lines. If you can't say what it's for or who decides, settle that before anyone starts.

Start with Supanova

Sources

AI Delegationhuman reviewAI evaluationAI QualityAcceptance Criteria