How to Give AI Better Feedback: Write a Change Order

Six · 9 min read · September 8, 2026

How to Give AI Better Feedback: Write a Change Order

Say you asked an AI tool to draft a one-page pricing comparison for Thursday's leadership meeting: your plan against your closest competitor's. It comes back looking finished. There's a short summary, a feature table, the two prices side by side and a recommendation to cut your price.

You check the feature table line by line, and it's right. Then you look at the prices. The competitor's $40 a seat is what it charges when a customer pays for the year up front. Its month-to-month price is $48. Your $50 is monthly. The page is comparing two different offers, and the recommendation to cut your price rests on a $10 gap that is really $2.

So you type what most people type: "The pricing is wrong. Fix it."

Picture the version that comes back. The competitor's price now reads $48. The feature table you already checked has been reworded, and the recommendation still says to cut your price. You'll have to read the whole page again, and Thursday is getting closer.

What "fix it" leaves out

"Fix it" tells the AI the work missed. It doesn't say where the miss is, why it matters, what the right answer is based on, what should stay the same or how you'll know the problem is gone. The model has to guess, and every guess is a chance to change something you'd already checked.

Researchers who study AI editing have a word for what happened to the feature table: over-editing, changing more than you asked for. The HyperEdit paper, in Findings of ACL 2026, says existing editing approaches "either over-edit or fail to apply modifications consistently." It says an editing system has to get two things right, and one is keeping the edit local. For you, that means fixing the price and leaving the table nobody mentioned as it was.

A thumbs-down has the same gap. Tap it on the pricing page and all it says is that you didn't like the result. It says nothing about the $40 being an annual price, or about the feature table being fine.

In some tools a rating isn't an instruction to the draft at all. It goes to the company that makes the product, and what happens next depends on the product and the account. Google says feedback on a Gemini response carries the conversation with it: your prompts, Gemini's responses and any files or images you included. Rate the pricing page there and your own $50 goes along. Microsoft says it uses thumbs feedback in Microsoft 365 Copilot Chat to improve Copilot, and doesn't use it to train the foundation models behind it. Check what your tool sends before you rate a page with real numbers on it.

Typing "the $40 is their annual price" gives the AI more to work with in that chat. Training the model is a separate job. NIST describes fine-tuning as a training step that starts from an already trained model and adds information for a particular task or field. Your correction is information for the draft in front of you, not for the model.

What fixes the pricing page is an instruction for the next version, written so the AI doesn't have to guess. We call it a change order, after the note a builder writes when a client changes a job partway through. It says what's changing, what that affects and what stays as agreed.

The five lines of a change order

Ours is a short note about the draft in front of you. It covers five things, and the pricing page needs all of them.

Miss: what's wrong, exactly

Point to the spot and say how it differs from what you expected. Leave out your theory of why the AI did it. "The competitor price in the table is $40, its annual-billing price, and our $50 is monthly" is a miss. "You got confused about pricing" is a guess about the model, and it gives the model nothing to change.

If you aren't sure something is wrong, say so. "This looks like their annual rate. Check it against their pricing page" is honest, and it still points to a place on the page.

Consequence: why it matters for this page

Add one sentence on what the miss breaks. "The recommendation to cut our price rests on a $10 gap. On monthly terms the gap is $2, so the recommendation may not hold." Now the AI knows the problem goes past one cell in a table. The number feeds a conclusion, and the conclusion has to be looked at again.

Skip this line when nothing depends on the miss. A typo in a heading doesn't need a consequence, and inventing one only gives the AI something else to act on.

Source or rule: what the right answer is based on

Give the AI something to correct against. Here that's the competitor's pricing page and a rule: compare both plans on the same billing term. Without it, the model may swap one wrong number for another.

A link isn't proof. Pricing pages change, and two sources can disagree. If they do, say which one governs, or say you don't know yet and ask the AI to show both. Never make up a source to fill the line.

What to keep: what must not change

Name what you've already checked. "Don't change the feature table or the first paragraph of the summary." It's the easiest line to leave out. Without it, nothing tells the AI the table was checked, which is how it got reworded last time.

Sometimes the part you want kept depends on the part you're fixing. If the summary's first paragraph mentions the $10 gap, that sentence has to change too. Say so in the change order, so the AI knows which sentence may move and which ones may not.

Check: how you'll know it's fixed

Say what you'll look for when the next version comes back. "Both prices are monthly, the gap is restated, and the recommendation either follows from the new gap or says it no longer does." That gives the AI a target, and it gives you a short list to work through when you review.

Put together, it runs longer than "fix it" and reads like a note to a colleague:

The competitor price in the table is $40, which is their annual-billing price. Our $50 is monthly, so the two aren't comparable. The recommendation to cut our price rests on that $10 gap; on monthly terms it's $2. Use the monthly price from their pricing page [link] and compare both plans on monthly billing. Don't change the feature table. The summary's first paragraph mentions the gap, so update that sentence and leave the rest alone. When you're done, both prices should be monthly, the gap should be restated, and the recommendation should either follow from the new gap or say that it doesn't.

A clear request can still miss

Writing all that down gives the AI a better shot at the pricing page. It doesn't guarantee one.

You found the $40 before you said anything about it, and one tested system works in that order too. In a Findings of EMNLP 2024 study, a separate model spotted errors in document-grounded summaries, each error got a fine-grained critique, and the summary was revised from those critiques. That kept the summaries truer to their documents than methods that revise the whole summary in one step. There a model did the spotting, not a person typing notes into a chat.

Feedback far more exact than a note about the $40 can still fall short. In Feedback Friction, presented at NeurIPS 2025, models including Claude 3.7 with extended thinking worked math, knowledge and science questions, and a feedback generator that could see near-complete correct answers told them what they got wrong. Their accuracy usually rose over the first few rounds, then leveled off below what they'd have scored had they used all of it. Sampling changes the authors tried helped, but didn't close the gap.

A study presented at ACL 2026 looked at repair in conversations about math questions, some solvable and some not, and found each model unreliable in its own way. Some were "almost completely resistant" to appropriate repair attempts. Others were "highly susceptible and easily manipulated." That second kind is one more reason the Miss line says to mark a hunch as a hunch. If you were wrong about the $40, a model that gives way to any correction would rebuild the page around your mistake.

Check the fix, then check the page

Start with your check line. Are both prices monthly? Is the gap restated? Does the recommendation follow from it? Then read the whole page again, faster. Look at the feature table you asked it to keep, the summary and anything that wasn't there before. A fix in one place can change a sentence in another, and the change order only covers what you thought to name.

The change order also leaves a record. If someone asks on Thursday why the gap moved from $10 to $2, you can show what you asked for and which page the number came from, and say you opened that page yourself and the $48 matched.

When to stop asking for another version

Say the second version gets the prices right but now recommends raising your price, with no reasoning behind it. You could ask again, but that isn't the only move.

If the AI is missing context, give it. It doesn't know your margin target, or that your head of sales wants to hold price this quarter, and the recommendation can't account for either. When the fix is one paragraph and you know what it should say, it's quicker to write it yourself.

A thread that has turned into a pile of corrections can be worth abandoning: open a fresh request with the corrected numbers and the billing-term rule. A recommendation that's really a call for finance or sales belongs with them. And sometimes the right move is to drop the recommendation and take the corrected numbers into the meeting on their own.

Make the lesson outlast the page

The rule you wrote, compare prices on the same billing term, will matter again the next time someone builds a pricing comparison, and that person won't be reading your old chat.

NASA's lessons learned guidance describes four stages: collect, record, disseminate and apply. Your billing-term rule has made it through the first two. NASA calls applying a lesson "a crucial step" and describes doing it through updates to processes, checklists, handbooks and formal policy.

NASA has also seen a lessons system fall out of use. A 2012 Office of Inspector General review looked at its Lessons Learned Information System and found that project managers didn't routinely search it for other projects' lessons or add their own. The OIG wrote that the system had been "marginalized in favor of other NASA knowledge sharing system components" and was of "diminishing and questionable value."

For the pricing page, applying the lesson is a small job. Put the billing-term rule in the brief or template your team uses for pricing comparisons, and name who owns that template. The next time someone builds a comparison, check that the rule was in the brief and that the page followed it.

Where Supanova fits

Say the pricing comparison ran on Supanova as a one-off task. When it comes back, you can rate it, and that rating changes who gets the next job. You can also send the task back for a revision with your notes, and the work runs again with them.

The rating and the notes do different jobs. The rating grades the version you got. The notes are where the change order goes: the $40 that should be $48, the $2 gap and what it does to the recommendation, the pricing page to correct against, the feature table to leave alone and what you'll check when the next version comes back.

The next time a draft misses, write those five lines before you type "fix it." Then decide whether another version is worth asking for at all.

See how it works at supanova.team.

Sources

autonomous-workforceAI RevisionsdelegationReviewing AI WorkAI Feedback