Ask AI to Prove It Can Handle the Material Your Business Uses

Six · 6 min read · March 31, 2026

Ask AI to Prove It Can Handle the Material Your Business Uses

Say you manage supplier renewals, and a vendor is showing you an AI tool that reads contracts. The demo goes well. It takes a clean sample agreement, finds the renewal date and lists the termination terms in a few seconds.

Then you think about the agreements your team actually works from. One is a scanned PDF with handwritten initials on every page. Another has its pricing schedule in a table that runs across three pages. A third still has tracked changes nobody accepted. The current prices live in a spreadsheet whose column names only make sense to your team.

The demo didn't show you whether the tool can read any of those. It showed you that it can read the vendor's sample.

So before you build a renewal process around it, run a small proof: one file that looks like yours, one result you already know the answer to, and the exact setup you plan to use.

This comes before testing a whole workflow. For now the only question is whether the tool can read your material and give back something a colleague can check, because if it can't, the rest of the renewal process isn't worth testing yet.

Start with one file and one result

Pick an agreement that's coming up for renewal and get permission to use it. That might mean a redacted copy, or a sample your legal team builds to look like the real thing. Don't upload a supplier agreement anywhere you aren't authorized to share it. A redacted copy can still show you whether the scanned pages read, whether the pricing table survives and whether clause references point to the right place.

Then decide what you want back. Here it's a short exception table: the renewal date, any clause that lets the price change, any commercial term you'd expect to see that's missing, and a page reference for each finding.

That gives the test an answer key. You already know when this agreement renews, that the supplier can raise prices once a year, and that nothing caps the increase. When the table comes back, you can check whether the tool read the file at all, whether it found those clauses, whether it said where it was unsure, and whether you could take the table into your next renewal review as it is.

Write the test down before you run it

A short card keeps the vendor's demo from setting the terms. Fill it in before anyone uploads anything.

Field What to write
Material The approved file, its format, and anything awkward about it (scanned pages, a long table, tracked changes).
Route Where the file goes in and what handles it: the workspace, the upload screen or another tool, and the model behind it.
Expected result The exact thing you want back, such as the exception table with page references.
Known facts Three to five things the reviewer already knows are in the file.
Stop rule What ends the test: the file won't open, a reference is missing, a value is made up, a connection isn't there, or the reviewer can't check the result.
Reviewer The person who decides whether the result is good enough to act on.

The route gets its own row because the same vendor can treat the same file differently depending on how it arrives. A file a chat window accepts may be handled differently when it's attached to a project or handed over through another tool. A different model behind the same screen may read it differently again. Text coming out of a PDF doesn't mean the table inside it was understood, and one good run tells you about this file, not the next one.

NIST's Generative AI Profile describes the same gap in AI testing generally. It says current testing approaches "often remain focused on laboratory conditions or restricted to benchmark test datasets," which may not carry over to real-world conditions. It adds that the gap may grow because a generative AI tool's answers shift with how it's asked, and because it gets used in so many different settings. The vendor's sample agreement is the laboratory. Your scanned contract with the three-page table is where you'd actually use the tool, and if the tool handles that one, it has earned a bigger test with more of your files.

Read the result one piece at a time

When the table comes back, record each part separately.

What you see What it tells you
The file opened and the sections you expected are there This route handled this file.
The known facts are in the table, with references you can follow The reviewer has something to check for this task.
A clause, table or reference is missing Something to investigate: the file, how it was read, the instruction, or the review.
A connection or outside action can't be confirmed That part of the work waits until the access or setup is in place and it can be run again.

Say the renewal date and the price-increase clause come back correct, with the right page numbers. The missing-terms column is empty, though, and the pricing tab shows the spreadsheet's header row as if it were a price. That's a useful result: the agreement read well and the spreadsheet didn't. Before you decide anything, check whether the empty column came from the file or from your request. If you never said which terms you expected to find, the tool had nothing to compare against. Add them to the request and run it again. If the spreadsheet still comes back wrong, you've found a real limit on that route.

Then write it down: the file, the route, the date, what came back and what the reviewer found. For this test, the note says the agreement read cleanly, with the renewal date and price clause on the right pages. It says the tool took the pricing tab's header row for a price, and whether the missing-terms column filled in once the request listed the terms. The next person who evaluates a tool can start from that.

Running the card on Supanova

Say you run the same card on Supanova. The project brief is the card written out in sentences: the renewal question, the table you want back and the three facts you already know. Attach the redacted agreement and the pricing sheet to the project, where files can give your atoms extracted text to work from. Whether the pricing sheet comes through cleanly is exactly what the card is there to test.

When the table is finished, it comes back for your review. The reviewer named on your card reads it against the agreement, follows the page references and looks at what happened to the pricing sheet. Whatever they find goes on the card, and it shapes the next brief, whether that's more files or the renewal process itself.

No tool can tell you whether you're allowed to use the real agreement, whether the redaction went far enough or whether a flagged clause matters commercially. Those calls stay with the reviewer and whoever owns the supplier relationship.

Bring the file that makes the decision real

Pick the agreement that would worry you most: the scanned one, or the one with the long pricing table. If yours is a monthly workbook or a client brief instead, the card works the same way. Put three facts you already know on the card, and decide before you start which miss would stop the test. Then run it on the exact route you plan to use.

If the table comes back right, you can move on to more files and, after that, the whole renewal process. If it doesn't, you've found the limit on one redacted contract instead of halfway through renewal season.

Bring one file and one expected result to Supanova.

Sources

Business DocumentsAI evaluationAI TestingAI procurement