Changing the Firm's AI Instructions or Tool? Re-Test on a Known File First

Someone at the Firm edits one line in an AI instruction, or the Firm moves to a new model version. The Firm keeps a fixed synthetic test file with accepted entries. Nobody has compared the new draft with those accepted entries.

To re-test AI instruction changes on a known file, the Preparer re-runs the AI pass on one fixed file whose entries the Firm has already accepted, then compares the new draft with those entries row by row. A named Firm person approves the change or holds it before it reaches a client file. The plugin drafts; the Firm's preparer prepares; the Reviewer signs off. None of that moves when an instruction does.

This piece gives a change log, a known-file comparison table and a go / hold rule the Firm can paste into its procedures. How to write and version the instruction itself is covered in writing AI instructions for tax documents.

What counts as a change

Under this approach, any of these sends the Firm back to the known file before the next client file:

• An edit to the words of an instruction, even one line.
• A new tool, a new account, or a new model version under the same tool.
• A change to the steps around the AI pass, such as how the working copy is set up before the pass runs.

If the Firm isn't sure whether something counts, it counts. A re-test takes one file. A doubt that surfaces at review touches every file drafted since the change.

Build the known file once

The known file is a synthetic client file the Firm builds once and keeps fixed: Doe names, made-up documents, and a numbered set of accepted entries the Reviewer has read and accepted as the right answer. If the Firm ran a five-file pilot before adopting the AI pass, one of its synthetic files and that file's reference version can become the known file.

A few rules keep the known file useful:

• Pick a file with some difficulty in it, such as a corrected document or a page that's hard to read, so a change has something to get wrong.
• Number every accepted entry, so the comparison runs row against row.
• Keep the known file read-only. It changes only by a logged decision, never during a re-test.
• Keep it out of the client folders, so nobody mistakes it for a live file.

The instruction change log

The log holds one row per change and sits with the Firm's procedures. The row opens before the re-test and closes when the approver decides.

If the Firm already keeps a change log for each instruction, these columns can sit beside it. Who approved is a named person, not a role, so anyone reading the log later can see who made the call.

Run the re-test

1. The Preparer opens a fresh working copy of the known file. The accepted entries stay closed until the draft is done.
2. The Preparer runs the AI pass on that copy with the changed instruction, tool or version, the same way staff run it on a client file.
3. The Preparer lines up the new draft against the accepted entries, one row at a time, and marks each row.
4. The Preparer hands the comparison and the log row to the approver.

The Preparer doesn't fix the draft during a re-test. A hand fix would hide the very difference the re-test is there to find.

The known-file comparison

Each accepted entry gets one line. A draft row with no accepted entry also gets a line, with the accepted column left blank and the row marked differs.

Match means the same entry, word for word, pointing to the same source page. Close isn't a match. A shortened name or a different page reference is a difference, and the approver sees it.

The go / hold rule

The Firm sets the rule before the first re-test and writes it at the top of the change log, so nobody sets it while looking at a result. Here's one rule the Firm can set:

• Go: every row matches. The named approver approves the change for use and records that decision in the log.
• Hold: any row differs or is missing. The change stays out of use. The approver sends it back for a revised version and a new re-test, or rejects it.
• While on hold: staff keep using the last version that passed. If that version can't be run any more, the Preparer works the files by hand until a version passes.

A difference that looks like an improvement is still a hold. If the approver thinks an accepted entry is wrong, the Reviewer looks at the known file again, and any change to it gets its own row in the log before the next re-test.

Who does what

Under this approach, the Preparer runs the re-test and marks every row. A named Firm person, such as the manager who owns the instruction, approves the change or holds it and records that decision in the log row.

The Reviewer's job doesn't change. On every client file drafted after the change, the Reviewer reviews the prepared file as before. Every file ends with the Reviewer's sign-off. The comparison records the result for that run.

This isn't a prior-year comparison. That compares two years of one client's file, while the re-test compares two versions of the Firm's own setup on a file that never changes.

A Doe example

The names, file and counts below are made up for illustration.

Doe & Co. CPA keeps a known file for the synthetic Doe Holdings client, with eight accepted entries, rows 1 to 8. Morgan Doe, a manager, owns the Firm's document-list instruction and approves changes to it. Alex Doe is a Preparer, and Pat Doe is the Reviewer.

Morgan edits the instruction so each drafted row names the page it came from. Alex opens a fresh copy of the known file, runs the pass with the new wording and lines the draft up against the eight accepted entries. Six rows match. Row 4 differs, because the drafted name is shorter than the accepted one. Row 7 is missing. Six plus one plus one is eight.

Under the Firm's rule, that's a hold. Alex logs the result as six of eight matched, hold, and staff keep using the old version. Morgan revises the wording, and Alex runs the known file again on a fresh copy. All eight rows match. Morgan approves, and the log row reads eight of eight matched, go.

Later, the Firm moves to a new model version under the same tool. The instruction's words didn't change, but the version did, so it gets its own log row. Alex re-runs the known file, all eight rows match, and Morgan approves.

Through all of it, Pat reviews each client file as before and signs off. The approver decides only whether the change goes into use.

Where TracePrep fits

We've built a TracePrep plugin that drafts these steps in your Firm's workflow, while your preparer decides and your Reviewer signs off. Join the waitlist and we'll tell you when plugin access opens.

Get on the list for plugin access

This is general information, not tax or legal advice. In TracePrep's workflow, the plugin drafts, the Firm's preparer prepares, and the Firm's Reviewer signs off. Access to the plugin is by waitlist. Keep client data inside the systems your Firm has approved.