Pilot Plan: Trying an AI Pass on Five Files Before the Firm Adopts It

A Firm weighing an AI drafting pass has a choice to make before the season: switch it on across the Firm, or try it small first. One way to try it small: five synthetic Doe files, a written plan, one log and a decision the Firm frames before anyone opens the first file.

This piece lays out one way to run that pilot. Throughout, the AI tool drafts an entry list, the Firm's Preparer checks it and prepares the file, and the Reviewer reads it back and signs off, as on live work. The pilot watches whether that chain holds on the Firm's own work.

A pilot asks a narrow question. It doesn't measure time, and it doesn't show the pass is right, on these files or any others.

Write the plan before the first file

The Firm can put the whole pilot on one page and date it. That page names the pilot lead, the five files, who checks each one, the log, the go/no-go questions, the stop triggers and the three calls the Firm might make at the end.

The Firm can date its questions and decision rule before the first draft, then record any later changes separately.

The pilot lead is one person, a manager or senior the Firm picks. Under this plan, the pilot lead isn't the Preparer or the Reviewer on any pilot file, so the comparison is a separate read. The Firm runs the pilot inside the systems it has approved, under its own policy, and reviews its own confidentiality and consent obligations before the pilot starts.

Pick five files of mixed difficulty

Under this plan, the pilot uses five synthetic Doe files the Firm builds, such as a Jane Doe file and a Doe Holdings file. For each one, the Firm also builds a finished reference version before the pilot starts and keeps it out of the draft checks.

Then the Firm can mix the five on purpose:

Two plain files. Few source documents, clean copies, nothing the Firm would call unusual.

Two middling files. More documents, plus one wrinkle the Firm knows well, such as a corrected document or two versions of the same statement.

One hard file. Many documents, at least one poor scan or handwritten page, and the kind of trouble that gets a file sent back on live work.

Triage is a different job. The Firm can use a triage rubric for client files to score files and choose their lanes. The Firm picks the pilot's five on purpose, including a hard one that triage might keep all by hand, because the pilot needs to see where the pass breaks.

Where the Firm can, each file goes to a Preparer who didn't build its reference version. Under this plan, the Firm asks the Preparer to check the draft against source documents rather than rely on memory.

Who checks what

In this plan, each role does one thing, in a fixed order:

The tool drafts the entry list from the client folder. Nothing more.

The Preparer checks the draft the way the Firm would on live work, without opening the finished file. Every change gets a short reason.

The Reviewer reads back the checked file as on live work and records any send-back.

Under this plan, the pilot lead opens the finished file only after the Reviewer is done, compares it with both the original draft and the checked version, and logs each original-draft difference with the stage that first caught it.

The order matters. Under this plan, the Firm keeps the finished file closed during the draft checks to reduce reliance on its contents.

During the pilot, the Reviewer can read back every row instead of a sample. If the Firm later chooses to sample rows on live work, it can write a risk-based sample plan for that separate workflow.

What to record per file

The pilot lead keeps one row per file. Here's a template the Firm can paste and adjust:

Under this plan, the pilot lead counts each original-draft difference once, under the first stage that caught it, so the three "caught" columns sum to the original-draft differences column. Every difference gets one line in the note, in the Firm's own terms: a wrong amount, a wrong label, a wrong page reference, a missing row or an extra row.

There's no minutes column. Under this plan, the Firm leaves time out of the pilot so the team can focus on checking the drafts. A Firm that wants to know about time can keep a two-column time log during the season, after it decides to go.

The go/no-go questions

The Firm picks its own questions. Here's a starting checklist it can paste into the plan and answer yes or no at the end:

• Was every difference caught by the Preparer or the Reviewer before the comparison?
• Can the Firm name a check to add or change for each comparison-only difference, recording yes on this line if there were none?
• Does every drafted row point to a source page the Preparer could open?
• Was every page the pass couldn't read flagged in the draft, not filled in?
• Was the hard file checked to the same standard as the plain ones?
• Can the Preparer and the Reviewer each describe their step without the pilot lead in the room?
• Are the instructions the tool ran with written down and dated, so a rerun starts from the same place?
• Did every file stay inside the systems the Firm approved?
• Did the pilot finish without a stop trigger firing?

The Firm decides in the plan which answers it needs for each call. One version the Firm could choose: a no on the client-data line means stop; otherwise, any no means change and rerun, and all yes means go. In this version, a go opens the next step the plan already named, such as a limited lane for plain files, not Firm-wide use on day one.

When to stop the pilot early

The Firm can write its stop triggers into the plan. Some it might pick:

• A difference caught only at the comparison, in a part of the file the Firm treats as high-stakes.
• A drafted row with no source page, or a page reference that points to the wrong document.
• The pass fills in a page it couldn't read instead of flagging it.
• Client data leaves a system the Firm approved.
• The team stops keeping the log, so a file's record is incomplete.

Under this plan, when a trigger fires, the pilot lead stops, writes the reason on the log with a date, and the Firm decides what to change. If the Firm restarts, it can use five new files, so the change is tested on work the pass hasn't seen. A stop still leaves the Firm a dated reason to work from.

A Doe example

The counts below are made up for illustration.

Doe & Co. CPA plans a pilot. Morgan Doe is the pilot lead. Jane Doe checks the drafts, and Pat Doe reads them back. The Firm builds five synthetic files and their reference versions before the pilot: two plain, two middling and one hard, the Doe Holdings file, with a faded scan.

The plain files show 0 and 1 differences. Jane catches the one. The middling files show 2 each. Jane catches both on the first, and on the second she catches one and Pat catches the other on readback.

Doe Holdings shows 2 differences. Jane catches one. The other is an amount from the faded page, filled in by the draft without a flag. Nobody catches it until Morgan's comparison.

Across five files, that's 7 differences: 5 caught by Jane, 1 by Pat and 1 only at the comparison. The faded-page row also fires a stop trigger the plan named, the pass filling in a page it couldn't read. It's the last file, so the pilot ends there anyway, and Morgan logs the reason and the date.

Three go/no-go answers come back no: the miss caught only at the comparison, the unflagged page and the fired trigger. None is the client-data line, so the call is change and rerun. The Firm adds an instruction for the pass to flag any page it can't read and a Preparer step to work flagged pages by hand. It plans a second pilot on five new files, with two hard ones this time.

Where TracePrep fits

We've built a TracePrep plugin that drafts these steps in your Firm's workflow, while your preparer decides and your Reviewer signs off. Join the waitlist and we'll tell you when plugin access opens.

Get on the list for plugin access

This is general information, not tax or legal advice. In TracePrep's workflow, the plugin drafts, the Firm's preparer prepares, and the Firm's Reviewer signs off. Access to the plugin is by waitlist. Keep client data inside the systems your Firm has approved.