Where AI Fails in Tax Prep: Five Failure Modes to Check First

Before the Firm relies on an AI tool for a task, someone should know how the tool fails on it. Compare each row with its source, even when its layout and page reference look right.

Start by testing these five failure modes on synthetic tax documents. Build a small known-answer test before you rely on the tool for a task. Record each mismatch and where it sits in the source, for the Reviewer.

This piece covers five failure modes and a known-answer test for each. Run the test on a known-answer folder first. Then the Firm decides, task by task, whether to rely on the tool, which human checks to require, and which work to keep manual.

How to test: a known-answer folder

A known-answer folder is a small set of documents where the Firm already knows every correct output.

1. Pick one task. Test a single task the Preparer would hand the tool, such as listing every figure on a set of income documents with the page it came from.
2. Use synthetic documents only. Build them for a made-up client, Jane Doe, and a made-up entity, Doe Holdings. Never copy a real client file into the test. Include the kinds of documents the task meets in real work: a typed statement, a scanned page, one long multi-page file, and a late attachment.
3. Write the answer key before you run anything. List every figure the task should return, the document and page it sits on, and every item that should come back "not found."
4. Plant the traps for each failure mode. The five sections below say what they are.
5. Run the task with the real instruction. Use the same wording the Preparer would use on a client folder.
6. Compare row by row. Mark each trap caught or missed. Note any other row that differs from the key.
7. Change one thing and run again. Adjust the instruction or add a check, then rerun the whole folder. Keep each run's output beside the key.

Keep the folder small enough to check by hand in one sitting.

Failure mode 1: a citation that doesn't hold

What it looks like. A row shows a figure and a page reference, and the cited page shows a different figure. Or the row names a document but no page. Or there's no citation at all.

The trap. On one page, put two figures that sit close together and look alike, such as a subtotal printed just above a total. The key names which one the task wants and its page. Add a second figure that only appears on an inside page of a statement, so a citation to the first page is wrong.

What to change. Tell the tool that every row cites a document and a page, and that a row it can't cite is marked uncited rather than dropped or guessed. The human check: for this test, compare every returned figure and citation with the answer key, including rows that involve no judgment. How to ask for and check citations in detail is covered in the citation piece.

Failure mode 2: the wrong document version

What it looks like. The figure is real, but it came from the wrong document. A prior-period statement was in the folder. A corrected form was there, and the tool read the original. Or another client's file was left behind.

The trap. Put an original and a corrected version of the same Jane Doe document in the folder, with the corrected one marked as corrected. Add a prior-period statement from the same payer. Drop one Doe Holdings document into the Jane Doe folder. The key uses only the corrected, current-period Jane Doe figures.

What to change. Tell the tool to list every document first, with its name, the person or entity, the period, and whether it's an original or a corrected version, and to flag duplicates and other names before pulling any figures. The human check: the Preparer confirms that document list and clears strays from the folder before the pass runs.

Failure mode 3: silent combining

What it looks like. One row shows a figure that appears on no single page. The tool added two documents together, netted one figure against another, or split one figure across two lines. Nothing on the row says so.

The trap. Include two current-period statements from the same payer for two different accounts, and have the key keep them as separate rows. Add one document where a credit sits next to a charge, and have the key list both rather than a net figure.

What to change. The instruction: return each requested figure separately, with its document and page. No adding, netting, or splitting figures. The human check: compare the returned rows with the key for the two accounts and for the credit and charge. Mark any extra, missing, merged, split, or incorrectly sourced row as a miss.

Failure mode 4: a filled blank

What it looks like. The output has a figure, a date, or an identifier where the folder has none. It looks plausible and it's in the right format. Compare it with the key. The format is not evidence.

The trap. Leave out one item the task asks for, so no document in the folder shows it. Add one document with a field left empty. The key marks both "not found."

What to change. Tell the tool that "not found" is an acceptable answer, and the required one when no page shows the item. Tell it never to estimate. The human check: every figure the task returns for an item needs a page behind it. A row that fills a blank without one goes to the Preparer, who finds the missing document or asks the client.

Failure mode 5: skipped pages or files

What it looks like. The output covers most of the folder, but a scanned page, the back half of a long file, or an attachment added last is missing. The output doesn't say anything was skipped.

The trap. Put one key figure on the last page of the longest document. Put another on a page that's an image only, with no text layer. Add one file after the others, named so it sorts last.

What to change. Have the tool list every file it read and the page count for each before it gives results, and flag any page it couldn't read. The human check: compare that list to the folder's own file list and page counts. If the long files keep losing pages, split the folder and run the task in parts.

After the test

Write down a result for each task you tested. A task passed: the output matched the key and caught every trap. A task passed with a named check: it missed a trap, the assigned person applied a named human check on a rerun, and the combined result matched the key. Record any mismatch that remains. A task stays manual: any miss is still unresolved after the rerun.

Keep that record with the test folder, the answer key, the exact instruction, and the date of each run. Run the folder again when the tool, the instruction, or the task changes.

The Firm decides which tasks to rely on the tool for. The test shows where to look; it doesn't make any output right. The Reviewer still concludes and signs off on the workpaper. When an entry list is finished, the Reviewer's pass over it is a separate, later check, laid out in the readback piece. What a Reviewer needs from the file is covered at for reviewers.

A one-screen test checklist

Adapt this to your Firm's procedures before you use it. Use synthetic documents only.

1. Name the one task under test and the exact instruction the Preparer would use.
2. Build the folder for Jane Doe, with one stray Doe Holdings document.
3. Include a typed statement, a scanned page, one long file, and a late attachment.
4. Write the answer key: every figure, its document and page, and every "not found."
5. Plant a lookalike figure and an inside-page figure for the citation trap.
6. Plant an original and a corrected version, plus a prior-period statement.
7. Plant two same-payer statements and a credit beside a charge.
8. Leave one requested item out and one field empty.
9. Put figures on the last page, on an image-only page, and in the last file.
10. Run the task, compare row by row, and mark each trap caught or missed.
11. Change one thing, rerun, and keep each run's output beside the key.
12. Record each task as passed, passed with a named check, or stays manual, with the date.

Where TracePrep fits

In TracePrep's Reasonable Compensation Study, every figure traces to its source, occupation, geography, and period, at field level. A Reviewer signs off on each Study, and the audit log records who prepared it, who changed what, and who approved it. Finalized evidence stays with the Firm.

This is general information, not tax or legal advice. Your licensed professional and your Firm's Reviewer are responsible for every return and every conclusion. Keep client data inside the systems your Firm has approved. As general guidance only, not advice: if you use an AI tool with client data, firms generally look for a business or commercial plan whose terms cover how that data is handled; read the vendor's current terms before you do.

Join the waitlist