Financial documents
AI that shows its working on financial reports
How we stopped an AI from doing mental arithmetic — and measured honestly how often it's still wrong.
The problem
Ask a chatbot a follow-up question about a financial report — "and by what percentage did that increase on the previous year?" — and it often loses track of which year you meant, or does the arithmetic in its head and gets it wrong. Confidently.
What we did
- 1Split the job: the AI reads the document and works out which numbers are needed; a separate, locked-down calculator does the maths.
- 2Make the AI write down its reasoning step by step before it gives an answer, so every answer can be checked.
- 3Design it to refuse rather than guess when a table is ambiguous — for example, when two columns have the same heading.
- 4Measure every change on 1,490 real questions, with proper statistics, rather than trusting a few good-looking examples.
Results
79.5%
of 1,490 questions answered correctly (strict scoring)
68.9%
the original 2022 research model on the same test
89.4%
human experts on the same test
The hardest cases — tables with duplicate column headings — scored only 53.1%. That's why the system now refuses to answer those instead of guessing.
The lesson
Well-engineered AI is genuinely useful on financial documents, and it is still wrong often enough that a qualified person must check anything that matters.
What this means for you
When we set up AI for your practice, we use the same principles: AI drafts and summarises, your software calculates, and a person reviews. We'll be straight with you about where it's reliable and where it isn't.
Further reading
Find out where you stand in 30 minutes.
Free, no obligation, and you'll leave with a one-page summary you can act on — whether or not you work with us.