Why ChatGPT Fails at Financial Analysis, and What Actually Works

ChatGPT fails at financial analysis for a specific, mechanical reason: it generates numbers as text rather than computing them, and it fills gaps in your context with the most statistically common assumption rather than asking. Both failures produce output that is fluent, confident, and occasionally wrong, and in finance, wrong and confident is worse than obviously broken.

This is not an argument that AI has no place in finance. It is an argument about which part of the work a language model should be doing.

The mechanical failure: arithmetic as prediction

A language model predicts the next token. When you ask it to sum a column, it is not running an addition: it is predicting what the answer probably looks like, based on patterns in its training data and the surrounding context.

You do not have to take our word for this, because the model vendors say it themselves. OpenAI's own guidance for its models is to use the code interpreter tool for any computation involving math rather than answering directly, on the grounds that executing a calculation gives more reliable answers than generating one. That is the same architectural argument made here, from the company with the most to gain from the opposite conclusion.

The benchmark evidence points the same way, with an important nuance. Scale AI's A Careful Examination of Large Language Model Performance on Grade School Arithmetic (Zhang et al., 2024) rebuilt the standard GSM8K grade-school maths benchmark from scratch as GSM1K, so no model could have seen the questions. Several model families dropped by up to 8 percentage points on the fresh set, which the authors attribute to systematic overfitting to the original benchmark. The paper is careful to conclude that models do genuinely generalise to unseen problems: this is not a claim that language models cannot do arithmetic. It is narrower and more useful than that: published accuracy figures overstate what you will actually get, and grade-school word problems are a far easier target than a novel calculation over your own ledger.

Most of the time that prediction is right, because the arithmetic is simple and well represented. Sometimes it is close but wrong. You can watch this happen: ask for a table with a total row and the total will occasionally fail to equal the sum of the rows directly above it, in the same response, with no indication that anything went wrong.

Modern tools mitigate this by wiring in a code interpreter, and where that is available and actually invoked, the arithmetic becomes reliable. But whether it fires is not always visible to you, and the model decides. A number produced by prediction and a number produced by execution look identical in the output.

The deeper failure: context it cannot have

The arithmetic problem is the one people notice. The context problem is the one that does real damage.

Ask a general-purpose model to calculate your net revenue retention and it will produce a defensible number using a standard formula. Whether it is your number depends on things the model has no way to know:

• Your fiscal year starts in April, so its Q2 is not your Q2.
• You classify customer success in COGS, which moves gross margin by several points against a peer who does not.
• Last October included a one-time migration fee that everyone internally knows to exclude.
• Two customers are on a legacy contract deliberately excluded from cohort analysis, for a reason agreed three years ago.
• Your commission plan recognises on collection, not on booking.

None of this is written down anywhere. It lives in the heads of two or three people, and it is applied automatically every time one of them builds an analysis. A model given the same raw data and none of the rules produces a technically correct answer to a question you did not ask.

It will not flag the gap. Faced with ambiguity, it resolves it and continues.

Why finance is different from other domains

In most domains, a wrong AI answer announces itself. Code does not compile. A summary contradicts the document. A recommendation sounds obviously off.

Financial output has none of those tells. It arrives formatted, internally consistent, and stated without hedging. A margin bridge with a wrong input still foots. A forecast built on a fabricated assumption still looks like a forecast. The failure mode is not that the output looks broken: it is that it looks exactly like the correct answer.

That has a consequence people underrate: a wrong AI number is more likely to reach a board than a wrong hand-built one, because the hand-built version carries the analyst's own uncertainty with it. The person who built it knows which assumption is shaky. The model expresses no uncertainty at all.

The five failure modes, concretely

Failure	What it looks like	Why you miss it
Generated arithmetic	A total that does not match its rows	Nothing in the output flags it
Undefined metrics	A standard formula applied to a non-standard business	The number is plausible and defensible in general
Silent assumptions	A missing month interpolated rather than reported	The gap is never mentioned
No traceability	A figure with no query, rows, or formula behind it	Verifying means rebuilding it yourself
Confident presentation	Unhedged prose around an uncertain number	Format signals reliability that is not there

What prompting does and does not fix

Careful prompting genuinely helps, and everything below is worth doing:

• Define every metric inline, with the formula you actually use.
• State the fiscal calendar and comparison basis explicitly.
• Instruct the model to stop and list what is missing rather than estimate.
• Require the formula and input values behind every number.
• Require arithmetic to be performed as code, and to show the code.
• Require assumptions in a separate section, before the conclusion.

These measurably reduce the error rate. They do not eliminate it. As long as the model is the thing producing the numbers, some proportion will be wrong, and none of them will look wrong. Prompting is a mitigation, not a fix.

The architecture that actually solves it

The fix is structural: stop letting the model touch the arithmetic.

A model is genuinely excellent at understanding an ambiguous question, planning an approach, and explaining a result in prose. It is unreliable at producing numbers. So separate those jobs:

1. The model plans. It interprets the question and determines what analysis is required.
2. Code executes. The actual calculation runs as code in a sandboxed environment with read-only access to source data: deterministic, inspectable, and identical every time.
3. Output is validated. Results are checked against constraints before being returned.
4. Everything is traceable. Every figure comes back with its formula, its query, and the rows it came from, so it can be verified without being rebuilt.
5. Company context is applied automatically. The metric definitions and business rules that live in someone's head are captured once and applied to every subsequent answer.

This is how DataWyse is built, and it is the reason we treat the trust question as an engineering problem rather than something to be handled with a disclaimer. The model never generates a number. It decides what to calculate; the calculation happens somewhere you can inspect.

Where a general-purpose model is genuinely useful today

Being accurate about the limits also means being accurate about the value. A general-purpose model is well suited to:

• Structuring an approach to an unfamiliar analysis.
• Explaining an accounting treatment or a metric definition.
• Drafting commentary once you have verified the numbers.
• Reviewing your logic and identifying what you have not considered.
• Writing the SQL or the formula, which you then run yourself.

Every one of those is a language task. The pattern is consistent: use it for language, not for arithmetic. The moment a number it produced is going to be acted on without being checked, you have crossed the line where the architecture matters.

What still works, and is worth using

None of this argues against using general-purpose models in finance. It argues for using them where prediction is the right mechanism:

• Structuring an approach before you build anything. Genuinely excellent, and the fastest way to find out what analysis you actually need.
• Explaining a concept or a formula you half-remember.
• Reviewing your own logic: describe the analysis you built and ask what you missed.
• Drafting commentary once the numbers are verified and final.

Every one of those is a language task, and language is what the mechanism is for.

The line to hold

Use it for the words. Do not use it for the numbers you intend to act on, not because the model is weak, but because a predicted figure and a computed one look identical in a chat window, and nothing in the output tells you which one you are reading.