GPT just predicts the next word, So how can it do math?

We explain it clearly in five levels, from intuition to principle.

1. The first level: it has "seen" a large number of mathematical texts

Like GPT-3, GPT-4 (from OpenAI) read during training:

• Mathematics textbooks
• Paper
• Solution
• Forum discussion
• StackOverflow
• code base
• Teaching aids

It has seen countless patterns like this:

问题：2 + 3 =
步骤：
2 + 3 = 5
答案：5

若 x + 2 = 5
则 x = 3

So the first step:

It learned "the linguistic structure of mathematical expressions."

But this is not enough.

2. Second level: Mathematics is also a “symbolic mode”

A lot of elementary mathematics is essentially pattern matching.

For example:

a + b = b + a

commutative law.

The model will see repeatedly during training:

2 + 3 = 5
3 + 2 = 5

Over time, it learned:

Addition is a commutative mode.

Note:

It is not about memorizing answers.

It is learning:

Statistical relationships between symbols.

3. The third layer: Transformer can model "step structure"

Here comes the key.

GPT is based on Transformer (paper: Attention Is All You Need).

The Attention mechanism allows:

• The current token focuses on all previous tokens
• Establish multi-step logical relationships

For example:

若 x + 2 = 5
两边减 2
x = 3

The model will learn:

• "Subtract 2 from both sides" usually follows the equation of "+2"
• After deformation, both sides of the equal sign must be consistent.

This is a kind of "step structure learning".

4. Level 4: It has learned the “reasoning format”

A very important point is:

Models are not just about learning mathematical results;

It also learned:

Language routines for solving problems.

For example:

• "First of all..."
• "Therefore..."
• "From this we can conclude..."
• "You can get it by substitution..."

When you ask:

Solve the equation 3x + 6 = 12

If the model directly outputs the answer, it is easy to make mistakes.

But if it outputs:

3x + 6 = 12
两边减 6
3x = 6
x = 2

The accuracy will be greatly improved.

This is called:

Chain-of-Thought

Research found:

When the model scale is large enough,

Just prompt "Please think step by step",

Mathematical skills will be significantly improved.

This is an "emergent ability."

5. The fifth layer: It forms an “abstract representation” internally

Transformer will encode numbers, operators, and variables into high-dimensional vectors.

These vectors are not simple numbers.

They include:

• Operational relationship
• grammatical role
• logical structure

For example, in vector space:

2 + 3 ≈ 3 + 2

It'll be close.

Both sides of the equation "align" in vector space.

This means:

Models internally form representations of algebraic structures.

6. But does it really “count”?

Let’s be clear here.

GPT is not a traditional calculator.

It doesn't:

• precise arithmetic unit
• symbolic reasoning engine
• mathematical proof system

What it does is:

Predict the most reasonable next symbol based on probability.

So:

• Decimal operations are easy to make mistakes
• Multiplication of large integers is prone to errors
• Super complex proofs will collapse

This means:

It is not calculated algorithmically.

It is approximated using "statistical structures".

7. Why is mathematics better when the scale is larger?

This involves an important phenomenon:

Scaling Law

When the number of parameters changes from millions → billions → tens of billions → hundreds of billions:

The model suddenly starts:

• Ability to perform multi-step reasoning
• Understand abstract variables
• Make a simple proof

This is called:

Emergent Ability

It is not added artificially.

Rather, it occurs naturally when the model is large enough.

8. Methods to truly improve mathematical abilities

Today's advanced models (such as GPT-4) often combine:

1. pre-training
2. Thought chain tips
3. reinforcement learning
4. Tool calls (such as calculator)

When the model calls the calculator:

It’s no longer about “guessing the answer”;

Instead:

Reasoning + precise calculation

This is why modern agent architectures incorporate tools.

9. An intuitive analogy

Imagine a student:

• Read millions of questions
• Read all the steps to solve the problem
• Be familiar with all expressions

When you give him a new question:

He will match similar question types in his mind,

Then copy the steps.

This is how GPT does math.

10. Final summary

GPT relies on five things to learn mathematics:

1. Massive mathematics text training
2. Learn symbolic patterns
3. Attention modeling step dependencies
4. Learn "reasoning expression format"
5. Emergent capabilities brought about by large-scale

But it is not a traditional symbolic computing system.

It's more like:

An extremely proficient "pattern reasoning machine".