Reasoning Model Architecture: How o3, DeepSeek-R1, and Thinking Models Work (2026)

Reasoning models — OpenAI o3, DeepSeek-R1, Google Gemini Flash Thinking, and Anthropic's extended-thinking Claude variants — represent a structural shift in how LLMs generate answers. Instead of producing a response in a single forward pass, they generate an internal chain of thought first, then produce the final answer. The result is dramatically better performance on math, code, and multi-step reasoning tasks — at the cost of higher latency and token spend.

Understanding reasoning model architecture helps you decide when to use them, how to route requests between reasoning and standard models, and how to diagram the systems you build with them.

The core architectural shift: thinking tokens

Standard LLMs (GPT-4o, Claude 3.5 Sonnet, Gemini Pro) produce output tokens directly. Given a prompt, the model runs a single autoregressive pass to generate the response. Reasoning models insert a thinking phase before the response:

1. Thinking phase: The model generates a scratchpad of internal reasoning — hypotheses, intermediate calculations, sub-problem decomposition, self-correction. These tokens are typically hidden from the end user but consume API quota and add latency. In some APIs (Anthropic extended thinking), they are returned as a separate thinking block.
2. Response phase: After the thinking budget is exhausted or the model decides it has reasoned enough, it generates the final answer conditioned on the thinking tokens. The response is typically shorter and more accurate than a standard model would produce without thinking.

The thinking tokens act as working memory — a scratch space where the model can be wrong, backtrack, and try again before committing to an answer. This is architecturally analogous to letting a human write rough notes before writing a final document.

Training architecture: how reasoning is learned

Reasoning capability is trained differently from standard instruction tuning. The two dominant approaches are reinforcement learning with process supervision and large-scale synthetic chain-of-thought generation.

Process reward models (PRMs)

A process reward model assigns a correctness score to each step of a reasoning chain, not just the final answer. This is in contrast to an outcome reward model (ORM), which only scores the final result.

• PRM advantage: The model learns which reasoning steps are valid, even when the final answer happens to be correct for the wrong reasons. This produces more robust reasoning.
• PRM challenge: Step-level labels require human annotators to evaluate intermediate reasoning steps, which is expensive. OpenAI used large-scale human annotation; DeepSeek used synthetic data generation to scale labeling.
• Training pipeline: A base LLM generates candidate reasoning chains → the PRM scores each step → the scores are used as rewards in RL fine-tuning (typically PPO or GRPO) → the fine-tuned model generates better chains → repeat.

Monte Carlo Tree Search (MCTS) at inference time

Some reasoning models (notably OpenAI o-series) use a search procedure at inference time rather than greedy decoding:

• The model generates multiple candidate next reasoning steps (branches).
• A verifier or value model scores each branch.
• The highest-scoring branch is expanded further (tree search).
• This continues until a terminal state (complete answer) is reached.

MCTS dramatically increases the compute used at inference time in exchange for better answers — this is the "test-time compute scaling" phenomenon. More compute at inference = better results, independently of model size.

Diagram prompt: reasoning model training pipeline

Inference architecture: test-time compute scaling

Standard LLMs have a fixed inference cost per output token. Reasoning models scale inference compute dynamically based on problem difficulty — harder problems consume more thinking tokens. This creates a new architectural consideration: compute budgeting.

Thinking budget control

Most reasoning APIs expose a parameter to control thinking depth:

• Anthropic extended thinking: budget_tokens sets the maximum thinking token allocation. Higher budgets = better accuracy on hard problems, higher cost and latency.
• OpenAI o-series: reasoning_effort (low / medium / high) controls how many internal reasoning tokens are used. High effort is best for complex coding and math; low effort suffices for most writing tasks.
• Dynamic routing: Well-architected LLM applications route easy requests to fast/cheap standard models and complex requests to reasoning models. A classifier or heuristic (query length, task type, confidence score from a cheap model) decides the routing.

Diagram prompt: reasoning model inference with dynamic routing

Reasoning model vs. standard model: when to use each

Architectural patterns for building with reasoning models

Pattern 1: Thinking-gated verification

Use a standard model for initial generation, then a reasoning model to verify the output before returning it to the user. This is especially effective for high-stakes outputs like financial calculations, legal analysis, or security reviews where a wrong answer is worse than a slow one.

Pattern 2: Decompose-then-reason

Use a standard model to decompose a complex task into sub-problems, then use a reasoning model on each sub-problem. The reasoning model handles the hard parts; the standard model handles orchestration. This reduces total thinking token spend compared to feeding the whole problem to a reasoning model.

Pattern 3: Streaming thinking for transparency

Anthropic's extended thinking API returns thinking tokens in a streaming response. Applications can surface these to users as a "showing work" feature — displaying the model's reasoning process while the final answer is being generated. This builds trust in high-stakes domains like medicine, law, and engineering.

Frequently asked questions

What is a reasoning model in LLMs?

A reasoning model is an LLM that generates an extended internal chain of thought before producing its final answer. The internal reasoning (sometimes called thinking tokens or scratchpad) allows the model to break down complex problems, check intermediate steps, and backtrack when it detects errors — producing more accurate answers on hard tasks at the cost of higher latency and token usage.

How is a reasoning model different from chain-of-thought prompting?

Chain-of-thought (CoT) prompting is a technique where you ask a standard model to "think step by step" in its output. The model generates visible reasoning as part of its answer. Reasoning models have CoT built into their architecture at a deeper level — the thinking is generated before the answer, with dedicated training (RL with process rewards) to make the thinking genuinely useful rather than just plausible-looking. A reasoning model trained with RL is more reliable than a standard model prompted to think step by step.

When should I use a reasoning model in my architecture?

Use reasoning models when: (1) the task has multiple interdependent steps where errors compound (math, multi-hop code, constraint satisfaction); (2) correctness matters more than speed; (3) you need the model to self-verify before committing. Avoid reasoning models for high-throughput, low-latency tasks like classification, simple Q&A, or real-time chat. Build a routing layer that selects the right model based on task complexity rather than always using the most expensive model.

How do I diagram a reasoning model system?

Describe the thinking phase and response phase as separate steps, show the thinking token budget as a configurable parameter, and illustrate how results flow through verification before reaching the user. Use ArchitectureDiagram.ai and paste one of the prompts above to generate a diagram instantly.

Related guides: LLM architecture diagrams, LLM evaluation architecture, AI guardrails architecture, and multi-agent orchestration patterns.