Shanghai AI Lab released InternLM3-8B-Instruct in January 2025, demonstrating that strong mathematical reasoning didn't require a 70B+ model. The 8B variant achieved 74.0% on MATH and 83.6% on MMLU-Pro — benchmark results that were competitive with Llama-3.1-70B at 8B inference cost.

The headline feature: Thinking Mode. InternLM3 can operate in two modes using the same model weights — as a reasoning model that produces extended chain-of-thought output, or as a direct-answer model without the overhead. This lets developers use one model for both latency-sensitive and accuracy-critical workloads.

What's New

What's New

Thinking Mode: InternLM3-8B supports a thinking_mode flag that switches the model between extended chain-of-thought reasoning and direct response. In thinking mode, the model produces <think>...</think> blocks with step-by-step reasoning before the final answer — similar to DeepSeek-R1's extended thinking, but in an 8B model. The same weights, two usage patterns.

8B reasoning efficiency: InternLM3 was specifically trained to maximize reasoning quality at the 8B scale. The training pipeline uses a mix of supervised fine-tuning on reasoning traces and RLHF to improve both accuracy and calibration. The result is an 8B model that substantially outperforms Llama-3.1-8B on math and science benchmarks.

Instruction following improvements: Beyond reasoning, InternLM3 focuses on reliable instruction following — following complex multi-step instructions, outputting in specified formats, respecting length constraints, and avoiding common failure modes like over-refusal. The RLHF pipeline specifically targets these production-critical behaviors.

Apache 2.0 license: All InternLM3 variants are Apache 2.0 — commercial use, fine-tuning, and redistribution. The 8B model runs comfortably on a single A100 or 3× A6000 GPUs.

Architecture

Architecture

InternLM3-8B uses a standard transformer architecture with several targeted improvements:

Specification Value
Parameters 8B
Architecture Dense transformer
Context window 32K tokens
Thinking Mode Yes (same weights)
License Apache 2.0
VRAM (inference) ~16GB

Thinking Mode implementation: When thinking_mode=True, the model generates a reasoning trace in <think> tags before the response. The model was trained with both thinking and non-thinking outputs — the same weights support both modes by conditioning the generation differently at the start. This is different from having two separate models; the reasoning capability is baked into the weights.

Group Query Attention: InternLM3 uses GQA (Group Query Attention) — sharing key/value heads across query groups — which reduces the KV cache memory footprint by ~4× compared to multi-head attention. At 8B scale with 32K context, this makes the model practical on 16GB VRAM with room for a reasonable context length.

InternLM3 family: Beyond the 8B-Instruct, the InternLM3 family includes:

Benchmarks

Benchmarks

InternLM3-8B-Instruct vs. comparable models (January 2025):

Benchmark InternLM3-8B Llama-3.1-8B Llama-3.1-70B Qwen2.5-7B Notes
MATH 74.0% 51.9% 68.0% 75.5% Mathematical reasoning
MMLU-Pro 63.6% 43.5% 66.4% 62.3% Knowledge breadth
GPQA-Diamond 42.4% 24.9% 46.7% 38.4% Graduate science
HumanEval 78.0% 72.6% 80.5% 84.8% Code generation
IFEval 85.3% 71.4% 82.5% 78.2% Instruction following

MATH 74.0% at 8B is the headline number — +22.1pt over Llama-3.1-8B (51.9%) and only -1.5pt behind Qwen2.5-7B (75.5%). For an 8B model, 74% on MATH represents a genuine step change from the previous generation of small models.

IFEval 85.3% (instruction following) is notably high — the RLHF work on instruction compliance shows clearly here. For production applications where the model needs to reliably follow structured instructions, InternLM3's IFEval score is competitive with 70B models.

GPQA-Diamond 42.4% vs Llama-3.1-70B's 46.7% shows the remaining gap to larger models on graduate-level science — the 8B capacity limit shows up in complex multi-step scientific reasoning.

Thinking Mode in Practice

Standard response:

User: What is the integral of x^2?
Assistant: x³/3 + C

Thinking mode response:

User: What is the integral of x^2?
Assistant: <think>
To find ∫x² dx, I apply the power rule:
∫xⁿ dx = xⁿ⁺¹/(n+1) + C for n ≠ -1
Here n=2, so: x²⁺¹/(2+1) = x³/3
Verify by differentiating: d/dx(x³/3) = x² ✓
</think>
x³/3 + C

Thinking mode adds latency but improves accuracy on harder problems. The same API endpoint, toggled by parameter — developers choose based on workload requirements.

What It Means for Developers

70B math quality at 8B cost: MATH 74.0% puts InternLM3-8B in range of Llama-3.1-70B (68.0%) and competitive with Qwen2.5-7B (75.5%). For math-heavy applications — tutoring, problem solving, scientific computing assistants — InternLM3-8B provides near-70B quality at 8B inference cost.

One model, two modes: Thinking Mode eliminates the need to maintain separate reasoning and non-reasoning models. Fast responses for simple queries, extended reasoning for hard problems — same deployment, same weights, same API.

IFEval for production reliability: 85.3% IFEval means the model reliably follows complex formatting instructions, length constraints, and output specifications. Important for production pipelines where structured output reliability is a requirement.

InternVL3 backbone: InternLM3-8B is the language backbone for InternVL3's smaller vision variants. If you're building on InternVL3, you're already building on InternLM3 capabilities.

Availability

Bottom Line

InternLM3-8B demonstrated that reasoning model training at 8B scale could close most of the gap to 70B models on mathematical reasoning. MATH 74.0% and IFEval 85.3% show strong math and instruction following at a model size that's deployable on a single GPU.

Thinking Mode's implementation — same weights, toggled behavior — is a practical design choice: one deployment serves both latency-sensitive and accuracy-critical workloads without model management overhead.

Resources

InternLM3 models are available on HuggingFace under Apache 2.0. Commercial use including fine-tuning.