DeepSeek released V3 on December 26, 2024 — quietly, while most of the Western AI industry was on Christmas break. The announcement included a full technical report, weights available on HuggingFace, and an API priced at $0.27/M input tokens.
The model matched GPT-4o and Claude 3.5 Sonnet on most public benchmarks. At a fraction of their API cost. As an open-weights model, freely available.
Three weeks later, DeepSeek would use V3 as the base for R1 — the reasoning model that triggered Nvidia's biggest single-day market loss ever. But V3 is worth understanding on its own terms, because its architectural innovations are what made R1 possible.
The Numbers
- 671B total parameters, 37B active per token (MoE)
- 14.8 trillion tokens pre-training data
- 128K context window
- Reported training cost: ~$5.5M on 2048 H800 GPUs (14.8T tokens / 2.788M GPU-hours)
- License: DeepSeek License (commercial use permitted with conditions)
The $5.5M training cost claim is the number that generated the most coverage — and the most skepticism. The consensus post-analysis: $5.5M is plausibly accurate for the final pre-training run, excluding experiments, failed runs, and the earlier V2 model development. The full cost of the entire lineage that produced V3 is higher — but probably not an order of magnitude higher.
At $0.27/M input tokens, V3 API pricing was approximately 9× cheaper than GPT-4o ($2.50/M) at launch. Cache hits cost $0.07/M.
Two Architectural Innovations
V3 builds on the V2 architecture (MLA + DeepSeekMoE) and adds two novel techniques:
1. Auxiliary-Loss-Free Load Balancing
Standard MoE training has a routing collapse problem: without intervention, the router tends to send most tokens to a few "popular" experts, starving the others and wasting model capacity. Previous solutions added auxiliary losses to penalize imbalance — but auxiliary losses hurt performance by pulling the model away from its primary objective.
DeepSeek's solution: bias correction. During training, maintain a running count of how much traffic each expert receives. At each step, add a small positive bias to the routing logits of underloaded experts and a negative bias to overloaded ones. The bias adjusts automatically, expert utilization stays balanced, and zero auxiliary loss penalty is incurred.
The result: better expert utilization and stronger final model quality compared to auxiliary-loss approaches at the same training budget.
2. Multi-Token Prediction (MTP)
Standard autoregressive training predicts one token at a time. V3 adds MTP heads that also predict the next 2 tokens in parallel with the primary prediction. The auxiliary MTP loss is used during training and discarded at inference.
This serves as a form of data augmentation — the model sees denser supervision signals from the same token stream. Empirically, it improves benchmark performance, particularly on tasks requiring lookahead reasoning. At inference time, MTP heads enable speculative decoding, where predictions for multiple tokens are generated and verified in parallel, increasing throughput.
V3 runs at roughly 60 tokens/second on DeepSeek's API at launch — comparable to GPT-4o output speeds, achieved with the full 671B MoE model.
Benchmarks
At December 2024 release, comparing against the then-current frontier:
| Benchmark | V3 | GPT-4o | Claude Sonnet 3.5 | Notes |
|---|---|---|---|---|
| MMLU | 88.5% | 87.2% | 88.3% | Knowledge |
| HumanEval | 89.3% | 90.2% | 92.0% | Coding |
| MATH-500 | 90.2% | 76.6% | 78.3% | Math |
| Codeforces | 51.6% | 23.1% | 20.6% | Competitive coding |
| AIME 2024 | 39.2% | 9.3% | 16.0% | Hard math |
The math story is the standout: V3 scores 90.2% on MATH-500 versus GPT-4o's 76.6%. On hard math (AIME 2024), V3 at 39.2% far exceeds both GPT-4o (9.3%) and Claude Sonnet (16.0%). This is not the reasoning model — that's R1. V3 is the base model that achieves these results through pretraining and SFT alone.
The FP8 Training Bet
V3 is trained in mixed FP8 precision — a more aggressive low-precision format than the BF16 standard used by most frontier models. FP8 reduces memory bandwidth requirements and increases GPU arithmetic throughput, but historically caused training instability.
DeepSeek's solution: apply FP8 only to compute-intensive operations (GEMM) while keeping accumulation and scaling in higher precision. Combined with tile-level quantization to reduce quantization error, this enables stable FP8 training at scale.
The payoff: H800 GPUs (which lack H100's full NVLink bandwidth) can achieve comparable effective throughput to H100 setups for this workload. This matters geopolitically — H800s are the highest-spec Nvidia chips available to Chinese companies under US export controls.
Why V3 → R1
When DeepSeek released R1 three weeks after V3, they initialized R1 from V3 checkpoints and applied reinforcement learning on top. The R1 paper explicitly notes that V3's architecture and training quality were prerequisites for R1's emergent reasoning behaviors.
Specifically: MLA's efficient KV cache management allows V3 (and by inheritance R1) to run long reasoning chains within its 128K context window without the memory overhead that would make extended thinking economically unviable at the API level.
V3 is the infrastructure. R1 is the application built on top of it.
What It Means for Developers
General-purpose coding and reasoning: V3 is competitive with GPT-4o for most practical tasks at ~9× lower cost. For applications where you're currently using GPT-4o or Claude Sonnet as a general-purpose backend, V3 is the first open-weights alternative worth serious evaluation.
Self-hosting economics: 671B MoE with 37B active parameters requires ~8×A100 80GB for reasonable throughput. At $8/hour for 8×A100 on major cloud providers, this is approximately $0.0067/M tokens at 2000 tokens/second — cheaper than the API for high-volume workloads.
Fine-tuning: The MIT-adjacent license permits fine-tuning and redistribution with attribution. V3's strong base makes it the preferred starting point for domain-specific fine-tunes — multiple labs have published V3-based fine-tunes for medicine, law, and coding.
V3 vs. V3.1 vs. V4: DeepSeek subsequently released V3.1 (March 2025, improved context and coding) and V4 (April 2026, multimodal). The V3 architecture in this article was superseded — but the architectural innovations (MTP, auxiliary-loss-free balancing) carried forward into all successors.
Bottom Line
V3 did three things simultaneously that seemed incompatible: matched frontier-model benchmark performance, offered open weights, and priced API access at $0.27/M tokens. The underlying mechanism — efficient MoE training via novel load balancing, FP8 compute, and MTP — represents genuine engineering advances, not shortcuts.
The model's importance is as much about timing as capability. Released 26 days before R1, V3 established that DeepSeek's training infrastructure could produce frontier-quality models at competitive cost. R1 then demonstrated that reasoning capability could be layered on top of that base through RL alone.
Resources
- Technical Report (arXiv 2412.19437) — architecture, training, and full benchmarks
- GitHub — code and model weights
- HuggingFace — V3 — model weights (671B)
- DeepSeek API — API access
- DeepSeek Chat — direct access
DeepSeek-V3 weights are available at huggingface.co/deepseek-ai/DeepSeek-V3. API access at platform.deepseek.com at $0.27/M input tokens (cache miss), $0.07/M (cache hit).
