StepFun released Step3 on July 25, 2025 — the arXiv technical report is titled "Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decoding." The title is the thesis. Step3's architecture is built around a specific question: how do you run a 321B multimodal model at frontier performance without frontier inference costs?

The answer is two co-designed innovations that operate at the hardware level, not just the model level. Multi-Matrix Factorization Attention (MFA) cuts per-token attention compute to 22% of DeepSeek V3's cost. Attention-FFN Disaggregation (AFD) physically separates attention and FFN computation onto specialized GPU subsystems. The combined result: 4,039 tokens per second per GPU throughput — unusual for a model this size. On the benchmark side, Step3 leads the open-source VLM category on MMMU (74.2%), MATH-Vision (64.8%), and most other vision-language reasoning benchmarks at release.

What's New: MFA + AFD Co-Design

What's New

Step3's architecture is built around decoding efficiency as a first-order design constraint. Most frontier model architectures optimize for benchmark performance first and then work backward to deployment efficiency. Step3 reverses this: the model architecture and deployment system are co-designed from scratch.

Multi-Matrix Factorization Attention (MFA) is the model-side innovation. Standard multi-head attention becomes computationally expensive at large scale because the KV cache grows with sequence length. MFA factors the attention matrices to reduce the KV cache footprint and attention FLOPs simultaneously. The key metric: MFA's per-token attention cost is approximately 22% of DeepSeek V3's — achieved by down-projecting query dimensions from the full hidden dimension (7168) to a lower rank (2048), followed by normalization and up-projection to 64×256.

Attention-FFN Disaggregation (AFD) is the system-side innovation. In standard transformer inference, attention layers and FFN (feed-forward network) layers execute sequentially on the same GPU memory. AFD physically decouples these: attention and FFN layers run on specialized, physically separate GPU subsystems. The disaggregated architecture is enabled by StepMesh, StepFun's custom GPUDirect RDMA library that handles inter-subsystem communication. For MoE models specifically, FFN layers are sparse (only a subset of experts active per token) while attention is dense — ADG lets each subsystem specialize for its compute pattern.

The combined effect of MFA + AFD: Step3 achieves 4,039 tokens per second per GPU throughput, making a 321B-parameter model economically viable for production deployment at scale.

Architecture: 321B MoE, 61 Layers, MFA Attention

Architecture

Full architecture specifications from the technical report and HuggingFace model card:

Config Value
Total parameters (VLM) 321B
Total parameters (LLM only) 316B
Activated parameters per token 38B
Number of layers 61 (including 5 dense layers)
Hidden dimension 7168
Attention mechanism MFA (Multi-Matrix Factorization Attention)
Low-rank query dimension 2048
Query heads 64
Head dimension 256
Number of experts 48
Selected experts per token 3 routed + 1 shared
Max context length 65,536 tokens (64K)
Tokenizer DeepSeek V3 tokenizer
Vision encoder 5B parameters (separate from LLM)

Training data:

The 5B vision encoder is separate from the LLM component and enables the multimodal grounding that differentiates Step3 from pure language models. MoE layers are applied to all FFNs except the first four and the last layer.

Benchmarks: Open-Source VLM SOTA at Release

Benchmarks

Step3 benchmarks against open-source VLMs and proprietary frontier models. Key results:

Vision-Language:

Reasoning and Math:

Proprietary comparison (for context):

The asterisked (*) scores in Step3's benchmark table indicate numbers reproduced by StepFun under the same evaluation conditions — not taken from original papers.

Throughput: 4,039 Tokens/s/GPU

The throughput result is the technical headline. 4,039 tokens per second per GPU for a 321B model is meaningfully higher than comparable open-source frontier models, and the MFA + AFD co-design explains why.

The LinkedIn summary (Embedded LLM analysis) notes the breakdown:

The practical implication: Step3 at 321B can be run in production at token generation speeds more typical of 70B-class models.

Access and Availability

Step3 is fully open-source:

Context window: 65,536 tokens (64K). The model accepts text and image inputs. Note: the vision encoder (5B params) is included in the VLM total of 321B; image input is supported via the VLM endpoint.

What It Means for Developers

Step3 is the model to reach for when both vision-language capability and inference throughput are requirements. The MMMU 74.2% and MATH-Vision 64.8% scores lead the open-source VLM category as of its July 2025 release. The 4,039 tokens/s/GPU throughput means the model can support production-scale deployments without disproportionate hardware costs.

The MFA + AFD co-design is also an engineering signal: StepFun built a custom RDMA library (StepMesh) to make disaggregated inference work. This level of infrastructure co-design is unusual for a lab of StepFun's scale — it suggests a longer-term commitment to the performance-efficient model architecture direction rather than a one-off release.

For developers choosing between open-source multimodal models:

The 64K context ceiling is a constraint for long-document processing use cases; models with 128K+ context (Llama 4 Scout, Qwen3 series) may be preferable there. For vision-heavy reasoning tasks within 64K context, Step3 is the open-source benchmark at release.

Bottom Line

Step3 delivers open-source VLM SOTA performance — MMMU 74.2%, MATH-Vision 64.8%, AIME 82.9% — at 4,039 tokens/s/GPU throughput. The MFA + AFD co-design directly addresses the cost barrier that makes 300B+ models impractical at production scale. MIT license, HuggingFace weights available. The technical architecture (StepMesh RDMA library, disaggregated GPU subsystems) signals that StepFun built this as infrastructure, not a demo.

Resources

Step3 weights available under MIT license at huggingface.co/stepfun-ai/step3. API access via StepFun platform and OpenRouter.