At NVIDIA GTC, Moonshot AI CEO Yang Zhilin made a statement that stood out among the demos and roadmaps: "We scaled and open-sourced." Then he said it again. Three times, each paired with a different name — MuonClip, Kimi Linear Attention, and Attention Residuals.
The last one caught Andrej Karpathy's attention. Elon Musk liked it on X. And if you look closely at what Attention Residuals actually does, the reaction is understandable: it replaces residual connections — the architectural pattern that has been unchanged in deep learning since ResNet in 2015 — with something meaningfully different.
This isn't theoretical. The paper reports Block AttnRes achieves the same loss as a baseline model that spends 1.25× more compute. Inference overhead: under 2%. It's drop-in replaceable with existing architectures.
The timing is worth noting: GTC announcement, active $1B funding round at $18B valuation, three consecutive architecture open-sources. Kimi is making a very specific kind of statement about what it is.
The 10-Year Problem with Residual Connections
Residual connections were introduced in ResNet (2015) to solve gradient vanishing in deep networks. The formula is simple: each layer's output equals the previous layer's output plus the current layer's transformation. This preserves an identity path — a "straight-through" channel that prevents gradient signals from dying in deep networks.
It worked, and it became the foundation Transformer was built on. The problem is that the solution came with an implicit constraint nobody questioned for a decade.
In standard PreNorm Transformer training, every layer's output contributes equally to the final state. Expand the recurrence mathematically, and the contribution weight for each layer is exactly 1, regardless of depth. The 3rd layer's features and the 97th layer's features carry identical weight in the final representation.
The paper calls this PreNorm dilution: as networks deepen, each new layer's contribution gets increasingly diluted by the accumulated sum of all previous layers. The total grows roughly linearly with depth L. Later layers have to produce larger-magnitude outputs just to maintain any influence — they're fighting against a growing pile of historical states with equal voting rights.
The analogy the paper uses: imagine meeting notes that are never deleted. Every meeting, new notes are appended. Old notes remain unchanged. The document grows indefinitely, and each new entry becomes a smaller fraction of the total — not because earlier entries were more important, but because there are more of them.
This problem has been known. It went unfixed because residual connections were simple, stable, and zero-cost. And everyone was busy fixing other things: attention mechanisms, activation functions, normalization strategies, MoE routing. The residual stream itself was treated as untouchable infrastructure.
Attention Residuals: Depth-Wise Attention
The core insight in Attention Residuals is a direct analogy to what Transformers did for sequence modeling.
In RNN-style sequence models, a single compressed state rolls forward in time — each position only sees a compressed summary of everything before it. Transformers replaced this with attention: every position can directly attend to all historical positions, dynamically weighting how much to take from each.
Residual connections do the sequence-model thing, but in the depth dimension. Every layer inherits one compressed total — the accumulated sum of all previous layers. Attention Residuals replaces this fixed equal-weight sum with a learnable, input-dependent attention across depth.
The mechanism: each layer l uses a learnable query vector w_l to compute attention scores against all previous layers' outputs. After softmax normalization, these scores become the weights for aggregating historical layer representations. The parameter count added per layer: just a single d-dimensional vector. Minimal overhead.
This means: layer 50 can look at all previous 49 layers and dynamically decide — for this particular input — which earlier representations are most relevant. Different inputs → different aggregation patterns at the same layer. Early semantic features that would normally get diluted by 50 layers of accumulated summation can be directly retrieved.
Full AttnRes (every layer attends to all history) is the theoretically cleanest version but has O(L²d) complexity and requires keeping all intermediate activations alive — expensive at scale with activation recomputation and pipeline parallelism.
Block AttnRes is the practical implementation: layers are grouped into N blocks. Within blocks, standard residual accumulation runs as before. Between blocks, attention aggregates block-level summary representations. Memory and communication cost drops to O(Nd). Empirically, ~8 blocks captures most of the Full AttnRes benefit.
Numbers: 1.25× Compute Efficiency
The paper's scaling law result: training a baseline model to the same loss as Block AttnRes requires approximately 1.25× more compute. In other words, Block AttnRes achieves in one training run what the baseline needs 1.25× the budget to reach.
The loss curves show Block AttnRes tracking closely with Full AttnRes across scales, both significantly below baseline. The improvement holds on downstream benchmarks — AttnRes matches or exceeds baseline on nearly every evaluation in the paper's same-recipe pretraining comparison.
Inference overhead: under 2% latency increase. The paper includes two engineering optimizations that make this possible: cross-stage caching (reduces redundant communication in pipeline parallelism) and two-phase computation (block attention computed in parallel, then merged with sequential within-block processing).
The comparison to DeepSeek's mHC is explicit in the paper. mHC expands the residual stream width — multiple parallel streams with learned mixing matrices constrained by Sinkhorn-Knopp to Birkhoff polytope, adding ~6.7% training overhead. AttnRes changes the aggregation rule depth-wise. The paper notes these approaches are roughly orthogonal, but flags that Block AttnRes has approximately 1/6 the memory access overhead of mHC for comparable training cost. Two Chinese labs, two different angles, both attacking the same 11-year-old primitive at roughly the same time.
The Kimi Valuation Signal
Attention Residuals is one of three architectural contributions Yang Zhilin listed at GTC: MuonClip (optimizer), Kimi Linear Attention, and Attention Residuals — covering the optimizer, attention mechanism, and residual connection, respectively. The three lowest-level architectural components of a Transformer. All three open-sourced, all three production-validated at scale.
The funding numbers are striking: Kimi's valuation was $4.3B three months before GTC. At time of the GTC announcement, they were closing a $1B round at $18B — a 4× increase in three months. The company reached $10B+ valuation faster than Pinduoduo (3+ years from founding) or ByteDance (4+ years). Moonshot did it in about two years.
The commercial signal is real: K2.5's revenue in its first 20 days post-launch exceeded all of 2025 combined. On Stripe's global payment rankings, Kimi went from outside the top 100 to #9 in two months — ahead of most Western AI API platforms.
The architectural open-sourcing is not unrelated to this. When a lab open-sources changes to optimizer, attention, and residual connections all in sequence, and simultaneously hits hypergrowth revenue numbers, the signal to investors is specific: this is not a lab that fine-tunes models and wraps them in products. The "we scale and open-sourced" framing at GTC was a deliberate statement of competitive identity.
What It Means for Developers
Attention Residuals is available as open-source research code. It is not yet a shipping API feature — you can't call a Kimi endpoint and get AttnRes-trained outputs today in a way that's explicitly documented.
What developers can do:
Implement Block AttnRes in custom training runs. The mechanism is straightforward: N block-level query vectors, attention aggregation across block summaries, standard residual within blocks. The engineering optimizations (cross-stage caching, two-phase computation) are described in the paper. On H100-class hardware, the <2% inference overhead claim is achievable with those optimizations in place.
Watch what Kimi trains next. The architecture open-source pattern (MuonClip → Linear Attention → Attention Residuals) suggests each new Kimi model will incorporate these primitives. If AttnRes becomes standard in production Kimi training, its capability advantages will be reflected in whatever API they ship.
Cross-reference with TileKernels. DeepSeek's mHC now has production kernel code in TileKernels. The paper says AttnRes and mHC are orthogonal — combining them is theoretically possible. Nobody has shipped a model with both yet.
The deeper implication: if attention mechanisms can be applied in the depth dimension with results comparable to what self-attention did for the sequence dimension, we're at the beginning of a re-evaluation of what Transformer depth actually does. That's what Karpathy's comment pointed at. The architecture of current LLMs may not be near a fundamental ceiling — it may be that the depth axis hasn't been properly exploited yet.
Bottom Line
Kimi open-sourced a drop-in replacement for the standard residual connection that delivers 1.25× compute efficiency at under 2% inference overhead. It works by applying attention across the depth dimension of a Transformer — the same insight that made self-attention powerful in the sequence dimension, now applied vertically. The mechanism is simple, the engineering is solved, and the paper's results hold at scale.
At $18B valuation and accelerating revenue, Kimi is betting that architecture-level differentiation compounds. Attention Residuals is the most concrete evidence that the bet has technical substance.
Resources
- Attention Residuals Paper — arXiv preprint (check for updated link)
- Kimi Official Site — main product
- Moonshot AI — company homepage
- Kimi API Platform — API access ($0.60/M input, $2.50/M output for K2)
- DeepSeek mHC / TileKernels — orthogonal approach, now with kernel code
Kimi K2 is available via API at platform.moonshot.cn at $0.60/M input tokens, $2.50/M output. Attention Residuals research code is open-source; production API availability based on AttnRes-trained models is not yet confirmed.
