Zhipu AI launched GLM-5.1-highspeed on May 22, 2026 — an API variant of their GLM-5.1 flagship model delivering 400 tokens per second in production. That's a claim worth examining carefully, because 400 tokens/s from a 754-billion parameter MoE model is not a benchmark cherry-pick or a small-model number: it's the same GLM-5.1 that scores 58.4% on SWE-Bench Pro, running at a speed that makes real-time coding collaboration feel genuinely interactive.

The technical story behind that number is about where inference latency actually lives, and why the standard answer — "optimize the kernels" — stopped being sufficient.

The Problem: Why Fast Hardware Runs Slow Models

GLM-5.1 High-Speed demo

An 8×H200 server has aggregate memory bandwidth approaching 38 TB/s. GLM-5.1 activates roughly 42 GB of parameters per decode step. Do the math: the theoretical upper bound on token generation is close to 1,000 tokens/s. Real production systems run at tens of tokens per second.

The gap isn't the GPU. It's everything happening between GPU operations.

Zhipu's TileRT team documented this precisely: GPU utilization metrics looked normal, theoretical FLOPS weren't bottlenecked, but per-token latency stayed high. The profiler showed a recurring pattern — kernels finishing before they were fully warmed up. The actual throughput limiter wasn't any single matrix multiply; it was the startup, synchronization, and memory overhead happening at every kernel boundary.

The conventional inference execution model works like this: a model is decomposed into independent operators, each launched separately, synchronized, and completing a full memory round-trip before the next one starts. In training workloads with large batches, this overhead gets amortized — the compute time per kernel is long enough that the boundary costs are invisible. In decode at latency-first settings (small batch, single token, multi-GPU TP), kernel lifetimes drop to microseconds, and those boundary costs dominate.

This is what TileRT calls the "execution gap": latency consumed not by computation but by the infrastructure connecting computations.

TileRT: Persistent Kernel Architecture

The core insight TileRT is built on: if runtime scheduling is entering the critical latency path, stop optimizing the runtime. Redesign the execution model.

TileRT compiles the entire model into a single persistent Engine Kernel at AOT (ahead-of-time) time. During inference:

TileRT tile-level task scheduling

Inside this persistent kernel, operators aren't executed sequentially. They're decomposed into tile-level micro-tasks — a granularity smaller than an operator, treated as the basic unit of scheduling. Compute, asynchronous IO, and inter-GPU communication are all represented as tiles and scheduled to run in a continuously advancing pipeline.

Warp specialization is how this pipeline stays full. Different warp groups within a CTA take on different roles simultaneously: some handle asynchronous data movement, some perform tensor computation, some manage communication overlap. In the conventional model, execution sequences through load → barrier → compute → barrier — each step waiting for the previous. In TileRT, those stages overlap at tile granularity. Intermediate results don't touch global memory between steps; they stay in registers, shared memory, and L2 cache and flow directly into the next tile's computation.

The multi-GPU dimension applies the same logic at scale. TileRT extends warp specialization from intra-SM coordination to the full 8-GPU NVLink topology. Rather than all GPU ranks executing identical code (the standard TP pattern), TileRT assigns different ranks specialized roles based on compute density and data dependency profiles — persistent heterogeneous workers across the whole inference cluster.

The published result: end-to-end 400 tokens/s in production, stable — not a peak measurement.

GLM-5.1: The Model Being Accelerated

Before GLM-5.1-highspeed makes sense as a capability, GLM-5.1 itself needs context. Released on April 7, 2026 under the MIT license, GLM-5.1 is a 754B-parameter MoE with 40B active parameters per token, 202,752-token context window (200K effective), and up to 131K output tokens.

The engineering focus is long-horizon agentic tasks. Unlike models that hit a plateau after initial task analysis, GLM-5.1 is explicitly designed to stay productive over hundreds of rounds and thousands of tool calls. On VectorDBBench (vector database optimization), GPT-5.4 plateaued around iteration 50; GLM-5.1 continued improving through 600+ iterations. The same pattern appeared in GPU kernel benchmarks and open-ended web application builds.

Benchmark numbers from the GLM-5.1 release:

The high-speed variant preserves GLM-5.1's full capability profile. This is the combination that makes the announcement technically significant: 400 tokens/s isn't a small distilled model — it's the flagship, accelerated.

What 400 Tokens/s Changes Operationally

The GLM-5.1-highspeed docs describe four target scenarios where the latency reduction is functionally transformative:

AI coding: Multi-turn code generation and large engineering refactors involve repeated model calls. When each call takes seconds, latency compounds through the task graph. At 400 tokens/s, a complex webpage can be generated in ~30 seconds; agent clusters can handle multi-persona parallel responses in real time. The demo shows a Coding Agent interaction pattern that looks like pair programming rather than waiting for a batch process.

Real-time interaction: Game generation, dynamic UI construction, live content feedback — contexts where model output needs to track user input continuously rather than arriving after a perceptible delay.

Business decision support: Real-time data analysis, operations Q&A, multi-agent parallel scenario modeling. The functional change is whether the model can participate in an active decision process or only deliver results after the fact.

Voice interfaces: In TTS pipelines, the language model's decode speed determines whether the voice response sounds natural or has an audible delay between recognition and synthesis. At 400 tokens/s, the model's generation is no longer the bottleneck in the audio pipeline.

There's a second-order effect that Zhipu explicitly notes: Test-Time Scaling becomes more useful when inference is fast. In a fixed latency budget, faster tokens-per-second means more rollouts, deeper reasoning paths, and stronger self-verification before the response arrives. Speed isn't only about user experience — it's about what the model can do within a time budget.

Access and Availability

GLM-5.1-highspeed is currently in selective enterprise access on Zhipu's BigModel MaaS platform — not generally available. API model name: glm-5.1-highspeed. Documentation at docs.bigmodel.cn.

Capabilities: streaming output, Function Call, context caching, structured output (JSON), MCP tool integration. Context: 200K tokens. Max output: 128K tokens.

GLM-5.1 base model: open-weight under MIT license on HuggingFace and GitHub. TileRT blog (technical detail on the inference architecture): tilert.ai/blog.

What It Means for Developers

The TileRT architecture description is more useful than the benchmark number for developers evaluating whether and how to use this. The persistent kernel approach — compile-time graph unrolling, resident GPU execution, tile-level pipeline, warp specialization — is a fundamentally different inference execution model from vLLM-style operator scheduling. It specifically targets the latency profile of real-time decode at small batch sizes, which is the regime that matters for interactive applications.

For developers building systems where the model needs to function as a real-time collaborator rather than a batch processor — coding agents, voice interfaces, interactive UI generation, live decision support — the question is whether the model's decode speed is the bottleneck. At typical frontier model speeds (30-80 tokens/s), it often is. At 400 tokens/s with full GLM-5.1 capability, it usually isn't.

The enterprise-only initial access is a constraint. The open-weight GLM-5.1 base model is available for self-hosted deployment, but the TileRT engine that produces the 400 tokens/s figure is not open-sourced — the inference acceleration is part of Zhipu's API service. Developers wanting the speed without the managed API will need to run their own inference optimization stack against the open weights.

Bottom Line

GLM-5.1-highspeed is the first production deployment where a Chinese frontier MoE reaches 400 tokens/s in a stable, non-peak measurement. The number is real because the architecture behind it — TileRT's persistent kernel, tile-level pipeline, warp-specialized multi-GPU execution — is attacking the right problem: not individual kernel speed, but the execution gap between kernels.

For the AI coding and real-time interaction use cases Zhipu is targeting, the combination of SWE-Bench Pro 58.4% and 400 tokens/s is a qualitatively different capability than those numbers would be independently.

Resources

GLM-5.1-highspeed is available to select enterprise customers via platform.bigmodel.cn. GLM-5.1 base weights available under MIT license at huggingface.co/zai-org/GLM-5.1.