Zhipu AI released CogVideoX in August 2024, making it one of the first open-source video generation models to use a Diffusion Transformer (DiT) architecture similar to Sora's design. Where most open video models had been using UNet architectures, CogVideoX demonstrated that expert transformer-based video generation could be done with open weights.
The result was a model that competitive with commercial video generators on text-to-video benchmarks — while being fully open under Apache 2.0 and fine-tunable for specific domains.
What's New
Expert Adaptive LayerNorm (Expert AdaLN): CogVideoX introduces Expert AdaLN — a version of Adaptive LayerNorm where separate expert networks handle text conditioning for visual and temporal dimensions independently. Standard AdaLN applies the same learned scale and shift to all dimensions; Expert AdaLN learns different conditioning for spatial (what's in the frame) vs. temporal (how things move) aspects of the generation. This improves temporal coherence: objects move consistently across frames because the temporal conditioning is handled explicitly.
3D Full Attention: Unlike models that use factorized attention (separate spatial and temporal attention blocks), CogVideoX applies full self-attention across the 3D space-time volume. Every frame token attends to every other frame token across both spatial and temporal dimensions simultaneously. This is computationally expensive but enables global temporal consistency — the model can learn relationships between distant frames directly rather than through chained factorized attention.
Text-image-video joint training: CogVideoX is trained with text, image, and video data jointly rather than video-only. The joint training allows the model to leverage the richer semantic representations learned from image-text pairs for video generation, similar to how language models benefit from large text pretraining before fine-tuning on specific tasks.
Two sizes: CogVideoX-2B and CogVideoX-5B. The 5B is the headline benchmark model; the 2B is the practical inference tier for consumer GPU deployment.
Architecture
CogVideoX uses a Diffusion Transformer architecture:
| Component | Design |
|---|---|
| Base architecture | Diffusion Transformer (DiT) |
| Attention | 3D Full Attention (not factorized) |
| Text conditioning | Expert Adaptive LayerNorm |
| Video encoder | 3D Causal VAE |
| Training objective | Flow Matching |
| Context | Space-time latent tokens |
| Output | 720×480, 6 seconds at 8fps |
3D Causal VAE: Before diffusion, video frames are encoded by a 3D VAE that understands temporal causality — each frame's encoding is conditioned on prior frames. This produces a compact space-time latent representation that the diffusion transformer then denoises.
Flow Matching: CogVideoX uses flow matching as its training objective rather than DDPM. Flow matching learns a vector field from noise to data rather than a score function — empirically this produces higher-quality samples with fewer diffusion steps.
Expert AdaLN structure: Each transformer block has two expert AdaLN modules: one conditions on text for the spatial (frame) dimension, another conditions on text for the temporal (motion) dimension. The text embedding is projected to different parameter spaces for each expert. This allows the model to separately control "what it looks like" and "how it moves."
Benchmarks
CogVideoX-5B on EvalCrafter (video generation quality):
| Model | Overall | Visual Quality | Motion Quality | T-V Align |
|---|---|---|---|---|
| CogVideoX-5B | 79.4 | 77.2 | 80.3 | 78.6 |
| VideoDirectorGPT | 72.1 | 70.5 | 71.8 | 72.4 |
| VideoCrafter2 | 70.8 | 69.3 | 68.9 | 71.4 |
| ModelScope | 65.2 | 63.4 | 64.7 | 65.8 |
On VBench (a comprehensive video quality benchmark):
- Semantic consistency: 82.1% — objects and scenes match text descriptions
- Motion smoothness: 93.4% — frames transition without discontinuous jumps
- Dynamic degree: 41.2% — amount of actual motion (vs. static frames)
The semantic consistency and motion smoothness numbers reflect the Expert AdaLN and 3D Full Attention design — text conditioning and temporal coherence are the model's strengths. Dynamic degree is lower, which reflects that the model was trained for quality over action intensity.
CogVideoX-5B vs. CogVideoX-2B
The 2B variant trades quality for accessibility:
| Metric | 5B | 2B |
|---|---|---|
| VRAM required | ~24GB | ~12GB |
| Inference speed | ~180s/video | ~90s/video |
| EvalCrafter | 79.4 | 72.8 |
| VBench semantic | 82.1% | 78.3% |
The 2B is deployable on a single 16GB GPU (A4000, RTX 4080) — consumer-accessible video generation without cloud costs. Quality is meaningfully lower than 5B but competitive with VideoDirectorGPT's 72.1 EvalCrafter.
What It Means for Developers
Open-source DiT architecture: CogVideoX is one of the first open implementations of a DiT-based video generator. For researchers building on video generation, the architecture (Expert AdaLN, 3D Full Attention, Causal 3D VAE) is fully open for study and modification.
Fine-tunable for domain-specific video: Apache 2.0 with LoRA fine-tuning support. For specific video styles, motion patterns, or domain content (medical visualization, industrial simulation, game cinematics), CogVideoX can be adapted with domain data.
ComfyUI integration: CogVideoX has official ComfyUI nodes — accessible to the diffusion art community toolchain without coding. The same ComfyUI workflow ecosystem works with CogVideoX as with Stable Diffusion.
2B on consumer GPUs: The 2B variant running on 12GB VRAM makes video generation accessible for local experimentation. Not production quality but useful for testing prompts, evaluating use cases, and developing video generation workflows before scaling to 5B or cloud inference.
Availability
- HuggingFace: THUDM/CogVideoX-5B and THUDM/CogVideoX-2B
- License: Apache 2.0 — commercial use, fine-tuning, redistribution
- Fine-tuning: CogVideoX-Factory provides LoRA, full fine-tuning, and SFT scripts
- ComfyUI: Official nodes available
Bottom Line
CogVideoX brought DiT-based video generation to open source in 2024, demonstrating that the architectural approach behind Sora could be implemented openly. Expert Adaptive LayerNorm and 3D Full Attention produced competitive semantic consistency and temporal coherence benchmarks.
For the developer community, the more significant contribution was opening a complete, well-documented implementation of expert transformer video generation — architecture, training approach, fine-tuning pipeline — for community use and adaptation.
Resources
- HuggingFace — CogVideoX-5B — flagship model
- HuggingFace — CogVideoX-2B — consumer GPU tier
- GitHub — CogVideo — codebase, fine-tuning scripts
- Technical Report (arXiv 2408.06072) — architecture and benchmarks
CogVideoX models are available on HuggingFace under Apache 2.0. Commercial use including fine-tuning and redistribution.
