Hunyuan-DiT: Tencent's Bilingual Text-to-Image Diffusion Transformer

Tencent released Hunyuan-DiT on May 14, 2024 — a diffusion transformer architecture for text-to-image generation, notable for its native Chinese-English bilingual capability and the multi-turn conversational interface for image refinement. Where most image generation models treat text prompts as single-shot inputs, Hunyuan-DiT enables multi-turn dialog: describe an image, see a result, refine it conversationally through follow-up instructions.

The technical foundation is a 1.5B parameter diffusion transformer that processes both English and Chinese prompts with equal quality — achieved through a dual encoder design rather than translation.

What's New

Native bilingual generation: Hunyuan-DiT uses two text encoders in parallel: a CLIP encoder (English-optimized) and a multilingual T5 encoder. Both encoders process the input simultaneously, and their outputs are fused before being fed to the diffusion transformer. Chinese cultural concepts, idiomatic descriptions, and proper nouns generate correctly without requiring translation to English first.

Diffusion Transformer (DiT) architecture: Hunyuan-DiT uses the DiT architecture (Scalable Diffusion Models with Transformers) rather than the UNet backbone used by Stable Diffusion. The transformer processes image tokens in a latent space, with text conditioning applied through cross-attention at each transformer layer. This architectural choice enables better scaling behavior.

Multi-turn dialog refinement: The model integrates with a dialog system that allows users to refine generated images through conversational follow-up: "make the sky more dramatic," "add clouds in the upper right," "change the clothing to red." The Hunyuan large language model handles the dialog management, translating conversational intent into image conditioning updates.

High-resolution generation: Supports 1024×1024 native generation with multi-aspect-ratio capability — portrait (768×1280), landscape (1280×768), and square (1024×1024) all natively supported without post-processing cropping.

Architecture

Hunyuan-DiT architecture components:

Dual encoder fusion: The CLIP encoder excels at English semantic alignment developed through contrastive training on image-text pairs. The multilingual T5 encoder adds Chinese language understanding from large-scale multilingual text pretraining. Outputs from both encoders are concatenated and fed to the DiT as conditioning signals — the model learns to use both representations together.

DiT vs UNet: Traditional diffusion models (Stable Diffusion, SDXL) use a UNet backbone with skip connections. Hunyuan-DiT uses a pure transformer with full self-attention across all spatial positions. This gives better global coherence for complex scene generation — all parts of the image can attend to each other throughout the denoising process.

MLLM dialog integration: The multi-turn refinement feature is powered by Hunyuan's multimodal large language model, which processes the conversation history and current image state to produce updated conditioning signals. The dialog model converts natural language instructions into structured edits without requiring users to learn prompt engineering.

Benchmarks

Hunyuan-DiT evaluation at May 2024 release:

Human preference evaluation for Chinese prompts placed Hunyuan-DiT first — ahead of DALLE-3 for Chinese-language image generation. This is the key metric for its intended market: Chinese-language applications where prompt input is predominantly Chinese.

TIFA 73.2 (text-image faithfulness alignment) places it near DALLE-3's 74.1, meaningful given DALLE-3's much larger scale. FID 18.2 on COCO shows strong general image quality.

What It Means for Developers

Chinese-language image generation without translation: Building an image generation feature for Chinese-speaking users previously required either translating prompts to English (losing nuance) or using proprietary Chinese image APIs. Hunyuan-DiT provides open-source Chinese image generation that handles Chinese idioms, cultural references, and proper nouns natively.

Conversational image editing: The multi-turn dialog capability enables a different UX pattern than single-shot generation: iterative refinement through natural language. For product design tools, creative applications, and e-commerce image generation, this iterative approach produces better results than prompt engineering alone.

Apache 2.0 for commercial use: The model weights are released under Apache 2.0, enabling production deployment in commercial applications. This is significant for Chinese developers: a production-grade Chinese text-to-image model with no API dependency or per-image cost.

Scalable DiT foundation: The DiT architecture scales better than UNet with model size. Hunyuan-DiT at 1.5B is a foundation that can be fine-tuned for specific styles, domains (product photography, anime, architectural visualization), or quality tiers — standard diffusion fine-tuning techniques apply.

Availability

• HuggingFace: Tencent-Hunyuan/HunyuanDiT
• GitHub: Tencent/HunyuanDiT
• License: Apache 2.0
• Paper: arXiv 2405.08748

Bottom Line

Hunyuan-DiT brought production-quality bilingual text-to-image generation to the open-source ecosystem. First-place human preference for Chinese-language prompts, Apache 2.0 license, and multi-turn dialog refinement — Tencent delivered a model that matches proprietary Chinese image generation APIs for the use cases they're built for. For developers building Chinese-language creative applications or e-commerce image tools, Hunyuan-DiT was the open-source answer that didn't require translating prompts through English first.

Resources

• GitHub — Tencent/HunyuanDiT
• HuggingFace — HunyuanDiT
• Technical Paper (arXiv 2405.08748)
• Hunyuan Platform — Tencent's AI services

Hunyuan-DiT is available on HuggingFace under Apache 2.0. Commercial deployment permitted.