Alibaba's Qwen team released Qwen2.5-VL in January 2025 as a major update to Qwen2-VL. While Qwen2-VL had solved the resolution bottleneck for document and image tasks, Qwen2.5-VL expanded in a different direction: making the model capable of acting in visual environments, not just describing them.
The key addition was native GUI agent support — the model can click buttons, fill forms, navigate browsers, and operate desktop applications by outputting pixel coordinates. Combined with improvements to video understanding and document parsing, Qwen2.5-VL became the most capable open-source VLM for agentic applications at the time of release.
What's New
Native GUI agent capabilities: Qwen2.5-VL can interact with graphical user interfaces by generating bounding box coordinates and action types (click, type, scroll, drag). This goes beyond visual QA — the model can operate browsers, desktop apps, and web forms. The output format is a structured action: {"action": "click", "coordinate": [x, y]}.
Absolute position encoding (M-RoPE upgrade): The position encoding was updated to use absolute pixel coordinates rather than relative position indices. The model outputs grounding coordinates like <|box_start|>(x1,y1),(x2,y2)<|box_end|> — precise pixel locations in the original image space. This is required for reliable GUI interaction and document element grounding.
Dynamic frame rate video: The video processing pipeline was upgraded to handle variable frame rates natively. Rather than sampling at a fixed rate, Qwen2.5-VL can process videos at their natural frame rate (or a dynamically chosen rate based on content speed). This improves temporal reasoning for fast-motion content and reduces redundant frames for static content.
Structured document parsing: Native output formats for complex documents — tables as structured data, mathematical formulas as LaTeX, code blocks as formatted code, charts as data structures. The model doesn't just read text from documents; it understands document structure.
Architecture
Qwen2.5-VL builds on the Qwen2-VL foundation with targeted upgrades:
| Feature | Qwen2-VL | Qwen2.5-VL |
|---|---|---|
| Position encoding | M-RoPE (relative) | M-RoPE (absolute pixel coords) |
| Video frame handling | Fixed rate sampling | Dynamic frame rate |
| GUI interaction | Not supported | Native (click/type/scroll/drag) |
| Max video length | 20 minutes | 1 hour |
| Model sizes | 2B / 7B / 72B | 3B / 7B / 72B |
| Context | 128K | 128K |
Absolute position encoding is the architectural change that enables GUI interaction. With relative positions, the model knows where elements are relative to each other but not their absolute pixel location. Absolute pixel coordinates allow the model to output (462, 128) meaning exactly that point in the original 1920×1080 screenshot.
Dynamic frame rate is implemented as a content-aware sampling strategy: the model receives video at whatever frame rate preserves the relevant content. An action video game gets sampled at 30fps; a slideshow presentation might get 1fps. This extends practical video length to 1 hour while maintaining reasonable context usage.
Benchmarks
Qwen2.5-VL-72B-Instruct vs. frontier models (January 2025):
| Benchmark | Qwen2.5-VL-72B | GPT-4o | Claude 3.5 Sonnet | Notes |
|---|---|---|---|---|
| DocVQA | 96.4% | 91.1% | 90.8% | Document QA |
| ChartQA | 89.5% | 85.7% | 90.8% | Chart understanding |
| MathVista | 74.8% | 63.8% | 67.7% | Visual math |
| OCRBench | 88.1% | 79.0% | — | Text recognition |
| Video-MME | 73.3% | 71.9% | — | Video understanding |
| ScreenSpot | 87.1% | — | — | GUI grounding |
| OSWorld | 35.2% | 38.1% | — | Desktop agent |
DocVQA 96.4% is the headline (+1.9pt over InternVL3, +5.3pt over GPT-4o) — the absolute position encoding and improved document structure understanding pushing the document benchmark to a new level.
ScreenSpot 87.1% is a GUI grounding benchmark measuring whether the model correctly identifies UI elements to interact with. This is the GUI agent capability made concrete: given a screenshot and "click the Submit button," does the model output the right coordinates?
OSWorld (desktop agent task completion) at 35.2% vs GPT-4o 38.1% shows that while the GUI grounding is strong, end-to-end desktop task completion remains harder — the grounding is there, but multi-step task planning in complex desktop environments is still challenging.
GUI Agent Capabilities
The GUI interaction format works as follows:
Input: screenshot + "Click the Submit button"
Output: {"action": "click", "coordinate": [462, 728]}
Input: screenshot + "Type 'hello@example.com' in the email field"
Output: {"action": "type", "coordinate": [320, 156], "text": "hello@example.com"}
This enables computer use applications: browser automation, form filling, desktop application control, web scraping with visual navigation. Qwen2.5-VL can be integrated into agent frameworks as the "eyes and hands" of a computer-using agent.
The model supports six action types: click, double_click, right_click, type, scroll, and drag. These cover most GUI interaction patterns without requiring screen recording or structured accessibility tree access.
What It Means for Developers
Document pipelines with structure preservation: DocVQA 96.4% combined with structured output for tables, formulas, and code means Qwen2.5-VL can extract not just text from documents but their structure. Invoice line items as JSON, research paper equations as LaTeX, data tables as CSV.
Computer use without API overhead: Unlike GPT-4's computer use capability (API-only, pay-per-call), Qwen2.5-VL is open-source. Self-hosted computer use for browser automation, UI testing, or desktop agent workflows at fixed inference cost.
Video analysis at 1 hour: Meeting recordings, lecture capture, instructional video — 1-hour video at dynamic frame rates with temporal reasoning. The 7B variant makes this viable at consumer GPU inference cost.
7B production tier: Qwen2.5-VL-7B-Instruct delivers strong document and GUI capabilities at 7B inference cost. For most document processing workflows that don't need the 72B's full capability, the 7B is the practical choice.
Availability
All models are available at huggingface.co/Qwen:
- Qwen2.5-VL-3B-Instruct, 7B-Instruct, 72B-Instruct
- Apache 2.0 license
- vLLM, transformers, and SGLang support
- Also available via Alibaba Cloud API and QwenLM platform
Bottom Line
Qwen2.5-VL took the resolution improvements from Qwen2-VL and added the ability to act — GUI interaction with pixel-level precision. DocVQA 96.4% pushed document understanding to a new benchmark ceiling while ScreenSpot 87.1% demonstrated that the same model could operate visual interfaces.
For developers building document understanding pipelines, the structured output capabilities alone justify the upgrade from Qwen2-VL. For those exploring computer use or visual agents, Qwen2.5-VL is the open-source baseline.
Resources
- Blog Post — release announcement
- Technical Report (arXiv 2502.13923) — architecture and benchmarks
- HuggingFace — Qwen2.5-VL-72B — 72B flagship
- HuggingFace — Qwen2.5-VL-7B — production tier
Qwen2.5-VL models are available on HuggingFace under Apache 2.0. All sizes including commercial use.
