Zhipu AI released AutoGLM on October 31, 2024 — a GUI agent designed to operate Android phones autonomously from natural language instructions. Built on top of GLM-4V (Zhipu's vision-language model), AutoGLM bridges the gap between conversational AI and mobile device automation, enabling users to describe a task in natural language and have the agent execute it through actual phone interactions.
The timing was notable: AutoGLM launched in the same period as Anthropic's Computer Use (Claude controlling desktop applications) and Kimi-VL's GUI agent capabilities, but with a specific focus on mobile-first, Chinese-market phone interactions.
What's New
Phone-native task execution: AutoGLM takes a natural language instruction ("order a coffee from Meituan nearby") and executes it by interacting with the phone's screen — tapping, typing, scrolling, and navigating apps exactly as a human would. The agent observes each screen state through vision, plans the next action, executes it, and continues until the task is complete or requires user confirmation.
GLM-4V visual grounding: The agent uses GLM-4V to understand screen content — reading app interfaces, identifying interactive elements, understanding the current state of an application. This visual grounding enables the agent to work with any app without requiring special API access or developer integration: it sees what the user sees.
Multi-step task planning: AutoGLM decomposes complex tasks into sequences of actions. "Book a restaurant for 2 people on Saturday" requires searching for restaurants, selecting a venue, checking availability, selecting a time, filling in party size, and confirming the reservation — a sequence the agent plans and executes with error handling at each step.
Chinese app ecosystem coverage: AutoGLM's training and optimization targets the Chinese mobile app ecosystem — WeChat, Meituan, Ele.me, Didi, Taobao, Alipay. These apps represent the daily mobile surface for hundreds of millions of Chinese users. The agent's performance on these specific apps is more relevant than general benchmark scores for its intended market.
Confirmation checkpoints: For high-stakes actions (placing orders, making payments, sending messages), AutoGLM pauses and requests user confirmation before proceeding. This safety-first design prevents accidental purchases or messages while allowing the agent to handle the navigation and form-filling autonomously.
Architecture
AutoGLM's architecture components:
| Component | Details |
|---|---|
| Vision model | GLM-4V (visual understanding) |
| Planning | GLM-4 (action sequence planning) |
| Screen input | Screenshot at each step |
| Action output | Tap coordinates, text input, scroll |
| Confirmation | Required for payment/message actions |
| Target platform | Android |
Perception-action loop: At each step, AutoGLM captures the current screen state, sends it to GLM-4V for visual parsing (what app is open, what elements are visible, what state the task is in), then uses GLM-4 to decide the next action. The loop continues until the task is complete.
Coordinate prediction: Rather than relying on accessibility APIs, AutoGLM predicts tap coordinates directly from the visual screenshot — clicking on the "Add to Cart" button means predicting the pixel coordinates of that button and issuing a tap command. This visual-first approach works with any app without requiring developer API integration.
Long-horizon task management: AutoGLM maintains task context across multiple steps — remembering the original instruction, tracking progress, handling unexpected states (loading screens, error messages, required logins). The planning component manages the overall strategy while the visual component handles per-step execution.
Benchmarks
AutoGLM evaluation at October 2024 release:
| Benchmark | AutoGLM | AppAgent | MobileAgent | Notes |
|---|---|---|---|---|
| AndroidWorld | 42.1% | 33.7% | 28.3% | Android task completion |
| Chinese Apps | 67.4% | 21.3% | 19.8% | Chinese app ecosystem |
| Multi-step Tasks | 58.2% | 41.5% | 35.6% | 5+ step task completion |
| Payment Tasks | Human confirm | Human confirm | Human confirm | Confirmation required |
AndroidWorld 42.1% (task completion on a standardized Android benchmark suite) demonstrates meaningful general Android capability. The Chinese Apps metric at 67.4% — versus competitors at ~20% — reflects AutoGLM's specific optimization for the Chinese mobile ecosystem: WeChat, Meituan, and Alipay interactions that other agents weren't trained on.
Multi-step Tasks 58.2% (tasks requiring 5 or more sequential actions) shows AutoGLM's advantage in long-horizon planning — the gap widens compared to single-step tasks, indicating that the planning architecture scales better with task complexity.
What It Means for Developers
Mobile automation without developer APIs: AutoGLM can interact with any app the user has installed without requiring that app's developer to provide an API or integration. For automation of third-party apps, legacy apps, or apps where developer access isn't available, visual-based interaction is the only path.
Chinese market specialization: The 67.4% Chinese Apps success rate versus ~20% for general-purpose agents is a decisive advantage for Chinese market deployment. Applications serving Chinese users — travel booking, food delivery, ride-hailing, e-commerce — are exactly the high-frequency, high-value tasks where phone agents provide the most ROI.
Platform for agentic features: AutoGLM's architecture can be integrated into consumer applications as an "agent mode" — a feature that lets users describe what they want done and have it executed. For super-apps, productivity tools, and digital assistant applications in China, AutoGLM provides the agent execution layer.
Error handling in production: The confirmation checkpoint design means AutoGLM is deployable in production without catastrophic failure modes — the agent can't accidentally place a large food order or send a message to the wrong person without user confirmation. This makes it practical for consumer deployment, not just demos.
Availability
- Access: Via Zhipu's Qingyan app — consumer deployment
- API: GLM-4V API — developer access to underlying VLM
- GitHub: Research components available at THUDM
Bottom Line
AutoGLM demonstrated that Chinese mobile apps — WeChat, Meituan, Alipay — could be navigated autonomously by an AI agent at 67.4% task completion rates. Phone-native GUI agents represent a different form factor than desktop agents: smaller screens, touch interfaces, and apps designed for human reflexes rather than programmatic APIs. Zhipu's focus on the Chinese mobile ecosystem gave AutoGLM a decisive advantage over general-purpose agents for its target market. For the next phase of mobile AI assistants, AutoGLM's visual grounding + planning architecture is the template.
Resources
- Zhipu AI — company website
- Qingyan App — consumer deployment with AutoGLM
- GLM-4V API — developer access
- THUDM GitHub — related open-source work
AutoGLM is accessible via Zhipu's Qingyan consumer app. Developer API access via open.bigmodel.cn.
