菜单

关于 🐙 GitHub
arXiv 提交日期: 2026-01-28
📄 Abstract - OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Execution

Graphical User Interface (GUI) agents show great potential for enabling foundation models to complete real-world tasks, revolutionizing human-computer interaction and improving human productivity. In this report, we present OmegaUse, a general-purpose GUI agent model for autonomous task execution on both mobile and desktop platforms, supporting computer-use and phone-use scenarios. Building an effective GUI agent model relies on two factors: (1) high-quality data and (2) effective training methods. To address these, we introduce a carefully engineered data-construction pipeline and a decoupled training paradigm. For data construction, we leverage rigorously curated open-source datasets and introduce a novel automated synthesis framework that integrates bottom-up autonomous exploration with top-down taxonomy-guided generation to create high-fidelity synthetic data. For training, to better leverage these data, we adopt a two-stage strategy: Supervised Fine-Tuning (SFT) to establish fundamental interaction syntax, followed by Group Relative Policy Optimization (GRPO) to improve spatial grounding and sequential planning. To balance computational efficiency with agentic reasoning capacity, OmegaUse is built on a Mixture-of-Experts (MoE) backbone. To evaluate cross-terminal capabilities in an offline setting, we introduce OS-Nav, a benchmark suite spanning multiple operating systems: ChiM-Nav, targeting Chinese Android mobile environments, and Ubu-Nav, focusing on routine desktop interactions on Ubuntu. Extensive experiments show that OmegaUse is highly competitive across established GUI benchmarks, achieving a state-of-the-art (SOTA) score of 96.3% on ScreenSpot-V2 and a leading 79.1% step success rate on AndroidControl. OmegaUse also performs strongly on OS-Nav, reaching 74.24% step success on ChiM-Nav and 55.9% average success on Ubu-Nav.

顶级标签: agents systems model training
详细标签: gui agents autonomous task execution multi-platform synthetic data generation policy optimization 或 搜索:

OmegaUse:构建用于自主任务执行的通用图形用户界面智能体 / OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Execution


1️⃣ 一句话总结

这篇论文提出了一个名为OmegaUse的通用智能体模型,它能像人一样在电脑和手机屏幕上自主操作完成任务,其核心是通过创新的数据构建方法和两阶段训练策略,让模型在跨平台任务中表现出色。

源自 arXiv: 2601.20380