OSWorld Verified
OSWorld is a benchmark designed to evaluate AI agents in real computer environments. These agents are AI systems capable of processing text, images, and other data, with benchmarks including open-ended tasks such as file management or software usage. OSWorld Verified is an upgraded version that provides more accurate testing results by addressing bugs and refining processes. It supports multiple operating systems, including Ubuntu, Windows, and macOS, and enables AI to learn through interactive tasks.
Limitations of Existing Benchmarks
Several current AI benchmarks rely on simulated environments rather than actual computers, which means their results don't accurately reflect real-world usage. Key issues include:
- Simulated setups cannot handle diverse software or OS files.
- Testing often requires manual checks, hindering automation and reproducibility.
- Tasks are narrowly focused, overlooking complex multi-software workflows.
These drawbacks make it challenging to assess AI performance in everyday computer tasks reliably.
Origins and Goals
OSWorld was developed through a collaboration between the University of Hong Kong, Salesforce Research, Carnegie Mellon University, and the University of Waterloo. Its first version was released in 2024, with the research paper presented at NeurIPS 2024. OSWorld Verified launched on July 28, 2025, incorporating AWS cloud support for faster testing and resolving over 300 community-reported issues. The benchmark aims to overcome evaluation gaps by assessing AI agents in authentic computer settings. Specific goals include:
- Creating a scalable environment that supports diverse operating systems.
- Using automated scripts for result verification to minimize manual effort.
- Designing open-ended tasks to evaluate AI's interface recognition, operational skills, and planning abilities.
Testing Methods and Workflow
OSWorld Verified leverages virtual machines to create realistic computer environments. AI agents analyze scenarios through screenshots and UI structure trees, then execute actions like mouse clicks or keyboard inputs. Outcomes are automatically evaluated via scripts that check file contents or software states. The benchmark comprises 369 tasks (361 when excluding eight that require network access for Google Drive). These tasks span multiple categories:
| Task Type | Count | Example |
|---|---|---|
| Chrome Browser | 46 | Browsing websites, adjusting settings |
| GIMP Image Editing | 26 | Modifying images |
| LibreOffice Calc | 47 | Calculating data |
| LibreOffice Impress | 47 | Creating presentations |
| LibreOffice Writer | 23 | Editing documents |
| Multi-Software Collaboration | 93 | Combining multiple programs to complete work |
| OS Operations | 23 | Managing files, configuring systems |
| Thunderbird Email | 15 | Sending and receiving emails |
| VLC Media Playback | 17 | Playing videos |
| VS Code | 23 | Writing code |
The testing process involves:
- Setting up task initial states with VM snapshots and scripts.
- Allowing AI agents to take up to 100 actions.
- Automatically computing success rates via scripts.
It supports both local execution and cloud-based parallel testing, typically completing within an hour. To ensure consistency, 134 verification scripts are included, along with tools for manual checks.
Performance of Leading AI Models
Based on data from February 2026, here's how selected models performed on OSWorld Verified (success rates from single runs with up to 100 steps):
| Rank | Model Name | Release Date | Success Rate (%) | Type |
|---|---|---|---|---|
| 1 | Claude Sonnet 4.6 (Anthropic) | 2026-02-17 | 72.5 | General-purpose |
| 2 | Claude Opus 4.6 (Anthropic) | 2026-02-17 | 72.7 | General-purpose |
| 3 | Kimi K2.5 (Moonshot AI) | 2026-01-30 | 63.3 | General-purpose |
| 4 | GPT-5.3 Codex (OpenAI) | 2025-12 (approx.) | 64.7 | Specialized |
Humans achieved a 72.36% success rate on the same tasks. AI performance has surged from an initial 12.24% in earlier versions.
Key observations include:
- AI agents still struggle with screen element recognition and software operation, causing some failures.
- Higher-resolution screenshots boost success by 5-10%.
- Textual action histories outperform screenshot-only approaches.
- While AI is sensitive to UI layout changes, it maintains consistency across different operating systems.
Additionally, models like ByteDance's Seed-1.8 reached 61.9%, highlighting the multi-task strengths of general-purpose models.
Value and Future Directions
OSWorld Verified advances AI agent development by leveraging authentic environments and automated evaluation. It uncovers flaws in task planning and execution, offering valuable data for refinement. Looking ahead, the benchmark could expand to additional operating systems and task types, while also facilitating research in learning methodologies and AI safety.
Loading...