Terminal-Bench 2.1

Terminal-Bench is a benchmark designed to evaluate AI agents' capabilities in real-world terminal environments, developed in collaboration between Stanford University and Laude Institute. Terminal-Bench 2.1 is an improved version of 2.0, optimized based on Z.ai's Terminal-Bench 2.0 Verified, and is currently active, though not all tasks have been uploaded yet.

This benchmark focuses on multi-step terminal tasks across domains such as software engineering, system administration, machine learning, data processing, and security, emphasizing end-to-end execution rather than mere code generation. Each task is set in an isolated Docker environment, with the final state verified through test scripts.

Current Challenges in Terminal Agent Evaluation

Many existing agent evaluation benchmarks suffer from issues such as: tasks that are not difficult enough to differentiate between frontier models; verification mechanisms that are too lenient, allowing tasks to be passed through cheating or non-reproducible methods; tasks that are disconnected from real-world workflows and fail to reflect actual deployment value; and susceptibility to training data contamination or overfitting.

The Terminal-Bench series aims to address these problems by providing more reliable signals through real container environments, rigorous human verification, and outcome-oriented evaluation.

Basic Information and Goals

Publisher: A collaborative project between Stanford University and Laude Institute, with GitHub repositories including harbor-framework/terminal-bench, among others. The official website is tbench.ai.

Version Timeline: Terminal-Bench 1.0 included approximately 80 tasks; version 2.0, released in 2025, contains 89 meticulously selected tasks; version 2.1 is a subsequent improvement focused on fixing issues and enhancing reliability.

Core Goal: To evaluate AI agents' ability to complete complex, long-horizon real-world tasks in terminals, such as code compilation, model training, server configuration, and debugging. These tasks are derived from actual workflows, aiming to assess agents' system-level reasoning, error recovery, and tool usage skills, rather than relying solely on pattern matching.

Main Approach and Evaluation Process

Task Structure: Each task includes:

• A natural language instruction;
• A pre-configured Docker environment;
• Test scripts to verify the final container state (outcome-driven, not checking specific commands);
• A human-written reference solution;
• A time limit.

Task Quantity and Categories: For version 2.0 (with 2.1 as an improvement), there are 89 tasks covering software engineering, system administration, data science, security, machine learning, etc. Task difficulty is estimated by contributors (based on time for experts and junior engineers) and undergoes multiple rounds of review.

Evaluation Method:

• Agents complete tasks within containers using shell commands or tool interactions;
• Runs using the Harbor framework, supporting various agent scaffolds (e.g., Codex CLI, Terminus 2, OpenHands, etc.);
• Each task is run multiple times to calculate the resolution rate, reporting confidence intervals;
• Emphasis on reproducibility and anti-cheating design.

Process: Contributors submit tasks → automated and manual multi-round verification (approximately 3 hours of review per task) → inclusion in the benchmark → agents submit runs → leaderboard ranking.

Current Mainstream Model Evaluation Results

Based on public leaderboard and paper data (primarily from version 2.0, with 2.1 results being updated):

Frontier models combined with optimized agents typically achieve success rates in the 50%-90% range, depending on the agent scaffold and computational resources.

Typical Performance (Example from 2.0 Data):

• Top combinations (e.g., specific agents paired with Claude Opus 4.7 or GPT-5.5 series) can exceed 80%;
• Other frontier proprietary models are mostly in the 50%-70% range;
• Among open-weight models, the better performers are around 36%;
• Smaller models are around 15%.

Key Insights:

• Model selection generally has a greater impact than the agent scaffold;
• Increasing computational time (timeout) can significantly boost success rates, indicating that some differences stem from resources rather than capabilities;
• Tasks exist that no model has solved, showing the benchmark still has discriminative power;
• Hardware resources (e.g., RAM) also affect results, necessitating standardization.

The leaderboard is updated in real-time, supporting filtering by model, agent, and organization, with verified submission requirements.

Summary

Terminal-Bench 2.1 continues the series' rigorous evaluation of terminal agents' real-world capabilities, providing a benchmark oriented toward practical deployment by improving verification processes and task quality. It highlights the performance disparities of current frontier models in long-horizon, multi-step terminal tasks, while offering reproducible tools and datasets for agent developers and researchers.

As tasks from version 2.1 are gradually released and version 3.0 is under development, this benchmark will continue to drive AI agents toward more reliable system-level applications. Developers can participate in testing or contribute through tbench.ai and the Harbor framework.

Editor's Take

Terminal-Bench 2.1 is an important evolution in measuring AI agents' real-world capabilities. But how well terminal operation benchmarks correlate with actual development utility is still unclear. A fundamental principle: benchmark design determines results.