ARC-AGI-3

The latest complete ARC-AGI-3 evaluation results are available on DataLearnerAI's ARC-AGI-3 benchmark page: https://www.datalearner.com/benchmarks/arc-agi-3

The ARC-AGI series of benchmarks, maintained by the ARC Prize Foundation, has long served as a key reference for measuring AI reasoning capabilities among major AI labs and academic researchers.

At its core, ARC-AGI can be understood as a collection of "inferring rules from examples" tasks. The system receives several input/output example pairs, each consisting of a small two-dimensional grid where each cell holds a discrete value (typically representing a color code). The model must infer the underlying transformation rule from these examples and apply it to new inputs to produce correct outputs.

Importantly, these "images" are not natural photographs but abstract grid structures (essentially 2D arrays). The tasks involve no real-world semantic understanding — they focus on structural induction, pattern composition, and rule extrapolation.

For example, a typical task might look like this:

Input:

0 0 0
0 2 0
0 0 0

Output:

0 0 0
0 3 0
0 0 0

Across multiple similar examples, the model must induce the rule (e.g., "replace color 2 with color 3") and apply it to new inputs. These tasks are essentially problems of program induction from limited samples.

On March 25, 2026, the third-generation ARC-AGI-3 was officially launched at Y Combinator in San Francisco — the most significant format overhaul since the series debuted in 2019.

Why ARC-AGI-3: The Saturation of Earlier Versions

To understand why ARC-AGI-3 exists, you need to know where its predecessors ended up.

Both ARC-AGI-1 and ARC-AGI-2 were image-in, image-out evaluations — though again, "images" meant abstract grids, not natural photographs. Given a set of input/output grid pairs, the system had to infer transformation rules and produce correct outputs for new instances.

These rules could involve color substitution, geometric transformations, object counting, or structural rearrangement — fundamentally a program induction problem from limited samples.

By 2025, frontier models had already surpassed 90% on the first version, prompting the team to build a second version with more complex compositional puzzles.

By February 2026, Gemini 3.1 Pro scored 77.1% on ARC-AGI-2, and Gemini 3 Deep Think reached 84.6% — approaching the practical ceiling of that benchmark. ARC-AGI-1 was essentially solved, with Gemini 3.1 Pro hitting 98%.

Behind the rapid saturation of both benchmarks lies a structural problem beyond just improved model capabilities: during Gemini 3's reasoning process, the model correctly referenced the integer-to-color mapping used in ARC-AGI tasks — information that was never explicitly provided to the model. This is strong evidence that benchmark data had been absorbed into training. Current frontier AI reasoning is fundamentally bounded by knowledge coverage, giving rise to a new form of benchmark contamination.

These issues drove the creation of the third generation.

ARC-AGI-3 at a Glance

Item	Details
Maintainer	ARC Prize Foundation
Key Authors	François Chollet, Mike Knoop, et al.
Official Launch	March 25, 2026
Preview Release	July 17, 2025
Technical Paper	arXiv 2603.24621, to appear at ICLR 2026
Competition Platform	Kaggle (ARC Prize 2026)
Total Prize Pool	Over $2 million (across three tracks)
Open Source Requirement	All winning solutions must be published under CC0 or MIT-0 license

The prize structure includes $850,000 for the ARC-AGI-3 track, with a grand prize of $700,000 for the first team to reach 100% on the evaluation set, plus guaranteed prizes for the top five finishers and milestone awards on June 30 and September 30. The ARC-AGI-2 track continues with approximately $1 million for the first open-source solution reaching 85% — a prize that went unclaimed in both 2024 and 2025.

Evaluation Design: From Static Puzzles to Interactive Environments

A Radical Format Shift

ARC-AGI-3 completely overhauls the evaluation format.

Unlike the first two versions, which provided inputs all at once and required direct outputs, ARC-AGI-3 uses an interactive evaluation pipeline. For each task environment, the model doesn't generate an answer in one shot — instead, it operates as an agent in a loop:

1. Observe the current environment state (presented as a grid or visual)
2. Output a discrete action (e.g., move, select, or manipulate an object)
3. Receive environment feedback (state update or termination signal)
4. Update internal inference based on the trajectory history and continue acting

This loop continues until the task is completed or a step limit is reached.

Each environment is a turn-based system with its own internal logic — no instructions, no descriptions, and no explicit win conditions. This means the model must simultaneously infer both the environment rules and the task objective during interaction, rather than simply executing a plan under a known goal.

A total of 135 environments were tested. All were successfully completed by human participants with no prior knowledge or instructions. Each environment contains 8–10 levels of increasing difficulty, progressively introducing new mechanics.

Human Baseline Data

Human baseline data was collected during the preview phase: the research team gathered data from over 1,200 human players across more than 3,900 game sessions. Each environment was tested by at least 10 participants under controlled conditions. The second-best human action count was used as the official baseline — the best performer was excluded to mitigate outlier effects.

Scoring Metric: RHAE

The core scoring metric is RHAE (Relative Human Action Efficiency). It measures the ratio of actions taken by the AI versus the human baseline to complete each level, normalized within each environment and averaged across all environments.

This metric penalizes brute-force search — strategies that rely on massive trial-and-error without building an effective environment model will score poorly. It also balances data efficiency and risk efficiency, and allows direct human-AI comparison.

The penalty mechanism uses a squared calculation: if a human needs 10 steps and the AI needs 100, the level score is 1%. Each level is capped at 1.0× the human baseline, and the AI's step limit per level is set at 5× the human average — exceeding this counts as incomplete.

The official leaderboard only accepts results from models accessed via API with a unified system prompt. It does not accept harness solutions customized for specific tasks — the rationale being that the evaluation aims to measure the model's general intelligence, not the intelligence human engineers invest in building task-specific systems.

Current Model Performance

Gemini 3.1 Pro Preview scored 0.37%, GPT-5.4 scored 0.26%, Claude Opus 4.6 scored 0.25%, and Grok-4.20 scored 0.00%.

Meanwhile, 100% of human testers passed all environments without any training or instructions.

The highest-scoring solution during the preview phase (July–August 2025) deserves special mention: 12.58% came from a non-LLM graph search approach (CNN + graph-structured state exploration), not from any large language model.

Another telling experiment came from a collaboration with Duke University: Claude Opus 4.6 scored 97.1% in known environments using a customized harness, but dropped to zero in environments it had never seen. This demonstrates that perceiving the game environment and API format isn't the bottleneck — customized strategies simply cannot transfer to unseen environments.

Discussion on the Benchmark Design

The ARC-AGI series has historically been relatively accurate at identifying AI capability inflection points: ARC-AGI-1 was likely the first benchmark to precisely detect breakthroughs in frontier reasoning systems like o3; ARC-AGI-2 subsequently captured the rapid progress of modern reasoning models and the rise of scaffolding techniques now deployed in production tools like Claude Code and Codex.

However, ARC-AGI-3's design has itself sparked debate. Critics point out that the squared efficiency penalty mechanism inherently produces low scores. Additionally, the official evaluation excludes models using extended thinking modes.

The Foundation's response: a community leaderboard exists alongside the official one for harness-driven results, using a self-reporting mechanism with an explicit warning that no community scores should be cited as evidence of AGI progress. The Foundation's position is that sufficiently general harness techniques will eventually be absorbed by models themselves — just as chain-of-thought evolved from an external technique into a built-in feature of o1.

One notable constraint in the competition: internet access is not allowed during Kaggle evaluation, meaning teams cannot call external inference APIs. Serious contenders must either run open-weight models locally or build non-LLM systems similar to the preview phase's winning approach.

The Bottom Line

In terms of evaluation format, the ARC-AGI series has evolved from "static rule-induction tasks" to "online learning and decision-making in dynamic environments."

ARC-AGI-3 differs fundamentally from its predecessors: it shifts from one-shot rule inference and output generation to building models through interaction in instruction-free environments; the scoring standard moves from accuracy to action efficiency aligned with human baselines.

The benchmark's core claim is straightforward: as long as a gap in learning efficiency exists between AI and humans, AGI has not been achieved. ARC-AGI-3 makes this gap quantifiable by testing intelligence over time, capturing planning horizons, memory compression, and the ability to update beliefs with new evidence.

Based on current data, frontier LLMs show an order-of-magnitude gap from humans on this dimension — a gap that cannot be attributed to engineering optimization shortfalls. ARC-AGI-3 will run throughout 2026 and can serve as a key reference point for observing the development of AI agents' autonomous adaptation capabilities. Technical reports and toolkits are open-sourced, and the latest complete evaluation results are available on DataLearnerAI's ARC-AGI-3 page: https://www.datalearner.com/benchmarks/arc-agi-3