With this post, we are open-sourcing our agent harness for the ARC-AGI-3 challenge on Kaggle. We summarize the design, explain the behavior of the agent, and provide a detailed technical overview linking the Kaggle resources needed to reproduce our results. At the same time, we also release the code on GitHub and compare the results to other published approaches.

Accompanying this post are two episodes on the Machine Learning Street Talk Podcast: one episode covers the technical details of our harness, and another goes deeper into broader topics around ARC-AGI-3.

ARC-AGI-3

ARC-AGI-3 is an abstract reasoning challenge designed to highlight what humans can do that machines still cannot — general continual learning under changing conditions and unclear goals. The benchmark consists of games, represented as sequences of 64x64 grids — an example of which you can find in the image below (source). The agent selects one of the possible actions to produce subsequent frames, which eventually leads to completing the levels. The goal of the puzzle is never explicitly stated and has to be inferred, tested, and verified by the agent. The solution path also changes in every level, so the strategy must adapt over time.

ARC-AGI-3 game example

Humans can solve these games, while current frontier reasoning models struggle. The optimized metric is not time but action efficiency: the agent must finish each puzzle in as few actions as possible, and the final score compares this against the median human baseline. This metric disincentivizes brute-forcing, which would score close to 0%.

Here at Tufa Labs, we believe this challenge gets to the core of what human reasoning is. Therefore, we see ARC-AGI-3 as a great challenge to build agents for and are convinced that working on the challenge will lead to interesting future research.

Large Language Models

The current top-scoring solutions on the public leaderboard (evaluated with unlimited compute on public games) are based on LLM coding agents. Notably, Symbolica’s solution achieved 36.08% within the first day after the release of ARC-AGI-3, using a recursive agentic orchestration system that interacts with the game through code. Even more convincingly, the RGB Agent built on OpenCode was the first agent to show near-human action efficiency on the 3 preview games. Additional prompt optimization to construct executable world models then brought performance up to 58.12%.

These promising results motivated us to construct our solution around coding/reasoning models. Similarly to the previous solutions, we encode the game as a programming problem, which lets us leverage the coding and reasoning abilities of pre-trained LLMs. Because evaluation on the semi-private/private test set on Kaggle is constrained to a single GPU, we are restricted to use small open-source models and small token throughput. Therefore, we focus on a simple harness architecture: a minimal coding harness with a Python interpreter that interacts with the game through a Read-Eval-Print Loop (REPL) with game perception through images of the grid.

Solution Architecture

In our solution, the LLM works in a REPL where all game observations are encoded as Python variables. It can inspect these variables with tool calls, evaluate pre-built helper functions, and take actions in the game environment. The actions are executed in the ARC-AGI-3 environment, which then updates the Python sandbox with fresh variables. The model uses both image and text representations of the grid and the context is kept short by automatically evicting the oldest messages. For more detailed information, we refer to the technical write-up on Kaggle.

Duck harness architecture

Results

The agent harness is built around Qwen 3.6 27B FP8. We run it on the 25 public games with 20 tries for each game. Our mean score across all public games is 1.6002 +/- 0.4475. We include the results directory on GitHub, including the automatically generated diagnostics.html that compares the runs in detail. The heatmap below shows the performance of the agent across games and runs, with attempts on the x-axis and games on the y-axis. The performance is rather uneven, with some games being solved consistently for over 40% of the levels, while for some not even the first level is solved.

Per-game, per-run levels heatmap

We also evaluate our harness with GPT 5.4 to compare the performance against the Executable World Models agent. In the figure below, the x-axis shows the cost per game and the y-axis the ARC score. For comparability, we display the best score of two tries for each game. We can see that the two harnesses solve a similar set of games, while the duck harness is an order of magnitude cheaper on each game. This hints at the solvability of a game being dependent on model capability, while the cost is mostly dictated by the harness.

Cost per game versus ARC score

We present the duck harness for pre-trained LLMs and show first signs of life on the challenge. By representing the task as a coding problem, we can exploit the model’s problem-solving abilities. However, we are ultimately limited by the constraints of the Kaggle environment. The harness can still benefit from many improvements, especially in context management and perception. With this, we are inviting the community to improve on our work. We will continue to work on the problem until the end of the competition and are looking forward to seeing you on the leaderboard!