Can a Coding Agent Drive a Robot? HexaShell and Triangulation Explained

Coding agents are good at a loop: write code, run it, read the error, fix it, and keep the fix. A new preprint, HexaShell: A Thin Harness for Exploring the Limits of VLM Agents as Robot Coding Agents (arXiv 2610.23479, 16 October 2026), asks what happens if the same agent runs that loop on a robot instead of a software project. The answer is interesting both for what works, a simple piece of geometry, and for what does not, which is touching things precisely.

Background: three ideas you need

• Vision language model (VLM) agents. A VLM is a language model that can also read images. An agent built on one can look through cameras and decide what to do. (For how language models work, see What Is a Transformer?)
• Learned robot policies. The usual way to control a robot is to train a model on many demonstrations so that it outputs motions directly. These have led robot manipulation leaderboards.
• Harness and tools. A harness is the software that connects an agent to the world. A tool is a small program the agent can call. The paper keeps the harness as thin as possible and lets the agent write the tools.

The test bed is RoboDojo, a simulated benchmark of 42 tasks for a robot with two arms, run in the Isaac Sim simulator. Tasks include pushing a T-shaped block onto a pad, playing a xylophone, folding clothes and inserting tubes.

The core idea: a thin harness and a repair loop

The agent is an unmodified coding agent running in a container. It gets one command, robo, a one-page manual and three camera images (one head camera and one on each wrist) with their calibration. There is no depth image. The harness does one job: it turns commands such as move or rotate into joint motion, and it makes no decisions for the agent.

What the agent learns is kept as code. After a failed attempt, a second agent of the same model reads the command log and camera frames as it would read a failing test, then writes or fixes a tool together with unit tests. A delivery check confirms the tools load and that the instructions do not mention the task or its objects. Then the same layout is tried again, up to five edits per layout. After ten layouts the tools are frozen. A repair round took a median of 6.3 minutes, and no robot training data and no change to the model were needed.

The rule that keeps this honest: every number about an object must be measured during the episode. Tools may not read object sizes or positions from the benchmark's files.

The geometry: triangulation instead of a depth network

To pick something up, the robot needs a 3D position. A pixel plus the camera's calibration gives a ray, a line from the camera through that pixel, but a ray alone is not a position. The tools add one more constraint, using plain OpenCV geometry:

• A known plane. If the point lies on the table, or on the top of a block whose height was measured, the position is where the ray meets that plane.
• A second view. If two cameras see the same point, it lies where their two rays meet. When the head camera is not enough, the robot moves a wrist camera to get a second view.

The model's job is only to pick the pixel on the target. The code then does the measuring. This split keeps the tools easy to check. The paper compares it with a monocular depth network (Depth Anything V2) that guesses depth from a single image: on 300 frames of eight tasks, that network's median error was 41 mm at the head camera and about 100 mm at the wrist cameras, and grasps need about a centimetre. Geometry has no such floor when its assumption holds. Moving the camera costs something, because every executed step counts against the task's step limit: viewing took 3.6% of executed steps with triangulation, against 0.7% with the depth network.

Worked example: a bug that was fixed in one round

In push T, a tool measured the height of the block's top from two cameras. On one layout it refused every request, because it estimated the block's area in each view by counting occupied 1 mm grid cells, and the coarse head-camera pixels fill only about a quarter of the cells that a close wrist camera fills. The two areas looked too different, so the tool rejected the height. The agent fell back to the table height, 0.765 m, while the block's top is at 0.780 m. That 15 mm error shifted the projected outline by about 10 mm, and the episode failed at 13.6 mm from the target (the limit is 7 mm).

The repair agent found the cause in the code and the tool's error messages, without looking at any image. It replaced the grid count with the sum of each pixel's footprint on the plane, added a test with two cameras of different resolution that failed before the fix and passed after, and the next attempt succeeded. The edit added 49 lines and removed 11, and the round took 7 minutes.

Results

• Headline. On the leaderboard's metric across all 42 tasks, HexaShell reaches 39.7% success and a progress score of 48.0. The best learned policies listed are VPP2 (32.3%, 39.3) and PhysicalRSI (31.4%, 36.3). The same model without tools, in the benchmark authors' fixed harness, scores 22.5%.
• Triangulation beats the depth network. With the same harness, loop and model, triangulation scores 8.7 points higher. Paired by layout, it turns 57 failures into successes and loses 23.
• Clutter. With 13 to 18 extra objects in the scene, HexaShell keeps about 95% of its success, because each tool measures only the object the agent points at. The 30 learned policies keep a median of 18%.
• Where it is strong and weak. It is ahead on memory tasks (65.0%) and open tasks (50.0%), and behind the best learned policy on precision and long-horizon tasks. Compared with VPP2 per task variant, it is higher on 23 and lower on 27.

Where it stops: contact, not reasoning

Success follows the contact a task needs. Pushing reaches 92.5% and memory tasks 65.0%, but tight insertion only 8.3%. Of 263 failed retest episodes on the development layouts, 57% ran out of steps (the agent could see the result was wrong and kept retrying), 30% ended with the agent declaring success while the checker disagreed, and 13% ended with the agent giving up early. The authors' reading is that a declared failure points to a gap in perception, and an exhausted budget to a gap in execution. In other words, the agent often knows it has failed and cannot make the motion work. They suggest the next step is tools that keep measuring during contact.

Caveats

• Simulation, not a real robot. All results are in the Isaac Sim simulator.
• A preprint by the system's authors, not yet peer reviewed. Several numbers come from their own re-runs.
• The comparison is not like-for-like. Leaderboard entries report 50 episodes on each of three seeds, while HexaShell's retest is one episode on each of ten new layouts of one seed per task. The tools were also built on the same seed, though on different layouts. Small samples mean the exact percentages are uncertain.
• The agent had extra context. In the retests its prompt includes the task description and score table from the benchmark's documentation, and for 25 tasks the repair agent read a short note on what the checker requires, written from the public task definitions.
• Cost and speed. A retest episode took a median of 739 seconds for 26 seconds of robot time, almost all of it model thinking time, which the benchmark does not charge for. It uses a frontier closed model (GPT-6-Astra at medium reasoning effort).

What it means if you are learning AI

The lasting idea is not the leaderboard score. It is that an agent's experience can be stored as small, tested, inspectable code instead of being buried in model weights, and that a plain geometric fact can beat a neural network when the geometry applies. If you want the foundations, see our robotics overview, how agents are tested in Learn2Play Bench, and how reinforcement learning rewards are shaped in Tracking-Value PPO. Our Introduction to AI in Robotics course covers the robotics side.

Source: Gao, Zhou, Zhu et al., HexaShell: A Thin Harness for Exploring the Limits of VLM Agents as Robot Coding Agents, arXiv:2610.23479, 16 October 2026. The authors' tools are released at github.com/HexaFuture/HexaShell.