←  Ian Zhang / Projects

Research report · 16 November 2026

Two GPT models drive a robot arm from two cameras.

GPT-6.1 Sol and GPT-5.6 Sol each chose every move of an AgileX PiPER arm, from three pictures and a JSON state per decision. The task was to put a bottle into a basket and take it out again. In a six-run benchmark, GPT-6.1 Sol completed all three of its cycles and GPT-5.6 Sol none of its three.

Full cycle · 16 Nov 13:00Arm · portal · cameras
3/3
cycles GPT-6.1 Sol completed in the benchmark
0/3
cycles GPT-5.6 Sol completed (it reached stages 4, 3 and 1)
13.1min
median time for a complete cycle by GPT-6.1 Sol
85%
of benchmark time the arm stood still while the model thought

Findings

The model chose every motion. The two models differed most in how they held the bottle.

The arm sits on a white table with a yellow hair-gel bottle and a black fabric basket. One camera looks down from about a metre up, and a second, with a depth sensor, rides on the wrist. For every decision the model gets three pictures and the arm's state as JSON, and answers with one action: move the tool to a point, move it by an offset, set the joints, open or close the gripper, look again, or say it is done. It plans the whole task itself. The software around it builds the pictures, checks every move against the table, a measured wall, the arm's reach and its joint limits, and runs the motion.

GPT-6.1 Sol completed 3 of 3 benchmark cycles; GPT-5.6 Sol completed 0 of 3. On a six-stage rubric, GPT-6.1 Sol reached the last stage every time and GPT-5.6 Sol reached stages 4, 3 and 1. Three runs per model is a small sample: a 3–0 split like this has a two-sided Fisher exact p of 0.10. The benchmark was meant to have five runs per model, and a bug in its start-of-run check ended it after six (details under limitations).

GPT-6.1 Sol was slower per decision and needed far fewer of them. Its median decision took 30.8 seconds against GPT-5.6 Sol's 18.5, but its runs needed a median of 21 decisions against 60 (the budget) and used 1.8 times fewer tokens: 396,661 per run on average against 698,256.

The arm stood still for 85% of the benchmark. The model was thinking for 8,071 of the 9,484 seconds; the arm moved for less than 13%. Once the moves were safe and the pictures were right, control was the easy part, and the model's latency set the pace.

GPT-5.6 Sol's failures were all about the grip. It dropped the bottle while carrying it out of the basket, closed on the basket's fabric instead of the bottle, and in one run lifted six times while the bottle stayed on the table. GPT-6.1 Sol made a similar mistake once, closing on the basket's edge, and noticed, put it back and tried again.

Most of the work was on what the model sees. The first run that worked came after eight that didn't, and after the depth camera's view was rebuilt as a top-down height map that remembers what scrolled out of view. The rest went into safety (a measured wall the wrist camera had scraped), a re-calibration, and a fix in the arm's driver for a loud buzz the motors made while holding still.

Figure 1

GPT-6.1 Sol completes the cycle every time, with 1.8 times fewer tokens per run

Stage reached against tokens per run · benchmark
GPT-6.1 SolGPT-5.6 Sol
Small dots are single runs, large dots each model's mean. Stage 5 means the bottle went into the basket, came out and was put down on the table. Tokens are input plus output for every call in the run, including the plan and the checker. Hover a dot for the run.

Same task, same speed

One finishes in 13 minutes. The other is still going at 24.

Both overhead views at 24 times real speed, starting together. On the left, GPT-6.1 Sol (run 14:11) takes the bottle from the table to the basket and back out, then holds still. On the right, GPT-5.6 Sol (run 13:46) gets the bottle into the basket and lifts it out, then loses it on the way to the table.

24× GPT-6.1 Sol GPT-5.6 Sol
Benchmark runs 20261116-141102 (left) and 20261116-134633 (right), overhead camera only, wall-clock time compressed 24 times.
moving only · 3×
GPT-6.1 Sol, the whole cycle. Only the 106 seconds the arm was moving, out of 13 minutes, at 3 times speed. The cut-out time is the model thinking.
moving only · 3×
GPT-5.6 Sol can do it too. Its one complete cycle came in the development runs, just before the benchmark (run 13:00, 28 decisions, 12.9 minutes).

Scoring

How far each run got

Each run is scored by the highest stage it reached, so a run that fails still records how far it got. A drop doesn't count as reaching the next stage.

I assigned stages from the run logs and the pictures, not from the model's own account. The evidence was the gripper's holding flag, the width at which the fingers stopped, and the overhead pictures before and after every release. The automatic checker's verdict is listed separately. It gave one on five of the six benchmark runs (the sixth ran out of decisions before saying done) and agreed with the stages on all five.

  1. No purposeful approach to the bottle
  2. Gripper around the bottle on the table, but no secure grasp
  3. Bottle grasped and lifted clear of the table
  4. Bottle released inside the basket, gripper withdrawn
  5. Bottle grasped again inside the basket and lifted clear of the rim
  6. Bottle put down on the table outside the basket and released under control

Figure 2 · Results

GPT-6.1 Sol reaches stage 5 three times; GPT-5.6 Sol stops at 4, 3 and 1

Stage reached per benchmark run
One bar per run, in the model's colour. The text is what ended the run. Hover a bar for the account of the run.

Results by model

ModelCompleteMean stageDecisions
median
Minutes
median / mean
Tokens per run
mean
s per decision
median
Refused or failed
moves
Thinking
share of time
GPT-6.1 Sol3 / 35.002113.1 / 22.4396,66130.82 of 8486.8%
GPT-5.6 Sol0 / 32.676031.2 / 30.3698,25618.512 of 14983.9%

Benchmark bench-20261116-133321, 16 November 13:33–16:20, three runs per model, models alternating, no notes carried between runs. Tokens are input plus output as reported by the API, for every call in the run. 60 decisions is the budget per run, so a median of 60 means most runs used all of it. GPT-6.1 Sol's mean time is pulled up by one 41-minute run, described under failures.

Figures 3 and 4 · Where the time goes

The arm moves for a few seconds after every half-minute of thought

Each row below is one benchmark run. Grey is the model thinking about its next decision, green is the arm carrying it out, a triangle is a gripper action, a red cross a move the arm refused or that failed, and the shaded spans are the time the gripper was holding something. Hover any decision to read what the model wrote before it.

Run timelines · benchmark
model thinkingarm movingholding▼ gripper✕ refused or failed
Time on the x-axis is minutes since the run started. Thinking time is the model call's own duration; moving time is the motion the arm reported for that decision.
Seconds per decision · benchmark
One dot per decision; the bar marks each model's median.

The two models spend their time differently. GPT-6.1 Sol thinks longer before each decision (a median of 30.8 seconds against 18.5) and makes far fewer of them. Both use about 13,000 tokens per decision, most of them the three pictures.

Over all 416 decisions in all 19 runs, the median model call took 20.7 seconds and one in ten took longer than 61.6. The arm's moves take about 2 to 9 seconds (a median of 5.7). A cycle that needs 21 decisions therefore takes about ten minutes however fast the arm is, and the obvious ways to shorten it are on the model's side: fewer, larger moves per call, or a smaller model for the steps where the answer is obvious.

The portal

Step through a run the way the model saw it

The runs are driven from a small web portal on the laptop that runs the arm. Below is a replay built from the run logs: pick a run and step through its decisions. Each one shows the three pictures the model received, what it wrote, the action it chose and what the arm reported back. Ticks under the slider mark gripper actions (dark) and refused or failed moves (red). The arrow keys step too.

gpt-arm portal · replay

The real portal, next to the camera

A screen recording of the portal during run 14:11 (GPT-6.1 Sol), with the overhead camera beside it, both at 16 times speed. Each decision appears in the portal's log as the arm carries it out.

16×

How it works

How a language model drives the arm, and what stops it doing damage

One decision, end to end
Overhead camera1600×1200 · about 1 m up Wrist cameracolour + depth 320×200 Arm jointsCAN feedback Perception and state overhead grid in mmtop-down depth map ±160 mmobjects 18 mm above floornearest wall, holding3 pictures + JSON GPT-6.1 Sol orGPT-5.6 Solone action as JSON~13k tokens · ~20-30 s Safety checks tips ≥ 8 mm, wrist ≤ 560 mmreach and joint limitswall: spheres 15 mm offdriver box, guardsrefuse → tell the model DriverIK + jerk-limited path · 100 Hz AgileX PiPER6 joints + gripper the next decision starts from new pictures on "done": a checker compares overhead pictures before, after each release, and after

What the model sees

The three pictures are the overhead photo with a grid in arm millimetres, the wrist camera's colour image, and a height map built from the wrist depth camera. The map is the one that mattered. The depth points are moved into the arm's frame using the arm's own pose, binned into 2 mm cells over ±160 mm around the tool point, and seen from straight above. Anything standing 18 mm or more above the local floor becomes an object, listed with its centre, height, length, width and long axis.

The depth camera sits 71 mm behind the gripper and looks ahead of it, and it can't measure closer than about 15 cm. So as the gripper comes over the bottle, the bottle leaves the camera's view. The map therefore keeps earlier frames, drawn dimmer, until the gripper opens or closes. On the right is the state that went with one decision of run 14:11, holding the bottle 250 mm above the basket just before lowering it.

What it can do

One action per decision: move the tool to a point with a lean, a yaw and a finger direction; move it by an offset; set the six joints; open or close the gripper to a width; take a fresh look; or say done. Two named shortcuts exist, a look from a parked pose and a scripted stand-up for a lying bottle, but the model chooses when to use them. At the start it writes a short plan of stages, each with a condition for being finished.

What the pipeline refuses

The fingertips never go below 8 mm above the table and the wrist never above 560 mm. Every target is checked for reach, since this arm can't turn its wrist more than about 50 to 60 degrees off the direction of its base, so the model sets the finger direction by turning the last joint instead. The wall behind the table was measured with the depth camera as a vertical plane at x = 485 mm (5 mm rms), and the camera, wrist, gripper and fingertips, modelled as spheres, must stay 15 mm off it, checked before every move and while it runs. The driver adds its own workspace box, a guard that holds the arm if a joint lags its command, and one that holds it if the inverse kinematics flips the wrist. A refused move costs a decision, and the model is told why.

State for decision #11 · run 14:11

{
  "tool_point_mm": [285, -211, 250],
  "tool_lean_deg": 25, "tool_yaw_deg": -36,
  "fingers_close_along_deg": 54,
  "gripper_mm": 42.8, "holding": true,
  "joints_deg": [-36.4, 85.4, -83.1, 0.0, 67.7, 0.0],
  "nearest_wall": { "wall": "backboard",
     "closest_part": "wrist camera", "clearance_mm": 92 },
  "wrist_depth": {
    "surface_under_tool_z_mm": 11,
    "fingertips_above_it_mm": 225,
    "objects_below": [{ "label": "D1",
      "centre_xy_mm": [326, -246], "top_z_mm": 109,
      "length_mm": 297, "width_mm": 233,
      "long_axis_deg": 84, "cut_off_by_view": true }]
  },
  "bottle_start_xy_mm": [152, 4],
  "steps_used": 8, "steps_left": 52
}

Trimmed from the run log. D1 is the basket: 109 mm tall at its rim, cut off by the edge of the camera's view. The model answered move_by (0, 0, -165), straight down into the basket.

Hardware

ArmAgileX PiPER, 6 joints, two-finger gripper opening to 75 mm
HostDell laptop, Ubuntu 24.04; the arm on CAN through a USB adapter
Overhead cameraSurface Pro 9 rear camera, 1600 × 1200, about a metre above the table
Wrist cameraOrbbec Dabai DC1: colour 1280 × 720, depth 320 × 200 at 10 fps
SceneWhite table with a measured backboard; hair-gel bottle about 125 mm long and 45–50 mm across; black fabric basket about 235 × 220 mm, rim about 110 mm

Software

ModelsGPT-6.1 Sol and GPT-5.6 Sol through the Codex CLI, reasoning effort medium, answers checked against a JSON schema
DriverOwn IK and a Ruckig jerk-limited trajectory, joint targets streamed at 100 Hz
PerceptionOpenCV and NumPy: overhead calibration from the arm itself (solvePnP), depth height map and objects
RunnerPython service with a web portal; every run logged as JSON with its pictures and a video
CheckerThe same model, judging success from overhead pictures before, during and after

Figures 5 and 6 · What changed from the first run to the last

Eight runs failed before the first one worked. The fixes were almost all on the perception side.

Every run in order · length in minutes
GPT-6.1 SolGPT-5.6 Solscripted● complete✕ failed□ stopped by hand┆ a fix went in
Runs on the evening of 15 November and through 16 November, in order. Not every run had the same instruction; hover for each one's task and what happened.

The first evening ended with two problems. A run that was asked to stand a lying bottle up did it, then used up its budget without saying so. And the USB hub in the laptop's dock dropped the camera and the arm's CAN adapter together, twice over the two days.

On Monday, between 10:44 and 12:18, five runs in a row went nowhere, and I stopped each by hand. In the one at 12:04 the gripper never got below 18 cm: 12 moves, all small corrections above the bottle, and the gripper never closed. Reading the logs, the cause was on my side. The wrist picture carried a fixed marker for where the fingers meet, which is only true at fingertip height. The depth view was an oblique picture the model couldn't measure from. And the nearest object in the state was the basket's rim, not the bottle inside it.

The top-down height map replaced both the marker and the oblique view at 12:15. The run at 12:18 still hovered. The depth camera looks ahead of the gripper, so the bottle left the map as soon as the gripper was over it, and the yaw the model wanted for a grip across the bottle was out of the arm's reach. Two changes followed: the map started remembering earlier frames, and the model got a finger-direction control that turns the last joint instead of the wrist. The run at 12:29 went down to 45 mm, closed at 42 mm on the bottle and lifted it out of the basket in 5.2 minutes.

Figure 6 shows the difference: the tool's height after every move in the run that hovered and in the one that worked.

Tool height after each move
Run 12:18 (dashed) and run 12:29 (solid), both GPT-5.6 Sol on "take the bottle out of the basket". Hollow points are gripper actions.
moving only · 2×
12:18, before the last two fixes. Twelve moves, all above 18 cm, and the gripper never closes.
moving only · 2×
12:29, the first run that worked. Eleven decisions: over the bottle, straight down, close, lift, carry, set down.

Three more things went wrong that day. In the late morning the wrist camera scraped along the backboard when the model sent the tool to the far edge of the table, which led to the measured wall. The overhead camera got nudged, so it was re-calibrated from 20 arm poses, of which 11 were usable, to 9.7 pixels rms. And after the arm had been moved by hand with its teach button, the controller stayed latched in teach mode with the light off. The next move swung the base joint back and forth by about 4 degrees every 3 seconds instead of settling, until a Stop cleared it. A run started in that state failed on its first move. The run after that failed too, for a different reason: the runner read the driver's record of that earlier stop as a new one. Only a stop during the current move counts now.

Failures, in detail

Three ways to lose a bottle

1.5×
Dropped on the way out · GPT-5.6 Sol, 13:46 · stage 4. It put the bottle into the basket by letting go from 170 mm, picked it up again at 110 mm with the fingers stopping at 34 mm, lifted to 270 mm, and lost it while moving to the table. It said done. The checker said the bottle had been dropped and gave it one more try, which ended in a second done and a second fail after 24 minutes.
1.5×
Closed on the basket · GPT-5.6 Sol, 14:24 · stage 3. The bottle was in the basket by 3.6 minutes. The re-grasp stopped at 56 mm, wider than the bottle, almost certainly on the fabric, and nothing came up with the lift. Its last 34 decisions were searching and small moves, including a scripted stand-up the pipeline refused because the bottle wasn't lying on the table, until the 60-decision budget ran out.
1.5×
Six lifts, no bottle · GPT-5.6 Sol, 15:49 · stage 1. The run started with the bottle lying close to the arm's base, and six of its first seven moves failed or were refused, most as out of reach. It then closed on the bottle at 43 mm height six times, with the fingers stopping at 32 to 35 mm, and every lift came up empty. On its last decisions the arm's feedback went stale for a moment and a guard stopped a move whose fingertips dipped below the table.
8×
The same mistake, recovered · GPT-6.1 Sol, 15:08 · stage 5. Its first attempt to take the bottle out closed on something 31 mm wide at the far edge of the basket. It lifted that by 180 mm, lowered it back and let go, then found the bottle and finished, 43 decisions and 41 minutes in. This is the run that pulls GPT-6.1 Sol's mean time up.

Every failure here comes down to the grip. The model only knows it is holding something from the width at which the fingers stopped closing, and a width of 31 to 56 mm can be the bottle, the end of it, or a fold of fabric. Nothing in the pipeline checks the grip after a short lift before the long move. That check is the first item under further work.

Figure 7 · The buzz

A loud buzz with nothing moving was a wrist motor that couldn't settle

During some holds the arm made a loud, high buzz while nothing visibly moved. To find which holds, I put the laptop's microphone next to the arm and logged the motors' own feedback off the CAN bus through a 13-minute run. That feedback is speed, current and position, about 100 times a second per motor.

The arm stands still while the model thinks, so every pause is a clean sample of one pose. Sorted by loudness, the worst hold was 16 dB above the quiet ones, and in it the J5 motor's speed had a standard deviation of 0.39 rad/s, against under 0.03 for a quiet hold. The joint's angle didn't change. The driver had already stopped sending commands, so this was the motor's own position loop failing to settle at that particular target.

On the arm, in three poses that buzzed, re-sending the same target didn't help (0.41 went to 0.33). Moving the hunting joint's target by half a degree stopped it in five of six tries. The driver now watches every motor while the arm is idle. If one oscillates for half a second after a one-second settle, it shifts that joint's hold target by 0.3 degrees, then 0.6 if needed, never further. At the fingertips 0.3 degrees on J5 is 1.2 mm.

Across the benchmark the driver did this 65 times: 31 on J6, 23 on J5, 10 on J4 and once on J3. Before the fix, 3 of 28 holds were above 0.15 rad/s; after it, none of 231. The runs and poses differ between the two sets; a controlled trial would repeat the same poses with the watch on and off.

Worst motor jitter per hold
One dot per hold (the time between two moves). Red dots are above the driver's 0.08 rad/s trigger. Before: run 13:00, the full cycle on the day. After: the six benchmark runs.

Technical specifications

The setup in one list

Embodiment
AgileX PiPER 6-joint arm with a two-finger parallel gripper (0–75 mm) on a white table; a black fabric basket and a hair-gel bottle.
Control
The model outputs one action per call: tool pose (x, y, z, lean, yaw, finger direction), relative move, joint angles, gripper width, look, or done. The driver turns moves into jerk-limited joint trajectories streamed at 100 Hz, at 20% of the arm's speed.
Observation
Overhead colour picture (sent at up to 1024 px), wrist colour picture (768 px), top-down depth height map (2 mm cells, ±160 mm), and the state as JSON: joints, tool pose, gripper width, holding, nearest wall, objects, steps used.
Policy
The model itself, prompted with a fixed description of the arm, cameras, actions and limits; it writes a plan of stages first. No examples of solved runs and, in the benchmark, no notes from earlier runs.
Models
GPT-6.1 Sol and GPT-5.6 Sol, reasoning effort medium, 180 s timeout per call, 2 retries.
Harness
Up to 60 decisions per run. When the model says done, the arm parks and a checker call judges success from overhead pictures before the run, after each release and after the run. If it says fail, the model gets one more go with the checker's critique.
Benchmark
bench-20261116-133321: "Pick up the hair gel bottle from the table and put it into the basket. Then take it back out of the basket and put it down on the table beside the basket. Standing or lying on the table are both fine." Models alternate; three runs each; each run starts where the last one left the bottle.
Measured
Stage reached (0–5), completion, decisions, minutes, tokens (input plus output), seconds of model time per decision, refused or failed moves, and the checker's verdict.

Limitations

What these numbers can't tell you

  • Three runs per model. The benchmark was set up for five each. After the sixth run, which ended with the gripper low in the basket, the benchmark's own start-of-run check tried to lift the arm, a guard stopped the lift, and the error ended the benchmark instead of pausing it. That is fixed. The remaining four runs weren't done because by then it was dark, and the cameras see a different scene under lamp light.
  • Starting positions varied. Each run started wherever the previous one left the bottle, standing or lying. GPT-5.6 Sol's worst run started with the bottle lying close to the arm's base, near the edge of what the arm can reach, which is harder than any start GPT-6.1 Sol got.
  • One task, one scene, one afternoon. The light changed through the afternoon, and there is one bottle, one basket and one table.
  • The grading isn't blind. I assigned the stages knowing which model ran, from logs and pictures. The automatic checker is the same model as the one driving, so it isn't independent either, although it agreed with the stages on every benchmark run it judged (five of six).
  • The prompt was developed mostly with GPT-5.6 Sol. 13 of the 18 direct-mode runs used it, so if the prompt favours either model it is likely GPT-5.6 Sol.
  • Tokens only. No prices are given, and reasoning effort was fixed at medium for both models.
  • Calibration. On the basket's side, the overhead camera's positions are good to about ±3 cm, because the gripper can't be picked out against the black basket during calibration.

Conclusion

A language model can run a pick-and-place loop on a real arm, slowly

With the right pictures and a pipeline that refuses unsafe moves, both models could plan and carry out a two-part manipulation task with every motion their own choice. GPT-6.1 Sol did it in all three benchmark runs, in a median of 21 decisions and 13 minutes. GPT-5.6 Sol could do it, and did once during development, but in the benchmark it lost the bottle three different ways.

The difference was in grasps and recovery. GPT-6.1 Sol made fewer, more deliberate decisions and noticed when it had picked up the wrong thing. On the engineering side, the decisive changes were all to what the model sees: a top-down depth map in the arm's own millimetres, a memory of what scrolled out of view, and a control that matches what this arm can physically do.

All runs

Every run, with its log and videos

All 19 runs from 15 and 16 November. Replay opens the run in the portal replay above; cameras and portal open the full recordings; log is the run's own JSON record.

Loading the run table…

Code and data

Everything is in the repository

The code, the raw logs of all 19 runs and the processed tables are at github.com/izhangfly/piper-gpt-arm (Apache-2.0 licence). results/ holds each run's JSON log, runs.csv (one row per run), decisions.csv (one row per model decision), holds.csv and hunts.csv for the buzz, and summary.json. Every figure on this page is drawn from those files. tools/export_results.py rebuilds them from a runs folder.

The arm driver, including the hunting watch, is in driver/. The microphone and CAN analysis is tools/buzz.py, and the screen recordings were made with tools/record_runs.py.