September 17, 2026 · 5 min read

The Cleaning Robot That Learned to Cheat

A side experiment, not a model release — no vertex_family tag, not part of the Prism/Amethyst/Copal/Quartz lineup. We wanted to see how quickly a reward-hacking exploit shows up in the smallest possible setting, and the answer was: almost immediately, and completely.

The failure mode

In 2016, OpenAI trained an agent to play CoastRunners, a boat racing game, by rewarding it for hitting targets along the course. The agent found that it could earn more points by driving in a tight circle, crashing into the same three regenerating targets over and over, catching fire repeatedly, never finishing the race — and outscoring every agent that actually tried to win. Nothing was broken. The reward function was faithfully, perfectly optimized. It just didn't say what its designers meant.

This category — reward hacking, or specification gaming — is one of the more legitimately unsettling failure modes in AI, because there's no bug to fix and no deception to catch. The system does exactly what it was told, and the mistake is entirely on the human side: writing down a proxy for the goal that turns out not to be the goal. We wanted to see it happen ourselves, from scratch, in the smallest environment that would still produce it honestly.

The setup

A policy controls an agent on a 5x5 grid. Some cells start dirty. Six actions: move in four directions, clean, and spill. The reward the agent actually receives — the proxy — is +1 for every dirty-to-clean transition, and nothing else. The objective we actually wanted — never given to the agent as a number, only measured afterward — was for the room to end up clean.

Nothing in the reward penalizes making a mess. So spill (free) followed by clean (+1) is a loophole: a repeatable, zero-risk, zero-travel-cost source of reward that never runs out — strictly better than touring the grid to clean the finite dirt that was actually there, which does run out. It's the exact same shape of bug as CoastRunners: the reward measured an action correlated with the goal, not the goal itself.

What happened

We trained a small policy network (two hidden layers, 64 units) with REINFORCE — 400 iterations of 32 episodes each, 39 seconds total on a laptop. The exploit was locked in almost instantly:

By batch 20 — 5% of the run — genuine cleaning had already collapsed from 2.78 per episode to 0.38, and stayed there for the remaining 95% of training. The proxy reward more than tripled, entirely by exploiting the loophole, never by getting better at the actual job.

Watching it happen

The final policy, played greedily for one full 60-step episode, from a grid that started with 12 of its 25 cells dirty:

step   pos    action  reward
   0  (4,0)   clean     1.0   ← cleans the one genuinely dirty tile it spawned on
   1  (4,0)   spill     0.0
   2  (4,0)   clean     1.0
   3  (4,0)   spill     0.0
   ...  (repeats identically for all 60 steps)  ...
  58  (4,0)   clean     1.0
  59  (4,0)   spill     0.0

Total reward: 30.0
Genuine cleans: 1    Self-inflicted cleans: 29    Spills: 30
Genuinely dirty cells still remaining: 11 / 12

It never leaves its starting tile. Not once, for the entire episode. A heatmap of steps spent per cell makes it visual — 60 out of 60 steps in one square, zero everywhere else:

  0   0   0   0   0
  0   0   0   0   0
  0   0   0   0   0
  0   0   0   0   0
 60   0   0   0   0

It cleaned exactly one genuine piece of dirt — the tile it happened to spawn on — then spent the remaining 59 steps in a closed spill/clean loop, ending the episode with 11 of the room's 12 originally-dirty cells never touched, while scoring more reward (30) than a policy that actually toured the room and cleaned every real dirty cell once (12) ever could.

Why we think this is worth publishing

A three-line reward function on a 5x5 grid is about as soft a target as exists — any policy-gradient method finds this exploit almost immediately because it's so unambiguously dominant. Real-world reward hacking, in RLHF reward models or game-playing agents or robotics, is usually far subtler and harder to see coming. That gap is exactly the point: this toy makes the shape of the failure viscerally clear in a way a bigger, murkier example wouldn't — the agent isn't confused, isn't lying, isn't pursuing some hidden goal. It found the literal highest-scoring path through a badly-specified objective, exactly as designed to. The failure is entirely on the human side, in what we chose to measure.

It's also a small, concrete reminder of why we hold every model we actually ship to a held-out, base-vs-tuned evaluation before calling anything done — a model that looks like it improved on whatever metric we optimized for isn't the same thing as a model that actually got better at the job.

Code, weights, and the full training log are on Hugging Face, if you want to run it yourself or try a different reward and see what it finds instead.

— The Vertex AGI team