01The reactive trap: robots that only learn when told
Today's most capable agentic robots are surprisingly good at one thing: given an instruction, they write an executable program that strings together perception and control primitives, watch it run, and revise across a few attempts. This is the Code-as-Policy (CaP) paradigm. The robot does not output low-level motor torques; it writes Python that calls segment, plan_grasp, goto_pose, and so on, then reads back what happened.
But all of that learning is reactive. The agent only starts working after it receives an explicit external instruction. Even when a successful execution gets saved as a reusable skill, that skill is an accidental leftover of solving an assigned task, not the product of any deliberate practice. The paper asks the question these systems never do: what should a robot practice and accumulate before anyone asks it to do anything?
Standard CaP synthesizes a program $\pi$ from a scene context $c$, a set of primitives $f$, and an instruction $l$. The whole loop is keyed on $l$. Remove $l$ and the system has nothing to do. The paper's move is to remove the instruction and replace it with a task the robot proposes to itself, turning play into a continual skill-learning phase that runs before deployment.
The alternative everyone reaches for, end-to-end vision-language-action (VLA) policies, folds perception, dynamics, and control into opaque weights. They generalize poorly under perturbation: on the robustness benchmark LIBERO-PRO, the strongest VLA tested (π_0.5) reaches only 12.8% average, and OpenVLA and π_0 both score 0.0. Something more inspectable and more reusable is needed.
02Children play before they're told: the developmental bet
A three or four year old is not handed a chore list. It picks objects up, opens drawers, stacks blocks, and pushes things off tables purely for fun. In doing so it discovers controllable effects and builds a toolkit of motor skills, near the edge of what it can already do, that it later deploys when finally asked to accomplish something. This is the long tradition of developmental robotics and intrinsic motivation: curiosity, learning progress, goal babbling, the Goldilocks effect from infant attention studies.
The bet of this paper is that these old ideas are newly powerful in the foundation-model era for one specific reason: a coding agent can express its goals in language, run them as programs, and save successes as callable code. Earlier intrinsic-motivation work explored fixed sensorimotor or feature spaces. RATs explores a space of language-described, program-executed tasks, and the skills it saves are inspectable functions, not weights.
Playful Agentic Robot Learning: an embodied coding agent uses self-directed play, self-proposed and self-executed tasks with no external instruction, as a continual skill-learning phase before downstream tasks are specified. The question shifts from how should the robot solve this task to what skills should it practice and stockpile before being asked.
03Three teams at play: the RATs architecture
RATs stands for Robotics Agent Teams, and the name is literal: it is a multi-agent system organized into three cooperating teams that turn play into an explicit skill-acquisition process. The whole thing runs in two phases. In Phase 1 (play), the system loops for $N$ iterations, maintaining a skill library $\mathcal{L}$ and a failure memory $\mathcal{M}$. In Phase 2 (evaluation), the library is frozen and reused on held-out tasks.
Each play iteration runs three steps. The Task Proposer team generates and selects a self-directed task $\tau_t$ using a curiosity rule, then builds the play environment. The Execution team runs that task through a bounded Write-Execute-Verify-Diagnose loop, planning, writing code, verifying both the goal and every step, and diagnosing failures, with a default budget of 5 attempts per task. The Memory-Management team distills successful code into the library, records compact lessons from failures, updates reliability tiers, and periodically curates everything.
The single idea that makes this work is dense, step-level feedback instead of a sparse task-level reward. A lone "task failed" signal cannot tell you which substep broke or what reusable behavior to save. By grading every step and diagnosing failures, RATs can preserve the substeps that worked, localize the capability that is missing, and turn recurring physical bottlenecks into named, reusable helper functions. Dense feedback is what makes play informative.
04Choosing what to practice: the Goldilocks proposer
Play is only useful if the robot practices the right things. The Task Proposer is prompted to role-play a curious toddler: it looks at what objects are visibly within reach, what skills it already has and how reliable they are, and what it recently tried, then brainstorms a handful of candidate tasks (typically $K=5$). The trick is in how it picks one. Each candidate $\tau$ is scored by the product of two terms, and the winner is the argmax.
The first term rewards trying things the robot has rarely drilled. It is a count-based novelty bonus over the object-skill pairs the task touches.
The second term is the competence frontier: a difficulty thermostat that peaks where the robot succeeds about half the time.
The widget below is the proposer's decision in miniature, wired to the paper's Table 7. Drag the tissue box's success rate $\bar r$ (and how many times it has been practiced) and watch its dot ride the competence parabola while the SELECTED badge jumps to whichever candidate currently maximizes $\mathcal{N}\cdot\mathcal{F}$. On load it reproduces the worked example: lifting a brown tissue box wins at about 0.62 over the two nearly-mastered drawer moves at 0.36.
Scene grounding proposes five candidate tasks; one (an unsupported cloth pick) is vetoed as unreachable, leaving four. Because none of these object-skill pairs has been attempted yet, all four carry the same novelty $\mathcal{N}=1.0$, so the choice is decided entirely by learnability $\mathcal{F}$. Lift brown tissue box scores $\mathcal{F}=4(0.1933)(0.8067)=\mathbf{0.6236}$ because its grasp-and-lift family has a Wilson-bounded reliability of only $\sim0.19$, squarely in the learnable zone. It beats two nearly-mastered drawer moves ($\bar r\approx0.9$, so $\mathcal{F}=0.36$) and a near-hopeless cloth place ($\bar r=0.05$, $\mathcal{F}=0.19$). The robot deliberately chooses the semi-familiar task over the ones it has already nailed.
The full deployed ranker adds two more terms, $s(\tau) = \mathcal{N}(\tau)\mathcal{F}(\tau) + w_B\,B_{\text{retry}}(\tau) - w_P\,P_{\text{fail}}(\tau)$: a small surprise bonus $B_{\text{retry}}$ for promising past near-misses worth re-attempting, and a penalty $P_{\text{fail}}$ for tasks too similar to recent failures. In a clean proposal turn both are zero and the score collapses back to pure $\mathcal{N}\cdot\mathcal{F}$.
Test yourself: why a product 𝒩·ℱ, and not a sum 𝒩+ℱ?
05Trying, failing, and fixing one step at a time
Once a task is chosen, an Environment Creator turns the abstract idea into a real, runnable scene (in LIBERO it writes a BDDL goal specification and instantiates a MuJoCo environment), sanity-checks it over two reset seeds, and only then hands it to the Execution team. That team runs the accepted task as a Code-as-Policy program inside a bounded retry loop with a clear division of labor.
The Planner writes an ordered plan and predicts likely failure points. A Planner Verifier checks the plan is grounded in what the camera actually sees. The Policy Writer turns the verified plan into Python using only the listed primitives and learned helpers. A Quality Checker statically lint-screens that code (syntax, unavailable APIs, unbounded loops) before any robot time is spent. After execution, two graders run: a Goal Verifier for overall success, and, the key piece, a Per-Step Verifier that issues a localized pass/fail verdict for each step using its objective, code slice, runtime values, and before/after visual evidence. On failure, a Failure Diagnoser names the failure category, the first failed step, and a concrete repair, and routes the fix.
A single "task failed" is like a teacher writing only "wrong" on a multi-step problem. The per-step verifier instead grades every line and the diagnoser writes "you set up the equation right but dropped a sign on line 3", so the next attempt fixes the exact broken step instead of redoing the whole thing. Dense feedback makes retries surgical; sparse feedback forces blind full rewrites.
The widget makes this concrete. A four-step manipulation (localize, grasp, lift, place) fails at the grasp. Toggle between Sparse and Dense feedback and run the retry: under Sparse only a final pass/fail is available, so every step gets rewritten; under Dense, per-step verdicts plus a "WHEN/WRONG/DO" diagnosis rewrite only the broken step and freeze the ones that already worked.
When one sub-action keeps failing, say a tricky grasp, the Diagnoser can spawn a SubAgent: a focused agent that drills just that move in isolation from the same reset state, often trying 2-3 distinct physical strategies in parallel (top-down vs side grasp, different localization paths), and returns the first visually verified helper. But that helper earns a permanent spot in the library only after it contributes to a real end-to-end success, keeping the library free of overfitted local fixes. It is a batting cage: you drill the missed swing off to the side, then only add it to your repertoire once it helps win an actual game.
06Building a trustworthy toolbox and a notebook of mistakes
The Memory-Management team is what makes play accumulate. On a successful end-to-end trajectory it extracts at most two self-contained, parameterized helper functions from the executed policy, with docstrings and type hints, never a "do-everything" composite, and never a duplicate of an existing skill. Each new skill enters the library $\mathcal{L}$ as experimental, storing its source, preconditions, expected effects, usage counts, success counts, and a reliability tier.
The lifecycle is yours to drive below. Pick a skill, log a success or a failure, and watch its tier move with its Wilson-bounded reliability: a fresh skill earns "verified" after three uses at 50%+, and a verified skill is demoted again the moment it starts failing.
Failures are not thrown away either. Each one is recorded in the failure memory $\mathcal{M}$ and distilled into a compact lesson in the form WHEN <condition> | WRONG <approach> | DO <correction>. The Planner retrieves these lessons by task and object overlap, so later attempts avoid old mistakes without replaying long traces. Every $K=5$ iterations a Memory Curator merges, rewrites, or deletes near-duplicate or contradicted lessons. And when recurring failures reveal a missing capability, an anticipatory Skill Proposer drafts a brand-new helper as a hypothesis, which enters as experimental and earns reliability only through later use.
07Freeze and reuse: does play pay off?
At test time the playing stops. Intrinsic exploration and memory updates are disabled, the library $\mathcal{L}$ is frozen, and held-out tasks arrive. The frozen toolbox is used two ways. In plug-and-play mode, each selected skill's signature, description, and source are simply dropped into the context of a vanilla single-agent baseline, CaP-Agent0, isolating the transfer value of the code itself. In full-system mode (RATs Exec.), the same frozen library is given to the entire execution team with its verification, diagnosis, and retry machinery intact.
The improvement shows up on the object splits most (Object Pos +34, Object Task +32 on LIBERO-PRO) but is present on goal and spatial splits too, evidence the learned skills generalize beyond the exact tasks practiced. The why is visible in the paper's side-by-side code panels: where CaP-Agent0 re-derives segmentation, grasp selection, approach geometry, and motion from low-level operations every single episode, RATs calls a named helper like execute_top_down_grasp_and_lift or grasp_and_pull_horizontal_handle.
CaP-Agent0 cooks every dish from raw ingredients (query points, segment, build a point cloud, plan a grasp, write waypoints) every time. RATs writes the recipe down once as a named function and just calls it, so it stops rediscovering "how to grasp from above" from pixels in every episode. That is what makes the skills both reliable and portable: they are parameterized code, not weights.
The whole thesis is that a skill is inspectable, parameterized code and not opaque weights, so it is worth seeing one. Here is a single learned skill beside the two ways to call on it: RATs makes one named call, while the baseline re-derives the same behavior from pixels every episode.
# promoted to verified: uses 7, succ 0.86 def execute_top_down_grasp_and_lift(obj, lift_z=0.15): """Grasp obj from above at its verified contact point, then lift it by lift_z. pre: obj visible & reachable, gripper empty post: obj grasped and raised lift_z m""" pts = segment(obj) grasp = plan_grasp(pts, approach="top_down") goto_pose(grasp.pre_grasp); close_gripper() goto_pose(grasp.pose.lift(lift_z)) return verify_held(obj)
# RATs: one verified, named call execute_top_down_grasp_and_lift("tissue_box") # CaP-Agent0: re-derives it every episode pts = query_points("tissue box") mask = segment(pts) pc = build_pointcloud(mask) g = plan_grasp(pc, approach="top_down") for p in waypoints(g.pre_grasp, g.pose): goto_pose(p) close_gripper(); goto_pose(g.pose.lift(0.15))
08Skills travel: cross-simulator, cross-embodiment, real robot
The strongest argument that code is a better substrate than weights is that the skills move. Skills learned during LIBERO-PRO play, plugged into CaP-Agent0 with no finetuning, lift performance on simulators and hardware the system never played in.
The ledger below makes the honesty interactive. Toggle the frozen library on and off: every transfer is measured against the same CaP-Agent0 baseline, the gains stacking up in green and the one regression (a mis-retrieved skill on the two-arm handover) dropping below in red.
The cross-embodiment result is the headline: skills practiced on the single-arm LIBERO-PRO robot transfer to a two-arm collaborative lift in a different simulator, a transfer that is hard to imagine for opaque policy weights but natural for inspectable, parameterized code. On the harder real-world rearrangement tasks, the standard baseline fails "Swap Cubes" entirely (0/30) while the skill-augmented agent succeeds 7/30. Not everything improves, though: a two-arm handover regressed by 4.0 points when a retrieved skill did not fit the task, an honest reminder that better retrieval still matters.
09Was it really the play? Ablations and honest costs
It would be easy to suspect the gains come from "more compute at test time" rather than from play. The paper rules this out carefully, and is candid about what play costs.
Curiosity matters; mere play does not
On LIBERO-PRO under the CaP-Agent0 baseline, random play adds only +1.5 pp (23.2 → 24.7) while curious play adds +9.1 pp (23.2 → 32.3). The Goldilocks selection, not the act of playing, drives the gain. Play-time skills and the test-time execution loop are also complementary: improving only execution gets you from 23.2 to 36.3, improving only play gets you to 32.3, and combining both reaches the best result of 44.3. Cramming at random the night before helps a little; practicing the right things, and banking them as reusable code, helps a lot.
Play is token-heavy: roughly 30M tokens for a 50-iteration phase, and the Failure Diagnoser alone consumes 40.5% of play tokens (policy writing another 28.8%), so the Write-Execute-Verify-Diagnose loop dominates cost. Other limits: skill diversity is bounded by what the simulator can express; improper skill reuse can hurt (the two-arm handover regression); reliability tiers are noisy early (3 uses to verify, 10 to deprecate); skills are extracted only from successful trajectories plus anticipatory proposals; the system is capped by primitive-level control APIs (no learned low-level geometric perception, since camera calibration is deliberately not exposed); and real-world evaluation remains preliminary (40 or 30 trials per task on a handful of tasks).
10Where this sits, and where it goes
RATs is, in effect, Voyager for physically-grounded robotics: it ports the automatic-curriculum and executable-skill-library idea from Minecraft to real manipulation, and operationalizes "sleep-time compute", spending effort before queries arrive. It extends the Code-as-Policy line (CaP, ProgPrompt, CaP-X) by bolting on a proactive skill-acquisition phase, and it directly instantiates classical developmental-robotics ideas (Schmidhuber-style curiosity, Oudeyer-Kaplan learning progress, the Goldilocks effect), but expresses goals in language and behaviors as programs rather than in fixed sensorimotor spaces.
The contrast with VLAs is the through-line of the whole paper. Where OpenVLA, π_0, and π_0.5 fold perception, dynamics, and policy into opaque weights that crumble under LIBERO-PRO perturbations, RATs keeps skills as inspectable, composable, verified code and never finetunes the backbone. The dense per-step verification and diagnosis loop is, conceptually, a reasoning-acting-reflecting agent specialized to embodied control.
The main RATs play uses gemini-3.1pro-preview as the backbone; the appendix token-cost analysis used gpt-5.5. Play runs 50 iterations on each of LIBERO-PRO and MolmoSpaces, with up to 5 attempts per task. The LIBERO library ends play with 47 learned skills, MolmoSpaces with 27; across 400 MolmoSpaces eval trials, 391 invoke at least one learned skill.
The one-sentence takeaway: a self-built library of verified, reusable code skills, accumulated through curiosity-driven play with dense per-step feedback, is a practical, finetuning-free path to more capable and more transferable agentic robots, a step toward machines that, like children, learn what to do before they are asked.