01The bottleneck: humanoid data is scarce, human video is not
The dream behind humanoid robots is general-purpose physical work: a robot that can walk over, pick up a box, kick a door shut, hand you a tool. The blocker is data. Teleoperating a humanoid through thousands of contact-rich behaviors is slow, expensive, and physically punishing on the hardware.
Meanwhile, there is an ocean of humans doing exactly these motions on video. If we could reliably convert human motion into robot motion, we could manufacture large synthetic datasets and sidestep the data bottleneck entirely. That conversion is called retargeting, and it is harder than it sounds, especially for loco-manipulation: tasks that combine walking with handling an object, where the system is underactuated in two places at once, the robot's own balance and the free-floating object it is pushing around.
A manipulation policy lives or dies by its contact information: exactly when and where the robot touches the object. Naive retargeting produces wrong contacts (a hand that floats a centimeter off the box, a foot that slides through the floor), and an RL policy trained on wrong contacts learns nonsense. Getting accurate contacts out of a kinematic retarget has been the field's stubborn weak link.
02Why copying the human pose breaks physics
Standard retargeting starts with inverse kinematics (IK): pick corresponding keypoints on the human and the robot, then solve for robot joint angles that put those keypoints in the right places, frame by frame. Methods like PHC add an unconstrained optimization step to nudge the result toward physical consistency; GMR extends PHC by matching keypoint rotations as well as positions. Both still produce motions that are only kinematically valid, not dynamically valid.
The difference matters. A kinematically feasible motion is just a sequence of geometrically legal poses. A dynamically feasible motion additionally has to be producible by real forces and contacts: no foot skating, no penetration, no object that changes velocity without anything touching it. Morphology mismatch between human and robot guarantees the kinematic version is riddled with these artifacts.
The key design decision is how stage 2 enforces physics. DynaRetarget does not write down equations of motion. It uses the MuJoCo simulator itself as the world model and optimizes through it: propose a control sequence, simulate it, and read off how well the robot and box ended up tracking the demo. Because every candidate is evaluated by actually simulating it, every candidate is automatically physically legal. The only thing left to search for is the one that best matches the reference.
03Two bad extremes
Optimizing a control sequence through a simulator is an old idea. The trouble is the horizon, how many timesteps you plan over at once. There are two obvious strategies, and both fail for long, contact-rich motions.
Extreme A: optimize everything at once (full-horizon)
Throw all the controls for the whole motion into one big optimization. The problem is that a sampling-based optimizer has no gradients to guide it, and the number of decision variables (plus the number of bad local minima) grows with the horizon. On a 4.6-second kick, optimizing the full motion at once simply does not converge: in the paper's test it failed all ten runs, with the robot falling over before it ever kicked the box.
Extreme B: optimize a short window and slide it forward (SBMPC)
This is what the prior state of the art, SPIDER, does, using a sampling-based model-predictive-control framework. At each step it solves only a short window (about one second of look-ahead), executes the first bit, then slides the window forward. It is fast, but it is myopic and greedy: once it executes an action it can never re-optimize it. If a short-sighted plan drops the box early, recovery is nearly impossible, because re-grabbing would mean deviating from the reference, which the tracking cost punishes. It also outputs jerky feedback-control trajectories that make the downstream RL harder to train.
The short-window plan can make the robot's joints track the demo beautifully and still fail the task. The object's position error keeps accumulating over the part of the future the short window never looked at. The robot looks right; the manipulation failed. The object, not the body, is the real objective.
04The idea: grow the horizon one knot at a time
DynaRetarget's central trick, Sampling-Based Trajectory Optimization (SBTO), sits between the two extremes. It starts by optimizing only the very first slice of the motion. Once that slice's solution has settled, it unlocks the next slice and re-optimizes everything so far, with the new, longer view of the future. It keeps extending the window one control point (one "knot") at a time until the whole motion is being optimized at once.
Because the earlier knots are already good, each extension is an easy, warm-started problem. But here is the beautiful part: those early knots keep getting re-tuned as the horizon grows. Each time a later knot is unlocked, the whole prefix is re-simulated against that longer future, so information about where the box has to end up propagates backward into the opening kick. After hundreds of re-solves the first knot has effectively seen roughly 3.4 seconds ahead, even though each step only formally adds a quarter-second. That look-ahead window is called the effective horizon, and it is far longer than any short-window method achieves. Drag σ_min in the widget below to set that gate yourself and watch the effective horizon, and the box tracking, grow or collapse with it.
Optimizing all controls at once drowns; optimizing a short window is short-sighted. Growing the horizon with warm-starts gets you the global, re-optimizable behavior of full-horizon planning while keeping each sampling problem small and well conditioned. The early decisions are set up correctly because they keep being re-solved against an ever-longer future.
It is how you learn a long piano piece: master the opening bar, then bars one-to-two, then one-to-three, replaying from the start each time so the earlier passages adapt to what comes next. You never try to sight-read the whole sonata at once (you would fall apart), and you never practice a single bar forever in isolation (you would never connect them).
05The machinery: knots, sampling, and the cost
Four pieces make SBTO work. None of them needs a gradient.
1. Single-shooting: let physics be the constraint
The problem is written as an optimal-control problem. Find the control sequence (a trajectory of PD motor targets) that minimizes a tracking cost, subject to the simulator's dynamics:
2. Knots: a handful of waypoints instead of every tick
Optimizing a motor command at every 10 ms tick would be an enormous search. Instead, SBTO optimizes K knots spaced every 0.25 s and interpolates between them to recover the dense control sequence.
3. The Cross-Entropy Method: guess, simulate, keep the best, repeat
The inner optimizer (called FHTO, fixed-horizon trajectory optimization) is the Cross-Entropy Method. Draw 1024 random knot vectors from a Gaussian centered on the reference motion, simulate each one, keep the best ~3% (the "elites"), recenter the Gaussian on the elites, and iterate. No gradients, so it sails straight through contacts and other non-smooth physics.
SBTO is the outer loop; FHTO is the inner one. FHTO solves a single fixed-horizon problem with the CEM above. SBTO is the schedule that decides how big a problem to hand it: unlock one knot, call FHTO to re-optimize every knot so far, unlock the next. FHTO is the engine, SBTO is the gearbox.
Take a single FHTO step on knot 3 of the kick. Draw 1024 knot vectors from $\mathcal N(\mu,\Sigma)$ with the spread still wide; roll each one through the simulator and keep the best ~31 (the top 3%), which cluster where the foot makes contact a beat later. Nudge $\mu$ toward that elite mean and shrink $\Sigma$ toward the elite spread, then repeat. Once $\mathrm{diag}(\Sigma)$ at knot 3 falls under $\sigma_{\min}\approx0.055$, unlock knot 4. The shrink is deliberately slow: collapse $\Sigma$ in one step and the gate fires before the window has actually explored. The widget above runs exactly this loop, live.
4. The cost: pin the object, free the body
The cost $J$ sums weighted tracking terms (joint, base, torso, foot, hand positions and velocities) plus collision penalties. The single most important number is the weight on object position: 40, dwarfing the weight on joint position (0.25). This is the director's note of the whole method:
The robot is allowed to improvise its own body motion, but the box absolutely must hit its marks. Physical feasibility is bought by letting the robot deviate while pinning the object to the demo's trajectory. Concretely, the weights square, so 40 versus 0.25 is a ~160× penalty ratio: a 1 cm slip of the box costs the optimizer as much as a ~13 cm drift of a joint, so it will gladly let the hip wander to keep the box within a centimeter of its line. And the velocity terms matter: tracking position alone lets the box teleport to the right spot; tracking velocity is what lets it be kicked with the right momentum.
06Why a short window can't take a move back
The cleanest way to feel the difference is to kick a box yourself with different amounts of look-ahead. Drag the planner's window below: a short window reads only the fast start of the motion, over-kicks, and the box sails past the target with no way to undo it. Widen the window and the very same kick is set up to land the box on its mark.
This irreversibility is exactly why the prior method, despite being a perfectly reasonable controller, leaves so much object-tracking error on the table. It is a chess player who only ever looks one move ahead and can never take a move back.
07Making it cheap: freeze the part that's done
Growing the horizon has a cost: naively, every time you extend the window you re-simulate the motion from the very beginning. But the opening of the motion usually converges long before the tail does. SBTO_skip notices when an early knot's variance has dropped below a tiny threshold ($\sigma_{\text{skip}}=10^{-4}$), freezes that settled prefix, and only re-simulates the still-changing tail.
Do not confuse the two knobs. $\sigma_{\min}\approx0.055$ decides when to unlock the next knot, which sets the effective horizon (the behavior knob). $\sigma_{\text{skip}}=10^{-4}$ is far tighter and decides when to freeze a settled prefix (purely an efficiency knob). $\sigma_{\min}$ shapes how far the planner looks; $\sigma_{\text{skip}}$ shapes how much it pays to look.
Test yourself: why does a lower σ_min give a shorter effective horizon?
08Does it work?
Tested on 285 motions from the OmniRetarget dataset (a Unitree G1 handling a box across pick-and-place, kicking, lifting, pushing, and dragging), against the SOTA short-window method SPIDER.
The win is the horizon-growth idea, not a richer cost. A stripped-down variant matched to SPIDER's position-only configuration (SBTO_pos) still scores 62.1%, comfortably beating SPIDER's 37.9%. So the gain comes from how the optimization is structured, not from extra cost terms.
The effective horizon claim is the conceptual heart of the results. By holding a fixed analysis time at $t^0=1$ s and watching the box's position error there, the authors show the opening of the motion keeps improving for hundreds of iterations, corresponding to about 3.4 s of look-ahead, even though each increment only adds 0.25 s. A parameter sweep confirms this effective horizon is governed mainly by $\sigma_{\min}$: it is a tunable design knob, not luck. That knob is exactly what the widget in section 4 puts in your hands: drag $\sigma_{\min}$ and watch the effective horizon, and the box tracking, grow or collapse with it.
09One demonstration, many objects
Because the refinement is driven entirely by the same tracking objective, a single human demonstration can be re-refined against different object physics and still come out feasible. From one box-handling demo, the authors generate dynamically valid motions across:
mass 0.1 – 8 kg size 0.2 – 0.4 m box cylinder chair shelf
This is the path from a handful of demos to a large synthetic dataset: collect a motion once, then let SBTO manufacture a whole family of physically consistent variations around it, attacking the data bottleneck from section 1 directly.
10From clean trajectories to a real robot
DynaRetarget trains a DeepMimic-style tracking controller with PPO and a residual action space, heavy domain randomization (object mass, friction, pose, external shoves), and, crucially, contact targets read straight from the physically consistent simulation, so no hand-designed contact heuristics or curriculum are needed.
Averaged over 1024 episodes across 8 motions, policies trained on SBTO references hit 97.1% success versus 79.4% for OmniRetarget references, with object-position error of 8.8 cm vs. 12.5 cm. Body-keypoint tracking is essentially tied (3.57 vs. 3.67 cm): the win is in object tracking and reliability, not body imitation. (Success here is strict: the object must stay within threshold of its target at every timestep.) The refined policies then transfer zero-shot to the physical G1, including the hardest behaviors like sliding the box and pushing it with the legs.
11Where it fits, and where it breaks
DynaRetarget is a clean instance of physics-as-the-world-model: instead of learning a neural dynamics model, it plans gradient-free through the simulator. That puts it in the same family of ideas as search-through-a-model planning (the AlphaZero / MuZero lineage), but with a known physics model and a sampling planner rather than learned dynamics and tree search. SBTO is the planning stage that manufactures clean targets for the learning stage, mirroring how a world model's rollouts can bootstrap a policy.
It refines, it does not repair from scratch: a demonstration with sudden contact changes or flipped object orientations can still defeat it. The optimization is heavy, but the skip trick (section 7) is what makes it practical, bringing the cost down to about 20 s of compute per second of motion on a 112-core CPU, roughly SPIDER's budget. The sampling distribution is a single Gaussian, so it captures one mode of behavior at a time; multi-modal sampling is future work. And validation is on one robot (G1) and mostly one object class. The downstream RL comparison is against OmniRetarget references only, because the short-window baselines did not succeed on enough motions to compare.
The one-sentence takeaway: don't optimize a long motion all at once, and don't optimize it one short window at a time, grow the horizon and keep re-tuning the start, and a black-box physics simulator becomes a good enough world model to turn messy human video into robot motion that survives contact with reality.