Robot data comes from three places that never agree on format. We bring captures, generated worlds and simulation rollouts into one structured stream of actions.
Three sources, three formats
Robot learning runs on three kinds of data, and each arrives in its own shape:
- Real — egocentric video, teleoperation and robot logs. Ground truth, but slow and expensive to collect.
- Synthetic — generated worlds, objects and edge cases. Endless variety, but only useful if it stays true to the original motion.
- Sim — rollouts, rewards and failures from physics. Cheap to repeat, and the only place a policy can fail safely.
Most teams keep these in three separate pipelines, with three sets of labels that never quite line up. Training on all of them means stitching them together by hand.
Everything passes the same gate
In World Actions every stream goes through the same validation step before it becomes training data. Each episode is cleaned, segmented into its steps and labeled — hands, objects, depth and the action itself — whether it was filmed, generated or simulated.
Because a generated variation keeps the original hand and timing, and a sim replay keeps the original trajectory, the labels transfer instead of being redone.
One action layer out
What comes out is one dataset with six aligned lanes per episode:
- Video — every camera, every frame
- Pose — hands and bodies
- State — the scene and the objects in it
- Action — what was done, step by step
- Trajectory — where it went
- Metadata — where it came from and how it was made
It exports straight to LeRobot, RLDS or MCAP, so a policy can train on real, synthetic and sim side by side without knowing which was which.
Three sources in. One action layer out.
This is a draft — edit it from the editor before sharing.