Real vs Simulated Robot Training Data

Every robotics team eventually has the same argument. Should you buy real data or generate it in simulation?

The argument is usually framed badly. It is not real against fake. Simulation is a serious tool, and the best simulation companies are doing serious engineering. The useful question is narrower. Which parts of the problem does simulation solve completely, and which parts does it not solve at all?

This is the second post in this series on the robot training data market. The first post covered the companies in the market.

What simulation genuinely solves

Start with what simulation does better than real capture, because it is a longer list than critics admit.

Volume. You can generate a million trajectories overnight. No rigs, no consent forms, no travel.

Labels. In a simulation, you know the position of every object to the millimetre, because you put it there. There is no annotation cost and no annotation error.

Control. You can change one variable and hold everything else fixed. Lighting, friction, object mass, camera position. That is very hard to do in the real world, and it is what you need for systematic testing.

Rare and dangerous events. You will never capture ten thousand examples of a robot nearly dropping a person, and you should not try. In simulation, you can.

Any team that dismisses simulation is giving up all four of these for no reason.

Lightwheel and the strongest counter-argument

The standard criticism of synthetic data is that it looks right and behaves wrong. Cables stretch in ways cables do not stretch. Surfaces slide without friction. Joints rotate without resistance. A policy trained on that learns physics that does not exist.

Lightwheel has given the most serious answer to this. Its recent work centres on physical measurement. It measures real objects for bending stiffness, torsional rigidity, friction coefficients and plastic deformation, then builds simulation assets from those measurements. The assets are structured in OpenUSD, so the same object description can be used for measurement, simulation and deployment.

This is not a marketing response. It is an engineering response, and it closes a real part of the gap. Anyone arguing about synthetic data who has not read Lightwheel’s work on this is arguing against a version of simulation that no longer exists at the top end.

Where simulation still breaks

Here is the part measurement does not fix.

The sim-to-real gap is not mainly a physics problem. It is a distribution problem.

Simulation reproduces the world you thought to model. Real deployment failures come from the world you did not think to model. The pallet that is stacked wrong. The floor that is wet in one corner. The worker who reaches across the robot’s path because that is how they have always done it. The shelf label that has been replaced with a handwritten sticker.

You cannot measure the friction coefficient of a situation you have never seen.

This is why the gap keeps reappearing after every round of improved physics. Better physics fixes the simulated scenarios. It does not generate the missing ones.

The asymmetry that decides the argument

There is one structural fact that settles where the line should sit.

You can build a simulation from real capture. You cannot recover real capture from a simulation.

Real-to-sim is a one-way process. Capture a real environment with accurate geometry, and you can reconstruct it as a simulation asset, then generate variations from it. Start in simulation, and there is no operation that gets you back to the real distribution.

That makes real capture the upstream asset. Simulation is downstream of it. This is not an argument against simulation. It is an argument about order.

What most serious programs actually do

The teams doing this well have stopped choosing. The pattern looks roughly like this.

Real capture defines the distribution. It tells you what actually happens in the target environment, how often, and in what variety.

Simulation multiplies it. Once you know the distribution, you can generate variations around it cheaply, and you can stress-test the edges.

Real data validates. Whatever the simulation says, the policy gets tested against real recordings of the real environment before it goes anywhere near a deployment.

Config is a good example of the last part. It trains and evaluates policies on its own data rather than shipping a dataset and stopping. Its stated view is that data is only validated by closing the loop in the physical world.

Where DreamVu sits

DreamVu captures real data with measured 3D geometry. That is not opposed to simulation. It is what good simulation should be built on.

Its 360-degree stereo capture produces real geometry of real environments, so the output can be reconstructed into simulation assets rather than modelled by hand. Teams use the data both ways. Some train directly on it. Some use it to build environments that match a specific site they are deploying into.

The trade-off is straightforward. DreamVu cannot generate a scenario that did not happen. A buyer who needs ten thousand variations of a rare failure needs a simulation vendor instead. What real capture provides is an accurate record of what the environment is actually like, which is the thing the simulation needs to be right about.