Your Robot Dataset Is Mostly Duplicates

Everyone selling robot training data is selling volume. Ten thousand episodes. A hundred thousand. Millions of frames. I understand why, it's the number that's easy to put on a slide, and it's the number buyers have been trained to ask for since the vision era. It's also close to meaningless.

If you generate ten thousand pick-and-place episodes by randomizing object position within a few centimeters, you don't have ten thousand training examples. You have one training example, sampled ten thousand times, plus some noise. The policy learns the thing you sampled. Then it meets a real kitchen, encounters the situation you never generated, and fails, and everyone blames the sim-to-real gap, as though it were weather rather than a consequence of what we chose to put in the dataset.

The metric that matters isn't how much data you generated. It's how much of the space you covered.

Coverage is a measurable property

This is the part that gets treated as philosophy when it's actually arithmetic. You can measure coverage directly, and once you can measure it, you can optimize for it.

Every scenario we generate gets an embedding vector: a point in a high-dimensional space where distance corresponds to how different two scenarios actually are. When a new scenario arrives, we compare it against everything already in the repository and find its nearest neighbor. If nothing is close to it, it's genuinely new. If something sits right on top of it, it's a duplicate wearing a different random seed.

N = 1 - max_i cos(e_new, e_i)

Novelty is one minus the highest cosine similarity found. Near 1 means nothing like this exists yet. Near 0 means you already have it. Scenarios that don't clear the bar get rejected — not stored, not counted, not sold.

That rejection step is the entire point, and it's the thing that feels wrong to people the first time they hear it. We throw away data we already paid the compute to generate. We do it because a dataset that counts its duplicates is lying about its own size, and a policy trained on that dataset inherits the lie in a way that only surfaces on real hardware, expensively, later.

The bar has to move

A fixed similarity threshold is wrong at both ends of a repository's life, and this is where most implementations quietly break.

Early on, when the repository is nearly empty, you want to be strict. Almost everything is genuinely novel at that stage, and letting near-duplicates through during the first stretch poisons the baseline you'll be comparing everything else against for the rest of the corpus's life. Later, when the repository is large and covers the space well, that same strictness starts rejecting legitimate variation, in a dense region, everything is somewhat close to something.

So the threshold relaxes as the repository matures. And this is the part thats interesting, it only relaxes if coverage is actually healthy. We track the rate at which deployed systems query for situations we don't have. If that rate climbs, the threshold snaps back to strict no matter how large the repository has grown. Size alone never earns leniency. A big repository with holes in it is a big repository with holes in it.

There's a second correction, for the fact that domains aren't equally dense. Raw novelty scores get normalized against a rolling baseline for their own domain. Without that, sparse domains look artificially novel and mature ones look artificially stale, and you end up rewarding whichever domain you happened to under-explore rather than the scenarios that genuinely add information.

Pointing generation at the holes

Measuring coverage is only half of it. The other half is doing something about the gaps you find.

When a deployed system queries for a situation and nothing adequate comes back, that's recorded as a gap, automatically, with no one needing to notice or file anything. Gaps cluster by similarity, and clusters that keep growing raise the priority of the region they point at. Sparse domains carry their own weighting, decaying as they fill in.

What this produces is a system that directs itself. Deployment tells you what's missing, scoring turns that into a priority, and generation moves toward the long tail instead of grinding out more of what you already have. The corpus grows toward what robots actually encounter rather than toward what's easy to simulate, and those two things diverge fast once you leave the lab bench.

Why publish any of it

The formula above is textbook. Anyone with an embedding model and an afternoon can implement nearest-neighbor rejection, which is exactly why I'm comfortable putting it in writing.

What's hard is everything the formula sits on top of. An embedding space where distance corresponds to physical difference rather than surface description. A physics pipeline whose validation artifacts can be trusted. Thresholds tuned against thousands of real accept and reject decisions, because the right values aren't derivable, you find them by watching the system behave and correcting it. And the corpus itself, the long tail, which by definition can't be scraped or shortcut because nobody else has generated it.

That's the honest version of a moat: the idea is free, the implementation took months, and the data can't be copied. I'd rather argue about the approach in public than have everyone quietly ship datasets whose row counts don't mean what they say.

The industry is going to spend the next few years discovering its datasets were smaller than they looked. I'd rather measure it now.