Proposal: one SimplerEnv task as a cross-simulator evaluation pilot

[omnilink-tech]

Hi SimplerEnv team,

I maintain OmniSim, an Apache-2.0 simulator built for agent-driven world authoring, execution, inspection, and reproducible evaluation over HTTP/MCP.

SimplerEnv's MMRV and Pearson-correlation framing is exactly the kind of evaluation discipline we want. We would like to implement one small SimplerEnv task in OmniSim and evaluate it through your existing methodology:

• you nominate one Google Robot or WidowX task with a stable observation/action/success contract;
• we handle the robot/task port and publish all scripts and asset-provenance notes;
• we run the same policy set where technically feasible and feed OmniSim's results into the existing MMRV/Pearson tooling;
• we report failures and mismatches, not just successful screenshots.

OmniSim is not photorealistic, so we would avoid presenting this as a visual-matching replacement. The useful question is whether a second simulator can preserve the task contract and policy ranking.

Which single task and policy pair would make the smallest credible pilot?

Best,
Ahmed Fetouh — OmniLink

Disclosure: this note was drafted with the same AI-agent workflow used to build OmniSim and reviewed/sent by me.

[xuanlinli17]

Hi Ahmed, you can try the google robot pick coke can and the move near tasks; in fact all of SimplerEnv tasks shouldn't be difficult to port to another simulator w/ agents.

[omnilink-tech]

Thank you -- that is helpful, and google_robot_pick_coke_can is a good
starting point.

One useful coincidence: we started a contract port of ManiSkill's PickCube-v1
in parallel (invited in mani-skill/ManiSkill#1485), and since SimplerEnv is built
on SAPIEN + ManiSkill2 the two share a stack. Two things we hit there apply
directly to the coke-can task, and both turned out to be missing capabilities on
our side rather than anything wrong with yours:

1. Per-link friction. The Google Robot's gripper pads and the object need
   different coefficients, exactly as ManiSkill's Panda sets mu=2.0 on
   panda_leftfinger/panda_rightfinger against a 0.3 default everywhere else.
   Our simulator had a single global friction value, so that ratio was not
   expressible at all. We shipped per-body friction today because of this.
   Related, and worth knowing if you ever compare contact behaviour across
   engines: we measured that MuJoCo combines two contacting geoms' friction by
   the elementwise maximum, so raising one side of a contact raises it while
   lowering one side does nothing. PhysX's combine mode is configurable and
   defaults differently, which is a real source of "same numbers, different
   grasp" between the two.

2. Gravity handling on the arm. ManiSkill disables gravity on every Panda
   link (base_agent.py, balance_passive_force defaults on) and its PD gains
   are tuned for that world. We had no per-body gravity control at all; that
   also shipped today. If SimplerEnv's Google Robot inherits the same
   convention, any port that simulates the arm under gravity with the original
   gains will diverge immediately, and it will look like a controller problem
   rather than a modelling one.

The question I would rather ask before porting anything, because I think it
decides whether a pilot is worth your time: SimplerEnv's contribution is the
real-to-sim correlation -- MMRV and Pearson against real rollouts -- and a
large part of what makes that work is visual matching, which we cannot
reproduce. So a straight port of the task contract would measure something
different from what your numbers measure.

Two framings that seem defensible to me, and I would rather you pick than have
me assume:

• (a) Ranking preservation only. Port the task contract and success
  predicate, run the same policy set, and report whether the ordering of
  policies is preserved even though absolute success rates are not comparable.
  That tests one specific claim -- that policy ranking survives a change of
  simulator -- and it fails honestly if it does not.
• (b) Dynamics isolation. Keep your images entirely, and use the second
  simulator only to ask whether the physical half of the task (grasp
  stability, object dynamics under the same commanded trajectory) is where
  sim-to-real error accumulates, versus the visual half.

(b) is the more interesting question to me and the less duplicative of your
existing work, but it is also more invasive of your pipeline, so (a) may be the
sensible first step.

Either way we would publish the failures and the interface gaps rather than a
success rate -- as above, most of what we found in the sibling port were
shortcomings in our own simulator, and those are worth writing down.

No rush on this; I know the repo has been quiet for a while.

[xuanlinli17]

I would prefer (a) Ranking preservation only ; but given that there are significant friction gaps and gravity handling gaps between the 2 simulators, and even if these are correctly handled in your sim, there are still gaps between the 2 simulators, so one always need to perform system identificantion (e.g., PD parameters) before performing evaluation.

[omnilink-tech]

Understood — (a), ranking preservation.

Your system-identification point is the part I want to get right, because it
cuts both ways and it decides what the result can honestly claim.

If we tune PD gains (and friction, and whatever else) until OmniSim reproduces
SimplerEnv's behaviour, then a preserved ranking is partly a measure of how hard
we tuned, not of whether the ranking is robust to a change of simulator. Taken
far enough, sysid can manufacture agreement — and a result that says "the
ranking survives, after we fitted our simulator to yours" is close to
uninformative.

But the opposite is not honest either: comparing an untuned port against a
tuned one measures our failure to calibrate, not the thing you care about.

So the protocol I would propose is to report both, and to publish exactly
what was fitted:

1. Un-identified arm. Physical parameters taken directly from the asset
   files, no fitting. Ranking reported with the honest caveat that absolute
   success rates are not comparable.
2. Identified arm. A named, minimal set of parameters fitted — PD gains
   first, since that is where you point — against a stated target (e.g.
   matching the arm's step response on a fixed trajectory, not matching task
   success). Publish the fitted values and the objective, so the fit is
   auditable rather than a free parameter.
3. Report the ranking under both, plus which policies changed rank and by
   how much. If the ranking only survives after fitting, that is a finding about
   how much sim-specific tuning a real2sim benchmark carries, and it is worth
   stating plainly rather than hiding in a success rate.

Fitting against a trajectory rather than against task success matters here — if
the objective is task success, sysid and the evaluation are the same quantity
and the result is circular.

Two things I would flag before starting, so the scope is realistic:

• Ranking has weak statistics with few policies. With a handful of policies
  and per-policy noise, a preserved ordering is not strong evidence. I would
  rather report per-policy success with confidence intervals and the rank
  correlation, and say plainly when the intervals overlap, than claim a
  preserved ranking that a re-seed would break.
• The gravity gap you allude to is the sharper one. ManiSkill disables
  gravity on every arm link and tunes its gains for that world. If SimplerEnv's
  Google Robot inherits that convention, then a port simulating the arm under
  gravity is not a tuning difference at all — it is a different mechanical
  problem, and sysid would be papering over it. Worth confirming that early,
  because it changes whether step 1 is even meaningful.

We shipped per-body gravity compensation and per-link friction on our side
yesterday, so both are now expressible rather than approximated — which is what
makes step 1 worth running at all rather than starting from a fitted model.

No timeline promises; I will come back with the un-identified arm first and the
interface gaps, and we can decide whether the identified arm is worth the effort
once there is a number to look at.

[xuanlinli17]

Tbh sysid is very crucial for good sim2real correlations (and in fact we didn't claim ranking preservation across different sims w/o good sysid on each; in fact what we mean is control gap matters a lot and visual gap also matter) - it's a one-time effort for each robot type (not per robot). Sysid is used match sim & real robot qpos & ee pose trajectories, not success rate.

[omnilink-tech]

That resolves it, and it corrects an assumption I had built into the proposal.

I had framed the un-identified arm as a fairness control — the idea being that
if we only report the tuned result, a preserved ranking is partly a measure of
how hard we tuned. But if the claim was never "ranking is preserved across
simulators without sysid", then that control is testing something you did not
assert, and reporting it as though it were the honest baseline would misrepresent
the benchmark rather than stress it.

So, revised, and I think this is now just your protocol rather than mine:

• Sysid first, per robot type, against qpos and end-effector pose
  trajectories — not against task success. Good to have that confirmed
  explicitly; it is the part that keeps the evaluation from being circular, and
  it is the thing most likely to get skipped by someone porting in a hurry.
• The ranking result is then reported only on the identified arm, with the
  fitted parameters and the fitting objective published so the sysid is
  auditable rather than a free parameter.
• The un-identified arm keeps a role, but a different one: as a measurement
  of the control gap you are pointing at, reported as "here is what this
  benchmark costs you if you skip calibration", not as a fairness baseline. If
  that number is large, it is an argument for sysid, which is your point.

One thing I will still flag, because it constrains what we can honestly conclude
either way: with a handful of policies, a preserved ordering is weak evidence on
its own. I would rather report per-policy success with intervals and the rank
correlation, and say plainly when the intervals overlap, than present a preserved
ranking that a re-seed would break.

Concretely, then, and without promising a timeline: pick coke-can-pick as you
suggested, do the arm sysid against trajectories, and come back with the
per-policy numbers plus whatever interface gaps the port surfaces. If the sysid
itself turns out to be the hard part, that is worth reporting too — it would say
something useful about how portable a real2sim benchmark actually is.

Thanks for being precise about this; it saved us running an experiment whose
headline would have been misleading.