[WIP] Two-Camera Fused Pose Estimation - #82 - #1749
Open
emilybunna wants to merge 121 commits into
Open
Conversation
- Add CausalProcess, DerivedPredicate, delay distributions to structs.py - Add ~286 new settings for PyBullet envs, process planning, agent SDK - Extend utils.py with process grounding, abstract states, parallel helpers - Add colorlog, torchvision dependencies; update CI config
- Extract common robot/finger methods to base PyBulletEnv - Add objects.py module for create_object/update_object helpers - Enhance controllers with move_to_pose tolerance, press, nudge - Add BiRRT path subsampling, collision diagnostics, IK validation - Add MobileFetch robot support
Add 161 URDF models, meshes, and textures for boil, circuit, coffee, domino, fan, float, grow, laser, and switch environments. Includes partnet_mobility models (bucket, fan, faucet, switch) and custom URDFs.
Implements generic pick, place, pour, push, move_to, and wait skills with phase-based trajectories, BiRRT motion planning, and canonical continuous parameters. Skills are reusable across all PyBullet envs.
New environments: coffee, grow, boil, domino, fan, balance, float, laser, ants, circuit, barrier, magic_bin, switch. Enhanced: blocks, cover. Each includes options, NSRTs, and process ground truth models with skill factory integration. Domino uses component-based architecture.
- planning_with_processes.py: FF/LM-cut heuristics, derived predicates, process scheduling, abstract process policies - Cluster-and-search process learner with LLM-guided condition selection - ELBO optimization with learnable delay distributions - Process planning approaches: oracle, param learning, predicate invention - Enhanced predicate score functions and grammar search for processes
- Restructure main.py into dedicated setup/demo/test functions - Add timestamped run directories for logs - Enhance cogman with process-aware action execution - Add option_model use_gui override and process model support - Extend explorers with plan enumeration and fallback selection - Add pretrained model interface OpenRouter compatibility
- agent_sdk/: MCP tools, local/Docker sandboxes, session management, proposal parsing, markdown log formatting - Agent approaches: planner, option learning, abstraction learning, closed loop planning with scratchpad and visualization support - Docker infrastructure for sandboxed agent execution
…aches - Human interaction approach with motion planning and scripted policies - Human low-level control approach with keyboard input - VLM online predicate invention with transition modeling - Classification approaches (VLM, DINO similarity)
- Mara-robosim adapter wrappers for 15 PyBullet environments - LLM/VLM prompt templates for predicate invention - Developer docs, random action GIFs, env guide - Experiment configs (ExoPredicator, predicator_v3, mara_bench) - Launch scripts, plotting utilities, scripted option policies - Tests for skill factories, agent SDK tools, controllers
Improve motion planning and skill factories
Add composable config system with includes support
Add agent bilevel approach
Improve bilevel planning infrastructure
## CI Infrastructure - Parallelize unit tests into 8 shards using pytest-split - Add pip caching and coverage merge job - Add .test_durations file for balanced sharding ## Dependency Fixes - Add missing deps: psutil, claude-agent-sdk, nest_asyncio, mara_robosim - Remove wandb and opencv-python ## Code Cleanup - Rename RawState to VLMState - Fix type annotations for PyBulletState image properties - Eliminate type: ignore[unreachable] comments - Extract grid component, remove old domino envs - Remove unused process_learning module - Clean up cogman.py ## Lint & Type Fixes - Fix mypy errors for LBFGS step() with newer torch stubs - Fix pylint issues across domino envs, ground truth models, tests - Autoformat fixes for yapf/isort ## Test Fixes - Default use_gui=False to prevent PyBullet hanging in headless CI - Make OpenAI key optional for mock LLM tests - Fix dataset loading tests for pytest-split compatibility - Ensure eval_trajectories/ and results/ dirs exist in _run_pipeline - Fix active sampler explorer test flakiness - Mark domino pick test as xfail for headless CI - Retry motion planning test with multiple seeds - Fix planning determinism and external oracle approach tests
…on (#19) * Parameterize Wait option with target atoms for noise-robust termination Wait previously terminated on any atom change, making it sensitive to incidental physics noise. Now Wait can be parameterized with specific target atoms (positive or negative) that must be satisfied before termination. Falls back to any-atom-change when no targets are specified. - Add check_wait_target_atoms, parse_wait_target_annotations, strip_wait_annotations, inject_wait_targets_for_option to utils.py - Update option_model.py termination + memory propagation through re-grounding - Update LLM prompts in agent_planner, agent_bilevel, agent_option_learning approaches to document -> {atoms, NOT atoms} annotation syntax - Parse and inject Wait targets in agent_planner and agent_bilevel approaches - Inject Wait targets from atoms_sequence in process planning paths - Add tests for target atom termination, including noisy-atom-ignored test * Fix formatting and exclude logs dir from mypy * Fix pylint unused-argument warning in _inject_wait_targets
* Refactor _validate_plan_forward to use option model directly Delegate option execution to option_model.get_next_state_and_num_actions instead of duplicating its termination logic (stuck detection, Wait atom-change checks) and directly accessing its simulator. * Unify backtracking refinement search into shared run_backtracking_refinement Extract the duplicated backtracking loop from run_low_level_search (SeSamE) and _refine_sketch (agent bilevel) into a single run_backtracking_refinement function in planning.py. Both callers now delegate to it with their own sample_fn and validate_fn callbacks, eliminating ~80 lines of duplicated loop/backtracking logic. * Simplify _validate_plan_forward to use run_backtracking_refinement Replace 60 lines of manual option-model execution with a call to run_backtracking_refinement using max_tries=[1] and a sample_fn that returns the pre-grounded options. Remove unused Any import.
* Add gymnasium-API benchmark wrapper and retire mara_adapter shim
Wrap predicators' native PyBullet envs in a standard gymnasium.Env so
the suite can be consumed as a benchmark independent of the planning
framework. Drop the predicators -> mara_robosim adapter layer, which
only existed to bridge two near-identical struct types.
New:
- predicators/envs/gymnasium_wrapper.py: PredicatorsBenchmarkEnv plus
register_all_environments(), make(), get_all_env_ids(). Registers
15 envs under mara/*-v0, with a cfg_overrides kwarg for per-make
CFG overrides.
- notebooks/getting_started.ipynb: interactive walkthrough.
- scripts/benchmark_getting_started.py: smoke test that mirrors the
notebook and resets every registered env.
- tests/envs/test_gymnasium_wrapper.py: 12 unit tests.
Removed:
- predicators/envs/mara_adapter.py (790 lines).
- mara_robosim git dependency from setup.py.
- _normalize_env_name_for_gt helper in ground_truth_models, inlined
env_name at all 7 callsites.
Docs:
- predicators/envs/README.md expanded into the benchmark README
(install, quick-start, env table, standalone API, walkthroughs).
- Top-level README points at the benchmark README.
* Rebrand the gym wrapper as MARA RoboSim and audit env status
User-facing rename: the Gymnasium wrapper now exports `MARARoboSimEnv`
(was `PredicatorsBenchmarkEnv`), the import idiom is
`from predicators.envs import gymnasium_wrapper as mara_robosim`
(was `as benchmark`), and the smoke script + tests are renamed to
`mara_robosim_getting_started.py` and `test_mara_robosim.py`. The
underlying file stays `predicators/envs/gymnasium_wrapper.py` so the
filename still describes the implementation.
Env table: `predicators/envs/README.md` gains three status columns
(Tasks / Skills / Demos) populated from a fresh oracle-planning audit:
- 4 envs solve oracle test tasks end-to-end (Ants, Blocks, Circuit,
Cover).
- 8 envs have non-empty option factories but oracle planning currently
fails on default config (Balance, Boil, Coffee, Domino, Fan, Float,
Grow, Laser).
- 3 envs ship empty option sets (Barrier, MagicBin, Switch).
Also tweak the top-level README section heading to "MARA RoboSim".
* Correct env status columns in MARA RoboSim README
Update the Tasks/Skills/Demos columns based on a manual audit:
several envs were previously marked ❌ for Demos but actually solve
oracle tasks under different configs (Boil, Domino, Fan, Float, Grow);
several others have a fixed task generator rather than randomized
init/goal sampling (Ants, Barrier, Circuit, Float, Laser, MagicBin,
Switch).
* Add walkthroughs for getting started with MARA RoboSim environments
* Move Walkthroughs section above the env table
Promotes the notebook + smoke-script links so readers see them
immediately after Quick Start, before scrolling through the 15-row
status table.
…DME (#27) The two most recent papers that use this codebase — VisualPredicator on neuro-symbolic predicate invention for robot planning and ExoPredicator on learning abstract models of dynamic worlds with exogenous processes.
) RoboDisco (Robot Model Discovery Benchmark) is a clearer name for the PyBullet manipulation suite, matching its positioning as an embodied world-model and causal-discovery benchmark. User-facing rename: - `MARARoboSimEnv` class → `RoboDiscoEnv` - Import idiom → `from predicators.envs import gymnasium_wrapper as robodisco` - Gymnasium env IDs → `robodisco/Blocks-v0` (and the other 14 envs) - Entry point string → `predicators.envs.gymnasium_wrapper:RoboDiscoEnv` - Smoke script → `scripts/robodisco_getting_started.py` (+ output dir) - Test file → `tests/envs/test_robodisco.py` - .gitignore, top-level README, predicators/envs/README, and notebook updated to match. The env table in `predicators/envs/README.md` also gains a link to the project page at https://yichao-liang.github.io/robodisco-site/. The module file stays `predicators/envs/gymnasium_wrapper.py` — the filename describes the implementation; the brand is applied at the class, env-ID, and import-alias layer.
* stop tracking CLAUDE.md * use pybullet-arm64 fork for arm64 compatibility --------- Co-authored-by: Yichao Liang <ycliang@Yichaos-MacBook-Air.local>
* Refactor _validate_plan_forward to use option model directly Delegate option execution to option_model.get_next_state_and_num_actions instead of duplicating its termination logic (stuck detection, Wait atom-change checks) and directly accessing its simulator. * Unify backtracking refinement search into shared run_backtracking_refinement Extract the duplicated backtracking loop from run_low_level_search (SeSamE) and _refine_sketch (agent bilevel) into a single run_backtracking_refinement function in planning.py. Both callers now delegate to it with their own sample_fn and validate_fn callbacks, eliminating ~80 lines of duplicated loop/backtracking logic. * Simplify _validate_plan_forward to use run_backtracking_refinement Replace 60 lines of manual option-model execution with a call to run_backtracking_refinement using max_tries=[1] and a sample_fn that returns the pre-grounded options. Remove unused Any import. * Refactor _current_observation/_current_state usage in pybullet_env Move the _current_observation assignment into _reset_state so callers don't need to remember the two-step pattern. Clarify the relationship between _current_observation (backing field) and _current_state (typed read accessor) in docstrings and comments. * Add CFG option to load plan sketch from file instead of LLM Adds agent_bilevel_plan_sketch_file setting that, when set to a file path, loads the plan sketch directly from that file, bypassing the foundation model query. Includes test data files and a unit test. * Remove redundant conditions from Place action in boil_plan_sketch * Scale target joint value based on switch_joint_scale in PyBulletBoilEnv * Refactor _terminal in option model to deduplicate wait-termination logic Extract repeated wait-termination check into _check_wait_termination helper and unify the three _terminal branches into a single definition with config checks inside the function body. * Refactor terminal state logging in _OracleOptionModel to simplify condition checks * Format docstring in get_observation method for improved readability * Refactor PyBulletEnv for readability and better naming - Remove dead/commented-out code and stale self-question comments - Add _VIRTUAL_OBJECT_TYPES constant to replace hardcoded type-name skip lists in _set_state and _get_state - Move env-specific _get_robot_state_dict branches to subclass overrides in pybullet_cover and pybullet_blocks - Extract _get_camera_matrices helper to deduplicate render methods - Extract _get_object_state_dict from _get_state for per-object logic - Move create_pybullet_block/sphere to pybullet_helpers/objects.py - Merge _create_task_specific_objects into _set_domain_specific_state - Rename: _reset_state -> _set_state, _reset_custom_env_state -> _set_domain_specific_state, _extract_feature -> _get_domain_specific_feature - Add docstrings explaining where each method is called from * Regroup PyBulletEnv methods by responsibility and update docstring Reorganize methods into labeled sections (Setup, Public API, Core Loop, State Write/Read, Grasp Management, Action Helpers, Rendering, Utilities) so related functions are adjacent. Update module docstring to document the main public API and state synchronization methods. * Refactor PyBulletEnv: extract _domain_specific_step from step() Add _step_base() and _domain_specific_step() to PyBulletEnv base class. step() now calls _step_base (robot control, physics, grasp) then _domain_specific_step (water filling, heating, etc.), gated by _skip_domain_specific_dynamics flag for kinematics-only mode. Migrate all 15 domain envs to override _domain_specific_step() instead of step(). Envs with pre-step logic (coffee, switch, blocks, cover) still override step() for the pre-step part only. * Update PyBulletEnv module docstring for step() refactoring Document the step_base → domain_specific_step → get_observation flow, _skip_domain_specific_dynamics flag, and _domain_specific_step as an optional override. * Add skip_process_dynamics constructor param to PyBulletEnv Replace direct access to private _skip_domain_specific_dynamics attribute with a public constructor parameter, so callers declare kinematics-only mode at creation time instead of mutating internal state after construction. * Extract run_query_sync helper to remove duplicated async-to-sync bridging Both AgentSessionMixin and AgentExplorer had near-identical wrappers that ran session.query() synchronously via nest_asyncio or asyncio.run. Move that logic into a module-level run_query_sync helper in session_manager and have both callers delegate to it. * Refactor main function: extract and modularize setup logic for clarity and maintainability * Rename agent explorer to agent_plan for clearer naming Distinguishes the grounded-plan explorer from upcoming bilevel variants. AgentExplorer -> AgentPlanExplorer, get_name() 'agent' -> 'agent_plan', file moved to agent_plan_explorer.py, and all callers / docstrings / YAML config examples updated accordingly. * Move AgentSessionMixin into agent_sdk package The mixin is pure agent-session plumbing (session creation, lifecycle, explorer factory) and has no approach-specific logic, so it belongs next to session_manager.py, tools.py, and the sandbox managers rather than in approaches/. * Add AgentBilevelExplorer for sim-learning experiments The explorer asks a Claude agent for a plan sketch, refines it against the approach's current (possibly learned) option model, and rolls the refined plan out in the real env. When the mental model disagrees with reality — e.g. the sketch expects JugFilled after a Wait but the mental model's process dynamics can't produce it — the explorer truncates the plan at the deepest unsatisfiable subgoal (inclusive) so the real-env rollout ends exactly where the disagreement occurs, maximising signal per experiment. Key pieces: - predicators/agent_sdk/bilevel_sketch.py: extracted the sketch build / parse / refine helpers from AgentBilevelApproach as module-level functions so both the approach (solve path) and the new explorer (exploration path) can share them. refine_sketch gains truncate_on_subgoal_fail: the on_step_fail callback snapshots the deepest subgoal failure seen during backtracking, and on exhaustion the captured prefix is returned as the experiment plan. - predicators/explorers/agent_bilevel_explorer.py: new explorer. Reads option_model from tool_context (synced by the approach), builds the sketch prompt via bilevel_sketch, runs refine_sketch with check_subgoals=True, check_final_goal=False, truncate_on_subgoal_fail =True, wraps the result in an option_plan_to_policy that converts OptionExecutionFailure into RequestActPolicyFailure so the episode cleanly terminates at the point of real-env divergence. Stashes the sketch subgoals/options on ToolContext for downstream diffing by the learning approach. - predicators/approaches/agent_bilevel_approach.py: shim methods over bilevel_sketch; behaviour unchanged. - predicators/approaches/agent_planner_approach.py: _create_explorer dispatches both "agent_plan" and "agent_bilevel" through the agent factory path and forwards CFG.explorer as the name. - predicators/explorers/__init__.py: factory branch merged for the two agent-session-backed explorers. - predicators/agent_sdk/tools.py: ToolContext gains last_sketch_subgoals / last_sketch_options fields, populated by the explorer and marked TODO for the learning approach to consume. - tests/explorers/test_agent_bilevel_explorer.py: happy-path, fallback, wait-memory-injection, and deepest-subgoal-failure truncation tests. * Add explorer-specific sample budget and experiment-plan logging - New setting agent_bilevel_explorer_max_samples_per_step (default 50), separate from the solve-path budget, so the explorer's backtracking cost is independently tunable. - Log the actual experiment plan (option names, objects, params) after refinement so the explorer's output is visible alongside the existing sketch/truncation log lines. - Test config updated to set both budgets explicitly. * Add sim-learning approach and synthesis tooling AgentSimLearningApproach extends AgentBilevelApproach to learn process dynamics online. Each cycle: the agent synthesizes parameterized process rules via Claude (using run_python / evaluate_simulator / test_simulator MCP tools), parameters are fitted via emcee MCMC, and the learned dynamics are composed with a kinematics-only PyBullet oracle into a combined option model for plan refinement. Key pieces: - predicators/approaches/agent_sim_learning_approach.py: the approach. Initialises with a kinematics-only option model (so AgentBilevelExplorer sees disagreements at process-dynamic subgoals like JugFilled/Boiled), and replaces it with the kin+learned model after each successful synthesis cycle. - predicators/agent_sdk/tools.py: create_synthesis_tools() builds the three MCP tools the synthesis agent uses; extra_mcp_tools field and get_allowed_tool_list(extra_names=) plumbing lets the approach inject them into the session. - predicators/code_sim_learning/: ParamSpec, fit_params (emcee MCMC), compute_mse, LearnedSimulator. - predicators/ground_truth_models/boil/gt_simulator.py: ground-truth process-dynamics simulator for the boil environment. - tests/: approach and param-fitting tests. * Update experiment configs for sim-learning - agents.yaml: comment out agent_bilevel preset, add agent_sim_learning with explorer=agent_bilevel and skip_test_until_last_ite_or_early_stopping. - common.yaml: disable failure/test video recording, set num_online_learning_cycles=1 for faster iteration. * Refactor sim-learning: extract primitives, add GT simulator factory Simulation primitives (code_sim_learning/utils.py): - apply_rules(state, rules, params) → ProcessUpdate - merge_updates(base_state, updates, process_features) → State - simulate_step(state, action, base_env, rules, params, features) → State These replace _build_fitted_step_fn, merge_process_updates, _sim_fn_from_rules, and the body of _build_combined_simulator. GT simulator factory (ground_truth_models): - GroundTruthSimulatorFactory ABC + get_gt_simulator(env_name) discovery, following the existing get_gt_options / get_gt_nsrts pattern. - PyBulletBoilGroundTruthSimulatorFactory registered in boil/. - Replaces the hardcoded _load_oracle_simulator in the approach. Oracle ablation flags (settings.py): - agent_sim_learn_oracle_sim_program: load GT rules, skip synthesis. - agent_sim_learn_oracle_sim_params: use GT param values, skip MCMC. Also: kin_env → base_env rename throughout, redundant self._types assignment removed, process_features computed once in __init__. * Fix formatting, pylint, and mypy issues for CI compliance - yapf + isort autoformatting applied to all touched files. - pylint: fix logging-not-lazy in agent_bilevel_explorer, add broad-except and reimported disables in agent_sim_learning_approach. - mypy: fix base/env variable name collision, add type: ignore on lambda inference, add return type annotations to GT factory methods. * Update test setup to use test tasks for boil environment and refine test description * Refactor combined model in GT simulator * Fix expected-atoms check to support DerivedPredicates Use utils.abstract to evaluate expected atoms in low-level search so that DerivedPredicates — which require a Set[GroundAtom] rather than a State — are handled correctly alongside regular predicates. * Skip kinematic reset in PyBullet when only non-kinematic state changed When sequential simulate calls differ only in process features (as in the combined kinematic+learned simulator), reapplying joint positions and tearing down/recreating grasp constraints causes visible arm jitter. Compare robot poses first and skip the kinematic reset path when they already match. * Support offline dataset learning in AgentSimLearningApproach Factor simulator synthesis into a shared _learn_simulator helper so that both learn_from_offline_dataset and learn_from_interaction_results can trigger it on their respective trajectory sources. Also create a separate headless env for parameter fitting so MCMC's thousands of _set_state calls don't thrash the GUI env during training. * Log periodic progress during MCMC parameter fitting Replace the silent run_mcmc call with a manual sample loop that logs step count and best log-probability roughly five times per run, and flushes handlers so the updates appear promptly under buffered logging. * Fix mypy and pylint errors for CI compliance Type-annotate **kwargs on PyBullet env __init__ overrides so mypy doesn't flag them. Initialize attrs used by _domain_specific_step in __init__ (pybullet_coffee, pybullet_switch) to silence defined-outside-init. Type-ignore the emcee import. Fix encoding, unused, protected-access, and redefined-outer-name warnings in the sim-learning tests and agent-SDK tooling. * Apply yapf, isort, and docformatter across the codebase * Inline approach configs into parent files in predicatorv3 * Preserve robot joint config across PyBullet state save/restore When a held object's grasp constraint is recreated via _set_state, the gripper frame must match the original world pose exactly — otherwise the recorded base_link->object offset is rotated and the object lands at the wrong world position when the gripper next moves. The State representation only carries (x, y, z, tilt, wrist), so IK during reset can pick a different wrist-roll solution and corrupt the constraint. Thread joint_positions from PyBulletState.simulator_state through reset_state so we skip IK and restore the exact arm configuration. Falls back to IK when joints aren't available (plain State). Also wire wait-termination so refinement and execution can stop Wait when expected atoms hold instead of running to max_num_steps_option_rollout: set _abstract_function on the option model in BilevelPlanningApproach (mirrors AgentPlannerApproach), pass abstract_function into option_plan_to_policy in BilevelProcessPlanningApproach, and inject wait_target_atoms per sample in run_low_level_search. * Add 'emcee' to the list of install_requires in setup.py * Force PyBullet FK refresh and skip redundant finger snap After resetJointState, PyBullet's getLinkState returns a stale link pose from the previous FK cache, producing 50-500μm drift in the EE pose readback. Pass computeForwardKinematics=1 so world poses are recomputed from current joints on every call. Also skip the explicit finger reset in reset_state when joint_positions are provided: arm_joints already includes the finger joints, so set_joints has restored them to their exact continuous values, and the subsequent loop was overwriting them with the discrete-snapped value from _fingers_state_to_joint. The finger reset still runs on the IK path where set_joints leaves fingers untouched. Together these eliminate the "Could not reconstruct state exactly in reset" warning noise (24 -> 0 on the boil-oracle run). * Apply yapf/docformatter to satisfy CI autoformat check * Configure predicatorv3 demos for offline-only sim-learning runs common.yaml: switch to one demonstration per task with no online learning cycle so launch_simp.py exercises only the offline pipeline. agents.yaml (agent_sim_learning): turn on oracle_sim_program with oracle_sim_params disabled so synthesis fits parameters but starts from the ground-truth program structure. * Add jug orientation handling in PyBulletBoilEnv * Revert getLinkState to PyBullet default (no computeForwardKinematics flag) Investigation found no measurable difference in reported Cartesian world position or orientation whether the flag is True or False, so the override introduced earlier was not needed. * Add lo/hi bounds to ParamSpec and skip-MCMC support in fit_params ParamSpec gains optional lo/hi fields for clamping sampled values. fit_params now reads num_steps from CFG.code_sim_learning_num_mcmc_steps by default; passing 0 (or setting the flag to 0) skips emcee entirely and returns the initial parameter values as the fit result. burn_in is also clamped to num_steps-1 to avoid emcee errors on very short runs. Adds a test covering the skip-MCMC path via CFG. * Build boil param specs dynamically from CFG with lo/hi bounds Replace the module-level BOIL_PARAM_SPECS list with _build_param_specs() so water_fill_speed is derived from CFG.boil_water_fill_speed at call time rather than import time. All specs now carry lo=0.0 to prevent MCMC from sampling physically invalid negative values. get_param_specs() is updated to call _build_param_specs() so per-run CFG values are always reflected. * Apply lo/hi clamping and configurable noise scale to oracle perturbation Oracle parameter perturbation now uses the relative scale from CFG.agent_sim_learn_oracle_sim_param_noise_scale (default 0.2) instead of a hard-coded 20 % figure, and clamps perturbed values to the lo/hi bounds declared in each ParamSpec. Also improves the log message when MCMC is skipped (num_mcmc_steps == 0) so it is clear no fitting occurred. * Update installation instructions and add macOS setup script for PyBullet * Update PyBullet version to 3.2.7 and simplify macOS setup script * Refactor liquid color update logic and rename related methods for clarity * Add more debug logging for CogMan and option execution flow * Handle PyBullet physics server crashes with env recreation and retry Converts _build_combined_simulator to an instance method so it can capture self, recreate the base env on pybullet.error, and retry once. Also catches pybullet.error in the oracle option model alongside OptionExecutionFailure. Updates agents.yaml config for testing. * Fix jug orientation handling in PyBulletBoilEnv by restoring rotation logic * Update installation instructions and dependencies; remove macOS setup script * Remove mara_robosim dependency from setup.py * Fix get_gt_simulator to use env_name instead of normalized name * Add before/after MSE, likelihood, and param-delta logging for parameter fitting * Use SSE loss and wider walker init so MCMC parameter fitting actually moves Switch the fitting loss from per-feature MSE to total SSE (drop the /count in compute_sse) so the Gaussian log-likelihood -0.5*SSE/sigma^2 is in its correct iid form. The previous MSE form silently rescaled per-observation noise by sqrt(count), making walker proposals indistinguishable from each other. Pair this with a wider walker initialization (0.5 * prior_sigma instead of 1% of init_value) so the swarm covers the prior support and emcee stretch moves can actually explore. * Move GT simulator components onto module-globals contract Unifies oracle and agent-synthesized simulators behind one loader: read_simulator_components pulls PROCESS_RULES, PARAM_SPECS, and PROCESS_FEATURES out of any namespace (module dict for oracle, exec_ns for agent), and get_gt_simulator now returns the triple including features. merge_updates no longer takes process_features since the rule producer owns that scope. * Soften boil parameter-dependent gates with sigmoid weights Replaces hard ``dist < threshold`` indicators in the boil rules with sigmoid-smoothed gates of width ``_SOFT_EPS``. Without smoothing, the LM finite-difference Jacobian is ~zero almost everywhere, and the Hessian identifiability diagnostic is uninformative; emcee also gets a non-flat likelihood as a side effect. State-dependent gates (faucet on/off, jug held) stay hard since they don't enter the parameter likelihood. * Add LM warm-start and Hessian identifiability diagnostic Adds fit_map_lm (Levenberg-Marquardt MAP estimate via SciPy TRF) and log_hessian_identifiability (eigendecompose J^T J/sigma^2 + prior precision to flag sloppy parameter directions). Both run as a single LM pass before MCMC; fit_params now centers walkers on theta_map when code_sim_learning_warm_start_with_lm is set, and short-circuits to it directly when num_mcmc_steps == 0. Also adds compute_residuals (per-feature residual vector LM consumes) and log_sse_breakdown (per-(type, feature) SSE so we can see which features dominate the loss). Two CFG flags gate the new behavior: warm_start_with_lm (default True), log_hessian_identifiability (default False). * Infer process-feature scope from base-sim residuals The agent now declares its own PROCESS_FEATURES alongside PROCESS_RULES and PARAM_SPECS, and the loss is scoped to that declaration (instead of every feature on every type). Before synthesis, the approach runs the base sim on each transition and flags (type, feat) pairs whose prediction diverges from the observation on at least min_hits triples; this set is sent to the agent as a starting hint and used as the eval/test scope until the agent overrides it. The base-sim prediction is precomputed once into base_pred_triples so MCMC's inner loop only evaluates the cheap apply_rules step. create_synthesis_tools now takes the precomputed triples plus the inferred hint, drops the live base_env, and reads PROCESS_FEATURES from exec_ns each call (falling back to the hint when undeclared). * Skip MCMC and use LM warm-start in boil agent config LM warm start alone matches the parameter fit for the current boil oracle program; emcee's MAP-of-walkers cannot improve on it in the time budgeted for 500 steps and routinely lands at higher SSE. Setting num_mcmc_steps to 0 and enabling warm_start_with_lm returns the LM theta_map directly. * Apply yapf and docformatter formatting Cleans up line-wrap and docstring drift across the sim-learning branch so the autoformat CI check is satisfied. Bundles the formatting-only changes for cogman, pybullet_boil, and utils that earlier branch commits left behind, plus minor wraps across the new sim-learning code. * Silence mypy on PyBullet client-id attribute access ``BaseEnv`` doesn't declare ``_physics_client_id`` (only PyBullet subclasses do), and ``_recreate_base_env`` reads it best-effort inside a try block. Bind to a local with type:ignore so mypy stops flagging the access without affecting runtime. * Mark unused action arg in sim_fn to satisfy pylint The simulator callback signature must match StepSimulatorFn's (state, action, params) shape even though apply_rules doesn't use the action. Renaming to _action signals intent and silences pylint's unused-argument check. * Use per-component diff in _set_state to eliminate robot jitter Replace the all-or-nothing kinematic-match gate with a per-component diff: robot pose, each object pose, and held-object identity are each compared against the live PyBullet world and only re-written when they actually differ. _robot_matches_state now compares at the joint level (the prior EE-quaternion path hard-coded roll=0, which spuriously mismatched whenever the wrist had any roll and forced a full reset on every simulate() call). reset_state honors caller-provided joint_positions only when they reconstruct the requested EE pose, falling back to IK otherwise. * Reposition recreated cups and plugs in coffee _set_domain_specific_state _remake_cups creates fresh PyBullet bodies that need to be teleported to their state-specified poses; the per-component diff in _set_state now skips objects whose pose already matches PyBullet, so the explicit _reset_single_object calls ensure freshly-recreated bodies land in the right place. Same treatment for plugs when coffee_machine_has_plug. * Look up predicates lazily in option-model _abstract_function The lambda used to capture predicates at __init__ time, which missed predicates invented later (grammar search) and broke subclasses whose _get_current_predicates depends on attributes not yet set during super().__init__(). * Rename 'kinematics-only' to 'base-sim-only' in docs and test names Terminology cleanup to match how skip_process_dynamics is described elsewhere; the env wraps the full base sim, not just kinematics. * Tighten _robot_matches_state atol so set_state hint forces reset The fast-path joint-match check used atol=1e-2, which let a caller's initial_joint_positions hint be silently treated as "already there" when live joints were within 1e-2 of initial — leaving the EE pose up to ~3e-3 off the requested state. State.allclose compares features at 1e-3, so the test then failed reconstruction. Match the State.allclose tolerance. Also pick up trailing yapf reformatting in two approach files. * Fix flaky test_glib_explorer and test_demo_dataset_loading under pytest-split Both tests pass on master and in isolation but fail on shards 6/8 of CI on this branch. The branch's new tests shifted pytest-split's least_duration distribution so existing tests landed in different shards than on master, exposing pre-existing fragility: - test_glib_explorer[Holding]: score_fn returned 0 (not -inf) for non-target goals, so they weren't filtered. With cover's 7-atom dynamic universe and 10 babbles, ~3.5% of seeds sample no Holding goal and the explorer falls through to a Covers goal, leaving the final state without Holding. Bumped glib_num_babbles to 100 and switched the test's score_fn to return -inf for non-target so the explorer never plans toward an off-target predicate. - test_demo_dataset_loading[10-True-oracle-...]: _ensure_cover_demo_ data_exists only checked file existence. test_demo_dataset's max_initial_demos block writes a 3-trajectory dataset under the cover__demo__oracle__7__... name; the [10-...] case then loaded 3 + generated 3 = 6, expected 10. Added a trajectory-count check so the helper regenerates partial files. * Add unit tests for _robot_matches_state atol and pybullet_helpers.objects - test_robot_matches_state_atol_forces_reset_on_small_drift: locks in the 1e-3 atol regression. A ~5e-3 joint drift (within the previous 1e-2 tolerance, outside the new 1e-3) must NOT be treated as "already there" by the fast-path; _set_state must move the robot back to the requested EE pose at State.allclose precision. - tests/pybullet_helpers/test_objects.py (new): coverage for sample_collision_free_2d_positions, used by 3 PyBullet envs but previously without direct tests. Covers no-overlap (circles and rectangles), bounds, reproducibility across seeds, RuntimeError on impossible packing, and ValueError on unknown shape_type. --------- Co-authored-by: Yichao Liang <ycliang@Yichaos-MacBook-Air.local>
Boxes created via `create_pybullet_block` were applying the same friction coefficient to lateral, spinning, and rolling channels. With `_obj_friction = 1.2`, a cube that landed on an edge or corner stayed frozen there: spinning/rolling friction supplied a contact torque equal and opposite to gravity's tipping torque, so the cube never settled onto a flat face. `create_pybullet_block` and `create_pybullet_sphere` now apply `friction` only to `lateralFriction`. Spinning and rolling friction are new optional kwargs, defaulting to 0.0 (PyBullet's own defaults). The fan ball and the domino ball — which actually want resistance to spinning around the contact normal — now pass `spinning_friction` explicitly to preserve their prior behavior. Other dynamic bodies benefit from the fix; static bodies are unaffected because PyBullet ignores friction torques on zero-mass bodies. Also add `if __name__ == "__main__":` GUI entry points to `pybullet_blocks.py` and `pybullet_cover.py` (the only concrete pybullet envs that lacked one) so they can be launched as scripts the same way as `pybullet_fan.py`.
) * Make human_low_level_control approach work across all pybullet envs The approach previously assumed (a) simulator_state is always the dict PyBulletEnv builds, (b) the robot type exposes x/y/z + tilt/wrist features, and (c) only fan/blocks/coffee/circuit exist. This crashed on pybullet_blocks (list-shaped simulator_state, pose_x/pose_y/pose_z features) and silently fell back to fan's robot for any other env, so IK ran against the wrong base pose. - Tolerate raw-list simulator_state; defer physics_client_id/robot_id lookup into the mobile-base branch where they are actually used. - Read the current EE pose via shadow_robot.forward_kinematics() so the policy is independent of env state-feature names (fixes blocks and cover, the latter being a 2D env with no y feature). Tilt/wrist deltas now apply on top of the FK orientation, preserving roll. - Look up the active PyBulletEnv subclass via the registry and build the shadow robot from a fresh DIRECT client + plane + the env's own _create_pybullet_robot, skipping each subclass's body-loading override (tables/blocks/fans/etc. are not needed for IK). Smoke-tested across all 14 pybullet envs (ants, balance, barrier, blocks, boil, circuit, coffee, cover, fan, float, grow, laser, magic_bin, switch). * Autoformat and lint fixes for human_low_level_control approach
* domino_real: fold perceived roll modulo pi (a domino is a box) Every domino in a captured scene was standing, but two of the four came back with roll = pi and read as Toppled before anything moved. A task whose goal is Toppled(target) was then satisfied in its own initial state: the planner returned a length-0 plan and the run reported SOLVED. A domino is a box, so turning it 180 degrees about its own width axis leaves it exactly where it was. Both orientations describe the same physical domino and a marker-based pose estimate returns either one arbitrarily -- in a single capture, some dominoes come back at roll 0 and others at roll +-pi. Roll is only meaningful modulo pi, so it is now folded into [-pi/2, pi/2): standing (0 or +-pi) folds to ~0, and knocked over (+-pi/2) keeps the magnitude that Toppled and Upright are defined on. Yaw is untouched, so the push direction is unaffected. This is a regression from the toppled-domino support in plan PR 6. Before it, both conversions hard-coded roll = 0, so a flipped estimate could not surface. The offline check against scenes 0000/0001 missed it because every domino in those two captures happened to come back near roll 0. The fold lives in the domino env rather than in domino_env_euler, so the geometry helper stays a faithful decomposition and the "a domino is a box" fact sits with the domain that knows it. Tests: the invariant that no domino starts toppled when built from a capture whose records are flipped, the same for a mid-episode observation, and the fold itself over eight angles. Verified against the real capture that exposed this -- all four dominoes now read 0.00 deg and the goal is no longer true at init. * configs: a Stage 1 launcher for oracle on the real domino scene There was no command that ran the oracle arm on this env: oracle.yaml un-skips fan, and launch_simp.py takes only -c, so running Stage 1 meant editing a shared config that fan runs also use. This is a thin launcher of the kind the repo already uses -- it only un-skips the env and arm it runs. It also documents the trap that sent Stage 1 off the rails: a bare `python predicators/main.py --env pybullet_domino_real --approach oracle` inherits none of the env's flags from envs/all.yaml, and this env does not work on the settings.py defaults. Most sharply, domino_use_domino_blocks_as_target defaults False, which sizes the domino component with the target held as a separate object -- so a 4-domino scene allocates 3 slots and task construction dies with "perceived 4 dominoes but only 3 slots". The scene path is set here rather than in settings.py because envs/all.yaml already sets it, and a config value beats the settings.py default -- so editing settings.py has no effect on a launcher run. Lands with the roll fold so that one branch both runs Stage 1 and answers it truthfully. * Drop the oracle domino_real launcher from this PR Narrows this PR to the roll fold alone. The launcher is preserved verbatim on the oracle-domino-real-launcher branch.
* domino_real: fold perceived roll modulo pi (a domino is a box)
Every domino in a captured scene was standing, but two of the four came
back with roll = pi and read as Toppled before anything moved. A task
whose goal is Toppled(target) was then satisfied in its own initial
state: the planner returned a length-0 plan and the run reported SOLVED.
A domino is a box, so turning it 180 degrees about its own width axis
leaves it exactly where it was. Both orientations describe the same
physical domino and a marker-based pose estimate returns either one
arbitrarily -- in a single capture, some dominoes come back at roll 0 and
others at roll +-pi. Roll is only meaningful modulo pi, so it is now
folded into [-pi/2, pi/2): standing (0 or +-pi) folds to ~0, and knocked
over (+-pi/2) keeps the magnitude that Toppled and Upright are defined
on. Yaw is untouched, so the push direction is unaffected.
This is a regression from the toppled-domino support in plan PR 6. Before
it, both conversions hard-coded roll = 0, so a flipped estimate could not
surface. The offline check against scenes 0000/0001 missed it because
every domino in those two captures happened to come back near roll 0.
The fold lives in the domino env rather than in domino_env_euler, so the
geometry helper stays a faithful decomposition and the "a domino is a
box" fact sits with the domain that knows it.
Tests: the invariant that no domino starts toppled when built from a
capture whose records are flipped, the same for a mid-episode
observation, and the fold itself over eight angles. Verified against the
real capture that exposed this -- all four dominoes now read 0.00 deg and
the goal is no longer true at init.
* configs: a Stage 1 launcher for oracle on the real domino scene
There was no command that ran the oracle arm on this env: oracle.yaml
un-skips fan, and launch_simp.py takes only -c, so running Stage 1 meant
editing a shared config that fan runs also use. This is a thin launcher of
the kind the repo already uses -- it only un-skips the env and arm it runs.
It also documents the trap that sent Stage 1 off the rails: a bare
`python predicators/main.py --env pybullet_domino_real --approach oracle`
inherits none of the env's flags from envs/all.yaml, and this env does not
work on the settings.py defaults. Most sharply,
domino_use_domino_blocks_as_target defaults False, which sizes the domino
component with the target held as a separate object -- so a 4-domino scene
allocates 3 slots and task construction dies with "perceived 4 dominoes but
only 3 slots".
The scene path is set here rather than in settings.py because envs/all.yaml
already sets it, and a config value beats the settings.py default -- so
editing settings.py has no effect on a launcher run.
Lands with the roll fold so that one branch both runs Stage 1 and answers
it truthfully.
* replay_plan: the oracle approach has no NSRTs for this env
replay_plan has been dead for pybullet_domino_real since the CogMan
rewiring landed:
NotImplementedError: Ground-truth NSRTs not implemented for env:
pybullet_domino_real
It built its CogMan with create_approach("oracle", ...) on the reasoning
that the approach is never consulted for control -- an override policy is
always set, so _reset_policy takes that branch and never calls solve. That
part is right, but OracleApproach builds ground-truth NSRTs in its
*constructor*, and this env has none: it is planned over processes, which
is why the process-planning oracle's _get_current_nsrts returns an empty
set. So the replay died before rendering a frame.
Uses random_options instead: a plain BaseApproach that needs nothing but
the option set, which is all a fixed-plan replay requires.
Verified by rendering a single Push on a real captured scene --
`steps=59 goal_reached=True`, MP4 written.
* Drop the oracle domino_real launcher from this PR
Narrows this PR to the roll fold alone. The launcher is preserved verbatim
on the oracle-domino-real-launcher branch.
…xes (#111) * Add exp config: high-friction condition on the real-scene twin Runs the friction mismatch from exp_domino.yaml's domino_high_friction_turn arm (true 0.5 / planning 0.1) against the reconstructed real scene instead of generated min-block tasks, with the margin AL agent. pybullet_domino_real overrides task generation, so only the physics half of that arm transfers. The min-block geometry flags, domino_test_turn_ratio, and domino_block_cost are deliberately omitted: the first two are read only by the min-block generator this env never reaches, and the third rides on a DominoEvaluator that _task_from_perceived does not attach. The file's header records this so the omissions do not read as oversights. The scene path is pinned because the envs/all.yaml default names a capture that is not checked in. * Run the domino system-ID arm on the Franka: new env + two Panda-only fixes Adds pybullet_domino_real_geometry: pybullet_domino's own generated tasks (min-block spans, turn legs, K*, DominoEvaluator) staged on the real scene's robot setup -- Panda on its short pedestal, extended table tile. The dominoes stay the SIMULATED ones, so exactly one variable moves against domino_high_friction_turn and a difference between the arms is attributable to the robot. RealSceneGeometryMixin is split out of PyBulletDominoRealEnv so the two axes of "real" compose independently: that env pairs the geometry with tasks rebuilt from a perceived scene, this one pairs it with generated tasks. The mixin is deliberately not a BaseEnv subclass -- create_new_env resolves envs by scanning get_all_subclasses(BaseEnv), so an intermediate env class would inherit PyBulletDominoEnv.get_name() and shadow pybullet_domino. Two bugs surfaced by the robot swap. Both are silent: no error, just a skill that reports success and does nothing. 1. Grasp. The Panda's default closed_fingers is 0.03 PER FINGER -- a 60mm gap around a 15mm domino. A grasp registers only when a finger comes within grasp_tol_small (0.5mm), so Pick ran all four phases, reported terminal success, and lifted air: is_held stayed 0 and the domino never moved. Fixed by pybullet_closed_fingers: 0.007 (clamps with ~1.6mm/side). Only the Panda reads this setting, which is why it never came up before. 2. Release. _get_robot_state_dict wrote the raw finger JOINT into the State's fingers feature without _fingers_joint_to_state, while _extract_robot_state applies _fingers_state_to_joint on the way out -- so the skills' fingers helper re-converted an already-joint value and every reading came back deflated. Finger targets became unreachable and Place spun in ClearFingers to max_num_steps_option_rollout (1000 actions) without ever releasing, blowing the 500-step horizon. Invisible on the Fetch, whose joint endpoints equal the State ones (0.01/0.04) so the map is the identity; pybullet_blocks already applied the inverse in its own override. Measured on the same plan, same params: Place went 1000 actions (capped, never released) -> 23, and a Pick+Place pair 1085 -> 96 steps. Across three seeds the arm then produced 7 certified train solves and one test solve (reward 0.80), with system-ID moving friction 0.1 -> 0.53-0.66 against a true 0.5; the pre-fix run had zero successes. Straight spans (0.29-0.31) were re-probed on this setup and transfer unchanged (true K*=1 vs believed 2, 3/3 reps). The TURN legs do not, and are left at the Fetch values with the blocker documented inline: recalibrating them is gated on a separate bug where straight_span_k_star depends on CFG.seed for the Panda (k=1 at seeds 0/1 vs 4/None at seed 2, converging once motion planning is off). Consequence: the test split is currently empty on this arm and reports "Tasks solved: 0 / 0"; training is unaffected. That work is parked on worktree-domino-calibration-probe with its evidence. * docformatter: rewrap PyBulletDominoRealEnv summary line CI runs docformatter 1.4 with --check; the class summary wrapped one word early after the docstring edits in the previous commit.
setup_sandbox_directory wrote CLAUDE.md, validate_sandbox.py, settings.json
and notes.md with bare Path.write_text(), which encodes using the locale's
preferred encoding. VALIDATE_SANDBOX_SCRIPT contains an em dash at index
3026, so under a C/POSIX locale the write died with
'ascii' codec can't encode character '—' in position 3026
Sandbox setup runs once per agent query, so this was not a single bad
query: every query and every final-submission nudge failed identically,
and the task ended with "no captured plan after 0 completed agent
queries" after exhausting all solve attempts. The traceback never named
the sandbox, because the approach catches the failure per query and logs
only the message.
The system-prompt write a few lines further down already passed
encoding="utf-8"; this makes the rest of the function consistent with it.
Verified by running setup_sandbox_directory under LC_ALL=C with C-locale
coercion disabled: origin/master raises the error above, this commit
completes and writes every file.
…arm (#112) * domino_real: a dry-arm rung, and carry sampled plans from sim to the arm Validating the domino skills on the real Franka had a gap on either side of the one tool that ships motion. replay_plan could only be run two ways: pure sim, or the arm moves. It took real_robot_dry from the config, where nothing sets it, so --execute went straight to metal. --dry adds the rung in between: the whole RealRobot minus the arm, so attachment, per-option chunking and the gripper split all run for real and nothing moves. It needs no hardware powered on, which is what makes it a rung you actually take. It also forced observe/human_reset off while leaving perception at the "zed" default -- and RealRobot opens its perception session at construction, so a replay that is hard-forced never to look was still holding both cameras open, and failed when they were busy or unplugged. Pin it to "none". probe_real_scene samples good option parameters with the oracle samplers and then dropped them on the floor; replay_plan wanted those same numbers typed in by hand. --dump-plan writes the grounded plan in replay_plan's format, so the plan that reaches the Franka is the one that was just watched working in sim rather than a fresh sample from the same sampler. simple_str() is parameter-free by design, so the line format is built here. Both scene defaults pointed at capture files that are no longer on disk (domino_real_0000.json); repoint them at domino_straight.json. * probe_real_scene: wrap a docstring line to 80 chars predicators lints at 80 (.predicators_pylintrc + .style.yapf), not the 100 the sibling babyrobot repo uses; one docstring line came in at 81 and failed the pylint gate. Verified with the repo's own rcfile this time, plus yapf, isort, docformatter and mypy.
… Place-in-a-sketch (#114) * probe_real_scene: ground the sketch lazily, and find InFront in the env Two bugs kept Place out of every sketch since #54, so the documented grammar 'Pick:1 Place:1@6 ...' never worked -- only Push/Pick/Wait did. 1. Every token's parameters were sampled against task.init before anything ran. _place_option_sampler requires a domino already held (is_held > 0.5), which does not exist until Pick has executed, so chaining Pick into Place died with 'expected one held domino, found 0'. Ground one token at a time instead, through option_policy_to_policy's existing hook: it asks for the next option only once the previous has terminated, and hands over the state at that moment -- exactly the state the next sampler needs. 2. InFront was looked up in the excluded-predicate-filtered set. The stock config (exp_domino_real.yaml) excludes InFront from the LEARNER's vocabulary, so the lookup silently yielded None, every @ref was dropped, and the placer then failed with 'no InFront subgoal references the held domino'. The subgoal is an instruction to the oracle sampler about where to aim, not part of the agent's vocabulary, so read it from env.predicates. Behaviour is unchanged for sketches that already worked: 'Push:start Wait' still grounds to Push(robot, domino_2)[0.0450, 0.0825] -- the same numbers, from the same seed, as the plan run on the Franka. Not fixed here, and worth knowing: _place_option_sampler only knows placements off a reference within 10 deg of cardinal (_is_cardinal), because its candidates come from the task generator's grid geometry. Perceived dominoes routinely miss that -- 2 of the 4 in domino_straight.json are ~12.5 deg off -- so @ref only works against a near-cardinal reference. * domino: correct Pick/Push heights for the hand actually in use Both z offsets position the TOOL frame, but what has to grasp or strike the domino is the hand hanging below it -- and the two arms differ. Measured at home (lowest finger-link AABB against the tool link): fetch 0.0320 m below the tool frame panda 0.0152 m The offsets were tuned on the Fetch, so the same 0.0825 puts the Fetch's fingertips at 84% of a 0.15 m domino's height and the Panda's at 95% -- the very top edge. On hardware the Panda barely caught the tops on a grasp, and on a push it skimmed over the domino entirely: every sim rollout on domino_straight.json reported toppled={all False}. Subtracting the difference (0.0168 m for the Panda, 0 for the Fetch) makes grasp and contact height a property of the domino rather than of the arm. Fetch values are unchanged by construction, so nothing tuned on it moves. With this, Push topples the start domino in sim for the first time: toppled={'domino_2': True}. * real robot: release at the drop pose, not at the top of the retreat _split_actions classified the hand by midpoint, so a commanded width had to travel halfway to open before the bridge called it a release. Place does not do that: with release_until_ungrasped it opens just enough for the simulator to drop the grasp, then a few millimetres to clear the object, and only fully opens in its final phase -- which runs AFTER the retreat, at transport height. The bridge therefore read 'closed' through the release, the clear and the whole retreat, and shipped its single open at the very end. On hardware that carried the domino up and dropped it from height. The real hand is binary, so any width meaningfully wider than closed is a release. Splitting a Place-shaped sequence now gives close, move(carry), open, move(release, clear, retreat) -- the open arrives before the arm leaves the table. * domino_real: home the wrist where the base domino env does The real-scene override set domino_real_robot_init_wrist to 0.0 while PyBulletDominoEnv uses -pi/2, so the real envs homed the hand a quarter turn from every other domino env -- the fingers opened across the domino's broad face instead of its thickness, on the arm and in the twin alike. Confirmed in simulation; not yet checked on hardware. * real robot: close on a grasp tighter than closed_fingers, and test as Panda Two fixes CI caught, both mine. The gripper split's new rule was symmetric -- abs(v - closed) <= tol -- which is wrong in the tighter-than-closed direction. A real Pick descends through finger values well BELOW closed_fingers (0.0 up to 0.0125 against a closed of 0.02 on the real scene), so every carry waypoint read as an open hand and the whole trajectory collapsed to one `open` and 75 waypoints: the gripper would never have closed at all. One-sided now, which is what the old nearest-value test was really protecting. Checked against a real Pick -> Place trajectory rather than a synthetic one; it splits as open, move(30) descend, close, move(33) carry, open, move(12) retreat i.e. the release lands at the drop pose and the retreat follows it. The real-scene tests built the env with the DEFAULT robot, though every shipped config sets pybullet_robot: panda and the env exists for a Panda on a pedestal -- one _config's docstring even claims it applies "the config this env is actually run with". That went unnoticed while the home orientation happened to suit both arms; with the wrist homed where the base domino env homes it, the Fetch's IK cannot converge and 20 tests failed. Set the robot in all four real-scene test configs. The nearest-value test is replaced by two: any widening is a release, and a grasp tighter than closed_fingers stays closed. * real robot: judge a release against the grasp width, not an absolute mark Both absolute rules tried here were wrong, and in the same way: they anchored the release test somewhere the release never has to cross. A nearest-value test kept the hand shut through the release AND the retreat, so the object was dropped from transport height. Anchoring at closed_fingers only moved the problem -- a firm grasp settles far tighter than closed_fingers (a real Pick on this scene bottoms out at 0.0 against a closed of 0.02), so the entire release still sat under the mark and the one `open` again landed at Place's FullyOpenFingers, at height. My check of that fix read the trailing move as the retreat; it was the full-open phase, and the retreat was inside the block before it. The release is grasp-relative, so the test has to be too: track the tightest width since the hand closed and treat a widening beyond _RELEASE_EPS as the release. Two guards fall out of the measured trajectory: * the closing motion is non-monotonic (0.010 -> 0.01183 -> tighter), so the epsilon sits above that wobble and below the skill layer's _RELEASE_OPEN_STEP, at half the step; * after releasing, the hold width is STILL under closed_fingers on a firm grasp, so re-closing needs a real tightening below where the hand let go. Without that the hand clamped shut again one action after opening. Verified against a rolled-out Pick -> Place with EE height, rather than by reading the segment shape: action 0 open ee_z=0.9273 action 30 close ee_z=0.5416 (grasp, at the table) action 51 open ee_z=0.5802 (release, 3.9cm up) end ee_z=0.7357 (retreated, after the release) _RELEASE_EPS is a local copy because skill_factories imports pybullet_helpers and importing it back would invert the layering; a test pins it against _RELEASE_OPEN_STEP so the two cannot drift. * real robot: tighten the release-rule comments Both were saying at length what one line says. * shorten comments * replay_plan: --observe, so the closed-loop rung has a vehicle The bring-up doc's closed-loop stage says to use replay_plan with real_robot_observe_at_option_boundary on, but the tool hard-forces it off -- and #112 pinned perception to "none" alongside it, because a run that never looks should not hold both cameras open. Between them that stage had no way to run at all. --observe opts into the mid-replay correction and takes perception with it. The default is unchanged: no look, no cameras, reproducible replay. Deliberately a flag rather than a default. Correcting the twin between options lets the option policies see states the recorded plan was never chosen against, so the run stops being a pure replay -- which is precisely what the rung is testing, but is wrong for every other use of this tool. * domino_real: perception must not move a domino the gripper is holding Perception snaps every domino to a resting pose on the table -- snap_stable sets the centre height from the known vertical extent -- so it cannot see one in the gripper and reports it lying where it would be if the hand let go. A real run showed all four dominoes at z=0.034 including the carried one. state_from_observation wrote that in regardless, which teleports the domino out of the hand; _set_state then treats the held object's pose having moved as a reason to rebuild the grasp constraint, so the twin ends up holding something that is not where the gripper is. Reproduced in simulation with replayed perception: the correction reports 0.222 m of divergence at the boundary. Downstream the run may survive or die on an IK timeout, which is why closing the loop over a Pick -> Place was stochastic. Skip held dominoes: the twin's belief wins for whatever it is carrying, which is the one thing perception structurally cannot check.
* real robot: a fixed-plan explorer, and a record of every look Three pieces for exercising the online loop on hardware, where the cost is wall-clock rather than compute. fixed_plan explorer. The planning explorers cost minutes per episode, so a full cycle is expensive to run for the sake of testing the loop around it. This one replays a plan file every episode and costs nothing, so a run that goes wrong is the loop's fault and not the planner's. It reads replay_plan's format, which means a plan dumped by probe_real_scene --dump-plan and already verified through replay_plan can be handed straight to the loop. Grounded per episode, because a human reset rebuilds the task and hands back fresh Objects. Every look is logged, not only the ones over tolerance. A run whose looks all behaved previously said nothing at all, so there was no way to distinguish "perception was healthy" from "nobody checked". The line carries the per-object breakdown, which is what tells a single knocked domino apart from a table-height offset shared by all of them -- the max alone cannot. Looks can be dumped to JSON (real_robot_observation_dump_dir, off by default). Each file records what was perceived beside what the twin predicted, so a session can be re-examined offline without the robot. Write failures are logged and swallowed: losing a dump must not take the arm down mid-episode. The learner is untouched. * configs: a cheap rehearsal of the Stage 6 loop Stage 6 proper is exp_domino_real.yaml -- live cameras, a live arm, a human reset per episode, a planning explorer costing minutes an episode. That is what to run once the loop is trusted, and the wrong thing to debug the loop with. This keeps the loop shape (explore, learn, test) and makes one pass survivable: the fixed_plan explorer replays a verified plan instead of planning, nothing moves, no camera opens, and every look is dumped. It runs with the robot powered down. It does not replace the learner, which is separately expensive; the header says so. * real robot: the boundary check must not disturb a stateful terminal Wait counts consecutive settled steps in its own memory, and its option policy already consults it once per step. OptionBoundaryBuffer consulted it a second time, so Wait judged the scene settled in a third of the steps it really takes. The check now restores the option's memory, making it a pure query. * real robot: a twin correction is a resync, not the scene moving Wait ends once the scene holds still for several consecutive steps. Writing perception into the twin moves objects without the scene having moved, so Wait counted every look as motion, zeroed its tally, and never saw the scene settle -- the episode then ran to the step cap instead of ending when the plan did, spending a look every few steps on the way. rebaseline_quiescence re-seeds the baseline and keeps the tally, so the jolt is skipped rather than counted. * rename rebaseline_quiescence to note_external_state_change The old name described the mechanism, so a reader had to already know what a quiescence baseline was to guess what the call did. The new one names the event instead: the state was set from outside rather than moved into, which is the thing the caller actually knows.
* domino_real: let a real task carry a DominoEvaluator A task built from the real scene carried no evaluator, so every episode scored reward 0.0 and an over-built chain was indistinguishable from a minimal one. That difference is exactly what the friction-mismatch experiments are read off, and it is what DominoEvaluator computes: each movable domino the cascade consumes costs domino_block_cost. Opt-in via domino_real_attach_evaluator, so existing runs score as before. The movable count comes from the perceived scene rather than domino_min_block_num_blues -- that flag is a generator budget, and a real scene stages whatever the person put on the table. The evaluator asserts that a success outscores any failure; that would fire mid-episode on the real robot, so the bound is checked at task construction and reports the scene's own numbers instead. * configs: Stage 6 launchers, and score the rehearsal's episodes stage6_domino_real.yaml is Stage 6 proper -- exp_domino_real.yaml with execution on. Everything else the stage needs (live cameras, a look at every option boundary, a human reset per episode) is already the settings.py default, so it sets only what differs. stage6_domino_real_underreach.yaml adds the REVERSE friction mismatch: the planner believes the dominoes are far more slippery than they are, under-estimates reach, over-builds, and pays for the surplus. Only domino_planning_friction moves -- the twin keeps modelling the real table. Its mirror does not transfer to hardware, where the twin is not the ground truth; the file says so. Both cap wait_option_max_steps, whose shipped bound is inf: a Wait that never sees the scene settle otherwise runs to the horizon and takes a human reset with it. The rehearsal now attaches the evaluator, which is the cheap place to learn whether the cascade certificate accepts an episode assembled from perception rather than physics. * real robot: a config asking for no cameras gets none utils.string_to_python_object maps both "None" and "none" to Python None, so real_robot_perception: "none" from a launcher config never arrives as the string _make_perception was matching on -- it arrived as None and fell through to the unknown-kind ValueError. The mode was reachable only from a direct reset_config call, which is why the existing test (which does exactly that) never caught it. Also adds the evaluator-check launcher: perception off and no looks, so the twin keeps what the plan built and the cascade can actually run. That is the cheap place to learn whether DominoEvaluator's certificate accepts a real-shaped episode, which the rehearsal cannot answer because scene_file perception undoes every Place. * domino_real: attach the evaluator on the generator's own terms Replaces domino_real_attach_evaluator with the condition DominoTaskGenerator._make_task already uses. What decides whether a verdict means anything is the certificate's causal model, not which env built the scene, so the real env should not need a flag the simulated ones do not have -- and defaulting it off meant the real path silently scored 0.0 while every sim path scored. Both of the generator's conditions hold for a real scene: every shipped domino config sets domino_use_domino_blocks_as_target, and a real scene is dominoes and nothing else, so there is no additional dynamic component to topple them without a Push. Also carries across the scoring text the generator appends to goal_nl. Its comment records why it exists (run_20260716_215533 burned its budget theorizing that any disturbed blue disqualifies a solve); an agent scored on the real robot would have hit the same misreading. * clean up configs * submodule: update BabyRobotPredicator to main Brings in the auto-launch (#59), so a restarted droid server binds its robot handles instead of failing every read with 'FrankaRobot object has no attribute _robot', and the safety box (#61), which confines the end-effector and resolves fail-closed. The pin was at a 2026-07-28 commit that predated both. Cherry-picked from master; its exp_domino_real.yaml scene line is dropped in favour of the newer one already on this branch. Note this does NOT include BabyRobotPredicator #63 (the measured table height), which is still open -- the submodule wants bumping again once it lands, or a scene captured through the submodule's own scripts will carry the 4 mm z error back.
Picks up the measured-table-height work: the capture path's -0.045 defaults now read SkillConfig.real_table_z_base (-0.041), so a recapture through the submodule no longer reproduces the 4mm error. No behaviour change for predicators itself -- real_robot_bridge passes table_z=CFG.domino_real_table_z explicitly, so DominoPerception's new default is a no-op here. It matters for the babyrobot capture CLI, which takes the defaults.
* real robot: one arrangement serves both task splits tasks_for consumed a single _reset_pending token, so whichever split asked first got the perceived scene and the other kept the captured-scene task. With the online loop off, main.py requests the train tasks during setup -- so the TEST task, the one that actually gets solved, silently stayed on the scene JSON while the arm executed in the real scene. Seen in run_20260806_132702: the agent's sim.state matched the JSON-derived pose to 1e-16 (bit-identical, where a camera would differ by millimetres). It tuned a single-bridge plan to 20/20 against those poses; on the arm the Push was refused before moving, its approach waypoint 9.4 mm inside the bridge domino it had just placed. The look is now kept and both splits rebuild from it. A physical reset arranges one scene and the person who arranged it meant it for whatever runs next, so train and test are the same task -- which is what a real bench means anyway. (cherry picked from commit 035a1e1) * real robot: refuse the captured scene while the cameras are live The scene JSON's poses are a snapshot. Planning against them while the arm works a scene nobody looked at plans for a world that is not there, and it is silent: planning succeeds, and the twin only jumps to the truth at the first option boundary. tasks_for now raises when live perception is configured and no look has happened, naming both ways out. Replaying a recorded plan is the one case that wants those exact poses -- the plan was written against them -- so replay_plan opts in via real_robot_allow_captured_scene_task, and the two test suites that deliberately run against a capture say so too. Does not remove the JSON: _scene_ids (the capture-id to slot map, without which a live observation names nothing) and the slot counts are read in __init__, before any camera exists. Dropping it entirely means capturing at construction, which is a separate change. * domino: narrow Pick's grasp box on short-fingered hands only The shipped grasp_z_offset box (0, 0.1) was drawn around the Fetch: its top edge IS the Fetch's reach edge, and the collision edge at 0.045 sits 45% up, leaving the top 55% feasible. Sweeping a real domino at 5 mm, the Panda collides at the same 0.045 -- that edge belongs to the domino, not the hand -- but stops reaching it past 0.080, so only 35% of the same box can work and a sampler spends most of its budget on offsets that cannot. Give the Panda the Fetch's proportions around its own reach edge rather than shrink-wrapping its band: same feasible fraction, same shape of learning problem, so a sampler compared across the two arms is comparing embodiment and not box width. The reach edge comes from _hand_z_correction, which is zero on the Fetch -- so the Fetch keeps the shipped box and its parameter description untouched, and a future hand gets bounds without another sweep. Panda: [0.0137, 0.0832], 35% -> 50% feasible. The tuned sampler value (0.0657) and every Pick in plans/*.txt stay inside it. create_pick_skill grows an optional param_defs that defaults to the canonical box, so coffee, bridge, grow and boil are unchanged -- their objects were never measured, and guessing bounds for them would be worse than leaving them. (cherry picked from commit bf9133d) * domino: pin the Place drop-height test to the constant, not 0.58 2403bf2 lowered _DOMINO_DROP_Z to 0.568 and left this assertion behind, so the sampler test has been failing since. What it means to check is that the sampler uses the canonical drop height, not that the height has one particular value. (cherry picked from commit 5923ce1) * domino_real: a look must not turn the opening push around Push takes its entire direction from the start domino's yaw (push.py: facing = (sin(yaw), cos(yaw))), and a domino is 180-degree symmetric, so perception returns whichever heading branch it likes. task_from_observation already knew that and flipped the start to face the target; state_from_observation deliberately did not, on the grounds that mid-episode the start may already have been pushed. But every option boundary before the push is a look, and each one wrote the raw branch back. A plan that stages a bridge and then pushes gets four looks first, so by the time Push read the yaw it was perception's arbitrary choice. On the real scene that choice is -1.727, which faces AWAY from the target: the arm pushed the start into open table. Canonicalize on the correction path too, but only while the start is still standing. That keeps the original reason intact -- once it has gone over there is no push left to orient, and flipping it would misreport which way it fell -- and it is the same fallen_threshold guard _task_from_perceived already uses. _target_xy prefers the target's pose from the current observation and falls back to the twin's last known one, matching this path's "absent means unchanged" policy: a target hidden behind the arm must not cost the start its heading. Measured on the real scene file: start yaw after a look goes from -1.7269 (away) to +1.4147 (toward). The replaced test pinned the old contract; the two new ones pin both halves of the new one. * domino: a domino set down crooked is standing, not falling Two real episodes toppled the target through a legitimate cascade and were both rejected at reward -0.15, the certificate reporting that a bridge domino "started falling at step 21" -- the step the gripper closed on it, 100 steps before the push. _topple_onset dates a fall by finding when the domino ended up on its side and walking back to the last moment it was standing within the upright band (5 degrees). The walk-back skipped held states with a `continue`, so it ran straight through the pick-and-place. A real placement does not land at exactly zero roll, so the domino rests a few degrees off plumb and never re-enters the band; the search therefore kept going past the grasp to the last time it stood untouched on the table, and dated a cascade that arrived long after the push to before it. Two changes. The walk-back now floors at the step after the domino was last released: it cannot have been falling since before the robot picked it up and carried it. And a domino that was staged and then rested below fallen_threshold until something reached it has its fall dated to the fall itself -- resting crooked is standing. The rule keeps its teeth. A robot that drops a domino over lands the full topple at the release and is still caught there; scenery is untouched (never held, so the floor stays 0); and a staged domino that genuinely starts tipping before the push, crossing the upright band but not the fallen one until after, is still dated to the tipping. Sim never showed this: a placed domino settles there at exactly zero roll, so the walk-back always found an upright state just before the real fall. It needed imperfect placements, i.e. hardware. * tests: isort's wrapping for the drop-height import CI runs isort --check, which wants the continuation on the second name rather than both. * Revert the Place drop-height test tidy-up (c6067cc, 1a47143) Those two were here only to keep the assertion working once #117 lowered _DOMINO_DROP_Z to 0.568, plus the isort wrapping that followed. #117 is closed -- the drop height stays 0.58 for every embodiment, having only been tested on the Panda -- so the literal is correct again and c6067cc's stated reason ("the sampler test has been failing since") no longer describes anything on master. Leaving it would put a message referencing a commit that never landed into the history of a PR that is otherwise only hardware fixes. Worth doing on its own merits later: what the test means to check is that the sampler returns the canonical drop height, not that the height is any particular number.
Slurm sbatch wrapper + launcher for the Engaging (ORCD) cluster; predicators imports deferred into _run() for fast CLI startup on login nodes.
Make the physics-margin arm the main al arm; run the fan al arm demo-free; 3 seeds per experiment; swap the active fan arm to agent_oracle_hybrid_sim; fan_test_num_pos_y 5 -> 6.
Read run liveness from squeue when serving on a login node; pin Slurm jobs and moved runs to their real directories.
…ract (#123) Explore prompts separate the delivery contract from the belief-model disclosure; solve makes one query per attempt with the delivery nudge only on the last.
… scoring (#128) Make the registry sweep opt-in via rollout=True; add phys_params point scoring so a probe can score a parameter vector without a full registry sweep.
…stack (#125) Mechanical rename (44 files): skip_process_dynamics -> skip_residual_dynamics, process_features -> residual_features, _process_rules -> _residual_rules, plus matching docs/tests/prompts. No behavior change.
…eck (#126) Fix switch-snap and fan-clearance sim/real divergences; expose blocker geometry as observed state and generalize the GT contact rule to use it; add the fan model-learning postmortem slide deck.
…ible base sim + hidden dynamics (#127) Learned residual rules can queue engine-executed physics commands (ApplyForce/ApplyTorque/SetVelocity) instead of re-deriving contact geometry as feature overwrites; the fan env splits into a visible base sim and hidden wind/task-gen/predicates, the env's own wind routes through the same command executor, and command rules fit via free-running rollout matching. Base-sim source provisioning is gated by agent_sim_provide_base_sim_source.
* cascade certificate: a topple must persist, not just occur once
A real domino run (20260813_112747) reached its goal and was rejected with
domino_0 started falling at step 31, before the green start block was
first pushed (step 129)
Step 31 is not a fall. It is the step the gripper closed on domino_0, ~100
steps before the push. Markerless perception of the same episode shows the
block never left the upright band before the push: 0.20 deg median for the
30 s it stood placed, peaking at 12.75 deg only while the gripper occluded
it, and zero frames past the 15 deg upright threshold anywhere before the
push. The cascade was real, arrived during the Push, and propagated down the
row in the intended order in 368 ms.
What fired instead was one state of release transient. `_topple_onset` took a
single non-held state at or past `fallen_threshold` as a topple, and the state
immediately after `is_held` clears catches the block mid-release, still
settling: measured at 4.9 deg against a 5 deg upright band, while a domino in
the gripper swings to 13.1 deg -- past `fallen_threshold` -- routinely, masked
only by `is_held`. One state at 10 deg therefore carries no information. The
spurious fall then landed before the push and the episode was rejected.
So require `_TOPPLE_MIN_STEPS` (3) consecutive non-held states at or past the
threshold. A real topple grows monotonically and stays there for tens of
states; the release transient decays back inside the upright band within about
two. The run restarts on a carry as well as on standing back up, since the
tilt of a block in the gripper says nothing about one on the table.
A run that reaches the end of the episode counts whatever its length. A topple
in the last states has no room to persist, and discarding it would leave
`onsets` empty and pass the episode unexamined -- trading this false reject
for a false accept, which is the worse direction for a certificate to fail in.
The onset is the first state of the run, so the existing backward search and
its carry floor are unchanged.
Six tests: the one-state transient and one just short of the minimum stay
quiet, a sustained fall after release still registers at its first state, runs
ending the episode register at every length below the minimum, a carry never
accumulates, and a regression case reproducing the run above -- pick, place
with a settling transient, push much later, cascade -- now certifies. The
first and last fail with the minimum set back to 1.
* shorten docstrings/comments
* panda: stop fingers crushing through grasped objects Skills command finger targets past a grasped object (Grasp targets closed_fingers - 0.01; 'closed' move phases nudge 1mm/step with no floor) and rely on contact to stall the fingers. That works on the Fetch (damped, light fingers) but the Panda URDF had no finger-joint damping and no inertials (PyBullet defaults each link to mass=1, identity inertia), so position control with PyBullet's default unlimited motor force drove the fingers straight through a grasped domino: ~400 N sustained squeeze, fingers fully closed inside the object, flaky transport (constraint jitter) and objects knocked over on release. - panda_arm_hand.urdf: real Franka hand/finger inertials (0.73 kg / 0.015 kg) and finger-joint damping 5.0 (must stay below 2*mass/dt = 7.2, or the explicitly-integrated damping oscillates once motor force is finite). - set_motors: new per-robot finger_motor_force cap, re-issuing the finger motors with a finite force. Panda uses its 20 N URDF effort limit; Fetch keeps None (its damping-100 fingers are only stable with unlimited motor authority, and it already stalls benignly). With pybullet_closed_fingers=0.007 in pybullet_domino_real_geometry, the Panda now grasps like the Fetch: constraint forms at first contact, ~5 N squeeze resting at the domino faces, zero-velocity upright release. Fetch behavior is unchanged. * domino configs: rest panda closed fingers at the domino faces With the finger force cap in place, retune pybullet_closed_fingers to the principled values: the joint value is per-finger travel from the centerline, so resting at the faces means half the domino's thin axis (0.015 m sim -> 0.008 with the 0.0005 detection tolerance; 0.029 m real -> 0.015). The previous values (0.007 / 0.02) commanded the resting fingers inside the object / short of a symmetric rest. Validated: E2E Pick in pybullet_domino_real_geometry with 0.008 grasps at first contact, 10-13 N squeeze, upright zero-velocity release; 0.015 against a 0.029 m box in the isolated harness settles at 7-8 N resting at the faces.
* domino Push: do not motion-plan the phases that are meant to make contact
BiRRT plans a COLLISION-FREE path, and the goal of the push stroke is the
object itself. Asked to reach a pose that touches the block without touching
anything, the planner does the only thing it can: it routes around. Measured on
a real captured scene (block centre (0.5255, 1.3091), standing top z 0.5495),
the 16-waypoint trajectory it returned for Waypoint_2 was
wp00 (0.4802,1.3106,0.5418) the behind point, correct side, contact height
wp01 (0.4928,1.3374,0.5565) lifts above the block top, swings out in +y
wp03 (0.5118,1.3700,0.5880) 59mm to the side, 39mm above the top
wp04..13 travels past the block, held out at y~1.35
wp14 (0.5752,1.3270,0.5500) back down to exactly the top
wp15 (0.5255,1.3091,0.5407) drops onto the goal -- moving -x
so the block is struck by that last 50mm hop, which travels AGAINST the push
direction. The block topples backwards, away from the row it is supposed to
start. Waypoint_3 (retreat) has the same problem from the other side.
The planner never had a chance to know better: the collision set is every
object in the state bar held ones (base.py), with no per-phase exclusion, so
the block being pushed is an obstacle like any other. ``expect_contact`` reads
like it covers this but does not -- it only suppresses collision diagnostics
and silences a planning failure, and here planning did not even fail.
A contact motion is exactly what a collision-free planner cannot express, so
the two contact phases now step IK directly. Measured on two captured scenes,
the start domino's displacement along the row goes
scene cam1: -102.7mm (away) -> +102.6mm (toward), roll +90
scene row2: -102.2mm (away) -> +100.2mm (toward), roll +90
and BiRRT calls per Push drop from 4 to 2, still covering both free-space
phases. ``make_move_to_phase`` gains a pass-through that defaults to None, so
every other skill keeps whatever the config says.
This reached hardware: the arm replays the twin's joint trajectory faithfully,
so it reproduced the backwards push exactly. The bug was never in the robot.
* domino Push: derive the approach distance from the domino actually in play
``_DOMINO_OFFSET_X = 0.045`` carried the comment ``domino_depth * 3``, and that
comment is only true of the SIMULATED domino: ``pybullet_domino/env.py`` sets
``domino_depth = 0.015``. The real blocks are 0.029 (``domino_real_domino_dims``,
applied at ``pybullet_domino_real.py`` where the component is built), so the
constant was never re-derived for them and the approach point sat 30.5mm clear
of the block's back face instead of the 37.5mm it was tuned to give.
Read the thickness at call time instead. Sim is unchanged (0.045); the real
scene now gets 0.087. ``_push_sampler`` and ``_push_option_sampler`` share it.
Separate from the motion-planning fix in the previous commit and NOT the cause
of the backwards push -- that symptom reproduces at both 0.045 and 0.087, in
sim and on hardware. Drop this commit if you would rather land the two apart.
* fixup: resolve use_motion_planning explicitly instead of via **kwargs
Passing the override through a conditionally-populated ``**kwargs`` dict hid
which keyword it was for, so mypy matched it against every remaining parameter
and reported it as both a bad ``terminal_fn`` and a bad ``finger_direction``.
Name it instead. ``None`` still means "whatever the config says": Phase's own
default_factory reads the same CFG flag, and both are evaluated when the Phase
is constructed, so this is the value the default would have picked. Verified
unchanged: 2 BiRRT calls per Push and the start domino still moves +101.8 mm
toward the row.
* fixup: docformatter wrapping in processes.py
docformatter 1.4 rewraps the _domino_depth description to its own width and
pulls the _push_option_sampler summary onto one line. No wording changed.
* domino Push: tell the planner which object it is allowed to hit
Supersedes the previous commit's approach of exempting the contact phases from
motion planning entirely. That fixed the push direction but threw away
collision checking on the stroke, so the gripper was no longer checked against
the OTHER dominoes or the table while driving forward.
Name the object instead. ``Phase.contact_object_index`` indexes the option's
``objects`` to say "this phase is allowed to drive into this one", and
``_plan_with_simulator`` drops that body from the collision set. Everything
else stays checked. No signature changed: that method already received both
``objects`` and ``phase``, and discarded the former with ``del objects``.
Why the existing machinery did not already cover it: run_motion_planning does
classify bodies near the robot at the start or goal as intended contact
partners, but only relaxes them to ``pybullet_birrt_contact_margin`` -- 1 mm of
tolerated penetration. A push has to bury the closed gripper centimetres into
the block, so the concept was right and the tolerance two orders of magnitude
short. Widening that margin globally is not the answer; it is what stops a
"collision-free" path from grazing a knife-edge object elsewhere.
The stroke BiRRT now returns is straight -- x advances 0.4803 -> 0.5255 in even
~4.5 mm steps with y and z flat, 11 waypoints against the 16-waypoint detour
that arced 59 mm sideways and 39 mm above the block's top. Measured:
scene cam1: -102.7mm (away) -> +95.7mm (toward), roll +90
scene row2: -102.2mm (away) -> +94.8mm (toward), roll +90
and all four Push phases are planned again, where the previous approach left
only two.
KNOWN GAP: the restricted Push variant grounds as ``[robot]`` alone, so there
is no object at index 1 and the exclusion silently does not apply -- that path
still detours. It finds its start block from state; wiring that up needs a
different hook than an index into ``objects``.
* docs: trim the push-geometry docstrings to what they need to say
The rationale for deriving the approach distance from the domino in play now
lives in the commit that introduced it, where it belongs; the docstrings keep
the statement of what the functions return. Code unchanged.
* push: correct the stale comment, drop the superseded escape hatch
The comment above the waypoint phases still described the first attempt at
this fix. It claimed the contact phases must NOT be motion-planned -- they are
-- and that BiRRT "fails outright and drops to incremental IK", which
measurement disproved: it succeeds, and the detour it returns is the bug.
Rewritten to say what the code now does and why expect_contact and the
contact-partner margin do not cover it.
Also removes the ``use_motion_planning`` pass-through added to
make_move_to_phase for that first attempt. Nothing calls it now -- pour.py
gets the same effect by constructing Phase directly -- so it was dead surface
that would only invite the question of why it exists. Phase's own default
applies again, and move_to.py no longer needs the CFG import.
Unchanged: 4 planner calls, 11-waypoint straight stroke, start domino +95.9 mm
toward the row, 55 domino tests pass.
* push: contact strokes step IK directly; revert the collision-set exemption
Reverts f9211c8 and d630549, restoring the approach of 02fd984: Waypoint_2
(the stroke) and Waypoint_3 (the retreat) are never motion-planned, and
make_move_to_phase regains the explicit use_motion_planning override.
The collision-set exemption kept BiRRT in the loop for the stroke and relied
on it returning the direct path. That leaves two holes. First, the planner
remains free to detour around a BYSTANDER near the stroke and strike the
pushed object from the wrong direction -- the original bug, one object over;
a stroke that cannot go straight should fail and be resampled, not rerouted.
Second, by its own commit message, the restricted Push variant grounds as
[robot] alone, so the objects-indexed hook silently does not apply and that
path still detours. And the checking it bought was thin: when BiRRT fails,
the expect_contact fallback runs unchecked incremental IK anyway.
The IK stroke is also the only behavior sim ever exercised. With the object
in the collision set, the stroke's goal config registers as colliding on
every sim scene, BiRRT fails the up-front goal check (utils.py RRT.query),
and the expect_contact fallback quietly ran exactly this IK stroke -- e.g.
2883 out of 2883 Push/Waypoint_2 planning attempts in one predicate-invention
run. Making it unconditional removes the knife-edge where scene and hand
geometry decide between "planner fails, fallback pushes correctly" (sim) and
"planner succeeds, block struck backwards" (the real captured scenes), so
sim and the real bench take the same code path.
The stale part of 02fd984's comment ("fails outright and drops to
incremental IK from a pose it has already contorted into"), which d630549
rightly flagged, is rewritten to what measurement actually showed. A new
unit test pins the contract: with skill_phase_use_motion_planning=True,
Waypoint_0/1 plan and Waypoint_2/3 do not.
---------
Co-authored-by: Yichao Liang <ycliang6@gmail.com>
…tion fit (#134) * plan: open-loop execution + markerless continuous perception The friction fit currently scores the twin against itself: of ~229 recorded states in the Stage-6 run, 6 were real camera looks and the rest are the twin integrating PyBullet at the true friction, so minimising that SSE recovers the true value by construction. The agent declined to declare, which was the right read of the evidence. This plan fixes the evidence. Three steps. Open-loop execution ships an episode's motion in one batch instead of one option at a time, which is safe because with observe=False shipping is a pure write-only side effect and deferring it leaves the twin trajectory bit-identical. Markerless perception supplies a dense real pose track, already measured at 1.03 deg median orientation error on the better camera. Scoring then moves to inter-domino propagation intervals, which are invariant to both the alignment offset and the ~25-38mm extrinsics calibration offset (measured displacements are accurate to ~1mm). Written against both remotes as of 2026-08-14, not against their docs. * domino real: correct the grasp offset, and sketch the bridge plans The agent's 0.082 grasp offset failed on the real arm in run_20260807_102548: look 2 put domino_4 0.1987 m from the twin's prediction while every other domino stayed under 9 mm, so the gripper closed on nothing and the twin went on believing it held the block. 0.082 puts the fingertips 8.2 mm below the top of a 150 mm domino; in sim that grips as well as anything, because a grasp is a JOINT_FIXED weld formed on 5e-4 proximity and grip depth costs nothing, and higher offsets clear BiRRT more easily -- so a sim optimiser drives this parameter to the worst real value. 0.0657 is _grasp_z_offset() for this hand. The bridge sketches take their start and target from domino_real_{start,target}_id rather than assuming them, and space the movables across the measured 0.331 m gap. The 3-movable one exists because 2 movables at 0.110 m pitch only reached the target 3/8 at friction 0.1 -- low friction shortens topple reach. These live in scripts/plan_sketches/ and are named for what distinguishes them -- scene, bridge size, and either the grasp offset or the fact that the roles come from the scene's own ids -- rather than for which run happened to produce them ("agent_best" ages badly, and said nothing about the 1-domino bridge or the grasp it was tuned for). Config: one exploration episode per cycle, since the fixed-plan explorer replays the same plan and a second episode costs a full hardware run plus a human scene reset for a near-duplicate. * submodules: bump BabyRobotPredicator to main (396094e -> b45ac97) Eight PRs of catch-up. The pointer sat at #63 ("use the measured table height"), which predates the entire perception stack, so predicators could not import any markerless or recorder code at all -- nothing downstream of it could even be prototyped. What this brings in: #66 ZedRecorderSession (open the ZEDs once, start/stop many SVO takes), #67 markerless pose estimation, #68 depth-free bundles (1.4 GB -> 48 MB, poses unchanged), #71 the stage-1 extrinsics check, #72 stage-3 fp16 + crop (20x -> ~3.2x real time) and the occlusion visibility gate, #73 markerless scene capture as the default, #75 the pre-5.3 pyzed fix that lets the recorder run on this machine's SDK 3.8.2 at all, and #76 output paths that stop derived artifacts overwriting the bundle they came from. The submodule is not checked out in this worktree, so this moves the gitlink only; run `git submodule update --init` to populate it. * real robot: ship an episode's motion in one batch, behind a flag Today the twin simulates an option, ships it to the arm, then simulates the next -- so the arm idles through the next option's motion planning before it moves again. real_robot_open_loop_episode (off by default) holds each completed option and ships the whole episode as a single request once it has all been simulated. Why deferring is safe, and not merely acceptable: with the boundary look off, execute_chunks(observe=False) returns [], the absorb loop never runs, and after_step already returns obs UNCHANGED. Shipping is a pure write-only side effect, so *when* it happens cannot be observed by the rollout -- the twin trajectory is bit-identical either way. A test asserts that equality directly, driving both paths with a distinct observation per step so it cannot pass by both sides being constant. The port needed a fourth method. ActionExecutor had tasks_for/after_reset/ after_step and no end-of-episode call, which is the only reason shipping had to happen inside after_step in the first place. after_episode(completed) closes that gap; BaseEnv.finish_execution is a no-op so cogman can end an episode without knowing whether the env drives anything, and PyBulletEnv delegates. completed=False drops the buffer instead of shipping it. What survives an abnormal end is a prefix -- half a bridge, or a transport with no place at the end of it -- and the arm would run it with nobody having decided that was a good idea. The information that it was partial exists only at that call, so it is the last place the judgement can be made. after_reset drops anything left over as a backstop, since a plan shipped against the next episode's scene is the same hazard one episode later. Mutually exclusive with real_robot_observe_at_option_boundary, asserted at construction: a boundary look has to happen between the two options it separates, and batching leaves no such moment. Batch start/end are logged from both time.monotonic_ns() and time.time_ns(); the wall stamp is what pairs with a recorder's own host stamp, the monotonic one survives an NTP step. Kept off by default because it is a real regression in supervisability: the arm runs the whole plan with the e-stop as the only intervention, where today a bad first option is visible before the second ships. Two duck-typed env mocks in test_cogman gain the no-op, since cogman now calls it on the BaseEnv it is declared to take. 9 tests. Verified they fail against the old behaviour: disabling the deferral reds 4 of them, and the bit-identical test reds when the deferred path hands back anything other than what it was given. Full suite 1497 passed, mypy clean, pylint 10.00/10. * real robot: record each episode to an SVO take for offline pose estimation Step 2 of the open-loop plan: the cameras record the whole execution and the poses are recovered afterwards, instead of six option-boundary looks. Nothing here estimates a pose. The markerless pipeline runs at roughly 3x real time, so a result cannot come back inside the episode that produced it -- which is the same fact that makes open-loop execution necessary rather than merely nice. real_robot_record_episodes (off by default) wires ZedRecorderSession onto the executor's lifecycle: open() once when the executor is built, start_take after each reset, stop_take at after_episode, close at exit. Opening once matters because a learning cycle is many episodes and per-episode camera init and warmup would otherwise be paid every time. Recording stops in a finally, shipping does not. They have opposite defaults on an abnormal end: a partial plan must NOT reach the arm, but a take left open records until the disk fills. Two tests pin the pair -- one where the episode did not complete, one where execute_chunks itself raises. Failures are asymmetric for the same reason. start_take raises: an episode that cannot record spends hardware time and a human scene reset for nothing, so it is better to say so before the arm moves. stop_take does not: by then the arm has already moved, and a recording problem must not destroy the run around it, so the take is logged and marked unusable. meta.json's errors list gets the same treatment -- a camera that dropped out mid-episode yields a short track that is perfectly well formed, which is exactly the failure worth being loud about. Recording and a live "zed" perception are refused together, before the env or the hardware is examined: both open the same cameras and a ZED admits one owner. This is a config contradiction, so it is reported as one. Defaults chosen from the pipeline's own measurements: HD720 (what it was measured on) at 60 fps, not 30. A real cascade's topple onsets came 6, 4 and 2 frames apart at 30 fps, and those inter-domino intervals are what the friction fit is scored on -- at 30 fps a one-frame detection error is half the shortest interval. HD720 already runs at 60. Exports stay off during a run; stop_take can write depth inline, but that is the expensive offline work and doing it in the episode loop would undo open-loop execution. babyrobot stays a lazy import, and the recorder gets its own module-level-import test rather than relying on the executor's: the executor imports this module at module level, so a top-level import here would break a submodule-less checkout just as surely, and the ZED recorder lives under pose_estimation rather than babyrobot. mypy.ini gains the matching stanza for the same reason. The recorder's own tests drive a stub session, which is what keeps them hardware-free and is also how a stub silently drifts. test_real_robot_bridge gains a contract test that pins the four calls and their keywords against the real ZedRecorderSession, skipping without the submodule like its neighbours. It passes against b45ac97. 12 tests. Verified load-bearing: removing the finally reds the raise case, and ignoring meta's errors reds the unusable case. Full suite 1510 passed, mypy clean, pylint 10.00/10. * real robot: rebuild each episode's task from a markerless snapshot Restores per-episode scene rebuild alongside episode recording, which the previous commit had to refuse: recording and a live "zed" perception both want the same cameras and a ZED admits one owner. The refusal gave up less than it appeared to. Live "zed" perception IS the marker pipeline -- _live_records loads a marker registry and a dictionary name and calls capture_frames -- and the 20mm ArUco markers are not resolvable at this camera distance: 1 of ~7 detected on 30264679 and 0 on 32294776. So the capability being sacrificed could not see these dominoes anyway. This replaces it with the thing markerless actually offers. The trick is that a snapshot opens no cameras. It is a second, short take on the recorder's already-open session, taken between episodes while no episode take is running -- so there is no second owner and nothing to arbitrate. A test pins the sequencing, and the stub session now refuses a concurrent take the way the real one does, so that test can fail. It plugs in without touching the reset flow. RealRobot.reset_env homes the arm, blocks until a human confirms the scene is arranged, and only THEN calls perception.observe() -- which is exactly when a snapshot should be taken. So MarkerlessSnapshotPerception duck-types the perception protocol and is injected through make_real_robot(perception=...); open() and close() are no-ops, because owning cameras here is the collision being avoided. attach_real_robot builds the recorder before the robot so the session exists to hand over. run_stages reads config.boxes and NOT config.boxes_json, so the runner resolves the file itself. Getting that wrong would not fail -- it would quietly fall back to the drag window on every episode, which is the difference between unattended and not. A contract test pins it, along with the MarkerlessCapture fields and run_stages' signature, and asserts run_stages' source mentions config.boxes and not config.boxes_json. z_mode defaults to "contact" here, opposite to the cascade default: a rebuild looks at a scene a human has just arranged upright on the table, where z from the table is right and better constrained. "free" is for dominoes at rest on each other. table_z is passed rather than measured per capture, because the twin's base->world transplant and the fit have to agree on where the table is. Snapshot take directories carry a counter, not just a timestamp: start_take makes the directory with exist_ok, so two snapshots in the same second would write into one and the second would inherit the first's frames. Found by a test that took two snapshots in a row. Snapshots are tracked separately from episode takes. ``takes`` is what the fit consumes; a snapshot is an input to a task, not a record of an execution. 10 tests, with the pipeline and scene loader injected so none of it needs a GPU, a camera, or the submodule. The two contract tests were run against the bumped submodule (b45ac97) via PYTHONPATH, since the installed babyrobot resolves to the main checkout's older working tree. Full suite 1519 passed, mypy clean, pylint 10.00/10. * style: docformatter and isort over the new real-robot modules CI runs docformatter 1.4 and isort as their own required checks, and I had run only yapf and pylint over these files. docformatter rewraps the summary lines and one-line docstrings its own way, and isort wanted the two new pybullet_helpers imports reordered in the executor's test. Produced by run_autoformat.sh's tools in its order (yapf, then docformatter, then isort), so the three agree rather than each undoing the last. No wording or behaviour changed. Full suite 1519 passed, mypy clean, pylint 10.00/10.
* sysID: score the friction fit against a markerless pose track
Step 3, and the point of the previous two. Under open-loop execution nothing
corrects the twin, so every recorded state is the twin's own PyBullet
simulation -- a per-step SSE over them recovers the twin's friction by
construction, which is exactly why the Stage-6 run could not learn one. The
track the cameras produce is the only real evidence in the episode.
code_sim_learning_rollout_score_observed_only (off by default) replaces the
per-step residuals with propagation intervals: when each domino started to
fall, relative to the first. The per-step loop is SKIPPED rather than added to
-- keeping it would let thousands of twin-against-twin terms outvote a handful
of real ones.
Intervals rather than poses, for two independently measured reasons. Absolute
base-frame position is 25-38 mm off while measured displacements are accurate
to ~1 mm, so scoring poses would score the extrinsics. And an interval is
invariant to however the track's clock is offset from the robot's -- which is
what lets alignment be a detected event rather than a clock reading, and is
why 3.2 of the plan needed no code at all. A test asserts that invariance.
Onsets are confirmed and then backdated because two spurious-fall mechanisms
have been measured on real takes and a naive threshold fires on both: gripper
occlusion produced a confident 29 deg topple 15 frames early, and orientation
drift wandered an untouched domino from 4.4 to 12.9 deg, across the 10 deg the
twin calls toppled. Neither reaches an unambiguous fall, so a fall is believed
only past 45 deg held for 3 samples -- the shape cascade_certificate._topple_
onset already uses -- and the onset is then walked back from there. Both traces
are tests, including the two combined, where the artifact must not drag the
true onset 15 frames early.
Notably NOT a rate or jump gate: those were measured upstream and rejected,
because during a real cascade the other dominoes translate 22-36 mm/frame,
overlapping the 48-67 mm of the artifacts. Speed cannot separate them;
visibility can, and the pipeline gates on it upstream.
Track ids are MATCHED to objects, not assumed. The ids are the order the
initialization boxes were drawn, which nothing makes agree with the env's
numbering. Every twin-point/track-point pairing is treated as a candidate
calibration offset and the one bringing the most dominoes within 40 mm wins.
Centroid cancellation was tried first and is wrong: one bad detection drags the
centroid and then NO pair matches, which a test pins.
A cascade that stalls on one side is penalised at the track's full duration
rather than skipped -- skipping would make a friction that stops the cascade
early look BETTER than one that reproduces it, by having fewer terms.
3.1: code_sim_learning_rollout_scope_types narrows the scored scope, empty by
default so the fidelity report is untouched. That report deliberately scores
"everything that moves"; identifying ONE parameter is a different question,
where the commanded arm reproduces at every candidate and can only dilute --
and with it in scope nothing ever rests, so rest-point segmentation can never
cut. _moving_feature_scope is now module-level (it captured nothing from its
closure) so the narrowing can be tested.
Flag on with no track: one WARNING and per-step scoring. Scoring zero residuals
would make every theta equally good and hand back the prior centre with a
confident-looking identifiability report.
* real robot: post-process each take into a track, and wait for it at fit time
The bridge between recording and scoring. Step 2 wrote .svo takes and Step 3
reads tracks; nothing turned one into the other, so the track path had to be
set by hand and could only ever name one episode.
real_robot_process_takes launches the markerless pipeline over each take as it
closes, detached, and joins the outstanding jobs when the run ends. Detached
because the pipeline runs about 3x the length of the take: inline it would
serialise post-processing into the episode loop and undo open-loop batching.
Launched per take rather than in one batch at the end because it parallelises
across takes at ~2 GB of a 24 GB card, so the work overlaps the next episode's
human scene reset instead of accumulating.
A manifest names each episode's take, its track, and whether the take was
usable. Rewritten in full as every take closes, so a run killed halfway leaves
a valid document rather than a truncated one. An unusable take is recorded and
marked, and deliberately NOT processed: a track fitted to a recording that lost
a camera mid-episode is a well-formed track of the wrong thing.
**The fit waits for the tracks the manifest promised**, up to
code_sim_learning_track_wait_s (15 min). This is the part that makes the flag
mean what it says. The manifest is written synchronously so it is always there,
but the tracks it points at are minutes behind, and the online loop fits as
soon as an episode ends -- so without the wait the fit would find nothing, warn,
and fall back to per-step scoring, silently reinstating the defect all of this
exists to remove. The distinction is that open-loop execution exists to stop
the ROBOT waiting on perception; the learner has no such excuse, because the
track is its data. The wait expires rather than hangs, and says the fit saw
less evidence than the run recorded.
Tracks are cached per path: a sweep evaluates the objective dozens of times,
and without it each candidate friction would re-parse a multi-megabyte JSON and
re-enter the wait.
z-mode is "contact" for episode tracks as well as scene captures. The scored
quantity is when each domino STARTS to fall, and at that moment it is still
standing on the table, so the mode that constrains z there is the right one.
A contract test pins run_markerless.sh's positional interface and the env vars
passed to it -- a shell contract with no type checker behind it, where a rename
would otherwise surface as a background job that fails silently and a track
that never appears.
* real robot: draw the prompt boxes once per run, and log each pipeline job
Two things that stood between the post-processing path and actually being
runnable.
**Boxes, once per run rather than once per take.** Stage 2 needs one box per
domino and is human-driven, which is the last thing gating unattended
operation. real_robot_pick_boxes_at_start takes a 5-frame snapshot when the
cameras open -- before any episode, while a human is still at the bench --
runs stages 1-2 with the drag window, and hands the boxes to the processor for
every take afterwards. Previously the only option was producing a boxes.json
out of band beforehand, which is a separate manual step nobody would remember.
This is valid because a fixed-plan replay trains and tests on ONE arrangement,
so boxes drawn on the scene as it stands are the right ones for every episode.
And it is self-checking rather than merely assumed: if the layout later shifts
far enough that a box no longer sits on its domino, stage 3's frame-0 identity
check aborts that take instead of tracking the wrong object. A failed draw is
not fatal -- the takes are still recorded for processing by hand.
Skipped entirely when real_robot_snapshot_boxes_json names an earlier run's
boxes, which is what makes a repeat run unattended.
**A log per job.** The pipeline runs detached, so its output had nowhere to go
and went to DEVNULL: a failed stage was a missing track and no reason, noticed
minutes later when the fit found nothing. Each job now writes stdout and stderr
to <bundle>/pipeline.log, and the failure message names that path rather than
just an exit code.
The log handle is attached to the process object rather than closed at launch:
the child writes to it for minutes after launch returns, so letting it be
collected would close the descriptor out from under a running stage. It is
closed when the job is joined.
Four tests. The logging one spawns a real script that writes to stderr and
exits non-zero, rather than stubbing the launcher -- which would have tested
nothing about the redirection. Full suite 1559 passed, mypy clean, pylint
10.00/10.
* domino real: configure the fixed-plan run for open-loop, recorded execution
Points exp_domino_real at everything the last three commits built, so the
experiment exercises record -> post-process -> fit end to end with the
fixed-plan explorer skipping straight to a real cascade.
The changes and why each is forced:
* open_loop_episode on, observe_at_option_boundary off. Mutually exclusive and
asserted at construction -- a boundary look has to happen BETWEEN two
options, and batching the episode leaves no such moment.
* perception "zed" -> "scene_file". "zed" is the MARKER pipeline, whose 20mm
tags do not resolve at this camera distance (1 of ~7 on one camera, 0 on the
other), and it would also fight the recorder for the same cameras. The
captured layout is what a fixed-plan replay wants anyway: the plan names
specific objects and a rebuild could renumber them.
* human_reset off, for the same reason.
* record_episodes and process_takes on, with pick_boxes_at_start so the one
human interaction happens at the start of the run.
* score_observed_only on. Without it the two flags above make the fit WORSE
rather than better: under open-loop nothing corrects the twin, so the
recorded states are the twin's own simulation and a per-step SSE over them
recovers the twin's friction by construction. Zero real observations instead
of six.
* scope_types ["domino"], dropping the commanded arm and the colour channels.
* track_path at the run manifest, and a 900s wait for tracks still being
post-processed.
num_online_learning_cycles 2 -> 1 for the first integration run: with
human_reset off there is no prompt between episodes, so a second cycle would
start on whatever the first left behind. Raise it once the path works, at
which point real_robot_snapshot_rebuild is what restores the between-episode
prompt without reopening the camera conflict.
Drops code_sim_learning_num_mcmc_steps: it does nothing for the rollout sysID.
The emcee branch was removed in 2026-07 and the flag is now read only by
fitting.py, so its comment here claimed an effect it has not had for a month.
* cogman: a plan that runs out has ENDED, not failed
run_20260817_160904 simulated all six of its options and shipped none of them.
The arm could not move at all under open-loop, in any configuration.
A fixed plan ends by raising OptionExecutionFailure("Option plan exhausted!"),
and that type is in the exploration loop's exceptions_to_break_on. The
end-of-episode verdict added for open-loop treated every break_on exception as
"this episode did not run to completion" -- so the normal terminus of every
fixed-plan episode was indistinguishable from an abort, and an executor that
defers its motion to the end of a completed episode discarded all of it:
Option plan exhausted after 6 options.
[CogMan] Finishing episode.
WARNING: real robot: dropping 6 buffered option(s) unshipped -- the
episode did not run to completion
The root cause is that one exception type carries two opposite meanings. So the
normal terminus is flagged where it is raised -- info={"plan_exhausted": True},
at both sites -- and only an unflagged break_on exception marks the episode
incomplete. Structural rather than matching on the message.
Why it took a hardware run to surface: with per-boundary shipping the motion
reached the arm DURING the episode, so the end-of-episode verdict decided
nothing. Open-loop is the first mode where that flag decides whether anything
moves, and no test covered a plan ending by exhaustion -- only by exception or
by step limit. The regression test does, and it reproduces the run: reverted,
it fails [False] == [True].
Also quiets zerorpc, the transport the arm is driven over. It logs a line per
channel at DEBUG, which at loglevel=DEBUG buries a hardware run's own output in
"--> new channel <uuid>". WARNING still surfaces a transport that is failing.
* real robot: fix the boxes handed to stage 2, and record only the motion
Three things the first hardware runs found.
Stage 2 died on int('id'). init_boxes.py WRITES records -- "id", "box",
"label" under a "boxes" key -- but the BOXES env it READS expects a bare list
of four-number lists. The records were handed over unchanged, so stage 2
iterated a dict and crashed minutes into run_20260817_162250, after the arm had
already executed the whole episode. The coordinates are unwrapped now, in list
order, which is the id order the writer enumerates. Verified against the
boxes.json that failed; both shapes are tested.
The take recorded the twin simulating. Under open-loop the arm does nothing
between the reset and the ship, so a take opened at the reset captures a static
scene -- 105 s of a 258 s take on run_20260817_165815. Trimming cannot recover
it: --trim-motion keeps everything between the first and last movement, and the
arm homing at the reset opens that window, leaving the still period in the
middle. The take is opened at the ship instead. Measured on the next run: 0.6 s
of dead air against 289 s of motion.
Per-boundary shipping still records from the reset, because there the motion is
spread through the episode. One existing test changed meaning rather than
behaviour and was rewritten to match: an aborted open-loop episode now leaves
nothing recording because it never STARTED a take, where before the stop was
what closed it. Same invariant; a companion test keeps the old assertion for
the per-boundary path.
The camera was whichever came first. real_robot_track_camera picks it, and
30264679 is the default: markerless is single-camera and the two are not
interchangeable -- on hand-measured ground truth this one is 6x better on
orientation (1.03 deg median against 6.29), which is what the topple onsets are
read off. Naming a camera the session does not record now raises up front
rather than failing per-episode with a missing file.
Also wires TRIM=1 (BabyRobotPredicator #78) so stage 1 drops the still lead-in
and tail. Worth having even though the take is now tight, and free: both of the
scan's failure modes keep frames rather than lose them. A driver that predates
the flag ignores it, so a contract test says which of the two is in front of us
instead of letting a silent no-op look like a working one.
Full suite 1572 passed, mypy clean over 539 files, pylint 10.00/10.
* domino real: point the experiment at the new scene, and bump the submodule
Submodule b45ac97 -> 603e4d4, two PRs:
#77 stops stage 4 deadlocking on fork, defaults -j 16, and makes NEURAL the
default depth model. The deadlock is in the exact path the background
post-processing drives, so this is the one that matters. NEURAL comes
along with it -- predicators cannot select a depth mode, because
run_markerless.sh forwards nothing to stage 1 for it, so the mode is
whatever stage 1 defaults to.
#78 adds --trim-motion, which the previous commit wires up.
The experiment config follows the scene the bench is actually set up for:
domino_row_20260817.json with four dominoes, capture id 3 as the green start
and id 0 as the purple target. envs/all.yaml's 6 / 5 are domino_straight.json's
ids and appear nowhere in this scene -- left in place the task has no target at
all and _task_from_perceived asserts on it.
The fixed plan follows the same scene. domino_straight_bridge2_scene_roles.txt
picks domino_4 and domino_5 and pushes domino_1, none of which mean the same
thing here: this scene has only domino_0..3, its movables are domino_1 and
domino_2, and its start is domino_3. The sketches are checked in beside the
config that names them, so the config does not reference a file that only
exists on one bench.
* tests: never let the executor suite write into the repository
The suite clobbered a live run's manifest. EpisodeRecorder's default track
directory is the RELATIVE "logs/zed_tracks", and it rewrites tracks.json there
whenever a take closes, so a test that constructs a recorder without passing
track_dir writes into the repo -- and, if a run is in flight, over its manifest.
That is not hypothetical. run_20260817_171402 recorded its episode, stage 4
produced a good track (60455 records over 15695 frames), and the fit then
reported "episode 1 still has no track at logs/zed_tracks/take_20260817_174618
_train0_ep001/dominoes_traj.json" and fell back to per-step scoring. The take_
prefix is _StubSession.start_take's return value, the timestamp is when the
suite ran, and the serial is the camera the run was not even using. The
experiment lost its evidence to its own test suite.
Isolating the working directory fixes every present and future test at once,
where passing track_dir at each call site fixes only the ones someone
remembers. Verified by checksumming the restored manifest across a full run of
both affected suites: unchanged.
* sysid: dump the fit data when it arrives, not only when a fit runs
_persist_fit_trajectories sat inside the physical sysID fit, so the branch that
never ran was the branch whose data mattered. A cycle where the agent DECLINES
to fit is exactly the one worth a post-mortem, and it left nothing behind.
run_20260817_171402 is the cost. Its sweep reported one identical SSE
(5.418e+06) for every value of five different physical parameters, against a
baseline of 39.37 with no override applied -- a five-order-of-magnitude
discontinuity that appears the moment an override is applied at all, and does
not vary with the value. Three of its five segments were already pinned at RMS
601, a saturation value rather than a measurement. The agent read that as "this
data cannot constrain it" and declined, which was the right call on the evidence
in front of it. Diagnosing WHY needs the trajectories, and they died with the
process.
So the dump now hangs off _learn_simulator, where a cycle's data arrives,
whether or not anything is fitted afterwards. The label in the filename says
which moment produced it: "recorded" for the data as it arrived, "fitted" for
the existing post-fit dump, whose payload's identified params only mean
something there.
The test asserts the wiring as well as the function. Written the obvious way it
passed with the call deleted -- it called _persist_fit_trajectories directly,
which is a test of the dump and not of the thing that was broken. Calling
_learn_simulator for real needs a whole synthesis session, so the wiring is
pinned on its source instead; reverted, the test fails.
Full suite 1576 passed, mypy clean, pylint 10.00/10.
* real robot: stop reading a carried domino as a release
Every Pick shipped close, open, close, and the hand visibly opened around the
domino it had just taken.
Grasp commands closed_fingers - 0.01 = 0.00000, deliberately past the block.
The fingers STALL on it at 0.00658 -- #131 gave them a finite motor force so
they rest at the faces instead of closing through -- and the carry phases that
follow nudge from the ACHIEVED width, commanding 0.00658 - 0.001 = 0.00558.
The splitter only ever sees commands, never achieved positions, so it compares
that against the grasp COMMAND of 0.00000: a 5.58mm rise, which cleared the
5mm release epsilon by 0.58mm and became a spurious "open", followed by a
re-"close" on the next step.
Pre-existing; #131 exposed it by changing where the fingers come to rest. NOT
caused by open-loop batching: the gripper dedup is session-wide state applied
per segment, and a test added earlier proves the per-chunk segments are
identical batched or shipped one at a time.
This raises the epsilon to 0.008, which is a FITTED constant and is commented
as one. A carried object measures 0.00558 on this bench and a genuine release
measures 0.0122, so 0.008 is simply a value between them.
It cannot be derived, and three attempts at a principled fix are why:
* Requiring the release to leave the closed band broke
test_split_actions_ignores_wobble_while_closing, which encodes a real
release at 0.0122 against a closed of 0.02 -- inside the band.
* Clamping the carry nudge to the tightest command so far was inert: the
rebound crosses the Grasp -> carry phase boundary, and Grasp is a
CHANGE_FINGERS phase that never runs that code.
* Seeding that clamp from Grasp's command fixed the symptom but pinned the
command at 0.00000 forever, so the fingers kept squeezing (achieved drifted
0.00485 -> 0.00447). That undoes what #131 built and broke
test_oracle_process_planning_solves_bridge_task.
The two cases are indistinguishable from the command stream -- both are
"widen after a grasp", differing only in magnitude. The real fix is to stop
inferring intent and carry the skill's own finger_status through on
Action.extra_info (unused today, and documented for exactly this), leaving
this constant to guard only actions that arrive without a stamp.
Three tests pin what the constant was fitted to, so a retune of the grasp
depth or the finger force names the two numbers to re-measure. Verified by
rolling a Pick out in the twin: gripper commands go ['open','close','open'] ->
['open','close'], with the commanded widths unchanged.
…137) * sysID: match the track to the twin in one frame, and once per episode Two defects between the pose track and the objects it is supposed to name. Together they meant NOTHING was ever matched, on either hardware run. The frame is the one that mattered. The markerless pipeline emits poses in the ROBOT BASE frame; a twin state is in the env's world frame, and for this env the two differ by a quarter turn about z plus (0.75, 0.72) -- exactly pybullet_domino.real_geometry.base_to_world_transform, which the env applies when it builds the task and nothing applied to the track. match_ids_by_position votes over candidate translations, which is what absorbs the 25-38 mm camera calibration error, but a rotation is not a translation: on run_20260818_092302 every pair sat 144-307 mm apart against a 40 mm tolerance. Fitting a rigid transform to that run's own data recovers 92.24 deg and (0.748, 0.718) against the constants' 90 deg and (0.75, 0.72), with per-domino residuals of 2.7-12.9 mm. So the transform is measured, not assumed. It is applied on the way out of the config-aware loader, so every track the objective sees is already in the env's frame, and it is IDENTITY by default: an env whose track already shares the twin's frame, and every test that builds both sides in one frame, must be untouched. This is what produced yesterday's meaningless numbers, and the arithmetic is exact. When a cascade propagates in one stream but not the other, _interval_residual_terms substitutes the track's own span, deliberately large. run_20260817_171402's track is 268.768 s and summary_weight is 5, so one penalised term is sqrt(5) * 268.768 = 600.98 -- the RMS 601 that appeared in three of five segments -- and fifteen of them are 5.418e+06, the SSE that came back IDENTICAL for every value of five different physical parameters. It was identical because the penalty is the track's duration and does not depend on theta at all. The earlier guess, that an override dropped the rollout into a shared failure regime, was wrong: the residual was never a function of the parameter being swept. Both runs' agents declined to declare on that, which was the right read of the evidence they were shown. The second defect is scope. match_ids_by_position ran per scored trajectory against the track's FIRST frame, and the two only agree at the episode's start: rest-point segmentation splits one episode into several trajectories, and this plan picks and places two dominoes before the push, so a later segment begins with them ~200 mm from where frame 0 saw them. _episode_id_maps now matches once per episode, anchored there. Trajectories that pair one-to-one with tracks are their own episodes and still anchor on themselves. On the run that failed, both fixes take matching from 1-2 of 4 dominoes to 4/4 on every segment, and from zero usable intervals to three: 0.217, 0.084 and 0.034 s. The leftover position error is 14-31 mm, inside the calibration budget the plan documented, which is the independent check that the transform is right rather than merely fitted. One test note. The obvious negative assertion -- that raw base-frame positions match NOTHING -- is false and was corrected: the winning offset is drawn from a candidate pairing, so that pair always matches itself. What a rotated frame costs is every other domino, and that is what the test now asserts. * sysID: a track is ready when it PARSES, not when the path exists The pipeline writes a track in one pass and a dense one is tens of megabytes, so between the path appearing and the last byte landing there is a window in which the file is real, growing, and not valid JSON. The wait tested os.path.exists, so it fell straight into that window: on run_20260818_092302 the fit logged "all episode tracks are ready" and then failed to parse at "Expecting ',' delimiter: line 427907 column 20 (char 11997567)" of a 28 MB document. Announcing readiness and then finding nothing usable is the worst of the available outcomes, because it looks like success in the log. track_is_complete parses instead. _settled puts a size check in front of it so the wait loop does not re-read the whole document every two seconds -- a file whose length changed since the last look is still growing and needs no parse to rule out -- but the parse is what decides, since a writer that stalled would hold its size steady while remaining truncated. The second failure was next to it and is arguably worse. load_tracks called load_track unguarded once the wait returned, so a truncated file raised out of the function, and _load_scored_track catches at the granularity of the WHOLE manifest. One half-written track therefore discarded the finished tracks either side of it and sent every episode to per-step scoring, which under open-loop scores the twin against itself. It is now skipped with a warning, exactly as an absent one already was. The give-up message changed with it: "still has no track" was accurate when absence was the only way to fail, and now has to cover "present but still being written" as well. The existing test that pinned that wording is updated. Three tests, all of which fail if track_is_complete is reverted to os.path.exists: a truncated file is not complete though it does exist, the wait runs to its deadline rather than declaring victory on it, and a finished episode survives a truncated neighbour in the same manifest. Full suite 1580 passed before this change; rerunning. mypy clean, pylint 10.00/10. * sysID: anchor the id matching on the cascade, not on the episode Matching is positional, so the two snapshots being compared have to be of the same MOMENT. The previous commit anchored the twin side on the episode's first state, which is right only if the recording covers the whole episode. It does today, and will not shortly: recording just the Push is being prepared as a post-processing speedup, and such a take's first frame is AFTER the plan has rearranged the scene. Measured on run_20260818_092302, by taking the real track's arrangement just before the cascade as a synthetic frame 0: whole take + episode-start anchor : 4/4 dominoes matched push only + episode-start anchor : 2/4 push only + pre-push anchor : 4/4 The plan relocates domino_1 and domino_2 before the push, so they sit ~150 mm from where the episode began. Note the failure is partial rather than clean, and a partial positional match is the dangerous kind: the offset is voted for over candidate pairings, so a shifted offset can match a subset exactly and assign every one of them to its neighbour. So both sides now anchor on the settled arrangement immediately BEFORE the cascade -- the twin's last state before any fall angle confirms, and the track's last frame before its first onset. That moment is identifiable in either stream whichever window was recorded, and it is the arrangement the propagation intervals are measured on. Neither "the episode's start" nor "each segment's start" is right in general; each was right only for one recording window. ObservationTrack carries pre_cascade_xy alongside first_xy, filled by a second pass over the parsed document once the onsets are known -- the onsets are not available during the first pass, and re-walking what is already in memory is cheap next to carrying every frame's positions. first_xy remains the fallback for a track in which nothing ever falls. The frame transform moves both sets, or the anchor would sit in a different frame from its own fallback. The twin side reads roll by object NAME, so it needs no id mapping and there is no circularity. match_ids_by_xy is split out of match_ids_by_position because the twin side is no longer a single State. Both recording windows now give the identity mapping on all four segments of the real run. The test builds an episode that places a domino 600 mm from where it started and a take that only ever sees the placed layout; with the anchor reverted to the episode's first state it fails.
Saving learned NSRTs and GNN weights intermittently dies with `TypeError: cannot pickle '_abc._abc_data' object` -- the C-level cache behind an abstract base class. It has blocked three PRs on three different shards this week and it predates all of them: the same nine-test cluster reproduces on unmodified master. WHAT IS ESTABLISHED Two call sites, both reached from the CI tracebacks: nsrt_learning_approach.py :114 (`pkl.dump(self._nsrts, f)`) and gnn_approach.py:207 (`pkl.dump(info, f)`). Every failure seen so far funnels through one of them. On CI it is DETERMINISTIC for a given test set: shard 8 failed twice on the same single test, and shard 6 twice on the same three, across re-runs of untouched jobs. Which shard is hit moves with pytest-split's packing, which is why adding tests to an unrelated PR appears to "cause" it. Locally it is stochastic -- 3 of 12 identical runs, with code, test order and PYTHONHASHSEED all fixed. The likely difference is that CI containers are uniform while a developer machine is not. WHAT IS NOT ESTABLISHED The root cause. An `_abc_data` holds WEAK references, so whether dill trips over one plausibly turns on collection timing -- the one thing that still varies with everything else pinned. That is the hypothesis this mitigation is aimed at, and it is a hypothesis: it is stated here rather than dressed up as a diagnosis. If the failures stop, that is also the evidence for it. If they do not, it rules the hypothesis out, which is worth knowing too. Ruled out along the way, none of them the cause: pytest-randomly (not installed -- `-p no:randomly` is a silent no-op here), execution order (progress lines match character-for-character between a red and a green run), hash randomisation (PYTHONHASHSEED=0 is exported), and any single polluting test (a bisection appeared to find one, then the control showed removing its neighbours worked equally well -- that bisection was invalid, having used single runs to measure a 25% event). WHY IT IS SHAPED THIS WAY Serialising to bytes before writing, rather than retrying into the handle: a dump that raises part-way has already written a prefix, and appending a retry to that leaves a corrupt file which only fails at LOAD time -- much worse than the error being fixed. A test pins the file being empty after a persistent failure. Any TypeError is retried, not just the `_abc_data` one. Matching on the message would break silently when it is reworded, and an object that is genuinely unpicklable fails the second time too and raises exactly as before -- so the broad catch costs one wasted attempt and hides nothing. Two tests, both mutation-checked: removing the retry fails the first and not the second, and dumping straight into the handle fails both. Full local gates: mypy clean, pylint 10.00/10, yapf/docformatter/isort applied. The previously failing shard passes locally, though a single local run is weak evidence at a 25% rate -- CI is the real test of this, and is why it is worth landing to find out.
…n overlay nobody reads (#138) * real robot: give stage 4 the cores it has, and stop rendering an overlay nobody reads run_20260818_092302 recorded a good episode and then could not use it. Post- processing took 1008 s against a 900 s fit deadline, so the fit skipped the take it had just paid a robot to produce and fell back to per-step scoring -- the twin-against-twin case the whole open-loop design exists to avoid. It missed by 108 s. The pipeline's own estimate says "about 3x the length of its take". Measured from the stage artifacts' timestamps it was 7.3x, and the discrepancy is not noise: the 3x figure was calibrated at 30 fps, and these takes record at 60 to resolve the propagation intervals. Cost scales with FRAMES, so doubling the frame rate doubles the multiple. 138 s of video, 7773 frames, 1008 s to a track: stage 1 replay -> frames.mp4 116 s stage 2 boxes (given) 0.5 crop -> frames_crop.mp4 40 stage 3 SAM-2 propagation 326 masks_overlay.mp4 render 163 <- nothing reads this stage 4 fit + emit 364 stage 5 poses overlay 60 (after the track; free) Two of those are addressed here. JOBS. Stage 4 fans out and ran 16 workers on a 32-core machine, which is the driver's default rather than a decision about this box. The arithmetic says it was cleanly core-limited -- 486 frames per worker over 364 s is 16.0 cores busy for the entire stage -- so it is the rare case where more workers really do translate. real_robot_track_jobs=0 keeps the driver's own choice; the domino config asks for 30, sized for this machine and deliberately not 32, since the pipeline runs in the background while the next episode is driving the robot. TRACK_VIZ. masks_overlay.mp4 is a debugging aid nothing downstream opens, and it is rendered BEFORE stage 4 -- so its 163 s is not tacked onto the end, it is paid straight out of time-to-track. Default off for automated runs, and easy to put back: that same run turned up a real id-matching problem, and the overlay is how id swaps get spotted. Needs BabyRobotPredicator#79; a driver that predates it renders the overlay as it always did rather than failing, which is what lets this land first. 0 means "leave the driver alone" for JOBS, and viz=True means saying nothing at all rather than passing TRACK_VIZ=1 -- an empty or contradictory env var is worse than an absent one when the far side is a shell. Both are asserted, and both assertions were checked against a mutant: dropping either guard fails its test. Expected effect, not yet measured on hardware: ~165 s from JOBS and 163 s from the overlay, so roughly 1008 s -> 11 min. That still exceeds no deadline by itself -- code_sim_learning_track_wait_s is untouched here and is the next thing to fix, along with reading a track only once it is completely written. * submodule: pick up TRACK_VIZ, so the overlay switch actually does something BabyRobotPredicator 603e4d4 -> ba4ee81 (#79). The previous commit already sets TRACK_VIZ=0 on every launch; against a driver without it that was simply ignored, which is what let the two land in either order. With this bump the 163 s masks_overlay.mp4 render is genuinely skipped and real_robot_track_viz stops being a no-op.
* real robot: open the take in front of the push, not the whole batch Only the cascade is scored. On run_20260818_092302 the first onset landed 107 s into a 131 s track, so roughly 80% of the video was the pick-and-place that arranges the row -- none of it evidence, all of it post-processing, which scales with frames rather than seconds. real_robot_record_from_option names the option the take opens in front of. _ship_episode sends the prologue as one unrecorded request, opens the take, then sends the rest. That costs one controller round trip and NOT a planner call: the twin has already simulated every option by the time _ship_episode runs, so this does not give back what open-loop batching bought. Splitting is safe for the same reason per-boundary shipping was -- _split_actions restarts its gripper tracking per call and RealRobot dedups session-wide -- and both requests still go out with observe=False, so the write-only side-effect argument that makes the twin trajectory bit-identical is untouched. The arm coming to rest at the boundary is a small gain of its own: the free-run is anchored at the last rest state before the push, and now there really is one. Empty, or a name the episode never runs, records everything and warns. Too much video is slow; too little is an episode whose first topple happened off camera, and the first onset is what every interval is measured against. STACKED ON #137. This moves the track's first frame from the episode's initial arrangement to the post-place one, and the id matching anchors on a position set. Against #137 as originally written -- anchored on the episode's first state -- that was fatal: the two placed dominoes sit ~150-200 mm from the anchor and drop out, taking the match from 4/4 to 2/4. 98e24cd moved both sides to the settled arrangement immediately BEFORE the cascade, which is independent of when recording starts, and push-only measures 4/4 again. This commit is only correct with that fix underneath it. An earlier version of this branch claimed recording from the push FIXES the id matching. That was wrong. The dominant cause was the base-frame/world-frame rotation #137 identified -- 144-307 mm against a 40 mm tolerance -- and starting the take later does not touch it. No lead-in, as a deliberate first cut. Two known hazards ride on that, neither fixed here and both likelier on a short take: load_track takes each domino's first OBSERVED centre, so a start domino occluded by the gripper for the whole pre-cascade window contributes no position and drops out of the match; and the offset vote's tie-break does not settle exact aliasing, so an evenly spaced row missing one domino can match completely and be shifted by one. If either bites, splitting one option earlier is the fix. Tests: five, plus three mutants -- no prologue split, recording opened before the prologue, and the unknown-option fallback recording nothing. The third initially PASSED, because asserting one shipment of three chunks cannot tell "everything recorded" from "everything shipped, then the take opened"; the ordering probe was added to pin it, and it fails under that mutant now. * real robot: draw the prompt boxes where the take opens, not at run start The boxes are SAM-2 prompts applied to the take's FIRST FRAME, so they have to describe the arrangement that frame shows. While the take began at the reset those were the same thing and the draw sat at run start. Recording from a later option makes them different, and nothing detected it: stage 2 does not report an empty box, it fits a mask to whatever is inside the one it was given. Observed on run_20260818_140211. boxes.json was written at 14:02:37, before any motion; the take opened at 14:05:37, after the prologue had picked and placed two dominoes. Two of the four prompts pointed at bare table. A human noticed it in the overlay, which is not a detection mechanism. So the draw moves to the recording boundary: after the prologue ships, before _start_recording. The arm is at rest and the row is in its final arrangement, and it is also the only moment a human can draw boxes that match the take. Where the take still opens at the reset -- no record_from_option -- the draw stays at run start, which was already right there. ensure_boxes is now idempotent. The boundary call site runs every episode and the draw must not: a fixed-plan replay arranges the same row each time, and a drag window opening per take is exactly what pick_boxes_at_start exists to avoid. This makes a real trade rather than removing a cost. The human is now needed ~50 s into the first episode instead of before it starts. The unattended alternative is projecting the twin's geometry into image space, which needs no human at all and is considerably more work. Three tests, on the ordering rather than the fact of the draw -- asserting that boxes were drawn cannot tell the right moment from the wrong one, which is the whole defect. The probe records what had shipped and whether the take was open at draw time, and asserts one shipment and no open take. Both mutations fail it: drawing at run start regardless, and deleting the boundary call.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is coupled with https://github.com/BasisResearch/BabyRobotPredicator/pull/82.
We use information from two cameras to do pose estimation, in order to avoid occluded or blocked views.
This PR is a WIP, blocked by: