From 9ba5a5992b15e93d24752bf2c3a9744a740b014d Mon Sep 17 00:00:00 2001 From: Zilong Date: Wed, 19 Aug 2026 21:53:08 -0500 Subject: [PATCH] double_pendulum: pay the stability bonus every step height_record_reward has three terms covering three phases: a height-record term that drives the swing-up, a bonus for records above 0.9, and a 0.1 stability bonus for being near the top and slow. The third was gated behind got_upright, a per-episode latch, so it paid once and never again. Once best_height saturates the record terms stop too, so from the first stable step onward the agent was holding the pendulum upright for the remaining ~450 steps of the episode while earning nothing. There was no gradient distinguishing "keep balancing" from "drift into the rail", and it drifted: every episode ended at the rail, at ~295 of 600 steps. Removing the latch makes the bonus recur. The predicate is unchanged and was already evaluated every step for the hold_time metric; only the payment changes. got_upright is now unused and removed. total_timesteps goes 50M -> 200M. The takeoff completes around 125M, so at 50M the shipped config could not learn the task at all. Measured over 6 seeds, hold_time out of 600 steps: upstream, 50M 0.12 x_threshold_termination 0.30 this change, 200M 510.8 +- 5.1 x_threshold_termination 0.000 range 501.5 - 514.9 best_height 1.000, upright_frac 0.87, every episode reaching the timeout. Runs in ~40s on one GPU. Co-Authored-By: Claude Opus 5 --- config/double_pendulum.ini | 2 +- ocean/double_pendulum/double_pendulum.h | 5 +---- 2 files changed, 2 insertions(+), 5 deletions(-) diff --git a/config/double_pendulum.ini b/config/double_pendulum.ini index c2e631a71d..b70c0d11c1 100644 --- a/config/double_pendulum.ini +++ b/config/double_pendulum.ini @@ -24,7 +24,7 @@ num_layers = 2 [train] gpus = 1 -total_timesteps = 50000000 +total_timesteps = 200000000 learning_rate = 0.005 anneal_lr = 1 min_lr_ratio = 0 diff --git a/ocean/double_pendulum/double_pendulum.h b/ocean/double_pendulum/double_pendulum.h index 803b55e3e8..1fea5c7335 100644 --- a/ocean/double_pendulum/double_pendulum.h +++ b/ocean/double_pendulum/double_pendulum.h @@ -52,7 +52,6 @@ struct Env { int max_upright_steps; int upright_count; float best_height; - int got_upright; int physics_failed; float cart_mass; @@ -178,7 +177,6 @@ void puf_reset(DoublePendulum* env) { env->max_upright_steps = 0; env->upright_count = 0; env->best_height = 0.0f; - env->got_upright = 0; env->physics_failed = 0; compute_observations(env); } @@ -305,9 +303,8 @@ float height_record_reward(DoublePendulum* env) { if (height > env->best_height && height > 0.9f) { reward += 0.05f * (height - 0.9f) / 0.1f; } - if (stable && !env->got_upright) { + if (stable) { reward += 0.1f; - env->got_upright = 1; } env->best_height = fmaxf(env->best_height, height); return reward;