Double pendulum env 5.0 fix - #652
Open
zdu863 wants to merge 1 commit into
Open
Conversation
height_record_reward has three terms covering three phases: a height-record
term that drives the swing-up, a bonus for records above 0.9, and a 0.1
stability bonus for being near the top and slow. The third was gated behind
got_upright, a per-episode latch, so it paid once and never again.
Once best_height saturates the record terms stop too, so from the first stable
step onward the agent was holding the pendulum upright for the remaining ~450
steps of the episode while earning nothing. There was no gradient distinguishing
"keep balancing" from "drift into the rail", and it drifted: every episode ended
at the rail, at ~295 of 600 steps.
Removing the latch makes the bonus recur. The predicate is unchanged and was
already evaluated every step for the hold_time metric; only the payment changes.
got_upright is now unused and removed.
total_timesteps goes 50M -> 200M. The takeoff completes around 125M, so at 50M
the shipped config could not learn the task at all.
Measured over 6 seeds, hold_time out of 600 steps:
upstream, 50M 0.12 x_threshold_termination 0.30
this change, 200M 510.8 +- 5.1 x_threshold_termination 0.000
range 501.5 - 514.9
best_height 1.000, upright_frac 0.87, every episode reaching the timeout.
Runs in ~40s on one GPU.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
Author
|
I made this new fix for 5.0 based on what you already have. i think it works better than the 4.0 pr from earlier. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
got_uprightlatched the 0.1 stability bonus to fire once per episode, so once the height records saturate nothing rewards staying up.total_timesteps50M -> 200M. Takeoff completes around 125M, so the previous budget could not learn the task.hold_time0.12 -> 510.8 +/- 5.1 out of 600,x_threshold_termination0.30 -> 0.000, ~40s on one laptop GPU.