RLPlays game env helps you build 2D games - pixel platformers, shooters etc - that you can then train using PufferLib.

• An editor to create levels/maps
• Ability to quickly create new characters/blocks/enemies and edit/preview them in the editor
• Pipeline to train using Puffer RL library
  • Optional advanced features such as self-play, curriculum learning
• Deploy the trained playable model on Web/Win/Lin/Mac
  • You can play against yourself (recorded actions from previous rounds)
  • You can also play against an RL agent
  • Or you can simply play as a single-player by yourself

Here is what the game looks like - with a fully trained RL agent plus enemies/reward/goal/blocks:

Check out my blog for some of the rationale behind this project and also to play the game live inside the web-browser.

NOTE: I am no longer working on this project - feel free to fork/use as you see fit. If you do find this useful, please donate to OSS / cite this project / send me a note.

Use the prebuilt docker image for a quick start:

docker pull rlplays/rlplays:latest
docker run -d -p 3399:3389 -p 2222:22 --gpus all --name rlplays_rdp rlplays/rlplays:latest

# Connect via RDP or SSH (Change the password after connecting!)
# ssh rlplays@localhost -p 2222 (password c)
# or rdp via localhost:3399 rlplays / c

cd rlplays/docker
docker build -t rlplays:latest .
docker run -d -p 3399:3389 -p 2222:22 --name rlplays_rdp rlplays:latest
# Connect via RDP or SSH
# ssh rlplays@localhost -p 2222 (password c)
# or rdp via localhost:3399 rlplays / c

# In docker, cd /home/rlplays/rlplays
cd game/

# Get the latest changes in case the docker image is stale:
git pull

# Build the C++ code and run the GUI env. Requires RDP or some Ubuntu GUI to view the env.
bash full-build.sh DEBUG EDITOR RUN

You should see a default map open up.

Press CTRL+T to toggle between the editor mode and the gameplay mode.

Press C to switch between the RL Agent and your Player. (On Round 2 and above, you can play against the RL agent!)

Start the editor/gameplay program using bash full-build.sh DEBUG EDITOR RUN

--Shortcuts--

CTRL+T to open/close the editor mode, check out the various files, edit the level etc and go back to playing it.

**Editor Mode**
Right click to select any block
 - Edit the block using the Block window
 - Use the Worlds window to load a map/level/world
 - Use the Blocks to add blocks, save/load world
 - Change the World-level parameters in the World window.

CTRL+Click/drag to resize the block 
SHIFT+Left-click/drag to move blocks
CTRL+D to delete a block
Change any property and you can preview with CTRL+T
Important: Make sure to Save World in the Blocks Window to save the file, otherwise, you will lose the changes!

DEBUG VIEW: CTRL+1 to show the blocks, move vectors, and grid collisions.

The coreloop is written with a headless game env in mind that can run multithreaded (often running 1000s of copies with different levels across the copies) with 100K+ FPS (or in RL parlance, SPS steps per seconds) across all the environments. All animations/scene transitions etc are optional and run only in GUI (raylib Draw) mode. (TODO: add sounds similarly in an optional manner outside of the core Update loop if you were to actually ship a game!).

To write your own blocks/characters, add them in game/plays. Here is a fully self-contained example for a brick block:

struct TBrickBlock : ABlock
{
  TTexture Tex;
  SerializerWithBase(TBrickBlock, ABlock, Tex)

  void Init(TContextPtr context) override { Traits = Solid; }
  void LoadContent(TContextPtr context) override { context->LoadTexture(Tex); }
  void Draw(TContextPtr context) override { context->DrawTexture(Tex, Box, WHITE); }
};

This shows up in the editor and you can position this block anywhere, load any texture and is available as an RL obs and is RL trainable automatically with no specialization needed.

You do need to add the enum/type for serialization: two-line change in tblock.h.

I found it useful to to use an LLM to create characters and blocks, as AGENTS.md already has this recipe ready including creating a block template for you to add via the editor easily.

Here is the overall structure in game/:

• coreloop/ contains core serialization, world/scene, main game, timing/scene transitions, input, context, grid related definitions/code/structures.

• data/ contains the current level's data including sprites, fonts etc.

  • game/ contains the JSON level(s), while worlds/ contains the overall list of levels.
  • models/ contains the final trained RL config+weights

• editor/ contains the ImGui-based editor invoked using CTRL+T.

  • alldata/ here contains the entire set of resources you might have (in the public GitHub repo, I have included some CC0-licensed assets from kenney.nl, but this is where you would have all your assets in one place that will then be converted/copied to data/ using the converter).

• gameui/ - ignore this, contains the overall CMake files, some raylib-level integration.

• plays/ - contains the core block definitions.

  • Create new block types, characters, enemies here and use the resources in `alldata/.

• rlplays/ - contains the core RL glue:

  • include/rl_env.h - contains the actual RL environment including converting to the obs and running the coreloop.
  • tests/ - contains the RL tests that I use to test the multi-threading and other CUDA-related stuff.
  • Also contains various scripts used to link with the thirdparty/PufferLib/ codebase including copying the RL config into pufferlib, training the RL weights and copying the RL weights back.
  • rlplays.h and rlplays.py link with the Puffer 3.0 (not 4.0) code.

  NOTE: rlplays.py should go away when integrated with 4.0, but the rlplays.h should work mostly the same with the 4.0 codebase.

• tests/ - contains GoogleTest (gtest) based tests. Tests out the core grid/actions as well as the play/utils code.

  • Installed automatically when using CMake.
  • To run, see below.

NOTE: Most of the game/RL code is created by hand or with minimal LLM use. Since game design etc were not a core part of this project, I used LLMs heavily to workaround my design/UI weakness.

I use Opus/Sonnet 4.x as well as GPT 5.x to create the blocks/characters - it's a lot of fun to use an LLM as it's very well suited to this task as well as adding things like confetti or scene transitions etc.

I added AGENTS.md to help with this. It's fairly easy to create these blocks manually but editing the various headers to add a new block has many manual steps - LLMs are way better at it and they produce quick/decent UIs.

I used this prompt below to generate this dropper_enemy_block.h:

• Create a block that drops a block on a player if it detects the player is below them vertically - the dropping block should crush them and after a certain time, it can reset so it can new blocks (should be similar to the bullet/weapon block customizable). Add block templates and wire up everything for both.

...any recent LLM should be able to use the AGENTS.md to generate the whole set of changes and is instantly RL trainable as well:

Here is what it looks like (code in dropper_enemy_block.h):

Jumping enemy block:

Hover block:

All the RL code is isolated so the game env itself is standalone/playable/tweakable. The RL code has an environment glue code to stitch the RL env via actions/obs/rewards/goal (in rl_env.h - it's a bit messy as I was experimenting with curriculum learning, self-play and various forms of obs).

Note: I have a forked version of pufferlib (3.0) with my own native multithreading code in game/thirdparty/PufferLib/. I haven't ported to 4.0 yet.

cd /home/rlplays/rlplays/game

# To just retrain (convert all the world files and ensure obs size etc is correctly setup in rlplays.ini):
bash rlplays/build_rl_train.sh BUILD CONVERTER TRAIN

# To use WandB - https://wandb.ai/settings#apikeys and run `wandb login` first
bash rlplays/build_rl_train.sh  WANDB BUILD CONVERTER TRAIN

You should see the training TUI like this:

TODO: This uses Puffer 3.0; porting to 4.0 requires a resweep (took me several days/almost a week for the 3.0 sweep!) which is the main time consuming step I haven't invested as I don't have a powerful enough GPU to train it.

Once the training is done, it automatically places the trained .bin file so you can start playing. You can also watch a 'ghost player' show the highlight reel from the training mode if you use the right file in ghost_player.h line 137. Use CTRL+ 1, CTRL + 2, CTRL + 3 to show the debug ghost player views - you need to open the correct level to see the ghost player.

If you add new block types/levels, bash game/full-build.sh CONVERTER processes all the assets and generates a final RL config as well for RL training. It also places a web-ready/playable version. This also converts old serialized blocks into new ones automatically across all levels/maps.

These are a few features I was toying with - I am not an RL researcher so these are mostly toy research expeditions that I am not fully confident is the right way to do things, but it turned out to be a lot of fun for myself mostly. YMMV.

You can add a second player (with a different player index) into a map and that can play against you (a) via an RL agent (b) or using your recorded actions from a previous round. (You can configure the rounds to try out self-play yourself before training with Puffer).

This self-play by itself (e.g. rlplays_level4.json) would show super human characteristics as it learns by battling itself. Adding more enemies, interesting blocks etc would result in some quite interesting behaviors emerge out of simply having the RL algorithm beat itself over a long period of time.

I did a dumb version of curriculum training by having a level ladder that an RL agent would have to climb as it trains on increasingly more difficult levels starting with a few simple blocks/enemies/rewards. You can see syllabus_index in the Puffer dashboard TUI (or in WandB) as it hill climbs the various levels. I also tried having simpler levels after training on harder levels to ensure it can master the levels correctly.

Here is how the curriculum learning looks like in WandB:

As you can see, the tougher levels take a longer time to train and you can clearly see the agent hillclimb the difficulty ladder over hundreds of millions of steps (~1.5-2 hours on a 2080 RTX Ti card).

NOTE: The resulting code is research-level crappy code unfortunately as I was experimenting quite a lot with various ways to understand how the RL agent behaves in a realistic playable environment.

Game tests are in tests and RL Tests are in rlplays/tests. Use bash full-build.sh TEST or bash full-build.sh PUFFER_TEST to run the relevant tests. I mainly added PUFFER_TEST as that was my primary way to test the multithreading/GPU batching code I was experimenting with.

I have included the minimal assets from Kenney.nl as this repo is strictly to understand/train using RL - you can download the full assets from www.kenney.nl.

Please consider donating/buying the full asset pack if you find it useful.

I have included the actual code instead of a complicated git submodule setup which is finicky.

Here are the actual thirdparty code in game/thirdparty/:

• PufferLib - Main RL training+eval - Using my forked 3.0 branch native multithreading here https://github.com/rlplays/PufferLib
• Raylib - for all the graphics, input and tons of amazing utils. https://raylib.com
• Dear ImGui - for the editor UI, integrated via raylib_imgui - https://github.com/ocornut/imgui/
• Kenney.nl - minimal pixel platformer assets included (CC0) - https://kenney.nl/assets
• json - serialization library - https://github.com/nlohmann/json
• emsdk - for the web build - https://github.com/emscripten-core/emsdk

Please check THIRDPARTY.md for the specific licenses and please donate/support these OSS as they are all awesome projects.