Skip to content

ritmo-devtools keyboard: walk a game's page by keyboard in Chromium - #234

Merged
SandroMaglione merged 2 commits into
mainfrom
devtools/keyboard-walk
Nov 16, 2026
Merged

SandroMaglione merged 2 commits into
mainfrom
devtools/keyboard-walk

Conversation

@SandroMaglione

Copy link
Copy Markdown
Member

What this adds

ritmo-devtools keyboard [--url <url>] [--route <file>] [--json] plays a game's page by keyboard in Chromium and fails on six generic invariants:

check fails when
focus-outside-menu a menu is open and focus is outside it
tab-leaves-menu Tab moves focus out of the open menu
unnamed-control a control Tab reaches has no accessible name
key-not-pressed Enter or Space on the focused control does not press it
focus-dropped pressing a control drops focus to <body>
unreachable-control a visible control is reached neither by Tab nor by the arrows of its group

What counts:

  • A menu is a dialog:modal, a shown aria-modal="true" element (the same checkVisibility test Keyboard.actions uses), or a shown dialog (<dialog open>, role="dialog", role="alertdialog").
  • Names and roles come from Chromium's own accessibility tree, through a CDP session. Route clicks match those same names, so a name copied from a report finds its control. I found this the hard way: still-time's "Play" button is "PLAY" to Chromium (its CSS uppercases it), and Playwright's getByRole didn't find it.

Never a false pass:

  • A check that cannot be decided is printed as not checked with the reason, never as passed: no menu open, nothing reachable by Tab, a custom control that showed neither a click nor a state change, a link to another site, a control marked noPress.
  • A failed route step or a page error is a failure, and that screen's checks are not checked.

Side effects:

  • key-not-pressed stops the click it caused at a window capture listener installed before the game's scripts. A custom control (e.g. role="switch") counts as pressed when its aria-checked, aria-pressed, aria-expanded or aria-selected changes.
  • focus-dropped presses each control for real, on a fresh load of its screen. The README says so, and a route marks what must not be pressed (noPress). Links to another site are never pressed for real, and a popup a press opens is closed and counted as not checked.
  • Elements are tracked through a WeakMap; the game's DOM is never changed.

Serving and output:

  • Without --url it serves the working directory with the game's own Vite config. verify's engine uses a bare server and can't serve a game's page.
  • The playwright and vite peers are the optional ones verify already uses; neither range changes (the walk uses only APIs from before 1.40).
  • One line per failure: check, screen, the control (role, name, selector), what was seen, and the fix. The fix points at ui-and-render.md steps 7 and 8, or at Keyboard.actions and ritmo/use-keyboard-for-game-keys.
  • --json carries the same facts. Exit codes: 0 nothing failed, 1 any failure, 2 usage, an invalid route, a missing tool, or a dev server that didn't start.

Where it lives: the walk is internal to the CLI (src/internal/keyboardWalk.ts, src/internal/keyboardPage.ts), like scan; there is no new TypeScript API.

Docs: the README gets a "Keyboard walk" section. The UI playbook's Check line names the command.

The three games (clones, not the game folders)

Fix texts are trimmed to … below; obvious "not checked" reasons (no menu, nothing reachable) are left out.

still-time (Ritmo 0.19.1), no route

FAIL focus-outside-menu [start] dialog (no name) at body > div:nth-of-type(4): this menu is open and focus is on <body>. Fix: …
FAIL focus-dropped [start] button "PLAY" at body > div:nth-of-type(4) > div > div > div:nth-of-type(2) > button:nth-of-type(1): Enter pressed it and focus fell to <body>. Fix: …
FAIL focus-dropped [start] button "ROOMS" at body > div:nth-of-type(4) > div > div > div:nth-of-type(2) > button:nth-of-type(2): Enter pressed it and focus fell to <body>. Fix: …

[start]
  FAIL focus-outside-menu (1 decided)
  pass tab-leaves-menu (1 decided)
  pass unnamed-control (7 decided)
  pass key-not-pressed (7 decided)
  FAIL focus-dropped (7 decided)
  pass unreachable-control (7 decided)

keyboard walk of http://127.0.0.1:51584/: 3 failures; checks 2 failed, 4 passed, 0 not checked on 1 screen

Expected: fails focus-outside-menu and tab-leaves-menu. It fails focus-outside-menu but passes tab-leaves-menu. On the title, every control outside the menu is hidden: I listed every focusable element, and only the menu's 7 buttons are visible. So Tab has nowhere to go. The expectation was wrong; the walk is right. It also fails focus-dropped, because Play and Rooms rebuild the menu with replaceChildren or close it into play.

still-time, with a route into play and pause

{ "screens": [
  { "name": "title" },
  { "name": "play", "steps": [{ "click": { "role": "button", "name": "PLAY" } }, { "press": "s" }, { "wait": 800 }] },
  { "name": "pause", "steps": [{ "click": { "role": "button", "name": "PLAY" } }, { "press": "s" }, { "wait": 800 }, { "press": "Escape" }] }
] }

(s skips the first room's cutscene, which fades the HUD out to opacity 0.)

FAIL focus-outside-menu [title] dialog (no name) at body > div:nth-of-type(4): this menu is open and focus is on <body>. Fix: …
FAIL focus-dropped [title] button "PLAY" at body > div:nth-of-type(4) > div > div > div:nth-of-type(2) > button:nth-of-type(1): Enter pressed it and focus fell to <body>. Fix: …
FAIL focus-dropped [title] button "ROOMS" at body > div:nth-of-type(4) > div > div > div:nth-of-type(2) > button:nth-of-type(2): Enter pressed it and focus fell to <body>. Fix: …
FAIL key-not-pressed [play] button "Menu" at #pause: Enter with it focused did not press it: a key listener called preventDefault() on its keydown. Fix: …
FAIL key-not-pressed [play] button "Menu" at #pause: Space with it focused did not press it: a key listener called preventDefault() on its keydown. Fix: …
FAIL key-not-pressed [play] button "Restart" at #restart: Enter with it focused did not press it: a key listener called preventDefault() on its keydown. Fix: …
FAIL key-not-pressed [play] button "Restart" at #restart: Space with it focused did not press it: a key listener called preventDefault() on its keydown. Fix: …
FAIL focus-outside-menu [pause] dialog (no name) at body > div:nth-of-type(4): this menu is open and focus is on <body>. Fix: …
FAIL focus-dropped [pause] button "RESUME" at body > div:nth-of-type(4) > div > div > button:nth-of-type(1): Enter pressed it and focus fell to <body>. Fix: …
FAIL focus-dropped [pause] button "RESTART" at body > div:nth-of-type(4) > div > div > button:nth-of-type(2): Enter pressed it and focus fell to <body>. Fix: …
FAIL focus-dropped [pause] button "ROOMS" at body > div:nth-of-type(4) > div > div > button:nth-of-type(3): Enter pressed it and focus fell to <body>. Fix: …
FAIL focus-dropped [pause] button "HOME" at body > div:nth-of-type(4) > div > div > button:nth-of-type(4): Enter pressed it and focus fell to <body>. Fix: …

[title]
  FAIL focus-outside-menu (1 decided)
  pass tab-leaves-menu (1 decided)
  pass unnamed-control (7 decided)
  pass key-not-pressed (7 decided)
  FAIL focus-dropped (7 decided)
  pass unreachable-control (7 decided)
[play]
  not checked focus-outside-menu
  not checked tab-leaves-menu
  pass unnamed-control (2 decided)
  FAIL key-not-pressed (2 decided)
  not checked focus-dropped
    not checked button "Menu" at #pause: Enter or Space does not press it (key-not-pressed)
    not checked button "Restart" at #restart: Enter or Space does not press it (key-not-pressed)
  pass unreachable-control (2 decided)
[pause]
  FAIL focus-outside-menu (1 decided)
  pass tab-leaves-menu (1 decided)
  pass unnamed-control (9 decided)
  pass key-not-pressed (9 decided)
  FAIL focus-dropped (9 decided)
  pass unreachable-control (9 decided)

keyboard walk of http://127.0.0.1:51646/: 12 failures; checks 5 failed, 10 passed, 3 not checked on 3 screens

Expected: key-not-pressed on the in-game menu button. It fails exactly there: on Menu and Restart, Enter and Space are taken by the game's own window key listener, which calls preventDefault().

tabletop-grand-prix (Ritmo 0.23.0)

?server=fake is the game's in-page stand-in server, which shows the leaderboard panel. l closes it.

{ "screens": [
  { "name": "start" },
  { "name": "board", "path": "?server=fake" },
  { "name": "board-closed", "path": "?server=fake", "steps": [{ "press": "l" }, { "wait": 600 }] }
] }
FAIL unreachable-control [start] link "Typeonce Games: more games by the maker of this one (opens in a new tab)" at body > a:nth-of-type(1): it is visible, and neither Tab nor the arrow keys of its group reach it. Fix: …
FAIL unreachable-control [board] link "Typeonce Games: more games by the maker of this one (opens in a new tab)" at body > div:nth-of-type(2) > div:nth-of-type(1) > a: it is visible, and neither Tab nor the arrow keys of its group reach it. Fix: …
FAIL unreachable-control [board] button "×" at body > div:nth-of-type(2) > div:nth-of-type(1) > div:nth-of-type(1) > button: it is visible, and neither Tab nor the arrow keys of its group reach it. Fix: …
FAIL unreachable-control [board] button "NEXT CLOSEST" at body > div:nth-of-type(2) > div:nth-of-type(1) > div:nth-of-type(5) > button:nth-of-type(1): it is visible, and neither Tab nor the arrow keys of its group reach it. Fix: …
FAIL unreachable-control [board] button "MY BEST" at body > div:nth-of-type(2) > div:nth-of-type(1) > div:nth-of-type(5) > button:nth-of-type(2): it is visible, and neither Tab nor the arrow keys of its group reach it. Fix: …
FAIL unreachable-control [board] button "OFF" at body > div:nth-of-type(2) > div:nth-of-type(1) > div:nth-of-type(5) > button:nth-of-type(3): it is visible, and neither Tab nor the arrow keys of its group reach it. Fix: …
FAIL unreachable-control [board] button "1" at body > div:nth-of-type(2) > div:nth-of-type(1) > div:nth-of-type(6) > button:nth-of-type(1): it is visible, and neither Tab nor the arrow keys of its group reach it. Fix: …
FAIL unreachable-control [board] button "5" at body > div:nth-of-type(2) > div:nth-of-type(1) > div:nth-of-type(6) > button:nth-of-type(2): it is visible, and neither Tab nor the arrow keys of its group reach it. Fix: …
FAIL unreachable-control [board] button "10" at body > div:nth-of-type(2) > div:nth-of-type(1) > div:nth-of-type(6) > button:nth-of-type(3): it is visible, and neither Tab nor the arrow keys of its group reach it. Fix: …
FAIL unreachable-control [board] button "PICK A NAME" at body > div:nth-of-type(2) > div:nth-of-type(1) > div:nth-of-type(7) > button: it is visible, and neither Tab nor the arrow keys of its group reach it. Fix: …
FAIL unreachable-control [board] link "Typeonce Games: more games by the maker of this one (opens in a new tab)" at body > a:nth-of-type(1): it is visible, and neither Tab nor the arrow keys of its group reach it. Fix: …
FAIL unreachable-control [board-closed] button "L BOARD" at body > div:nth-of-type(2) > button: it is visible, and neither Tab nor the arrow keys of its group reach it. Fix: …
FAIL unreachable-control [board-closed] link "Typeonce Games: more games by the maker of this one (opens in a new tab)" at body > a:nth-of-type(1): it is visible, and neither Tab nor the arrow keys of its group reach it. Fix: …

[start]
  not checked focus-outside-menu
  not checked tab-leaves-menu
  pass unnamed-control (1 decided)
  pass key-not-pressed (1 decided)
  not checked focus-dropped
    not checked link "Made with PixiJS 3D, drawn with WebGL (opens in a new tab)" at body > a:nth-of-type(2): a link to another site is not pressed for real
  FAIL unreachable-control (2 decided)
[board]
  not checked focus-outside-menu
  not checked tab-leaves-menu
  pass unnamed-control (1 decided)
  pass key-not-pressed (1 decided)
  not checked focus-dropped
    not checked link "Made with PixiJS 3D, drawn with WebGL (opens in a new tab)" at body > a:nth-of-type(2): a link to another site is not pressed for real
  FAIL unreachable-control (11 decided)
[board-closed]
  not checked focus-outside-menu
  not checked tab-leaves-menu
  pass unnamed-control (1 decided)
  pass key-not-pressed (1 decided)
  not checked focus-dropped
    not checked link "Made with PixiJS 3D, drawn with WebGL (opens in a new tab)" at body > a:nth-of-type(2): a link to another site is not pressed for real
  FAIL unreachable-control (3 decided)

keyboard walk of http://127.0.0.1:51788/: 13 failures; checks 3 failed, 6 passed, 9 not checked on 3 screens

Expected: unreachable-control. It fails on every leaderboard button, the "L BOARD" opener and the maker's link, all of which the game sets to tabIndex = -1. Its other bug, a stopSpace listener that prevents Space on every non-field target, is hidden behind this one: no button can take focus, so key-not-pressed can't be judged.

landkeep ui-sdk branch (f32babb, Ritmo 0.24.0)

The screens are the ones its own scripts/keyboard.ts visits:

{ "screens": [
  { "name": "title" },
  { "name": "settings", "steps": [{ "press": "Enter" }, { "wait": 1500 }, { "press": "ArrowDown" }, { "press": "Enter" }, { "wait": 700 }] },
  { "name": "map-card", "path": "?map=8&seed=7" },
  { "name": "choosing", "path": "?map=8&seed=7", "steps": [{ "press": "Escape" }, { "wait": 900 }] },
  { "name": "dusk", "path": "?moments&moment=downpour&map=8&hold", "steps": [{ "wait": 2500 }] },
  { "name": "end", "path": "?moments&moment=end-fail&map=8", "steps": [{ "wait": 5000 }] }
] }
FAIL focus-dropped [dusk] button "Got it" at #coach > button: Enter pressed it and focus fell to <body>. Fix: …

[title]
  pass focus-outside-menu (1 decided)
  pass tab-leaves-menu (1 decided)
  pass unnamed-control (4 decided)
  pass key-not-pressed (4 decided)
  pass focus-dropped (3 decided)
    not checked link "Discord ↗" at #modal-body > div > div:nth-of-type(2) > a: a link to another site is not pressed for real
  pass unreachable-control (4 decided)
[settings]
  pass focus-outside-menu (1 decided)
  pass tab-leaves-menu (1 decided)
  pass unnamed-control (6 decided)
  pass key-not-pressed (4 decided)
  pass focus-dropped (2 decided)
    not checked link "Discord ↗" at #modal-body > div > div:nth-of-type(2) > a: a link to another site is not pressed for real
  pass unreachable-control (4 decided)
[map-card]
  pass focus-outside-menu (1 decided)
  pass tab-leaves-menu (1 decided)
  pass unnamed-control (3 decided)
  pass key-not-pressed (3 decided)
  pass focus-dropped (3 decided)
  pass unreachable-control (3 decided)
[choosing]
  pass focus-outside-menu (1 decided)
  pass tab-leaves-menu (1 decided)
  pass unnamed-control (6 decided)
  pass key-not-pressed (6 decided)
  pass focus-dropped (6 decided)
  pass unreachable-control (10 decided)
[dusk]
  not checked focus-outside-menu
  not checked tab-leaves-menu
  pass unnamed-control (32 decided)
  pass key-not-pressed (32 decided)
  FAIL focus-dropped (32 decided)
  pass unreachable-control (38 decided)
[end]
  pass focus-outside-menu (1 decided)
  pass tab-leaves-menu (1 decided)
  pass unnamed-control (4 decided)
  pass key-not-pressed (4 decided)
  pass focus-dropped (4 decided)
  pass unreachable-control (4 decided)

keyboard walk of http://127.0.0.1:51095/: 1 failure; checks 1 failed, 33 passed, 2 not checked on 6 screens

Expected: a pass. It passes 33 checks and fails one, which is real. On the dusk screen, "Got it" on a coach line hides the line by removing a CSS class. The focused button disappears, Chromium drops focus to <body>, and nothing moves it to the land. landkeep's own script clicks "Got it" with the mouse, so it never sees this. Nothing here is a failure that #232 fixed: the game is on 0.24.0.

What landkeep's own script checks that the generic walk does not:

  • what a key does on each screen: Esc goes on to the cards, Esc on the end card shows the land, Esc again brings the card back;
  • that a pressed card reads as pressed afterwards (aria-pressed="true"), and that focus lands on the canvas specifically. The walk only checks that focus isn't on <body>;
  • that the arrows move between cards in a group. The walk uses the arrows only to reach controls Tab missed;
  • game hotkeys (digits, Z) working while a control has focus;
  • the slider's arrows changing its value, and Space flipping the Sound switch. The walk counts the switch as pressed when its aria-checked changes, but doesn't check the value;
  • its own patchHtml helper, and its dev warning about a region rewritten whole. The walk sees page errors, not console warnings.

Findings I'd question

Possible false positives:

  • still-time, focus-dropped on Play and Resume. These close the menu into play, and focus falls to <body>. still-time reads keys with its own window listener, so the player can still play. But Tab then starts from the top of the page, and the Keyboard JSDoc asks for focus on the canvas. I'd keep it as a failure.
  • tabletop, "L BOARD". The l key opens the board, so a keyboard player can reach it without the button. The walk can't know a control has a shortcut.

Known false negatives:

  • Tab checked only where something is reachable. tab-leaves-menu can only fail where something focusable is visible outside the menu, so a non-modal menu over a page with nothing else to reach passes. That is still-time's case.
  • One key per control for real. focus-dropped presses each control with its first key only (Enter, or Space for a checkbox).
  • No visual check. A control hidden only by transform or by being off-screen still counts as visible.
  • Unreachable controls checked only inside an open menu. unreachable-control looks only inside the open menu when there is one: the page behind a modal is inert by design.

Left out

  • WebKit and Firefox.
  • Screenshots.
  • Console warnings.
  • Asserting where focus lands (the canvas) beyond "not <body>".
  • Settle time as a flag: a route's wait steps cover a slow screen.

Speed: landkeep's six screens took 5½ minutes, because each control pressed for real reloads its screen (32 controls on the dusk screen). noPress on a screen skips that.

Tests and CI

  • test/KeyboardWalk.test.ts:
    • route parsing (runs everywhere);
    • the CLI's argument and route errors (runs everywhere, no browser);
    • fixture pages in Chromium: one that passes all six checks with a roving toolbar and a modal <dialog>, one per failing check, a page error, a failed route step, noPress, the Vite-served start page, and --json.
  • The Chromium tests skip where Chromium isn't downloaded, like PlaywrightEngine.test.ts. They are added to the "Browser engine tests" step of the CI job that already installs browsers. They take about 45 s.
  • Verified locally:
    • pnpm run check: 1,999 tests, including the Chromium ones.
    • pnpm pack:check.
    • pnpm --dir website build: there is no website change; I ran it because the CLI is public.
    • the built dist/cli.js keyboard run on the passing fixture.

🤖 Generated with Claude Code

SandroMaglione and others added 2 commits November 17, 2026 06:36
A new CLI command plays the page by keyboard on each screen of a route and
fails on six generic invariants: a menu open with focus outside it, Tab
leaving an open menu, a control with no accessible name, Enter or Space not
pressing the focused control (its click stopped, so nothing changes), a
press that drops focus to <body> (pressed for real on a fresh load), and a
visible control neither Tab nor its group's arrows reach.

A check it cannot decide is printed as not checked with the reason, never
as passed; a failed route step or a page error is a failure, and that
screen's checks are not checked. Each failure names the check, the screen,
the control (Chromium's role and name, a selector), what was seen, and the
fix. Without --url it serves the working directory with the game's Vite
config. Routes are validated with messages that name the bad field.

The tests run fixture pages in Chromium where it is downloaded (the CI
browser job) and skip elsewhere, like the engine tests. The UI playbook's
check line names the command.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…arrow-reached controls

- A screen on which no check could be decided is a failure (nothing-checked),
  so a route that misses the game's controls never exits with 0.
- unnamed-control covers every control the keyboard reaches, by Tab or by
  the arrows of its group; names are read before the arrow keys are pressed.
- The focus-dropped fix covers a control that hides itself, and no longer
  cites a playbook step that does not say it.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@SandroMaglione

Copy link
Copy Markdown
Member Author

Review pass (commit 4b3a348), after rerunning the walk on the still-time, tabletop-grand-prix, and landkeep ui-sdk clones (same results as the description) and on two games it was not designed against (shape-brawl, emberworks):

  • A screen with nothing checked now fails (nothing-checked, exit 1). shape-brawl's start page has no DOM control, and the walk printed six "not checked" lines and exited with 0, which a check script reads as a pass.
  • unnamed-control covers controls reached by a group's arrow keys, not only Tab stops: an icon button inside a roving toolbar was never named-checked. Names are read before the arrow keys are pressed, since a game that takes them may redraw.
  • The focus-dropped fix text covers a control that hides itself (landkeep's "Got it"), and no longer cites playbook step 8 for something only the Keyboard.actions JSDoc says.

emberworks passed with no false alarm. pnpm run check passes locally (2,000 tests, Chromium ones included).

🤖 Generated with Claude Code

@SandroMaglione
SandroMaglione merged commit 6cd38d6 into main Nov 16, 2026
5 checks passed
@SandroMaglione
SandroMaglione deleted the devtools/keyboard-walk branch November 16, 2026 17:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant