Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/devtools-keyboard-walk.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@typeonce/ritmo-devtools": minor
---

`ritmo-devtools keyboard [--url <url>] [--route <file>] [--json]` plays the game's page by keyboard in Chromium and fails on what breaks keyboard play: a menu open with focus outside it, Tab leaving an open menu, a control with no accessible name, Enter or Space not pressing the focused control, a press that drops focus to `<body>`, and a visible control the keyboard cannot reach. Each failure names the check, the screen, the control (role, name, selector), what was seen, and the fix; a check it cannot decide is printed as not checked, never as passed, and a screen on which nothing could be checked fails. Without `--url` it serves the working directory with the game's own Vite config. A route file lists the screens to visit and how to reach them; without one, the start page is the only screen. It presses controls for real to check focus, so a route marks what must not be pressed. It uses the optional `playwright` and `vite` peers `verify` already uses, at the same versions.
5 changes: 5 additions & 0 deletions .changeset/ui-playbook-keyboard-check.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@typeonce/ritmo": patch
---

The `ritmo` skill's UI playbook (`ui-and-render.md`) checks a page with DOM controls with `ritmo-devtools keyboard`, on a route through every screen that shows them.
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ jobs:

# The check job skips these without browsers.
- name: Browser engine tests
run: pnpm vitest run packages/devtools/test/PlaywrightEngine.test.ts
run: pnpm vitest run packages/devtools/test/PlaywrightEngine.test.ts packages/devtools/test/KeyboardWalk.test.ts

# Pull requests only: on main the change is already merged, so a result
# there could not stop it.
Expand Down
2 changes: 1 addition & 1 deletion packages/core/skills/ritmo/playbooks/ui-and-render.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,4 @@
7. Build each DOM control (a button, a toggle, a slider) once and update its text and attributes in place; never rewrite a region that holds one (`innerHTML`, `replaceChildren`): the rewrite deletes the focused control and drops focus to `<body>`. Give every control a name a test can find it by: its text, or `aria-label` on an icon.
8. A menu that stops play is a modal: `showModal()` on a `<dialog>`, which moves focus in, keeps it there, and makes the page behind inert; an element with `aria-modal="true"` must do all three itself. The menu's own keys (arrows, letters) listen on that element. Handle a `<dialog>`'s `cancel` and `close` events as intents and show it again from the game's state, because the browser closes a dialog on a repeated Escape even when `cancel` is prevented. A splash or wipe that must cover a menu is a `popover="manual"` shown after it, never a `z-index`; a layer the player presses (a crash panel) is its own modal dialog, since everything outside the newest modal is inert.

**Check:** the simulation's headless tests pass unchanged, and the game's check passes.
**Check:** the simulation's headless tests pass unchanged, and the game's check passes. When the page has DOM controls, `ritmo-devtools keyboard` passes with a route through every screen that shows them (the `@typeonce/ritmo-devtools` README).
81 changes: 80 additions & 1 deletion packages/devtools/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ pnpm add -D @typeonce/ritmo-devtools
The main import (`Session`, `Fuzz`, `Invariant`, `Replay`, `Lint`, ...) and
`ritmo-devtools lint` need no peer dependency. Each optional peer is for one
feature: `playwright` and `vite` for runs in browser engines
(`PlaywrightEngine`, `ritmo-devtools verify`), `vite` for `ReplayVitePlugin`,
(`PlaywrightEngine`, `ritmo-devtools verify`, `ritmo-devtools keyboard`), `vite` for `ReplayVitePlugin`,
and `rolldown` and `oxlint` for `ritmo-devtools scan`.

The package reads the core `Debug` handle, which exists only on runtimes made
Expand Down Expand Up @@ -528,6 +528,85 @@ ritmo-devtools scan src/simulation/index.ts
missing tool). `--json` prints
`{ entry, files, packages: [{ name, files, findings: [{ file, line, column, rule, snippet }] }] }`.

## Keyboard walk

`ritmo-devtools keyboard` plays the game's page by keyboard in Chromium and
fails on what breaks keyboard play, whatever the game. Run it after a change
to the page's DOM controls or menus:

```sh
ritmo-devtools keyboard --route keyboard.json
# FAIL focus-outside-menu [title] dialog "Paused" at #menu: this menu is open and focus is on <body>. Fix: move focus into the menu when it opens: ...
# FAIL key-not-pressed [play] button "Menu" at #pause: Space with it focused did not press it: a key listener called preventDefault() on its keydown. Fix: ...
#
# [title]
# FAIL focus-outside-menu (1 decided)
# pass tab-leaves-menu (1 decided)
# ...
# not checked focus-dropped
# not checked link "Discord" at #links > a: a link to another site is not pressed for real
#
# keyboard walk of http://127.0.0.1:52011/: 2 failures; checks 2 failed, 9 passed, 1 not checked on 2 screens
```

Each screen gets six checks:

| check | fails when |
| --- | --- |
| `focus-outside-menu` | a menu is open and focus is outside it |
| `tab-leaves-menu` | Tab moves focus out of the open menu |
| `unnamed-control` | a control the keyboard reaches (by Tab, or by the arrow keys of its group) has no accessible name |
| `key-not-pressed` | Enter or Space on the focused control does not press it |
| `focus-dropped` | pressing a control drops focus to `<body>` |
| `unreachable-control` | a visible control is reached neither by Tab nor by the arrow keys of its group (a roving group with one Tab stop passes) |

A menu is a `<dialog>` opened with `showModal()`, a shown
`aria-modal="true"` element, or a shown dialog (`<dialog open>`,
`role="dialog"`). Names and roles are Chromium's own, from its accessibility
tree. A check that cannot be decided on a screen (no menu open, nothing Tab
reaches) is printed as not checked with the reason, never as passed. Each
failure line names the check, the screen, the control (role, name, and a
CSS selector that finds it), what was seen, and the fix.

`key-not-pressed` stops the click it caused, so it changes nothing a click
would; a custom control (a `role="switch"`) counts as pressed when its
`aria-checked`, `aria-pressed`, `aria-expanded`, or `aria-selected` changes.
`focus-dropped` presses each control for real, on a fresh load of its
screen, so it does what a player's press does: mark what must not be pressed
in the route. A link to another site is never pressed for real.

Without `--route` the start page is one screen. A route is a JSON file:

```json
{
"screens": [
{ "name": "title" },
{ "name": "play", "path": "?seed=7", "steps": [{ "click": { "role": "button", "name": "Play" } }, { "press": "s" }, { "wait": 800 }] },
{ "name": "end", "path": "?moment=end", "steps": [{ "wait": 5000 }], "noPress": [{ "name": "Start over", "role": "button" }] }
]
}
```

- `path` is appended to the URL. `steps` run in order after the page
settles: `press` takes a key as Playwright names it, `click` a role and
the accessible name the report prints, `wait` milliseconds.
- `noPress` is `true` (press nothing on that screen) or a list of controls
by name and optional role.
- A step that cannot run, or an error the page throws, is a failure, and
that screen's checks are not checked. A screen on which no check could be
decided is a failure too (`nothing-checked`): the route did not reach the
game's controls, or the page has none the walk can see.

- Without `--url` it serves the working directory with the game's own Vite
config; `--url` walks a page already served. It needs `playwright` and its
Chromium (`npm install --save-dev playwright vite` and
`npx playwright install chromium`); a missing one is printed with its
install command.
- Exit codes: 0 nothing failed, 1 a check failed, a route step failed, the
page threw, or a screen had nothing to check, 2 usage error (arguments, a route that is not valid, a missing
tool, a dev server that did not start). `--json` prints
`{ ok, url, screens, failures: [{ check, screen, control?, saw, fix }], outcomes: [{ check, screen, status, decided, undecided }] }`.

## Fuzzing

`Fuzz.run` finds a bug before anyone plays into it: seeded random input,
Expand Down
127 changes: 124 additions & 3 deletions packages/devtools/src/cli.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
* ritmo-devtools lint <entry>
* ritmo-devtools verify <entry> <replay> [--engines node,chromium,webkit,firefox] [--json]
* ritmo-devtools scan <entry> [--json]
* ritmo-devtools keyboard [--url <url>] [--route <file>] [--json]
*
* `lint` imports `<entry>` (a module exporting `lintTarget`, see the `Lint` module),
* prints every lint of its schedules with its fix, and exits with 1 when one
Expand Down Expand Up @@ -54,9 +55,22 @@
* and the game's `oxlint` with `@typeonce/oxlint-plugin-ritmo`; a missing one
* is printed with its install command and exits with 2.
*
* `keyboard` plays the game's page by keyboard in Chromium and checks, on
* each screen of a route, what a keyboard player relies on: focus moves into
* an open menu and Tab stays there, every control has a name, Enter and
* Space press the focused control, a press keeps focus off `<body>`, and
* every visible control is reached by Tab or by its group's arrow keys.
* Without `--url` it serves the working directory with the game's own Vite
* config. `--route <file>` names the screens to visit (JSON, see the README);
* without it the start page is one screen. It needs `playwright` and its
* Chromium, and `vite` when there is no `--url`; a missing one is printed
* with its install command.
*
* Exit codes: 0 ok (and every `scan` that ran), 1 lint warning, uncovered
* module, mismatched versions, or an engine that differs or did not run, 2 usage error (for `scan`
* also an unreadable entry, an import that does not resolve, or a missing tool).
* module, mismatched versions, an engine that differs or did not run, or a
* keyboard failure, 2 usage error (for `scan` also an unreadable entry, an
* import that does not resolve, or a missing tool; for `keyboard` an invalid
* route, a missing tool, or a dev server that did not start).
* Requires Node 22.15 or later for `module.registerHooks`; on older Node it
* prints that coverage was not checked.
*
Expand All @@ -68,6 +82,7 @@ import { resolve } from "node:path"
import { pathToFileURL } from "node:url"

import * as DependencyScan from "./internal/dependencyScan.ts"
import * as KeyboardWalk from "./internal/keyboardWalk.ts"
import * as PackageVersions from "./internal/packageVersions.ts"
import * as SimulationCoverage from "./internal/simulationCoverage.ts"
import * as SkillLink from "./internal/skillLink.ts"
Expand All @@ -79,9 +94,11 @@ const usage = [
"usage: ritmo-devtools lint <entry>",
" ritmo-devtools verify <entry> <replay> [--engines node,chromium,webkit,firefox] [--json]",
" ritmo-devtools scan <entry> [--json]",
" ritmo-devtools keyboard [--url <url>] [--route <file>] [--json]",
" lint: <entry> exports lintTarget: () => { runtime, schedules, describe?, hostEvents? } (see the Lint module)",
" verify: <entry> exports replayTarget: (playback) => { runtime, schedules, hostClock } (see the Verify module)",
" scan: <entry> is the simulation's entry module; every file it imports, node_modules included, is checked for Math, clock, and randomness calls"
" scan: <entry> is the simulation's entry module; every file it imports, node_modules included, is checked for Math, clock, and randomness calls",
" keyboard: plays the page at --url (or the working directory served with its Vite config) by keyboard in Chromium; --route is a JSON file of screens"
].join("\n")

const isRecord = (value: unknown): value is Record<string, unknown> => typeof value === "object" && value !== null
Expand All @@ -101,6 +118,7 @@ const main = async (args: ReadonlyArray<string>): Promise<number> => {
if (command === "lint" && rest.length === 1) return lint(rest[0]!)
if (command === "verify") return verify(rest)
if (command === "scan") return scan(rest)
if (command === "keyboard") return keyboard(rest)
console.error(usage)
return 2
}
Expand Down Expand Up @@ -223,6 +241,109 @@ const scan = async (args: ReadonlyArray<string>): Promise<number> => {
return 0
}

/** The value of `--name value` or `--name=value` at `index`, and how many arguments it took; undefined when `args[index]` is not that flag. */
const flag = (args: ReadonlyArray<string>, index: number, name: string): { readonly value: string | undefined; readonly taken: number } | undefined => {
const arg = args[index]!
if (arg === name) return { value: args[index + 1], taken: 2 }
if (arg.startsWith(`${name}=`)) return { value: arg.slice(name.length + 1), taken: 1 }
return undefined
}

const keyboard = async (args: ReadonlyArray<string>): Promise<number> => {
let url: string | undefined
let routeFile: string | undefined
let json = false
for (let index = 0; index < args.length;) {
if (args[index] === "--json") {
json = true
index++
continue
}
const given = flag(args, index, "--url") ?? flag(args, index, "--route")
if (given === undefined || given.value === undefined || given.value === "") {
console.error(usage)
return 2
}
if (args[index]!.startsWith("--url")) url = given.value
else routeFile = given.value
index += given.taken
}
if (url !== undefined && !/^https?:\/\//.test(url)) {
console.error(`--url must be an http(s) URL of the game's page, such as http://localhost:5173/; got ${url}`)
return 2
}
let route = KeyboardWalk.startRoute
if (routeFile !== undefined) {
let data: unknown
try {
data = JSON.parse(readFileSync(resolve(routeFile), "utf8"))
} catch (error) {
console.error(`${routeFile} could not be read as JSON: ${error instanceof Error ? error.message : String(error)}`)
return 2
}
const parsed = KeyboardWalk.parseRoute(data)
if (!parsed.ok) {
console.error(`${routeFile} is not a keyboard route: ${parsed.error}`)
return 2
}
route = parsed.value
}
let playwright: typeof import("playwright")
try {
playwright = await import("playwright")
} catch {
console.error("keyboard needs playwright, which is not installed: npm install --save-dev playwright && npx playwright install chromium")
return 2
}
let browser: Awaited<ReturnType<typeof playwright.chromium.launch>>
try {
browser = await playwright.chromium.launch()
} catch (error) {
const message = error instanceof Error ? error.message : String(error)
console.error(/Executable doesn't exist|playwright install/i.test(message)
? "keyboard needs Chromium, which is not downloaded: npx playwright install chromium"
: `Chromium did not launch: ${message.split("\n")[0]}`)
return 2
}
let server: { readonly close: () => Promise<void> } | undefined
try {
if (url === undefined) {
let vite: typeof import("vite")
try {
vite = await import("vite")
} catch {
console.error("keyboard serves the game with its own Vite config and needs vite, which is not installed: npm install --save-dev vite (or pass --url of a running dev server)")
return 2
}
try {
const dev = await vite.createServer({
root: process.cwd(),
logLevel: "silent",
clearScreen: false,
server: { host: "127.0.0.1", port: 0, strictPort: false, hmr: false, watch: null }
})
server = dev
await dev.listen()
const address = dev.httpServer?.address()
if (address === null || address === undefined || typeof address === "string") {
console.error("the Vite dev server did not report its port")
return 2
}
url = `http://127.0.0.1:${address.port}/`
} catch (error) {
console.error(`the game's Vite dev server did not start: ${error instanceof Error ? error.message : String(error)}`)
return 2
}
}
const report = await KeyboardWalk.walk(browser, { url, route })
console.log(json ? JSON.stringify(report, null, 2) : KeyboardWalk.format(report))
return report.ok ? 0 : 1
} finally {
await browser.close().catch(() => undefined)
await server?.close().catch(() => undefined)
}
}

const withoutText = ({ text: _text, ...error }: Verify.VerifyError) => error

process.exitCode = await main(process.argv.slice(2))
Loading
Loading