Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .changeset/devtools-layout-audit.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
"@typeonce/ritmo-devtools": minor
---

`ritmo-devtools layout [--url <url>] [--route <file>] [--screenshots <dir>] [--json]` opens each screen of a route in Chromium at several screen sizes (a small phone and an iPhone held sideways, an iPhone upright with its notch and home bar, and a desktop window, unless the route names its own) and reports what is broken in the layout the player sees. Errors fail the run: two things drawn over each other (`overlap`), text a box cuts with no ellipsis (`cut-text`), text or a control past the screen's edge (`off-screen`) or in the notch or home bar (`safe-area`), a tap area under 44 px (`small-target`, measured by hit tests, so a `::after` that extends it counts), something that moves while nothing is pressed (`layout-shift`), and a layout that never holds still (`never-settled`). Warnings never fail: text shortened with an ellipsis (`truncated-text`), a control half hidden at a scroll edge (`scroll-edge`), and what breaks only when every text is 40% longer. The buttons of a `Touch.actions` layout are checked too. A route's `allow` leaves out what the game means; `--screenshots` saves each screen with its findings boxed and numbered.

The route file is shared with `ritmo-devtools keyboard`: it gains a `type` step (types text into the focused field), and `viewports`, `minTarget`, and `allow`, which only `layout` reads. A `click` step finds its control in the open menu first.
5 changes: 5 additions & 0 deletions .changeset/touch-layout-reader.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@typeonce/ritmo-browser": patch
---

While attached, `Touch.actions` leaves a function on its target that returns the layout at the target's current size, under `Symbol.for("@typeonce/ritmo-browser/Touch.layout")`, so `ritmo-devtools layout` can check the buttons the canvas draws. `dispose()` removes it. Nothing changes for a game.
5 changes: 5 additions & 0 deletions .changeset/ui-playbook-layout-check.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@typeonce/ritmo": patch
---

The `ritmo` skill's UI playbook (`ui-and-render.md`) checks the page with `ritmo-devtools layout` on the keyboard walk's route, at every screen size the game targets, and has the agent look at its screenshots.
11 changes: 11 additions & 0 deletions packages/browser/src/Touch.ts
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,12 @@
* press opens is cancelled while a tracked finger is down. `dispose()`
* restores the styles.
*
* While attached, the target also carries a function that returns the
* layout at its current size, under `Symbol.for("@typeonce/ritmo-browser/Touch.layout")`:
* `ritmo-devtools layout` reads it to check the buttons the canvas does not
* show to the page (too small to tap, under the notch, under a panel).
* `dispose()` removes it.
*
* @module Touch
* @docGroup browser
*
Expand Down Expand Up @@ -77,6 +83,9 @@ import { anyUsed } from "./internal/used.ts"
import type { ActionState } from "./internal/state.ts"
import type { Vector } from "./internal/vectors.ts"

/** Where `ritmo-devtools layout` reads the layout of an attached target (see the module header). */
const layoutKey = Symbol.for("@typeonce/ritmo-browser/Touch.layout")

/** A position in CSS pixels relative to the element's top-left corner. */
export interface Position {
readonly x: number
Expand Down Expand Up @@ -551,6 +560,7 @@ export const actions = <const B extends string, const V extends string = never,
const style = target.style
const previousStyles = style === undefined ? [] : noBrowserGestures.map((property) => [property, style.getPropertyValue(property)] as const)
for (const property of noBrowserGestures) style?.setProperty(property, "none")
Reflect.set(target, layoutKey, () => layout(sizeOf()))
target.addEventListener("pointerdown", onDown)
target.addEventListener("pointermove", onMove)
target.addEventListener("pointerup", onUp)
Expand Down Expand Up @@ -610,6 +620,7 @@ export const actions = <const B extends string, const V extends string = never,
blurHost.removeEventListener("blur", onBlur)
// An empty value removes the property, so a style the element did not have is not left behind.
for (const [property, value] of previousStyles) style?.setProperty(property, value)
Reflect.deleteProperty(target, layoutKey)
}
}
}
Expand Down
10 changes: 10 additions & 0 deletions packages/browser/test/Touch.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -179,6 +179,16 @@ describe("Touch.actions", () => {
expect(withMouse.snapshot().actions.attack.held).toBe(true)
})

it("lets ritmo-devtools layout read the layout while attached", () => {
const fake = makeTarget()
const key = Symbol.for("@typeonce/ritmo-browser/Touch.layout")
const touch = Touch.actions(fake.target, fake.blurHost, layout)
const read = Reflect.get(fake.target, key)
expect(typeof read === "function" ? read() : undefined).toEqual(layout(size))
touch.dispose()
expect(Reflect.has(fake.target, key)).toBe(false)
})

it("cancels the long-press context menu only while a tracked finger is down", () => {
const fake = makeTarget()
Touch.actions(fake.target, fake.blurHost, layout)
Expand Down
2 changes: 1 addition & 1 deletion packages/core/skills/ritmo/playbooks/ui-and-render.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,4 @@
7. Build each DOM control (a button, a toggle, a slider) once and update its text and attributes in place; never rewrite a region that holds one (`innerHTML`, `replaceChildren`): the rewrite deletes the focused control and drops focus to `<body>`. Give every control a name a test can find it by: its text, or `aria-label` on an icon.
8. A menu that stops play is a modal: `showModal()` on a `<dialog>`, which moves focus in, keeps it there, and makes the page behind inert; an element with `aria-modal="true"` must do all three itself. The menu's own keys (arrows, letters) listen on that element. Handle a `<dialog>`'s `cancel` and `close` events as intents and show it again from the game's state, because the browser closes a dialog on a repeated Escape even when `cancel` is prevented. A splash or wipe that must cover a menu is a `popover="manual"` shown after it, never a `z-index`; a layer the player presses (a crash panel) is its own modal dialog, since everything outside the newest modal is inert.

**Check:** the simulation's headless tests pass unchanged, and the game's check passes. When the page has DOM controls, `ritmo-devtools keyboard` passes with a route through every screen that shows them (the `@typeonce/ritmo-devtools` README).
**Check:** the simulation's headless tests pass unchanged, and the game's check passes. When the page has DOM controls, `ritmo-devtools keyboard` passes with a route through every screen that shows them, and `ritmo-devtools layout` passes on the same route at every screen size the game targets, with its screenshots looked at (the `@typeonce/ritmo-devtools` README).
106 changes: 103 additions & 3 deletions packages/devtools/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,8 @@ pnpm add -D @typeonce/ritmo-devtools
The main import (`Session`, `Fuzz`, `Invariant`, `Replay`, `Lint`, ...) and
`ritmo-devtools lint` need no peer dependency. Each optional peer is for one
feature: `playwright` and `vite` for runs in browser engines
(`PlaywrightEngine`, `ritmo-devtools verify`, `ritmo-devtools keyboard`), `vite` for `ReplayVitePlugin`,
(`PlaywrightEngine`, `ritmo-devtools verify`, `ritmo-devtools keyboard`,
`ritmo-devtools layout`), `vite` for `ReplayVitePlugin`,
and `rolldown` and `oxlint` for `ritmo-devtools scan`.

The package reads the core `Debug` handle, which exists only on runtimes made
Expand Down Expand Up @@ -588,8 +589,11 @@ Without `--route` the start page is one screen. A route is a JSON file:
```

- `path` is appended to the URL. `steps` run in order after the page
settles: `press` takes a key as Playwright names it, `click` a role and
the accessible name the report prints, `wait` milliseconds.
settles: `press` takes a key as Playwright names it, `type` types text
into the focused field, `click` a role and the accessible name the report
prints (in the open menu first), `wait` milliseconds. Each screen starts
from a fresh page with nothing saved, so a game that asks for a name
first starts its steps with `{ "click": { "role": "textbox", "name": "Name" } }, { "type": "Ada" }`.
- `noPress` is `true` (press nothing on that screen) or a list of controls
by name and optional role.
- A step that cannot run, or an error the page throws, is a failure, and
Expand All @@ -607,6 +611,102 @@ Without `--route` the start page is one screen. A route is a JSON file:
tool, a dev server that did not start). `--json` prints
`{ ok, url, screens, failures: [{ check, screen, control?, saw, fix }], outcomes: [{ check, screen, status, decided, undecided }] }`.

## Layout audit

`ritmo-devtools layout` opens each screen of the same route in Chromium at
several screen sizes and fails on what is broken in the layout the player
sees, whatever the game. Run it after a change to anything the page shows
over the game (menus, HUD, touch controls):

```sh
ritmo-devtools layout --route route.json --screenshots .layout
# FAIL small-target [settings @ phone-landscape] control "Close" at div.screen.settings > header.topbar > span.topbar__end > button.round: a tap reaches it over 40 x 50 px (at least 42 x 42). (.layout/settings@phone-landscape.png #1) Fix: ...
# FAIL safe-area [arena @ phone-landscape] touch button "dash" on #game > canvas:nth-of-type(1): the touch button "dash" reaches 4 px into the space the device keeps (a notch, a home bar). (.layout/arena@phone-landscape.png #1) Fix: ...
# warn truncated-text [practice @ phone-landscape] text "Training Dummy" at div.cards > button.card:nth-of-type(1) > span.card__foot: "Training Dummy" is shortened with an ellipsis: 55 of its 82 px show. (.layout/practice@phone-landscape.png #1) Fix: ...
# warn safe-area [home @ phone-landscape, longer text] control "Play Pl" at div.home__play > button.pill:nth-of-type(2): with every text about 40% longer, it reaches into the space the device keeps on the right (47 px: a notch or the home bar). (...) Fix: ...
#
# [settings]
# small-phone-landscape: FAIL small-target (1), pass 8 more
# phone-landscape: FAIL small-target (1), pass 8 more
# ...
# layout audit of http://127.0.0.1:5199/: 55 errors, 31 warnings on 12 screens at 3 viewports
```

It measures what the browser has already laid out, never a picture: each
line of visible text (the text itself, not the box around it), each
visible control (as the keyboard walk finds them), and each button of a
`Touch.actions` layout, which the canvas draws and the page cannot see
(`Touch.actions` leaves a reader on its target for this). Boxes are in
screen pixels after every transform, so a UI scaled as one piece is
measured as shown. Before measuring, it turns on reduced motion and waits
until the layout holds still. When an open `<dialog>` or `aria-modal`
element owns the screen, only what is inside it is measured.

Errors fail the run; each is a measurement a game never means:

| check | fails when |
| --- | --- |
| `overlap` | two things are drawn over each other and neither is part of the other: text over text, text or a button over a button, the page over a Touch button, two Touch buttons. Hit tests at the shared area confirm both show there with nothing drawn above both, so a control that draws nothing (a touch area over the stage), something behind a scrim, a badge inside its card, and a button on a control under the whole of it ("tap anywhere", a clickable panel) are not overlaps |
| `cut-text` | the edge of a box cuts text, with no ellipsis to say text is missing |
| `off-screen` | part of a text or control is past the edge of the screen (a box that spans the screen cuts at the screen's edge, so that counts) |
| `safe-area` | part of a text or control is in the space the viewport's `insets` keep (a notch, the home bar), set through Chromium so `env(safe-area-inset-*)` reads them. A control over the whole safe area ("tap anywhere to start") is meant to fill the screen, and only its text is judged |
| `small-target` | the area that takes a tap is under `minTarget` either side, found by hit tests through the control's centre (a `::after` that extends it counts); a Touch button narrower than `minTarget` |
| `layout-shift` | something moved while nothing was pressed, from Chromium's own layout-shift entries over a still screen |
| `never-settled` | the layout kept moving, so nothing else was measured (a transform animation is motion, not layout, and settles) |

Warnings are printed and never fail, because a game may mean them:

| check | warns when |
| --- | --- |
| `truncated-text` | text is shortened with an ellipsis or a line clamp |
| `scroll-edge` | a control is half hidden at the edge of a scrolling area, or a scrolling area runs past the screen |

Then each screen is loaded again with every text about 40% longer (each
text followed by the start of itself, so the width is the font's own), and
what breaks only then (`overlap`, `cut-text`, `off-screen`, `safe-area`) is
a warning marked "longer text": a layout with no room to spare for a longer
name or a translation.

The route is the keyboard walk's, with three fields only the audit reads:

```json
{
"screens": [
{ "name": "home", "steps": [{ "click": { "role": "button", "name": "Play" } }], "allow": [{ "check": "overlap", "selector": ".card .badge" }] }
],
"viewports": [
{ "name": "phone", "width": 844, "height": 390, "touch": true, "insets": { "left": 47, "right": 47, "bottom": 21 } },
{ "name": "desktop", "width": 1280, "height": 800 }
],
"minTarget": 44,
"allow": [{ "check": "truncated-text", "selector": ".card__name" }]
}
```

- `viewports` default to a small phone held sideways (667 x 375), an
iPhone held sideways (844 x 390, notch and home bar) and upright
(390 x 844), and a desktop window (1280 x 800). A game played one way
only lists its own.
- `minTarget` is the smallest tap area in CSS pixels (44 by default). A UI
scaled to the screen is smaller on a small phone: set the game's own rule
as it comes out at the smallest size.
- `allow`, on the route or on a screen, leaves out a finding of `check`
whose element (or the other element of an overlap) matches `selector` or
is inside a match. The summary counts what it left out.
- `--screenshots <dir>` saves each screen and viewport with findings as
`<screen>@<viewport>.png` (and `-longer-text.png`), each finding boxed and
numbered as the report names it (red for errors, amber for warnings).
- It needs what `keyboard` needs, and serves the game the same way. Exit
codes: 0 no error (warnings allowed), 1 an error, a route step that
failed, a screen that never settled or showed nothing to measure, 2 usage
error. `--json` prints `{ ok, url, screens, viewports, findings: [{ check,
severity, screen, viewport, pass, element?, other?, box?, saw, fix,
screenshot? }], outcomes: [{ check, screen, viewport, status, count,
reason? }], allowed }`.

What it cannot see: anything drawn on the canvas other than Touch buttons,
screens the route does not reach, and how Safari or Firefox lay out text.

## Fuzzing

`Fuzz.run` finds a bug before anyone plays into it: seeded random input,
Expand Down
Loading
Loading