wayland: fixes to frame insets (C2b), each with a suite assertion

[foxnne]

Part of fizzyedit/fizzy#280 (step C2b). Run by fizzyedit/sdl_zig#4.

Patch 6 ("Wayland: frame insets", 2d6efde) before it is proposed upstream. A test commit first, so CI can show it failing on fizzy-3.4, then one fix per problem, each described "squash into 6 at the next rebase".

Commits

1. test: the frame-insets suite checks what patch 6 still gets wrong. New FizzyWayland assertions, read from the WAYLAND_DEBUG trace under headless weston (details below). Test files only.
2. A window made again keeps its frame insets (C2b 1). SDL_RecreateWindow calls CreateSDLWindow(_this, window, 0), so the insets were gone after an OpenGL renderer was attached to a shown window, or after a compositor reconnect. The insets asked for are now also kept in the window's own properties under internal names (SDL.internal.wayland.frame_inset.*, as the GPU drivers and renderer keep per-window data), which outlive the window data. No public setter: see "Upstream" below.
3. An opaque window's opaque region is its frame, not its shadow (C2b 2). Was (0, 0, w, h).
4. Clamp a framed window to the toplevel bounds by its frame (C2b 3). configure_bounds bound the window geometry (the frame), but were compared with the surface. Untested: weston 14 advertises xdg_wm_base v5 but never sends configure_bounds (nothing in libweston/desktop/xdg-shell.c does), so under headless weston the clamp never runs. The floating 0x0 configure that reaches it does happen. Checked by reading.
5. The input band round a framed window is a creation property (C2b 4). New SDL_PROP_WINDOW_CREATE_WAYLAND_FRAME_INPUT_MARGIN_NUMBER ("SDL.window.create.wayland.frame_input_margin"), default 8, 0 for the frame alone, each side still at most its inset. It is kept on recreate as the insets are. Documented beside the insets in SDL_video.h. Fizzy keeps the default.
6. Publish a window's frame insets from creation on (C2b 5). Wayland_CreateWindow publishes the insets the window will float with (zero for libdecor). The first configure corrects them if the window comes up maximized, tiled or fullscreen.
7. A framed window shown again is framed again (found while doing this; not in C2b's list). Hiding destroys the xdg_surface and its geometry. Shown again at the same size, nothing looked changed and set_window_geometry was never sent to the new xdg_surface. Hiding now forgets the applied insets, and ApplyFrameInsets skips a window with no xdg_surface.

What the new assertions check

• Every floating check (Floating, Maximized/Fullscreen before and after, Recreated, ShownAgain, input-margin tests): the opaque region is the frame, (24, 6, 346, 254) for 400x300. Maximized or fullscreen: the whole surface.
• wayland_frameInsetsBeforeShow: a window made hidden publishes 24/6/30/40.
• wayland_frameInsetsRecreated: after SDL_CreateRenderer(window, "opengles2") makes the shown window again (flags go from VULKAN to OPENGL), the new objects' geometry, input region and opaque region are the frame's. The trace is read from a mark set before the recreate, because libwayland reuses the ids of destroyed objects.
• wayland_frameInsetsShownAgain: hide, show; the new xdg_surface's geometry is the frame. The window is released from the GPU device while hidden and claimed again after showing. A first version kept it claimed, and the run hung with nothing printed.
• wayland_frameInsetsInputMarginWide (12) and ...None (0): the input region follows the property.

Trace marks now also go to the real stderr, so a test that hangs while tracing shows how far it got.

Verified

• Green, this branch (011a2dc5) via sdl_zig#4's pin: https://github.com/fizzyedit/sdl_zig/actions/runs/38001031572. In test-fizzy-wayland-insets at scale 1 and 2, and in test-fizzy-linux (wayland) (where libdecor is installed), all 9 FizzyWayland tests pass, 0 asserts fail. The run concludes success. The build (master, …) jobs fail as they do on fizzy itself: Zig master, continue-on-error.
• Red, the test commit alone (1eb90b75, fizzy-3.4 + suite) on a throwaway sdl_zig branch, since deleted: https://github.com/fizzyedit/sdl_zig/actions/runs/38000927122. At both scales every new assertion fails, and Popup alone passes:
  • opaque region (0, 0, 400, 300) instead of (24, 6, 346, 254) in every floating check (2);
  • created hidden: all four insets -1 (not published) (5);
  • recreated: no set_window_geometry, input region never set, opaque region the whole surface (1);
  • shown again: no set_window_geometry, input and opaque region never set (the hide/show fix);
  • margin 12: input region (16, 0, 362, 268) instead of (12, 0, 370, 272); margin 0: (16, 0, 362, 268) instead of (24, 6, 346, 254) (4).
• An earlier red run (https://github.com/fizzyedit/sdl_zig/actions/runs/37984303044, on the test commit's first version) showed the same failures for 1, 2 and 5, then hung in wayland_frameInsetsShownAgain with the window still claimed by the GPU device. That is why the test now releases and reclaims the window, and why the jobs have a timeout.
• 3 is untestable under weston, as above. Checked by reading.
• Compiles clean with -Wall -Wextra -Wmissing-prototypes -Wmissing-variable-declarations -Wshadow for x86_64-linux and aarch64-macos (suite), and for Linux through sdl_zig (zig build test-fizzy -Dtarget=x86_64-linux-gnu). The macOS runners skip the suite (Wayland only) and pass.

Upstream

Upstream main now has its own version of this feature: libsdl-org/SDL 5d7167cb "wayland: Add opt-in border inset window properties" (2026-09-24), with cleanups in 86b89cff and dff05080. It isn't in release-3.4.x. There an application opts in with SDL_PROP_WINDOW_CREATE_WAYLAND_ENABLE_INSETS_BOOLEAN and then sets SDL_PROP_WINDOW_WAYLAND_BORDER_INSET_{LEFT,TOP,RIGHT,BOTTOM}_NUMBER on the window. SDL reads them each time it configures, so they are their own setter. Upstream does the window geometry, configure sizes, min/max and popup anchoring, as patch 6 does. Where they differ:

• Upstream has no input region. The whole surface, shadow included, takes input. Patch 6 sets frame plus band.
• Its opaque region is still the whole surface (C2b 2 applies to it), and its bounds clamp compares with the surface (C2b 3 applies to it too).
• Tiled windows keep their insets upstream. Patch 6 zeroes them.
• No "in effect" values are published. The app infers them from the maximized/fullscreen flags.
• enable_insets comes from the creation properties, so upstream loses it on SDL_RecreateWindow the same way (C2b 1). The inset values themselves survive, since window properties outlive the window data.

So this PR adds no setter. Upstream already has the setter shape, settable window properties, and a fork-only one would be a third API to reconcile at the next rebase. The upstream path for patch 6 is to adopt upstream's API when fizzy rebases onto an SDL release that has it, and to propose what it lacks as fixes to upstream's feature: input region plus band (with this margin property), opaque region, bounds clamp, re-framing after hide/show, and keeping enable_insets across a recreate.

The SDL commit needs a permanent tag before sdl_zig's pin on it merges.

🤖 Generated with Claude Code