diff --git a/docs/architecture.md b/docs/architecture.md index 9a29a89..93499f3 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -146,6 +146,14 @@ keeps the profiles' passwords in the KeyStore. | `hiddenInputWanted` | `hiddenInput` | | `commandHistory()`, read at `profileLoaded` and when a UI attaches | `history` | | `commandLineTextRequested` (`printCmdLine()` and the like) | `commandLine` | +| `scriptWindows()`, read at `profileLoaded`, then at each `scriptWindowsChanged` | `windows`, only the windows that changed and those `removed`; all of them, with `full`, to a UI that attaches. A close sends an empty list | +| `windowLines()`, the last 300 rows, at `profileLoaded` and at each `windowTextChanged` | `windowText`; any rows on request with `getWindowText` | +| `setMainWindowSize()`, then `mainConsoleGrid()` and `mainConsoleTextRect()` at `profileLoaded` and at each `mainConsoleAreaChanged` | `setMainWindowSize`, kept and given again to each profile as it loads; `mainArea` | +| `setUserWindowGeometry()`, `closeUserWindow()` | `placeUserWindow`, `closeUserWindow` | +| `submitWindowCommandLine()`; `windowCommandLineTextChanged` | `sendWindowCommandLine`; `windowCommandLine` | +| `labelContent()` of each label, read with `scriptWindows()` | `label` after the `windows`, for each label whose content changed; every label to a UI that attaches | +| `pointer()` | `pointer`, answered only when refused | +| none: the engine reads the file itself, with `QImageReader` or as a `QIcon` | `getImage`, answered by `image` as a PNG | The tail is sent once per pass of the event loop, so a flood sends its lines and a single tail. @@ -195,6 +203,17 @@ Android options (`scripts/build-libmudlet-desktop.sh`) to run it. It checks: - the input line: a game's `WILL ECHO` gives `hiddenInput`, `sendHidden` reaches the game and no frame or log, `dismissHidden`, `history` to a UI that attaches and after a restart, `complete` of a word the game sent and from the history, and `printCmdLine()` as `commandLine`; +- script windows: a Geyser `MiniConsole` as `windows` with its geometry, and its `cecho()` as `windowText` with its + colours; a Geyser `UserWindow` placed by `placeUserWindow`, which raises `sysUserWindowResizeEvent`; the size from + `setMainWindowSize` before the profile loads, and `setBorderLeft()`, as `mainArea`; `enableCommandLine()` with + `printCmdLine()` as `windowCommandLine` and `sendWindowCommandLine` running `setCmdLineAction()`'s function; + `getWindowText`; hiding, `closeUserWindow`, a `full` `windows` and each console's text to a UI that attaches, and + `removed`; +- labels: a Geyser `Label`'s `echo()`, `setStyleSheet()`, `setAccessibleName()`, tool tip and click callback as + `label`, its click callback run by a `pointer` press, and `pointer` refused off the window; a new `echo()` as + `label` alone; `setBackgroundImage()` as the label's `image`, and that file as a PNG from `getImage`, scaled; an + SVG one with `setSvgTint()` and `setSvgRotation()`; a Geyser `Gauge`'s three labels, whose front width is its value + and whose text label is transparent; every label to a UI that attaches; `deleteLabel()` as `removed`; - TLS (see "TLS" below): a self-signed certificate refused, TLS to a port that does not speak it, then the same certificate accepted once the profile accepts self-signed ones. diff --git a/docs/protocol.md b/docs/protocol.md index ba1f394..21bb6d4 100644 --- a/docs/protocol.md +++ b/docs/protocol.md @@ -10,7 +10,7 @@ How the UI process talks to the engine process, as implemented by: the same messages and fields, and that all of them, and the Python client, use the same protocol version. Change this file in the same commit as the code. -The current version is **3**. +The current version is **8**. ## Transport @@ -65,10 +65,17 @@ message, in backquotes, with `?` after an optional one, and the fields of the ob | `saveProfile` | `name`, `host`, `port`, `tls`, `login`, `autoReconnect`, `id`? | Creates the profile `name`, or updates the one of that name, ignoring case; an update keeps the name it was created with. All the fields are needed: `host` not empty, `port` from 1 to 65535, `tls` and `autoReconnect` booleans, `login` a string, which may be empty. A new `name` must be one Mudlet can make a folder of: only the letters A to Z, the digits, spaces and `. _ ( ) - # &`, no space at either end, and neither `.` nor `..` in it. A running profile read its details when it started, so a change takes effect at its next start. | | `deleteProfile` | `name`, `id`? | Deletes the profile, with everything Mudlet keeps for it. Refused while it is running, being opened or closed, or waited for by a restore. | | `openProfile` | `name`, `password`?, `connect`?, `id`? | Starts the profile `name`. One already running is first closed and saved, as Mudlet closes a profile. `password` is for this run only: Mudlet's auto-login sends the profile's `login` and then this password once it connects, and nothing stores it. `connect`, true if absent, connects the profile to its own address. Opening the profile that is running leaves it running, with the password it was started with, and connects it if `connect` and it is not connected. While a profile is being opened, `connect`, `disconnect` and `runLua` are for it. | +| `setMainWindowSize` | `w`, `h` | The main window's size in Mudlet's pixels, whole numbers above 0. The borders scripts set take their room from it, and the main console's font decides the grid in what is left (`mainArea`). libmudlet gives it to the running profile at once (`THeadlessSession::setMainWindowSize()`, which raises `sysWindowResizeEvent`) and keeps it for each profile that loads later, which may still refuse it: `mainArea` then gives 0 for it. | +| `getWindowText` | `window`, `first`, `count`, `id`? | Up to `count` rows, at most 1000, of the console `window` (a mini console, user window or MXP frame; `main` is the main console, as in Mudlet) from row `first`, counted from 0. A negative `first` counts back from the end, so `-count` is the last `count` rows. Answered by `windowText`, or `error` for a name that is no console's. | +| `placeUserWindow` | `name`, `x`, `y`, `w`, `h`, `floating` | Where the UI put the user window `name`, in the main window's pixels, docked or `floating` (`setUserWindowGeometry()`). A new size raises `sysUserWindowResizeEvent`. `w` and `h` above 0. | +| `closeUserWindow` | `name` | The dock's close button: hides the user window, as `hideWindow()` does. | +| `sendWindowCommandLine` | `name`, `text` | Enter on the command line `name`, a console's own from `enableCommandLine()` or one from `createCommandLine()`, holding `text` (`submitWindowCommandLine()`): the line's action, or else the aliases, take each line of it. | +| `pointer` | `window`, `type`, `x`, `y`, `button`?, `buttons`?, `modifiers`?, `wheelY`? | What the UI's pointer did at `x`, `y` (numbers, which may have fractions) in the pixels of the window `window`, from its top left; `main` is the main console's area (`THeadlessSession::pointer()`). `type` is `press`, `release`, `doubleClick`, `move`, `enter`, `leave` or `wheel`. `button` is the one that changed, for a press, release or double-click, and `buttons` those held after it, as Qt's flags: 1 left, 2 right, 4 middle. `modifiers` are Qt's: 0x02000000 Shift, 0x04000000 Control, 0x08000000 Alt, 0x10000000 Meta. `wheelY` is the wheel's turn in eighths of a degree, 120 a notch, positive away from the player. Each of those four is 0 when absent. A label's callback runs as the GUI's would, with the event table it gives (`button` `LeftButton` and the rest, `x`, `y`); a press or release no label takes goes on to the console under it. | +| `getImage` | `path`, `w`?, `h`?, `frame`?, `id`? | The image file at `path`, as a label's `image` or `movie` names it, as a PNG, answered by `image` or `error`. With `w` and `h`, which go together, from 1 to 4096, it is scaled to that size; a path that is no image file is then taken as an icon file, or the name of one in the icon theme, at that size. `frame`, from 0, is that frame of an animated image. Without `w` and `h`, an image larger than 4096 pixels a side is refused, as is a PNG too large for one frame. | Any request may carry `id`. The engine copies it, as it came, into the `re` of every `status` it sends while -handling that request, of the `profiles` or `error` that answers it, and into `pong`. Statuses that come later, such -as `connection` once a connect succeeds, carry no `re`. +handling that request, of the `profiles`, `windowText`, `image` or `error` that answers it, and into `pong`. +Statuses that come later, such as `connection` once a connect succeeds, carry no `re`. Each profile request is answered with `profiles`, or with `error` whose `message` is the request's name, a colon and why it was refused: a bad name or field, a profile that does not exist, or one that is in use. An `openProfile` @@ -78,6 +85,12 @@ exist, as in version 1; after one, it connects the profile opened. The stand-in `profiles.json` beside its socket, has no login, so a password goes nowhere, and connects a `connect` to the address it names whatever profile is open. +The script windows' requests are answered only when refused, with `error` whose `message` is the request's name, a +colon and why; `getWindowText` and `getImage` are always answered. libmudlet refuses all but `setMainWindowSize` and +`getImage` while no profile is open, and `pointer` as `THeadlessSession::pointer()` does: for a window that is not +shown, a point outside it, or another window than the one a held button went down on. The stand-in has no script +windows and refuses all seven. + ### Engine to UI | Message | Fields | Meaning | @@ -96,6 +109,70 @@ it names whatever profile is open. | `hiddenInput` | `wanted` | The game hides what the player types, as for a password (`wanted` true), or has stopped: the UI masks its input line and sends what is typed there with `sendHidden`. libmudlet sends it from `hiddenInputWanted`, which a disconnect or the profile closing also ends, and never while the profile shows passwords as they are typed; the stand-in from the game's telnet `WILL ECHO` and `WONT ECHO`. Sent on each change, and to each UI that attaches. | | `history` | `lines` | The input line's history, newest first, each line once, as Mudlet's up arrow walks it (`commandHistory()`): the lines `send` sent, as their aliases left them, and those the profile's earlier sessions kept, but nothing sent while input was hidden. Sent to each UI that attaches, and by libmudlet when a profile has loaded. In between, the UI adds the lines it sends itself; the next attach corrects what it guessed. The stand-in's is the lines `send` sent since it started. | | `commandLine` | `text` | A script set the whole input line, as `printCmdLine()`, `appendCmdLine()` and `clearCmdLine()` do, or Mudlet emptied it after a `send` (`commandLineTextRequested`): the UI replaces what is on its line. Not kept for a UI that attaches later. Only libmudlet sends it. | +| `windows` | `full`?, `windows`: `name`, `kind`, `parent`, `parentKind`?, `x`, `y`, `w`, `h`, `visible`, `shown`, `stack`, `bg`?, `bgAlpha`?, `fontFamily`?, `fontSize`?, `inDock`, `autoDock`?, `dockArea`?, `dockFloating`?, `dockTitle`?, `scrolledBackTo`?, `columns`?, `rows`?, `scrollBar`?, `timeStamps`?; `removed`?: `name`, `kind` | The windows the profile's scripts made ("Script windows" below). With `full` true, all of them, replacing what the UI has; otherwise those that changed, and in `removed` those gone. | +| `windowText` | `window`, `lineCount`, `first`, `lines`: `text`, `runs`, `prompt`, `wrapped`; `re`? | Rows of a console: it holds `lineCount`, and `lines` starts at row `first`. Each row is a line after the console's wrap; `wrapped` is true for one that carries on from the row above. Sent with the last 300 rows when they may have changed, and for each console to a UI that attaches; with `re`, the answer to `getWindowText`. | +| `mainArea` | `w`, `h`, `columns`, `rows`, `textX`, `textY`, `textW`, `textH` | The main window's area: the size `setMainWindowSize` gave (0 for none), the main console's grid in characters, and the rectangle its text goes in, in Mudlet's pixels, which is the area less the borders scripts set. Sent as a profile loads, on each change and to each UI that attaches. Only libmudlet sends it. | +| `windowCommandLine` | `name`, `text` | A script set the command line `name`, as `printCmdLine()` and the rest do given its name, or Mudlet emptied it after a command: the UI replaces what is on it. Not kept for a UI that attaches later. Only libmudlet sends it. | +| `label` | `name`, `text`, `image`, `svg`, `svgTint`?, `svgTintAlpha`?, `svgRotation`?, `svgShearX`?, `svgShearY`?, `movie`, `moviePaused`, `movieFrame`, `movieSpeed`, `movieW`?, `movieH`?, `toolTip`, `clickThrough`, `accessibleName`, `accessibleButton`, `styleSheet` | What the label `name` shows ("Labels" below), after the `windows` that has it, each time it changes, and for each label to a UI that attaches. Only libmudlet sends it. | +| `image` | `path`, `png`, `w`, `h`, `re`? | The answer to `getImage`: `png` is the image as a PNG, in base64, of `w` by `h` pixels. | + +### Script windows + +What `windows` and `windowText` carry, from libmudlet's `THeadlessSession::scriptWindows()` and `windowLines()`: + +- A window is known by its `name` and `kind`, as a console's own command line has its console's name. `kind` is + upstream's: `miniConsole`, `userWindow`, `label`, `commandLine`, `textBox`, `scrollBox`, `mapper` or `mxpFrame`. + The app leaves out a kind it does not know. +- They come parents first, each parent's windows from the lowest up, those of the main window's area before the + docks. `stack` counts from 0 among a parent's windows; a higher one is drawn over a lower one. +- `parent` is the user window or scroll box a window is in, the console whose command line it is, or the MXP frame it + was opened in, with its kind in `parentKind`. It is empty, with no `parentKind`, for the main window's area, a dock + and an external MXP frame. +- `x`, `y`, `w` and `h` are in the parent's pixels, a dock's in the main window's. They are all 0 for a console's own + command line, which goes along the console's bottom. +- `visible` is the window's own flag, as `hideWindow()` and `showWindow()` set it; `shown` is whether it and every + window it is in are visible. +- `bg` is a console's background colour, or a label's, as a run's colour. `fontFamily` and `fontSize`, in points, are + what a console, label or text box draws its text in. +- `inDock` is true for a dock beside the main window's area rather than a window in it. A user window has `autoDock` + (whether the player may dock it, or only float it), `dockArea` (`left`, `right`, `top` or `bottom`), + `dockFloating` and `dockTitle`. +- A console (a mini console, user window or MXP frame) has `columns` and `rows`, the grid of text it has room for, + `scrollBar` and `timeStamps`, and `scrolledBackTo`, the row it is scrolled back to, while it does not follow new + lines. +- The runs of a console's rows are a `line`'s, with two differences: `fg` is always a colour, never -1, as the UI is + not told a console's own; a `bg` of -1 is the window's `bg`. Their `link` ids are the window's own. + +The engine keeps the list. Between attaches it sends one `windows` for each turn of its event loop that changed +some, with only what changed. libmudlet sends every window of a profile as it loads, and `removed` for all of them +as it closes. `windowText` is not kept: each console's last 300 rows come again on attach. + +A label's `bg` may be partly or wholly transparent: `bgAlpha`, from 0 to 254, is its alpha, absent for an opaque +one. A `Geyser.Gauge` is three labels in the gauge's parent, `_back`, `_front` and `_text`, drawn +in that order: the front's width (or height, for a vertical gauge) against the back's is the gauge's value, and the +text label, transparent, holds the text. + +### Labels + +What `label` carries, from libmudlet's `THeadlessSession::labelContent()`, which scripts set with `echo()`, +`setLabelStyleSheet()`, `setBackgroundImage()` and the rest: + +- `text` is the rich text `echo()` or `setLabelText()` gave it, which Qt's rich text draws; it is empty while an + image or movie shows in its place. +- `image` is the file `setBackgroundImage()` gave it while that shows: a raster image, which text or a movie takes + down, or with `svg` true an SVG drawn under the text. An SVG may have `svgTint`, a colour as a run's with its alpha + in `svgTintAlpha` when below 255, and `svgRotation` in degrees, `svgShearX` and `svgShearY`, each absent for 0. +- `movie` is the file `setMovie()` gave it, shown in place of the text. `moviePaused` is true once `pauseMovie()` + paused it or it played to its end, `movieFrame` the frame it then shows (-1 while it plays), `movieSpeed` its + speed in percent, and `movieW` and `movieH` the size `scaleMovie()` gave it, absent for the file's own. +- `toolTip` is the label's tool tip. With `clickThrough` true the pointer passes through it to what is under it. +- `accessibleName` is what a screen reader says for it, and `accessibleButton` is true once it has a click or release + callback, so a screen reader is told it is a button. +- `styleSheet` is what `setLabelStyleSheet()` gave it, which the engine does not interpret; drawing it is the UI's. + +The files are the engine's: the UI asks for one with `getImage`. A label's cursor is not sent. The engine keeps each +label's content and sends a `label` only when it differs from what it last sent; a label that goes is in the +`windows` message's `removed`, with no `label`. ### Status keys @@ -167,7 +244,8 @@ A colour (`fg`, `bg`) is one of: | fast blink | 128 | | concealed | 256 | -`link` is libmudlet's id of the link the run is part of, 0 for none. Nothing can follow a link over the protocol yet. +`link` is libmudlet's id of the link the run is part of, 0 for none; in a script window's rows, among that window's +links. Nothing can follow a link over the protocol yet. ## Attaching: hello, instance and lastSeq @@ -175,7 +253,9 @@ The engine keeps the last 2000 lines in a ring buffer, its current tail, and the a UI says `hello`, the engine sends, in this order: 1. its own `hello`; -2. every status, each as a `status` without `re`, in no particular order, then `hiddenInput` and `history`; +2. every status, each as a `status` without `re`, in no particular order, then `hiddenInput` and `history`, then + `mainArea` if the engine has sent one, `windows` with `full` true, a `label` for each label, and a `windowText` of + each console's last 300 rows; 3. `clear`, if the console was emptied after the line the UI says it has, or ever, for a UI that has none; 4. `gap`, if lines the UI does not have already fell out of the ring; 5. the lines in the ring after the UI's, each with `replay` true. That is all of them unless the UI's `instance` @@ -200,23 +280,29 @@ The engine never waits for the UI, as Android can freeze a backgrounded UI proce - When more than 4 MiB is queued unsent, the UI is lagging. The engine stops writing lines, tails, clears, notifies and statuses to it, but keeps them in the ring and its status table. -- Once the queue drains below 256 KiB, the engine sends every status, `hiddenInput` and `history` again, then the lines the UI has not had - (with `clear` and `gap` as on attach, but without `replay`), then the tail. -- Answers are always written, even to a lagging UI: `pong`, `profiles`, `error`, and a `status` that carries `re`. +- Once the queue drains below 256 KiB, the engine sends every status, `hiddenInput`, `history` and the script + windows' state again, as on attach, then the lines the UI has not had (with `clear` and `gap` as on attach, but + without `replay`), then the tail. +- Answers are always written, even to a lagging UI: `pong`, `profiles`, `error`, the `windowText` that answers + `getWindowText`, `image`, and a `status` that carries `re`. A `notify` sent while the UI lags is not sent again, but the engine has already posted the notification on Android; -nor is a `commandLine`. +nor is a `commandLine` or a `windowCommandLine`. `synced.seq` is the newest line when greeting even if lag cut the replay short; the rest follows once the UI drains, and `EngineLink` then tracks `lastSeq` from those lines. ## Versioning and compatibility -As implemented in version 3, which added the input line's requests (`sendHidden`, `dismissHidden`, `complete`) and -messages (`hiddenInput`, `history`, `commandLine`); version 2 added the profile requests and `profiles`: +As implemented in version 8, which added the labels' messages (`label`, `image`, and `bgAlpha` in `windows`) and +requests (`pointer`, `getImage`). Version 7 added the script windows' requests (`setMainWindowSize`, +`getWindowText`, `placeUserWindow`, `closeUserWindow`, `sendWindowCommandLine`) and messages (`windows`, +`windowText`, `mainArea`, `windowCommandLine`). Versions 4 to 6 are kept for changes still in review, so no engine speaks them. Version 3 added +the input line's requests (`sendHidden`, `dismissHidden`, `complete`) and messages (`hiddenInput`, `history`, +`commandLine`); version 2 added the profile requests and `profiles`: - The UI sends its version in `hello` `v`, and the engine sends its own in its `hello`. - The engine accepts only a `hello` whose `v` equals its own. Otherwise it sends - `error` (`expected hello with v=3`) and closes the connection. A first frame that is not `hello` is refused the + `error` (`expected hello with v=8`) and closes the connection. A first frame that is not `hello` is refused the same way. - The app does not check the engine's `v`. A refused `hello` reaches `EngineLink` as an `error` where it expects `hello`, so it reports the link lost and keeps retrying. diff --git a/docs/roadmap.md b/docs/roadmap.md index 1490a1e..85f2ab4 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -27,11 +27,12 @@ repository uses. What Phase E needs, and where it stands: | Lines with colours and styles | `lineShown`, `localTextStyled`, `mainConsoleCleared` | `line`, `tail`, `clear` | | A line changed after it was shown | None: `lineShown` sends each line once | No | | Links (MXP, OSC 8, `echoLink()`) | Ids in runs; `link()`, `followLink()`, `linksChanged` | Ids in runs only | -| Screen size for NAWS | `setScreenSize()`, `setMainWindowSize()` | No: the game is told 90 by 25 | +| Screen size for NAWS | `setScreenSize()`, `setMainWindowSize()` | `setMainWindowSize` (version 7), which regrids the main console the game is told of; the app does not send it yet, so the game is told 90 by 25 | | Default colours, font, wrap | `consoleColors*`, `consoleFont()`, `consoleWrap()` and their signals | No | | Command history, completion | `commandHistory()`, `completeCommandLine()`, `completeCommandLineFromHistory()` | `history`, `complete` (version 3) | | Password prompts | `hiddenInputWanted`, `submitHiddenInput()` | `hiddenInput`, `sendHidden`, `dismissHidden` | | Scripts setting the input line | `commandLineTextRequested`, `setCommandLineText()` | `commandLine` | +| Script windows (Phase F) | `scriptWindows()`, `windowLines()`, `mainConsoleGrid()`, `setUserWindowGeometry()`, `closeUserWindow()`, `submitWindowCommandLine()` and their signals; `labelContent()`, `pointer()` | `windows`, `windowText`, `mainArea`, `windowCommandLine`, `getWindowText`, `placeUserWindow`, `closeUserWindow`, `sendWindowCommandLine` (version 7); `label`, `image`, `pointer`, `getImage` (version 8); the app draws none of it yet | | Screen reader announcements | `announcementRequested` (Lua `announce()`) | No | | Backgrounded app | `setApplicationActive()`, `setWindowMinimized()`, `saveAsync()`, `attentionRequested` | No | | Notifications | `notificationRequested` | `notify`, posted by the engine | diff --git a/native/api/IEngine.h b/native/api/IEngine.h index 0b5f1d0..e04f798 100644 --- a/native/api/IEngine.h +++ b/native/api/IEngine.h @@ -1,11 +1,44 @@ #pragma once +#include "IFrontend.h" #include "Profile.h" +#include #include +#include +#include +#include #include #include +#include + +// The UI's pointer on a script window, as THeadlessSession::WindowPointerEvent describes it +struct PointerEvent +{ + // As upstream names them on the wire + enum class Type { press, release, doubleClick, move, enter, leave, wheel }; + Type type = Type::move; + // In the window's pixels, from its top left corner + QPointF position; + // Qt's values: the button that changed, for a press, release or double-click, and those held after it + int button = 0; + int buttons = 0; + // Qt::KeyboardModifiers + int modifiers = 0; + // The wheel's turn in eighths of a degree, as QWheelEvent::angleDelta().y() + int wheelY = 0; +}; + +// An image as PNG, and its size in pixels +struct PngImage +{ + // The largest side IEngine::image() scales to or reads, so that a huge file is not decoded whole for a UI that + // could not take it in one frame + static constexpr int maxSide = 4096; + QByteArray png; + QSize size; +}; // What the host asks of an engine: the stand-in, or libmudlet behind MudletEngine class IEngine @@ -33,4 +66,21 @@ class IEngine virtual QString deleteProfile(const QString& name) = 0; // Starts that profile, first closing and saving another that runs. The password is for this run only. virtual QString openProfile(const QString& name, const QString& password, bool connect) = 0; + + // The script windows'. The main window's size in pixels, kept for the next profile too. + virtual QString setMainWindowSize(int width, int height) = 0; + // Up to count rows of a console from row first, which counts back from the end below 0; none for a name that is + // not a console's, or with no profile open + virtual std::optional windowText(const QString& window, int first, int count) const = 0; + // Where the UI put a user window, in the main window's pixels, docked or floating + virtual QString placeUserWindow(const QString& name, const QRect& geometry, bool floating) = 0; + // The dock's close button: hides the user window + virtual QString closeUserWindow(const QString& name) = 0; + // Enter on a window's command line holding text + virtual QString sendWindowCommandLine(const QString& name, const QString& text) = 0; + // What the UI's pointer did on a window + virtual QString pointer(const QString& window, const PointerEvent& event) = 0; + // The image file, or icon, at path as PNG, scaled to size unless it is empty, or that frame of an animated one; + // else why there is none. Size is at most PngImage::maxSide a side. + virtual std::variant image(const QString& path, const QSize& size, std::optional frame) const = 0; }; diff --git a/native/api/IFrontend.h b/native/api/IFrontend.h index eb11357..52b0b32 100644 --- a/native/api/IFrontend.h +++ b/native/api/IFrontend.h @@ -1,9 +1,13 @@ #pragma once #include +#include +#include #include #include +#include + struct ColorRun { int start = 0; @@ -19,6 +23,116 @@ struct ColorRun bool operator==(const ColorRun&) const = default; }; +// A window a profile's scripts made, as THeadlessSession::ScriptWindow describes it, for the UI to draw +struct ScriptWindowInfo +{ + QString name; + // Upstream's enum name: miniConsole, userWindow, label, commandLine, textBox, scrollBox, mapper or mxpFrame + QString kind; + // The window it is in, and that window's kind; both empty for the main window's area, a dock or an external frame + QString parent; + QString parentKind; + // In the parent's pixels; empty for a console's own command line + QRect geometry; + bool visible = false; + bool shown = false; + int stackOrder = 0; + // As ColorRun's, -1 for none; its alpha from 0 to 255, which a label's may have below 255 + int background = -1; + int backgroundAlpha = 255; + // Empty and 0 for none + QString fontFamily; + double fontSize = 0; + bool inDock = false; + // A user window's + struct Dock + { + bool autoDock = true; + // left, right, top or bottom + QString area; + bool floating = false; + QString title; + bool operator==(const Dock&) const = default; + }; + std::optional dock; + // A mini console's, user window's or MXP frame's + struct Console + { + // The row it is scrolled back to, none while it follows new lines + std::optional scrolledBackTo; + int columns = 0; + int rows = 0; + bool scrollBar = false; + bool timeStamps = false; + bool operator==(const Console&) const = default; + }; + std::optional console; + bool operator==(const ScriptWindowInfo&) const = default; +}; + +// Rows of a mini console, user window or MXP frame, after its wrap +struct WindowText +{ + struct Row + { + QString text; + // A fg is always a colour, as the UI does not know the console's own; a bg of -1 is the window's + QList runs; + bool prompt = false; + // Carries on from the row above + bool wrapped = false; + }; + // How many rows the console holds, and the one rows starts at, from 0 + int lineCount = 0; + int first = 0; + QList rows; + // The last rows a window's text is sent with unasked + static constexpr int recentRows = 300; +}; + +// What a label shows, as THeadlessSession::LabelContent describes it: its geometry, background and font are its +// ScriptWindowInfo's +struct LabelInfo +{ + QString name; + // Qt rich text; empty while an image or movie shows + QString text; + // The file of a raster image while it shows, or of an SVG one, drawn under the text with the tint, rotation and shear + QString image; + bool svg = false; + // As ColorRun's, -1 for none, with its alpha from 0 to 255 + int svgTint = -1; + int svgTintAlpha = 255; + double svgRotation = 0; + double svgShearX = 0; + double svgShearY = 0; + // The file of a movie while it plays or is paused; the frame it shows while paused, -1 while it plays + QString movie; + bool moviePaused = false; + int movieFrame = -1; + // In percent of the file's own speed + int movieSpeed = 100; + // Invalid for the file's own + QSize movieSize; + QString toolTip; + bool clickThrough = false; + QString accessibleName; + bool accessibleButton = false; + // The label's own, which Mudlet does not interpret + QString styleSheet; + bool operator==(const LabelInfo&) const = default; +}; + +// The main window's area: the size the UI gave (empty for none), the main console's grid in characters, and where its +// text goes in pixels, which is the area less the borders scripts set +struct MainArea +{ + QSize size; + QSize grid; + QRect textRect; + bool operator==(const MainArea&) const = default; +}; + // Everything the engine tells its host. Every method must be implementable // over a byte stream, so arguments are plain values only. class IFrontend @@ -38,4 +152,13 @@ class IFrontend virtual void history(const QStringList& lines) = 0; // A script set the whole input line virtual void commandLine(const QString& text) = 0; + // Every window the profile's scripts made, parents first, and what each of its labels shows: whole each time, + // empty with no profile open + virtual void windows(const QList& windows, const QList& labels) = 0; + // A console's last WindowText::recentRows rows, as they may have changed; IEngine::windowText() is read at other times + virtual void windowText(const QString& window, const WindowText& text) = 0; + // Sent when it changes + virtual void mainArea(const MainArea& area) = 0; + // A script set a window's command line, or Mudlet emptied it after a command + virtual void windowCommandLine(const QString& name, const QString& text) = 0; }; diff --git a/native/enginehost/MudletEngine.cpp b/native/enginehost/MudletEngine.cpp index c600de7..bfefef0 100644 --- a/native/enginehost/MudletEngine.cpp +++ b/native/enginehost/MudletEngine.cpp @@ -2,6 +2,7 @@ #include "THeadlessSession.h" +#include #include #include #include @@ -13,10 +14,15 @@ #include #include #include +#include +#include +#include +#include #include namespace { +using ScriptWindow = THeadlessSession::ScriptWindow; using TextRun = THeadlessSession::TextRun; using TextStyle = THeadlessSession::TextStyle; @@ -52,12 +58,14 @@ int attributeBits(THeadlessSession::TextStyles styles) return attributes; } -// Runs that are the profile's own colours with no style or link are left out, as the stand-in leaves them -QList colorRuns(const QList& runs) +// Runs that are the profile's own colours with no style or link are left out, as the stand-in leaves them. +// exactForeground: the default foreground as its colour too, as for a script window, whose own the UI does not know. +QList colorRuns(const QList& runs, bool exactForeground = false) { QList out; for (const TextRun& run : runs) { - const ColorRun converted{run.start, run.length, colourCode(run.foreground, run.defaultForeground), colourCode(run.background, run.defaultBackground), attributeBits(run.styles), run.link}; + const ColorRun converted{ + run.start, run.length, colourCode(run.foreground, run.defaultForeground && !exactForeground), colourCode(run.background, run.defaultBackground), attributeBits(run.styles), run.link}; if (converted.length > 0 && (converted.fg != -1 || converted.bg != -1 || converted.attributes != 0 || converted.link != 0)) { out.append(converted); } @@ -81,6 +89,107 @@ QList sliceRuns(const QList& runs, int from, int to) return out; } +QString kindName(ScriptWindow::Kind kind) +{ + switch (kind) { + case ScriptWindow::Kind::miniConsole: + return QStringLiteral("miniConsole"); + case ScriptWindow::Kind::userWindow: + return QStringLiteral("userWindow"); + case ScriptWindow::Kind::label: + return QStringLiteral("label"); + case ScriptWindow::Kind::commandLine: + return QStringLiteral("commandLine"); + case ScriptWindow::Kind::textBox: + return QStringLiteral("textBox"); + case ScriptWindow::Kind::scrollBox: + return QStringLiteral("scrollBox"); + case ScriptWindow::Kind::mapper: + return QStringLiteral("mapper"); + case ScriptWindow::Kind::mxpFrame: + return QStringLiteral("mxpFrame"); + } + return {}; +} + +QString dockAreaName(Qt::DockWidgetArea area) +{ + switch (area) { + case Qt::LeftDockWidgetArea: + return QStringLiteral("left"); + case Qt::TopDockWidgetArea: + return QStringLiteral("top"); + case Qt::BottomDockWidgetArea: + return QStringLiteral("bottom"); + default: + return QStringLiteral("right"); + } +} + +bool isConsole(ScriptWindow::Kind kind) +{ + return kind == ScriptWindow::Kind::miniConsole || kind == ScriptWindow::Kind::userWindow || kind == ScriptWindow::Kind::mxpFrame; +} + +ScriptWindowInfo windowInfo(const ScriptWindow& window) +{ + ScriptWindowInfo info; + info.name = window.name; + info.kind = kindName(window.kind); + info.parent = window.parent; + if (!window.parent.isEmpty()) { + info.parentKind = kindName(window.parentKind); + } + info.geometry = window.geometry; + info.visible = window.visible; + info.shown = window.shown; + info.stackOrder = window.stackOrder; + info.background = colourCode(window.background, false); + if (info.background >= 0) { + info.backgroundAlpha = window.background.alpha(); + } + // The other kinds have a default QFont, which draws nothing + if (isConsole(window.kind) || window.kind == ScriptWindow::Kind::label || window.kind == ScriptWindow::Kind::textBox) { + info.fontFamily = window.font.family(); + info.fontSize = std::max(window.font.pointSizeF(), 0.0); + } + info.inDock = window.inDock; + if (window.dock) { + info.dock = ScriptWindowInfo::Dock{window.dock->autoDock, dockAreaName(window.dock->area), window.dock->floating, window.dock->title}; + } + if (isConsole(window.kind)) { + info.console = ScriptWindowInfo::Console{window.scrolledBackTo, window.grid.width(), window.grid.height(), window.scrollBar, window.timeStamps}; + } + return info; +} + +LabelInfo labelInfo(const QString& name, const THeadlessSession::LabelContent& content) +{ + LabelInfo info; + info.name = name; + info.text = content.text; + info.image = content.image; + info.svg = content.svg; + info.svgTint = colourCode(content.svgTint, false); + if (info.svgTint >= 0) { + info.svgTintAlpha = content.svgTint.alpha(); + } + info.svgRotation = content.svgRotation; + info.svgShearX = content.svgShearX; + info.svgShearY = content.svgShearY; + info.movie = content.movie; + info.moviePaused = content.moviePaused; + info.movieFrame = content.movieFrame; + info.movieSpeed = content.movieSpeed; + info.movieSize = content.movieSize; + info.toolTip = content.toolTip; + info.clickThrough = content.clickThrough; + info.accessibleName = content.accessibleName; + info.accessibleButton = content.accessibleButton; + info.styleSheet = content.styleSheet; + return info; +} + QString settleConfigDir() { // As the GUI's, as the session reads Mudlet.ini and the keychain under them, and AppDataLocation follows them @@ -221,6 +330,14 @@ void MudletEngine::connectSession() mFrontend->status(QStringLiteral("profile"), QStringLiteral("%1 loaded").arg(profile)); mFrontend->status(QStringLiteral("profileLoadMs"), QString::number(mStartClock.elapsed())); mFrontend->history(mSession->commandHistory()); + mLoaded = true; + if (!mMainWindowSize.isEmpty() && !mSession->setMainWindowSize(mMainWindowSize)) { + qWarning("MudletEngine: the main window's size %dx%d was refused: %s", mMainWindowSize.width(), mMainWindowSize.height(), qPrintable(mSession->errorString())); + // So mainArea does not give it as the profile's + mMainWindowSize = {0, 0}; + } + pushMainArea(); + pushWindows(true); }); connect(session, &THeadlessSession::hiddenInputWanted, this, [this](const QString&, bool wanted) { mFrontend->hiddenInput(wanted); @@ -228,6 +345,25 @@ void MudletEngine::connectSession() connect(session, &THeadlessSession::commandLineTextRequested, this, [this](const QString&, const QString& text) { mFrontend->commandLine(text); }); + // The script windows' signals come from the event loop, so these may call the session + connect(session, &THeadlessSession::scriptWindowsChanged, this, [this] { + pushWindows(false); + }); + connect(session, &THeadlessSession::windowTextChanged, this, [this](const QString&, const QStringList& windows) { + for (const QString& window : windows) { + pushWindowText(window); + } + }); + connect(session, &THeadlessSession::windowCommandLineTextChanged, this, [this](const QString&, const QStringList& commandLines) { + for (const QString& name : commandLines) { + if (const std::optional text = mSession->windowCommandLineText(name)) { + mFrontend->windowCommandLine(name, *text); + } + } + }); + connect(session, &THeadlessSession::mainConsoleAreaChanged, this, [this](const QString&, const QSize& grid, const QRect& textRect) { + mFrontend->mainArea({mMainWindowSize, grid, textRect}); + }); connect(session, &THeadlessSession::connecting, this, [this] { mFrontend->status(QStringLiteral("connection"), QStringLiteral("connecting")); }); @@ -275,6 +411,7 @@ void MudletEngine::connectSession() mKept = {}; persist(); mRestoring = false; + onProfileClosed(); }); } @@ -391,6 +528,7 @@ void MudletEngine::start(const QString& host, quint16 port, bool connect, std::o mKept.tls = tls; persist(); mState = State::starting; + mLoaded = false; mFrontend->status(QStringLiteral("profile"), QStringLiteral("%1 starting").arg(mProfile)); mStartClock.start(); mSession->startAsync(options); @@ -700,6 +838,7 @@ void MudletEngine::onShutdownFinished(bool saved, const QString& error) return; } mState = State::idle; + onProfileClosed(); if (saved) { mFrontend->status(QStringLiteral("profile"), QStringLiteral("%1 closed").arg(mProfile)); } else { @@ -731,6 +870,164 @@ void MudletEngine::keepPending() persist(); } +void MudletEngine::onProfileClosed() +{ + mLoaded = false; + mFrontend->windows({}, {}); +} + +void MudletEngine::pushWindows(bool withText) +{ + QList windows; + QList labels; + for (const ScriptWindow& window : mSession->scriptWindows()) { + windows.append(windowInfo(window)); + if (window.kind != ScriptWindow::Kind::label) { + continue; + } + if (const std::optional content = mSession->labelContent(window.name)) { + labels.append(labelInfo(window.name, *content)); + } + } + mFrontend->windows(windows, labels); + if (!withText) { + return; + } + for (const ScriptWindowInfo& window : std::as_const(windows)) { + if (window.console) { + pushWindowText(window.name); + } + } +} + +void MudletEngine::pushWindowText(const QString& window) +{ + if (const std::optional text = windowText(window, -WindowText::recentRows, WindowText::recentRows)) { + mFrontend->windowText(window, *text); + } +} + +// Also when only the size the UI gave changed, which mainConsoleAreaChanged does not tell +void MudletEngine::pushMainArea() +{ + mFrontend->mainArea({mMainWindowSize, mSession->mainConsoleGrid(), mSession->mainConsoleTextRect()}); +} + +QString MudletEngine::setMainWindowSize(int width, int height) +{ + const QSize before = std::exchange(mMainWindowSize, QSize(width, height)); + // Otherwise taken as the profile loads + if (!profileOpen()) { + return {}; + } + if (!mSession->setMainWindowSize(mMainWindowSize)) { + mMainWindowSize = before; + return mSession->errorString(); + } + pushMainArea(); + return {}; +} + +std::optional MudletEngine::windowText(const QString& window, int first, int count) const +{ + const std::optional lines = mSession->windowLines(window, first, count); + if (!lines) { + return std::nullopt; + } + WindowText text{lines->lineCount, lines->first, {}}; + for (const THeadlessSession::WindowLine& line : lines->lines) { + text.rows.append({line.text, colorRuns(line.runs, true), line.isPrompt, line.wrapped}); + } + return text; +} + +QString MudletEngine::placeUserWindow(const QString& name, const QRect& geometry, bool floating) +{ + if (!profileOpen()) { + return QStringLiteral("no profile running"); + } + return mSession->setUserWindowGeometry(name, geometry, floating) ? QString() : mSession->errorString(); +} + +QString MudletEngine::closeUserWindow(const QString& name) +{ + if (!profileOpen()) { + return QStringLiteral("no profile running"); + } + return mSession->closeUserWindow(name) ? QString() : mSession->errorString(); +} + +QString MudletEngine::sendWindowCommandLine(const QString& name, const QString& text) +{ + if (!profileOpen()) { + return QStringLiteral("no profile running"); + } + return mSession->submitWindowCommandLine(name, text) ? QString() : mSession->errorString(); +} + +QString MudletEngine::pointer(const QString& window, const PointerEvent& event) +{ + if (!profileOpen()) { + return QStringLiteral("no profile running"); + } + static constexpr std::pair types[] = { + {PointerEvent::Type::press, THeadlessSession::WindowPointerEvent::Type::press}, + {PointerEvent::Type::release, THeadlessSession::WindowPointerEvent::Type::release}, + {PointerEvent::Type::doubleClick, THeadlessSession::WindowPointerEvent::Type::doubleClick}, + {PointerEvent::Type::move, THeadlessSession::WindowPointerEvent::Type::move}, + {PointerEvent::Type::enter, THeadlessSession::WindowPointerEvent::Type::enter}, + {PointerEvent::Type::leave, THeadlessSession::WindowPointerEvent::Type::leave}, + {PointerEvent::Type::wheel, THeadlessSession::WindowPointerEvent::Type::wheel}, + }; + THeadlessSession::WindowPointerEvent sessionEvent; + for (const auto& [ours, theirs] : types) { + if (ours == event.type) { + sessionEvent.type = theirs; + break; + } + } + sessionEvent.position = event.position; + sessionEvent.button = Qt::MouseButton(event.button); + sessionEvent.buttons = Qt::MouseButtons::fromInt(event.buttons); + sessionEvent.modifiers = Qt::KeyboardModifiers::fromInt(event.modifiers); + sessionEvent.angleDelta = QPoint(0, event.wheelY); + return mSession->pointer(window, sessionEvent) ? QString() : mSession->errorString(); +} + +std::variant MudletEngine::image(const QString& path, const QSize& size, std::optional frame) const +{ + QImage image; + QImageReader reader(path); + if (reader.canRead()) { + const QSize own = reader.size(); + if (!size.isEmpty()) { + reader.setScaledSize(size); + } else if (own.width() > PngImage::maxSide || own.height() > PngImage::maxSide) { + return QStringLiteral("%1 is larger than %2 pixels a side: ask for a size").arg(path).arg(PngImage::maxSide); + } + // Read up to it, as QMovie reaches a frame, since few formats jump to one + for (int read = 0; read <= frame.value_or(0); ++read) { + if (!reader.read(&image)) { + return read > 0 ? QStringLiteral("no frame %1 in %2").arg(*frame).arg(path) : QStringLiteral("%1: %2").arg(path, reader.errorString()); + } + } + } else if (!size.isEmpty() && !frame) { + // An icon file, or a name in the icon theme + const QIcon icon = QFileInfo::exists(path) ? QIcon(path) : QIcon::fromTheme(path); + image = icon.pixmap(size).toImage(); + } + if (image.isNull()) { + return QStringLiteral("%1: %2").arg(path, reader.errorString()); + } + QByteArray png; + QBuffer buffer(&png); + buffer.open(QIODevice::WriteOnly); + if (!image.save(&buffer, "PNG")) { + return QStringLiteral("%1 could not be made a PNG").arg(path); + } + return PngImage{png, image.size()}; +} + void MudletEngine::setViewWidthPx(int px) { const int columns = mAdvancePx > 0 ? px / mAdvancePx : 0; diff --git a/native/enginehost/MudletEngine.h b/native/enginehost/MudletEngine.h index 0ba9584..7c192eb 100644 --- a/native/enginehost/MudletEngine.h +++ b/native/enginehost/MudletEngine.h @@ -8,6 +8,7 @@ #include #include #include +#include #include #include @@ -40,6 +41,13 @@ class MudletEngine : public QObject, public IEngine QString saveProfile(const Profile& profile) override; QString deleteProfile(const QString& name) override; QString openProfile(const QString& name, const QString& password, bool connect) override; + QString setMainWindowSize(int width, int height) override; + std::optional windowText(const QString& window, int first, int count) const override; + QString placeUserWindow(const QString& name, const QRect& geometry, bool floating) override; + QString closeUserWindow(const QString& name) override; + QString sendWindowCommandLine(const QString& name, const QString& text) override; + QString pointer(const QString& window, const PointerEvent& event) override; + std::variant image(const QString& path, const QSize& size, std::optional frame) const override; private: // stopping: the profile closes, for the one in mPendingOpen to start @@ -93,6 +101,13 @@ class MudletEngine : public QObject, public IEngine void onHeartbeat(); void flushCounters(); void markCountersDirty(); + // Loaded, and not closing: the script windows' calls reach it + bool profileOpen() const { return mLoaded && mState != State::stopping; } + void onProfileClosed(); + // The script windows, and with withText each console's last rows + void pushWindows(bool withText); + void pushWindowText(const QString& window); + void pushMainArea(); void onLineShown(const QString& text, bool isPrompt, const QList& runs, bool continuesLine); void onLocalText(const QString& text, const QList& runs, bool continuesLine); @@ -115,6 +130,10 @@ class MudletEngine : public QObject, public IEngine // What mKept was before a switch began, for a close that is refused SessionStore::Session mKeptBeforeSwitch; State mState = State::idle; + // The profile has loaded, since its start + bool mLoaded = false; + // The main window's size the UI gave, which each profile takes as it loads; empty for none, or once a profile refused it + QSize mMainWindowSize{0, 0}; // The TLS choice the profile was started with, unset when that start made none std::optional mProfileTls; // Lua (connects included) asked for while the profile starts, run once it has diff --git a/native/enginehost/SocketBridge.cpp b/native/enginehost/SocketBridge.cpp index cdb49df..2e86017 100644 --- a/native/enginehost/SocketBridge.cpp +++ b/native/enginehost/SocketBridge.cpp @@ -10,7 +10,11 @@ #include #include +#include #include +#include +#include +#include #include #include @@ -19,6 +23,20 @@ #include "AndroidNotifier.h" #endif +namespace { + +// A JSON number that is a whole number an int holds +std::optional wholeNumber(const QJsonValue& value) +{ + const double number = value.toDouble(); + if (!value.isDouble() || number != std::floor(number) || std::abs(number) > std::numeric_limits::max()) { + return std::nullopt; + } + return int(number); +} + +} // namespace + SocketBridge::SocketBridge(QObject* parent) : QObject(parent) , mServer(new QLocalServer(this)) @@ -152,6 +170,28 @@ void SocketBridge::sendState() if (mEngine) { sendFrame(historyMessage(mEngine->history())); } + if (mMainArea) { + sendFrame(mainAreaMessage(*mMainArea)); + } + QJsonArray windows; + for (const ScriptWindowInfo& window : std::as_const(mWindows)) { + windows.append(windowJson(window)); + } + sendFrame({{QStringLiteral("t"), QStringLiteral("windows")}, {QStringLiteral("full"), true}, {QStringLiteral("windows"), windows}}); + for (const LabelInfo& label : std::as_const(mLabels)) { + sendFrame(labelMessage(label)); + } + if (!mEngine) { + return; + } + for (const ScriptWindowInfo& window : std::as_const(mWindows)) { + if (!window.console) { + continue; + } + if (const std::optional text = mEngine->windowText(window.name, -WindowText::recentRows, WindowText::recentRows)) { + sendFrame(windowTextMessage(window.name, *text)); + } + } } void SocketBridge::sendLinesAfter(qint64 seq, bool replay) @@ -232,6 +272,20 @@ void SocketBridge::handleMessage(const QJsonObject& message) answerProfiles(type, mEngine->deleteProfile(message.value(QLatin1String("name")).toString())); } else if (type == QLatin1String("openProfile")) { answerProfiles(type, openProfile(message)); + } else if (type == QLatin1String("setMainWindowSize")) { + refuse(type, setMainWindowSize(message)); + } else if (type == QLatin1String("getWindowText")) { + getWindowText(message); + } else if (type == QLatin1String("placeUserWindow")) { + refuse(type, placeUserWindow(message)); + } else if (type == QLatin1String("closeUserWindow")) { + refuse(type, mEngine->closeUserWindow(message.value(QLatin1String("name")).toString())); + } else if (type == QLatin1String("sendWindowCommandLine")) { + refuse(type, mEngine->sendWindowCommandLine(message.value(QLatin1String("name")).toString(), message.value(QLatin1String("text")).toString())); + } else if (type == QLatin1String("pointer")) { + refuse(type, pointer(message)); + } else if (type == QLatin1String("getImage")) { + getImage(message); } else { sendError(QStringLiteral("unknown type %1").arg(type)); } @@ -285,6 +339,150 @@ void SocketBridge::complete(const QJsonObject& message) status(QStringLiteral("completion"), mEngine->complete(message.value(QLatin1String("text")).toString(), fromHistory.toBool(false))); } +QString SocketBridge::setMainWindowSize(const QJsonObject& message) +{ + const std::optional width = wholeNumber(message.value(QLatin1String("w"))); + const std::optional height = wholeNumber(message.value(QLatin1String("h"))); + if (!width || !height || *width < 1 || *height < 1) { + return QStringLiteral("w and h must be whole numbers above 0"); + } + return mEngine->setMainWindowSize(*width, *height); +} + +void SocketBridge::getWindowText(const QJsonObject& message) +{ + const QString window = message.value(QLatin1String("window")).toString(); + const std::optional first = wholeNumber(message.value(QLatin1String("first"))); + const std::optional count = wholeNumber(message.value(QLatin1String("count"))); + if (!first || !count || *count < 1) { + sendError(QStringLiteral("getWindowText: first and count must be whole numbers, count above 0")); + return; + } + // An empty name would be the main console's in Mudlet + const std::optional text = window.isEmpty() ? std::nullopt : mEngine->windowText(window, *first, std::min(*count, maxWindowTextRows)); + if (!text) { + sendError(QStringLiteral("getWindowText: there is no console named \"%1\"").arg(window)); + return; + } + sendReply(windowTextMessage(window, *text)); +} + +QString SocketBridge::placeUserWindow(const QJsonObject& message) +{ + const std::optional x = wholeNumber(message.value(QLatin1String("x"))); + const std::optional y = wholeNumber(message.value(QLatin1String("y"))); + const std::optional width = wholeNumber(message.value(QLatin1String("w"))); + const std::optional height = wholeNumber(message.value(QLatin1String("h"))); + const QJsonValue floating = message.value(QLatin1String("floating")); + if (!x || !y || !width || !height || *width < 1 || *height < 1) { + return QStringLiteral("x, y, w and h must be whole numbers, w and h above 0"); + } + if (!floating.isBool()) { + return QStringLiteral("floating must be true or false"); + } + return mEngine->placeUserWindow(message.value(QLatin1String("name")).toString(), QRect(*x, *y, *width, *height), floating.toBool()); +} + +QString SocketBridge::pointer(const QJsonObject& message) +{ + static constexpr std::pair types[] = { + {QLatin1StringView("press"), PointerEvent::Type::press}, + {QLatin1StringView("release"), PointerEvent::Type::release}, + {QLatin1StringView("doubleClick"), PointerEvent::Type::doubleClick}, + {QLatin1StringView("move"), PointerEvent::Type::move}, + {QLatin1StringView("enter"), PointerEvent::Type::enter}, + {QLatin1StringView("leave"), PointerEvent::Type::leave}, + {QLatin1StringView("wheel"), PointerEvent::Type::wheel}, + }; + PointerEvent event; + const QString type = message.value(QLatin1String("type")).toString(); + const auto known = std::find_if(std::begin(types), std::end(types), [&type](const auto& entry) { + return entry.first == type; + }); + if (known == std::end(types)) { + return QStringLiteral("type must be press, release, doubleClick, move, enter, leave or wheel"); + } + event.type = known->second; + const QJsonValue x = message.value(QLatin1String("x")); + const QJsonValue y = message.value(QLatin1String("y")); + if (!x.isDouble() || !y.isDouble()) { + return QStringLiteral("x and y must be numbers"); + } + event.position = QPointF(x.toDouble(), y.toDouble()); + // Absent is 0; the buttons and modifiers are Qt's flags, so never negative + const std::pair optional[] = { + {message.value(QLatin1String("button")), &event.button}, + {message.value(QLatin1String("buttons")), &event.buttons}, + {message.value(QLatin1String("modifiers")), &event.modifiers}, + {message.value(QLatin1String("wheelY")), &event.wheelY}, + }; + for (const auto& [value, target] : optional) { + if (value.isUndefined()) { + continue; + } + const std::optional number = wholeNumber(value); + if (!number || (target != &event.wheelY && *number < 0)) { + return QStringLiteral("button, buttons and modifiers must be whole numbers from 0, wheelY a whole number"); + } + *target = *number; + } + return mEngine->pointer(message.value(QLatin1String("window")).toString(), event); +} + +void SocketBridge::getImage(const QJsonObject& message) +{ + const QString path = message.value(QLatin1String("path")).toString(); + if (path.isEmpty()) { + sendError(QStringLiteral("getImage: path must not be empty")); + return; + } + const QJsonValue widthValue = message.value(QLatin1String("w")); + const QJsonValue heightValue = message.value(QLatin1String("h")); + const QJsonValue frameValue = message.value(QLatin1String("frame")); + QSize size; + if (!widthValue.isUndefined() || !heightValue.isUndefined()) { + const std::optional width = wholeNumber(widthValue); + const std::optional height = wholeNumber(heightValue); + if (!width || !height || *width < 1 || *height < 1 || *width > PngImage::maxSide || *height > PngImage::maxSide) { + sendError(QStringLiteral("getImage: w and h go together, whole numbers from 1 to %1").arg(PngImage::maxSide)); + return; + } + size = QSize(*width, *height); + } + std::optional frame; + if (!frameValue.isUndefined()) { + frame = wholeNumber(frameValue); + if (!frame || *frame < 0) { + sendError(QStringLiteral("getImage: frame must be a whole number from 0")); + return; + } + } + const std::variant image = mEngine->image(path, size, frame); + if (const QString* problem = std::get_if(&image)) { + sendError(QStringLiteral("getImage: %1").arg(*problem)); + return; + } + const PngImage& png = std::get(image); + const QByteArray base64 = png.png.toBase64(); + // With room for the path, each of its characters escaped at worst, and the other fields + if (base64.size() + path.size() * 6 + 1024 > qsizetype(maxFrameBytes)) { + sendError(QStringLiteral("getImage: %1 as a PNG is too large for a frame").arg(path)); + return; + } + sendReply({{QStringLiteral("t"), QStringLiteral("image")}, + {QStringLiteral("path"), path}, + {QStringLiteral("png"), QString::fromLatin1(base64)}, + {QStringLiteral("w"), png.size.width()}, + {QStringLiteral("h"), png.size.height()}}); +} + +void SocketBridge::refuse(const QString& request, const QString& problem) +{ + if (!problem.isEmpty()) { + sendError(QStringLiteral("%1: %2").arg(request, problem)); + } +} + void SocketBridge::answerProfiles(const QString& request, const QString& problem) { if (problem.isEmpty()) { @@ -365,6 +563,116 @@ QJsonObject SocketBridge::historyMessage(const QStringList& lines) return {{QStringLiteral("t"), QStringLiteral("history")}, {QStringLiteral("lines"), QJsonArray::fromStringList(lines)}}; } +QJsonObject SocketBridge::windowJson(const ScriptWindowInfo& window) +{ + QJsonObject json{{QStringLiteral("name"), window.name}, + {QStringLiteral("kind"), window.kind}, + {QStringLiteral("parent"), window.parent}, + {QStringLiteral("x"), window.geometry.x()}, + {QStringLiteral("y"), window.geometry.y()}, + {QStringLiteral("w"), window.geometry.width()}, + {QStringLiteral("h"), window.geometry.height()}, + {QStringLiteral("visible"), window.visible}, + {QStringLiteral("shown"), window.shown}, + {QStringLiteral("stack"), window.stackOrder}, + {QStringLiteral("inDock"), window.inDock}}; + if (!window.parentKind.isEmpty()) { + json.insert(QStringLiteral("parentKind"), window.parentKind); + } + if (window.background != -1) { + json.insert(QStringLiteral("bg"), window.background); + if (window.backgroundAlpha < 255) { + json.insert(QStringLiteral("bgAlpha"), window.backgroundAlpha); + } + } + if (!window.fontFamily.isEmpty()) { + json.insert(QStringLiteral("fontFamily"), window.fontFamily); + } + if (window.fontSize > 0) { + json.insert(QStringLiteral("fontSize"), window.fontSize); + } + if (window.dock) { + json.insert(QStringLiteral("autoDock"), window.dock->autoDock); + json.insert(QStringLiteral("dockArea"), window.dock->area); + json.insert(QStringLiteral("dockFloating"), window.dock->floating); + json.insert(QStringLiteral("dockTitle"), window.dock->title); + } + if (window.console) { + if (window.console->scrolledBackTo) { + json.insert(QStringLiteral("scrolledBackTo"), *window.console->scrolledBackTo); + } + json.insert(QStringLiteral("columns"), window.console->columns); + json.insert(QStringLiteral("rows"), window.console->rows); + json.insert(QStringLiteral("scrollBar"), window.console->scrollBar); + json.insert(QStringLiteral("timeStamps"), window.console->timeStamps); + } + return json; +} + +QJsonObject SocketBridge::labelMessage(const LabelInfo& label) +{ + QJsonObject json{{QStringLiteral("t"), QStringLiteral("label")}, + {QStringLiteral("name"), label.name}, + {QStringLiteral("text"), label.text}, + {QStringLiteral("image"), label.image}, + {QStringLiteral("svg"), label.svg}, + {QStringLiteral("movie"), label.movie}, + {QStringLiteral("moviePaused"), label.moviePaused}, + {QStringLiteral("movieFrame"), label.movieFrame}, + {QStringLiteral("movieSpeed"), label.movieSpeed}, + {QStringLiteral("toolTip"), label.toolTip}, + {QStringLiteral("clickThrough"), label.clickThrough}, + {QStringLiteral("accessibleName"), label.accessibleName}, + {QStringLiteral("accessibleButton"), label.accessibleButton}, + {QStringLiteral("styleSheet"), label.styleSheet}}; + if (label.svgTint != -1) { + json.insert(QStringLiteral("svgTint"), label.svgTint); + if (label.svgTintAlpha < 255) { + json.insert(QStringLiteral("svgTintAlpha"), label.svgTintAlpha); + } + } + if (label.svgRotation != 0) { + json.insert(QStringLiteral("svgRotation"), label.svgRotation); + } + if (label.svgShearX != 0) { + json.insert(QStringLiteral("svgShearX"), label.svgShearX); + } + if (label.svgShearY != 0) { + json.insert(QStringLiteral("svgShearY"), label.svgShearY); + } + if (label.movieSize.isValid()) { + json.insert(QStringLiteral("movieW"), label.movieSize.width()); + json.insert(QStringLiteral("movieH"), label.movieSize.height()); + } + return json; +} + +QJsonObject SocketBridge::windowTextMessage(const QString& window, const WindowText& text) +{ + QJsonArray lines; + for (const WindowText::Row& row : text.rows) { + lines.append(QJsonObject{{QStringLiteral("text"), row.text}, {QStringLiteral("runs"), runsJson(row.runs)}, {QStringLiteral("prompt"), row.prompt}, {QStringLiteral("wrapped"), row.wrapped}}); + } + return {{QStringLiteral("t"), QStringLiteral("windowText")}, + {QStringLiteral("window"), window}, + {QStringLiteral("lineCount"), text.lineCount}, + {QStringLiteral("first"), text.first}, + {QStringLiteral("lines"), lines}}; +} + +QJsonObject SocketBridge::mainAreaMessage(const MainArea& area) +{ + return {{QStringLiteral("t"), QStringLiteral("mainArea")}, + {QStringLiteral("w"), area.size.width()}, + {QStringLiteral("h"), area.size.height()}, + {QStringLiteral("columns"), area.grid.width()}, + {QStringLiteral("rows"), area.grid.height()}, + {QStringLiteral("textX"), area.textRect.x()}, + {QStringLiteral("textY"), area.textRect.y()}, + {QStringLiteral("textW"), area.textRect.width()}, + {QStringLiteral("textH"), area.textRect.height()}}; +} + void SocketBridge::sendFrame(const QJsonObject& message) { if (!mClient || mClient->state() != QLocalSocket::ConnectedState) { @@ -455,3 +763,80 @@ void SocketBridge::commandLine(const QString& text) sendFrame({{QStringLiteral("t"), QStringLiteral("commandLine")}, {QStringLiteral("text"), text}}); } } + +// Only what changed is sent, the windows keyed by name and kind and the labels by name, each label after the windows +// so that the UI knows its window first; a UI that attaches or drains gets them all +void SocketBridge::windows(const QList& windows, const QList& labels) +{ + using Key = std::pair; + QHash before; + for (const ScriptWindowInfo& window : std::as_const(mWindows)) { + before.insert({window.name, window.kind}, &window); + } + QJsonArray changed; + for (const ScriptWindowInfo& window : windows) { + const ScriptWindowInfo* was = before.take({window.name, window.kind}); + if (!was || !(*was == window)) { + changed.append(windowJson(window)); + } + } + QJsonArray removed; + for (const ScriptWindowInfo& window : std::as_const(mWindows)) { + if (before.contains({window.name, window.kind})) { + removed.append(QJsonObject{{QStringLiteral("name"), window.name}, {QStringLiteral("kind"), window.kind}}); + } + } + QHash labelsBefore; + for (const LabelInfo& label : std::as_const(mLabels)) { + labelsBefore.insert(label.name, &label); + } + QList changedLabels; + for (const LabelInfo& label : labels) { + const LabelInfo* was = labelsBefore.value(label.name); + if (!was || *was != label) { + changedLabels.append(labelMessage(label)); + } + } + mWindows = windows; + mLabels = labels; + if (!clientLive()) { + return; + } + if (!changed.isEmpty() || !removed.isEmpty()) { + QJsonObject message{{QStringLiteral("t"), QStringLiteral("windows")}, {QStringLiteral("windows"), changed}}; + if (!removed.isEmpty()) { + message.insert(QStringLiteral("removed"), removed); + } + sendFrame(message); + } + for (const QJsonObject& label : std::as_const(changedLabels)) { + sendFrame(label); + } +} + +// Not kept: a UI that attaches or drains gets every console's from IEngine::windowText() +void SocketBridge::windowText(const QString& window, const WindowText& text) +{ + if (clientLive()) { + sendFrame(windowTextMessage(window, text)); + } +} + +void SocketBridge::mainArea(const MainArea& area) +{ + if (mMainArea == area) { + return; + } + mMainArea = area; + if (clientLive()) { + sendFrame(mainAreaMessage(area)); + } +} + +// Not kept, as commandLine is not +void SocketBridge::windowCommandLine(const QString& name, const QString& text) +{ + if (clientLive()) { + sendFrame({{QStringLiteral("t"), QStringLiteral("windowCommandLine")}, {QStringLiteral("name"), name}, {QStringLiteral("text"), text}}); + } +} diff --git a/native/enginehost/SocketBridge.h b/native/enginehost/SocketBridge.h index 6ade0d1..4208d49 100644 --- a/native/enginehost/SocketBridge.h +++ b/native/enginehost/SocketBridge.h @@ -10,6 +10,7 @@ #include #include #include +#include class IEngine; class QLocalServer; @@ -26,11 +27,13 @@ class SocketBridge : public QObject, public IFrontend Q_OBJECT public: - static constexpr int protocolVersion = 3; + static constexpr int protocolVersion = 8; static constexpr quint32 maxFrameBytes = 16 * 1024 * 1024; static constexpr qsizetype scrollbackLines = 2000; static constexpr qint64 highWaterBytes = 4 * 1024 * 1024; static constexpr qint64 lowWaterBytes = 256 * 1024; + // The most rows one getWindowText is answered with + static constexpr int maxWindowTextRows = 1000; explicit SocketBridge(QObject* parent = nullptr); void setEngine(IEngine* engine) { mEngine = engine; } @@ -44,6 +47,10 @@ class SocketBridge : public QObject, public IFrontend void hiddenInput(bool wanted) override; void history(const QStringList& lines) override; void commandLine(const QString& text) override; + void windows(const QList& windows, const QList& labels) override; + void windowText(const QString& window, const WindowText& text) override; + void mainArea(const MainArea& area) override; + void windowCommandLine(const QString& name, const QString& text) override; private: struct StoredLine @@ -60,7 +67,7 @@ class SocketBridge : public QObject, public IFrontend void handleMessage(const QJsonObject& message); void greet(const QJsonObject& hello); void sendLinesAfter(qint64 seq, bool replay); - // Every status, then the input line's state, as a UI attaching or catching up needs them + // Every status, then the input line's state and the script windows', as a UI attaching or catching up needs them void sendState(); void sendFrame(const QJsonObject& message); // An answer to the request being handled: it carries that request's id as re, and is sent even to a lagging UI @@ -68,9 +75,16 @@ class SocketBridge : public QObject, public IFrontend void sendError(const QString& message); void sendProfiles(); void answerProfiles(const QString& request, const QString& problem); + // An error naming the request, for a problem that is not empty + void refuse(const QString& request, const QString& problem); QString saveProfile(const QJsonObject& message); QString openProfile(const QJsonObject& message); void complete(const QJsonObject& message); + QString setMainWindowSize(const QJsonObject& message); + void getWindowText(const QJsonObject& message); + QString placeUserWindow(const QJsonObject& message); + QString pointer(const QJsonObject& message); + void getImage(const QJsonObject& message); void dropClient(const QString& reason); bool clientLive() const { return mClient && mClientGreeted && !mClientLagging; } bool replying() const { return !mCurrentRequestId.isUndefined() && !mCurrentRequestId.isNull(); } @@ -79,6 +93,10 @@ class SocketBridge : public QObject, public IFrontend QJsonObject tailMessage() const; QJsonObject hiddenInputMessage() const; static QJsonObject historyMessage(const QStringList& lines); + static QJsonObject windowJson(const ScriptWindowInfo& window); + static QJsonObject labelMessage(const LabelInfo& label); + static QJsonObject windowTextMessage(const QString& window, const WindowText& text); + static QJsonObject mainAreaMessage(const MainArea& area); QLocalServer* mServer = nullptr; QPointer mClient; @@ -89,6 +107,11 @@ class SocketBridge : public QObject, public IFrontend IEngine* mEngine = nullptr; QHash mStatus; bool mHiddenInput = false; + // As IFrontend::windows() last gave them + QList mWindows; + QList mLabels; + // None until the engine has sent one + std::optional mMainArea; std::deque mScrollback; StoredLine mTail{0, {}, {}, false}; qint64 mNextSeq = 1; diff --git a/native/standin/Engine.h b/native/standin/Engine.h index f0d571e..e45e68b 100644 --- a/native/standin/Engine.h +++ b/native/standin/Engine.h @@ -32,11 +32,21 @@ class Engine : public QObject, public IEngine QString saveProfile(const Profile& profile) override; QString deleteProfile(const QString& name) override; QString openProfile(const QString& name, const QString& password, bool connect) override; + // No scripts make windows here + QString setMainWindowSize(int, int) override { return noWindows(); } + std::optional windowText(const QString&, int, int) const override { return std::nullopt; } + QString placeUserWindow(const QString&, const QRect&, bool) override { return noWindows(); } + QString closeUserWindow(const QString&) override { return noWindows(); } + QString sendWindowCommandLine(const QString&, const QString&) override { return noWindows(); } + QString pointer(const QString&, const PointerEvent&) override { return noWindows(); } + std::variant image(const QString&, const QSize&, std::optional) const override { return noWindows(); } // Called from Lua's notify() void luaNotify(const QString& title, const QString& text); private: + static QString noWindows() { return QStringLiteral("the stand-in engine has no script windows"); } + enum class TelnetState { Data, Iac, Option, Sub, SubIac }; void onReadyRead(); diff --git a/native/tests/protocol_client.py b/native/tests/protocol_client.py index fa212e2..22d35e4 100644 --- a/native/tests/protocol_client.py +++ b/native/tests/protocol_client.py @@ -5,7 +5,7 @@ import struct import time -PROTOCOL_VERSION = 3 +PROTOCOL_VERSION = 8 class Closed(Exception): @@ -69,12 +69,12 @@ def wait_for(self, pred, timeout=5.0, collect=None): if pred(msg): return msg - def ask(self, msg, timeout=5.0, collect=None): - """Send a request with a fresh id and return the profiles or error frame that answers it.""" + def ask(self, msg, timeout=5.0, collect=None, answers=("profiles", "error")): + """Send a request with a fresh id and return the frame of one of those types that answers it.""" self.next_id += 1 request_id = f"ask{self.next_id}" self.send(dict(msg, id=request_id)) - return self.wait_for(lambda m: m["t"] in ("profiles", "error") and m.get("re") == request_id, timeout, collect) + return self.wait_for(lambda m: m["t"] in answers and m.get("re") == request_id, timeout, collect) def close(self): self.sock.close() diff --git a/native/tests/test_engine.py b/native/tests/test_engine.py index 7873ed4..54d1c0d 100644 --- a/native/tests/test_engine.py +++ b/native/tests/test_engine.py @@ -145,6 +145,38 @@ def input_line(c): return c +def script_windows(c): + """The stand-in has no script windows or labels: it refuses their requests, once their fields are right.""" + refusal = "the stand-in engine has no script windows" + side = {"name": "side", "x": 0, "y": 0, "w": 300, "h": 200, "floating": False} + for request in ({"t": "setMainWindowSize", "w": 800, "h": 600}, dict(side, t="placeUserWindow"), + {"t": "closeUserWindow", "name": "side"}, {"t": "sendWindowCommandLine", "name": "chat", "text": "hi"}, + {"t": "pointer", "window": "box", "type": "press", "x": 1.5, "y": 2, "button": 1, "buttons": 1}, + {"t": "getImage", "path": "/no/such.png", "w": 16, "h": 16, "frame": 0}): + m = c.ask(request, answers=("error",)) + check(m["message"] == f"{request['t']}: {refusal}", f"{request['t']} is refused: {m}") + m = c.ask({"t": "getWindowText", "window": "chat", "first": -300, "count": 300}, answers=("error", "windowText")) + check(m["t"] == "error" and m["message"] == 'getWindowText: there is no console named "chat"', f"getWindowText answers error: {m}") + for request, message in ( + ({"t": "setMainWindowSize", "w": 0, "h": 600}, "w and h must be whole numbers above 0"), + ({"t": "setMainWindowSize", "w": 800.5, "h": 600}, "w and h must be whole numbers above 0"), + (dict(side, t="placeUserWindow", x="0"), "x, y, w and h must be whole numbers, w and h above 0"), + (dict(side, t="placeUserWindow", floating="no"), "floating must be true or false"), + ({"t": "getWindowText", "window": "chat", "first": 0, "count": 0}, "first and count must be whole numbers, count above 0"), + ({"t": "pointer", "window": "box", "type": "tap", "x": 1, "y": 2}, "type must be press, release, doubleClick, move, enter, leave or wheel"), + ({"t": "pointer", "window": "box", "type": "press", "x": "1", "y": 2}, "x and y must be numbers"), + ({"t": "pointer", "window": "box", "type": "press", "x": 1, "y": 2, "button": -1}, + "button, buttons and modifiers must be whole numbers from 0, wheelY a whole number"), + ({"t": "pointer", "window": "box", "type": "wheel", "x": 1, "y": 2, "wheelY": 0.5}, + "button, buttons and modifiers must be whole numbers from 0, wheelY a whole number"), + ({"t": "getImage", "path": "a.png", "w": 16}, "w and h go together, whole numbers from 1 to 4096"), + ({"t": "getImage", "path": "a.png", "w": 16, "h": 5000}, "w and h go together, whole numbers from 1 to 4096"), + ({"t": "getImage", "path": "a.png", "frame": -1}, "frame must be a whole number from 0"), + ({"t": "getImage", "path": ""}, "path must not be empty")): + m = c.ask(request, answers=("error",)) + check(m["message"] == f"{request['t']}: {message}", f"a bad field is refused first: {request} -> {m['message']}") + + def main(): faulthandler.register(signal.SIGUSR1) port = fake_telnet.start_in_thread() @@ -162,6 +194,9 @@ def main(): check(st.get("connection") == "disconnected", "initial connection status replayed") check(int(st.get("advanceM", "0")) > 0, f"font check: family={st.get('fontFamily')!r} advanceM={st.get('advanceM')} platform={st.get('platform')}") results["font"] = (st.get("fontFamily"), st.get("advanceM")) + windows = [m for m in hs if m["t"] == "windows"] + check(windows == [{"t": "windows", "full": True, "windows": []}], f"a UI that attaches is told there are no script windows: {windows}") + check(not any(m["t"] in ("mainArea", "windowText", "label") for m in hs), "nor a main area, window text or label") # Wrong protocol version is refused bad = Client(SOCK, version=99) @@ -251,6 +286,7 @@ def main(): c.send({"t": "unknownThing", "id": 3}) m = c.wait_for(lambda m: m["t"] == "error") check(m == {"t": "error", "message": "unknown type unknownThing", "re": 3}, f"an error carries the request's id: {m}") + script_windows(c) # Latency samples = [] diff --git a/native/tests/test_mudlet_engine.py b/native/tests/test_mudlet_engine.py index 4ab17a2..5b8ea6b 100644 --- a/native/tests/test_mudlet_engine.py +++ b/native/tests/test_mudlet_engine.py @@ -8,6 +8,7 @@ usage: test_mudlet_engine.py """ +import base64 import glob import os import signal @@ -17,6 +18,7 @@ import sys import tempfile import time +import zlib sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) import fake_telnet # noqa: E402 @@ -255,6 +257,8 @@ def until_last_line(m): tls_checks(port) profile_requests(port) input_line() + script_windows() + script_labels() if failures: print(f"\n{len(failures)} FAILED; engine log: {LOG}") with open(LOG, errors="replace") as f: @@ -451,5 +455,333 @@ def input_line(): profile_checks.password_kept_out(check, secret, seen, LOG) +def window_of(msgs, name, kind): + """The last state the windows messages gave of a script window, or None.""" + found = None + for m in msgs: + if m["t"] == "windows": + found = next((w for w in m["windows"] if w["name"] == name and w["kind"] == kind), found) + return found + + +def windows_with(name, kind, pred=lambda w: True): + return lambda m: m["t"] == "windows" and any(w["name"] == name and w["kind"] == kind and pred(w) for w in m["windows"]) + + +def seen_or_next(c, pred, seen, since, timeout=5.0): + """The first message from seen[since:] that pred takes, or else the next one to come. The session sends a turn's + main console lines as it runs and its script windows, their text and command lines from the event loop after it, + so a wait for one of those has already collected a line the same turn wrote. On a timeout, says what came.""" + found = next((m for m in seen[since:] if pred(m)), None) + if found is not None: + return found + try: + return c.wait_for(pred, timeout=timeout, collect=seen) + except TimeoutError: + came = [str(m)[:200] for m in seen[since:]] + raise TimeoutError(f"message not received; since the request: {came}") from None + + +def script_windows(): + """What scripts draw: a Geyser MiniConsole and UserWindow as windows, with the console's text as windowText; the + main window's size from setMainWindowSize, and the borders a script sets, as mainArea; placeUserWindow raising + the resize event a script handles; a console's own command line both ways; hiding, closing and deleting; and all + of it again for a UI that attaches.""" + game = fake_telnet.start_in_thread() + proc, c = start_engine("Window Test") + seen = [] + c.wait_for(lambda m: m["t"] == "synced", collect=seen) + check({"t": "windows", "full": True, "windows": []} in seen and not any(m["t"] == "mainArea" for m in seen), + "with no profile open, a UI that attaches is told of no script windows, and no main area") + m = c.ask({"t": "getWindowText", "window": "chat", "first": 0, "count": 5}, answers=("windowText", "error"), collect=seen) + check(m["t"] == "error", f"getWindowText with no profile open is refused: {m}") + c.send({"t": "placeUserWindow", "name": "side", "x": 0, "y": 0, "w": 10, "h": 10, "floating": False}) + m = c.wait_for(lambda m: m["t"] == "error", collect=seen) + check(m["message"] == "placeUserWindow: no profile running", f"placeUserWindow with no profile open is refused: {m}") + + # Kept until the profile loads, which takes it + c.send({"t": "setMainWindowSize", "w": 1000, "h": 800}) + c.send({"t": "connect", "host": "127.0.0.1", "port": game}) + m = c.wait_for(lambda m: m["t"] == "mainArea", timeout=30, collect=seen) + check(m["w"] == 1000 and m["h"] == 800 and (m["textX"], m["textY"], m["textW"], m["textH"]) == (0, 0, 1000, 800) + and m["columns"] > 0 and m["rows"] > 0, f"the profile takes the size the UI gave before it loaded: {m}") + columns = m["columns"] + wait_status(c, "connection", lambda v: v == "connected", seen) + + since = len(seen) + c.send({"t": "runLua", "code": "\n".join([ + 'chat = Geyser.MiniConsole:new({name = "chat", x = 0, y = 0, width = "30%", height = "40%"})', + 'chat:cecho("red plain\\n")', + 'side = Geyser.UserWindow:new({name = "side"})', + 'registerAnonymousEventHandler("sysUserWindowResizeEvent", function(_, width, height, window)', + ' echo(string.format("resized %s %dx%d\\n", window, width, height))', + 'end)', + 'local mw, mh = getMainWindowSize()', + 'echo(string.format("main window %dx%d\\n", mw, mh))', + ])}) + m = seen_or_next(c, lambda m: m["t"] == "windowText" and m["window"] == "chat" and m["lines"] + and m["lines"][0]["text"] == "red plain", seen, since) + check("re" not in m and m["first"] == 0 and m["lineCount"] >= 1, f"chat:cecho() gives the console's text: {m}") + runs = m["lines"][0]["runs"] + check(bool(runs) and runs[0][:3] == [0, 3, 0x1000000 | 0xFF0000], f"with its colours: {runs}") + m = seen_or_next(c, lambda m: m["t"] in ("line", "tail") and m["text"].startswith("main window "), seen, since) + check(m["text"] == "main window 1000x800", f"scripts read the size the UI gave as the main window's: {m['text']}") + chat = window_of(seen, "chat", "miniConsole") + check(chat is not None and (chat["x"], chat["y"], chat["w"], chat["h"]) == (0, 0, 300, 320) and chat["parent"] == "" + and chat["shown"] and chat["visible"] and not chat["inDock"] and chat.get("columns", 0) > 0 and chat.get("fontSize", 0) > 0, + f"a Geyser MiniConsole is a miniConsole at its place in the main window: {chat}") + side = window_of(seen, "side", "userWindow") + check(side is not None and side["inDock"] and "dockArea" in side and "columns" in side, + f"a Geyser UserWindow is a userWindow in a dock: {side}") + + since = len(seen) + c.send({"t": "placeUserWindow", "name": "side", "x": 10, "y": 20, "w": 250, "h": 400, "floating": False}) + seen_or_next(c, shown("resized side 250x400"), seen, since) + seen_or_next(c, windows_with("side", "userWindow", lambda w: w["w"] == 250), seen, since) + side = window_of(seen, "side", "userWindow") + check((side["x"], side["y"], side["w"], side["h"]) == (10, 20, 250, 400), + f"placeUserWindow places it, and raises sysUserWindowResizeEvent: {side}") + since = len(seen) + c.send({"t": "placeUserWindow", "name": "chat", "x": 0, "y": 0, "w": 10, "h": 10, "floating": False}) + m = seen_or_next(c, lambda m: m["t"] == "error", seen, since) + check(m["message"].startswith("placeUserWindow: ") and "chat" in m["message"], f"a mini console is no user window: {m}") + + since = len(seen) + # The profile's mudlet-base-ui package keeps its container attached to the right border, so the text ends there + c.send({"t": "runLua", "code": 'setBorderLeft(100)\necho(string.format("right border %d\\n", getBorderRight()))'}) + m = seen_or_next(c, lambda m: m["t"] in ("line", "tail") and m["text"].startswith("right border "), seen, since) + right = int(m["text"].split()[-1]) + m = seen_or_next(c, lambda m: m["t"] == "mainArea", seen, since) + check(m["w"] == 1000 and (m["textX"], m["textW"]) == (100, 900 - right) and m["columns"] < columns, + f"setBorderLeft() moves the main console's text: {m}, right border {right}") + since = len(seen) + c.send({"t": "setMainWindowSize", "w": 1, "h": 1}) + m = seen_or_next(c, lambda m: m["t"] in ("error", "mainArea"), seen, since) + check(m["t"] == "error" and m["message"].startswith("setMainWindowSize: "), f"a size with no room for a character is refused: {m}") + since = len(seen) + c.send({"t": "setMainWindowSize", "w": 1200, "h": 800}) + m = seen_or_next(c, lambda m: m["t"] == "mainArea", seen, since) + check((m["w"], m["h"], m["textX"], m["textW"]) == (1200, 800, 100, 1100 - right), + f"a size given while the profile runs is taken at once: {m}") + + since = len(seen) + c.send({"t": "runLua", "code": "\n".join([ + 'enableCommandLine("chat")', + 'setCmdLineAction("chat", function(text) echo("acted " .. text .. "\\n") end)', + 'printCmdLine("chat", "typed")', + ])}) + m = seen_or_next(c, lambda m: m["t"] == "windowCommandLine" and m["text"], seen, since) + check(m == {"t": "windowCommandLine", "name": "chat", "text": "typed"}, f"printCmdLine() on a console's line: {m}") + line = window_of(seen, "chat", "commandLine") + check(line is not None and line["parent"] == "chat" and line.get("parentKind") == "miniConsole" and line["shown"], + f"enableCommandLine() gives the console a commandLine: {line}") + since = len(seen) + c.send({"t": "sendWindowCommandLine", "name": "chat", "text": "look"}) + seen_or_next(c, shown("acted look"), seen, since) + check(True, "sendWindowCommandLine runs the line's action") + + m = c.ask({"t": "getWindowText", "window": "chat", "first": 0, "count": 10}, answers=("windowText", "error"), collect=seen) + check(m["t"] == "windowText" and m["window"] == "chat" and m["first"] == 0 and m["lines"][0]["text"] == "red plain", + f"getWindowText answers with the rows: {m}") + m = c.ask({"t": "getWindowText", "window": "nonesuch", "first": 0, "count": 10}, answers=("windowText", "error"), + collect=seen) + check(m["t"] == "error" and "nonesuch" in m["message"], f"and refuses a console there is not: {m}") + + since = len(seen) + c.send({"t": "runLua", "code": "chat:hide()"}) + seen_or_next(c, windows_with("chat", "miniConsole", lambda w: not w["shown"]), seen, since) + chat = window_of(seen, "chat", "miniConsole") + line = window_of(seen, "chat", "commandLine") + check(not chat["visible"] and line is not None and line["visible"] and not line["shown"], + f"hiding it hides what it holds: {chat} {line}") + since = len(seen) + c.send({"t": "closeUserWindow", "name": "side"}) + seen_or_next(c, windows_with("side", "userWindow", lambda w: not w["visible"]), seen, since) + check(True, "closeUserWindow hides the user window") + + c.close() + c = Client(SOCK) + msgs = [] + c.wait_for(lambda m: m["t"] == "synced", collect=msgs) + seen += msgs + full = [m for m in msgs if m["t"] == "windows"] + keys = {(w["name"], w["kind"]) for w in full[0]["windows"]} if full else set() + check(len(full) == 1 and full[0]["full"] and {("chat", "miniConsole"), ("chat", "commandLine"), ("side", "userWindow")} <= keys, + f"a UI that attaches gets every window: {full}") + texts = {m["window"]: m for m in msgs if m["t"] == "windowText"} + check("chat" in texts and "side" in texts and texts["chat"]["lines"][0]["text"] == "red plain", + f"and each console's text: {list(texts)}") + areas = [m for m in msgs if m["t"] == "mainArea"] + check(len(areas) == 1 and (areas[0]["w"], areas[0]["textX"]) == (1200, 100), f"and the main area: {areas}") + + since = len(seen) + c.send({"t": "runLua", "code": 'deleteMiniConsole("chat")'}) + m = seen_or_next(c, lambda m: m["t"] == "windows" and m.get("removed"), seen, since) + check({"name": "chat", "kind": "miniConsole"} in m["removed"], f"a deleted console is removed: {m}") + code = stop_engine(proc) + check(code == 0, f"the engine exits cleanly on SIGTERM: {code}") + + + +def png_file(path, width, height): + """A PNG of width by height opaque red pixels.""" + def chunk(kind, data): + return struct.pack(">I", len(data)) + kind + data + struct.pack(">I", zlib.crc32(kind + data)) + rows = b"".join(b"\0" + b"\xff\0\0" * width for _ in range(height)) + with open(path, "wb") as f: + f.write(b"\x89PNG\r\n\x1a\n" + chunk(b"IHDR", struct.pack(">IIBBBBB", width, height, 8, 2, 0, 0, 0)) + + chunk(b"IDAT", zlib.compress(rows)) + chunk(b"IEND", b"")) + + +def png_size(data): + """The width and height in a PNG's header, or None for what is not a PNG.""" + if data[:8] != b"\x89PNG\r\n\x1a\n" or data[12:16] != b"IHDR": + return None + return struct.unpack(">II", data[16:24]) + + +def label_of(msgs, name): + """The last label message for a label, or None.""" + return next((m for m in reversed(msgs) if m["t"] == "label" and m["name"] == name), None) + + +def script_labels(): + """What labels show: a Geyser.Label's text, stylesheet, image and accessible name as label messages, the image as + a PNG from getImage, and its click callback run by pointer; a Geyser.Gauge as three labels whose geometry gives + its value, with its text label transparent; a deleted label removed; and every label again for a UI that + attaches.""" + picture = os.path.join(TMP, "dot.png") + png_file(picture, 4, 3) + vector = os.path.join(TMP, "square.svg") + with open(vector, "w") as f: + f.write('') + game = fake_telnet.start_in_thread() + proc, c = start_engine("Label Test") + seen = [] + c.wait_for(lambda m: m["t"] == "synced", collect=seen) + m = c.ask({"t": "getImage", "path": picture}, answers=("image", "error"), collect=seen) + check(m["t"] == "image" and (m["w"], m["h"]) == (4, 3) and png_size(base64.b64decode(m["png"])) == (4, 3), + f"getImage needs no profile, and answers with the file as a PNG: {str(m)[:200]}") + c.send({"t": "pointer", "window": "main", "type": "move", "x": 1, "y": 1}) + m = c.wait_for(lambda m: m["t"] == "error", collect=seen) + check(m["message"] == "pointer: no profile running", f"pointer with no profile open is refused: {m}") + + c.send({"t": "setMainWindowSize", "w": 1000, "h": 800}) + c.send({"t": "connect", "host": "127.0.0.1", "port": game}) + wait_status(c, "connection", lambda v: v == "connected", seen, timeout=30) + + since = len(seen) + c.send({"t": "runLua", "code": "\n".join([ + 'box = Geyser.Label:new({name = "box", x = 20, y = 20, width = 120, height = 40})', + 'box:echo("hello label")', + 'box:setStyleSheet([[QLabel { border: 2px solid red; }]])', + 'box:setClickCallback(function(event) echo(string.format("clicked %s %d,%d\\n", event.button, event.x, event.y)) end)', + 'box:setAccessibleName("Hit me")', + 'box:setToolTip("a tip")', + 'pic = Geyser.Label:new({name = "pic", x = 200, y = 20, width = 40, height = 30})', + 'pic:echo("covered")', + f'pic:setBackgroundImage([==[{picture}]==])', + 'ico = Geyser.Label:new({name = "ico", x = 260, y = 20, width = 32, height = 32})', + f'ico:setBackgroundImage([==[{vector}]==])', + 'ico:setSvgTint(0, 128, 255)', + 'ico:setSvgRotation(90)', + ])}) + m = seen_or_next(c, lambda m: m["t"] == "label" and m["name"] == "box" and m["accessibleName"] == "Hit me", seen, since) + check("hello label" in m["text"] and m["styleSheet"] == "QLabel { border: 2px solid red; }" and m["accessibleButton"] + and m["toolTip"] == "a tip" and m["image"] == "" and not m["clickThrough"] and m["movie"] == "", + f"a Geyser.Label's echo(), stylesheet, click callback, accessible name and tool tip: {m}") + first_label = next(i for i, m in enumerate(seen) if m["t"] == "label" and m["name"] == "box") + first_window = next((i for i, m in enumerate(seen) if windows_with("box", "label")(m)), len(seen)) + check(first_window < first_label, "the label comes after the windows message that has it") + box = window_of(seen, "box", "label") + check(box is not None and (box["x"], box["y"], box["w"], box["h"]) == (20, 20, 120, 40) and box["shown"], + f"and is a label window at its place: {box}") + m = seen_or_next(c, lambda m: m["t"] == "label" and m["name"] == "pic" and m["image"], seen, since) + check(m["image"] == picture and m["text"] == "" and not m["svg"], f"setBackgroundImage() shows the file in place of the text: {m}") + m = seen_or_next(c, lambda m: m["t"] == "label" and m["name"] == "ico" and m.get("svgRotation"), seen, since) + check(m["image"] == vector and m["svg"] and m.get("svgTint") == 0x1000000 | 0x0080FF and "svgTintAlpha" not in m + and m["svgRotation"] == 90, f"an SVG one with its tint and rotation: {m}") + + m = c.ask({"t": "getImage", "path": picture, "w": 8, "h": 6}, answers=("image", "error"), collect=seen) + check(m["t"] == "image" and (m["w"], m["h"]) == (8, 6) and png_size(base64.b64decode(m["png"])) == (8, 6) + and m["path"] == picture, f"getImage scales it to the size asked: {str(m)[:200]}") + m = c.ask({"t": "getImage", "path": picture, "frame": 0}, answers=("image", "error"), collect=seen) + check(m["t"] == "image" and (m["w"], m["h"]) == (4, 3), f"a still image's frame 0 is the image: {str(m)[:200]}") + m = c.ask({"t": "getImage", "path": picture, "frame": 1}, answers=("image", "error"), collect=seen) + check(m["t"] == "error" and m["message"] == f"getImage: no frame 1 in {picture}", f"and it has no frame 1: {m}") + m = c.ask({"t": "getImage", "path": os.path.join(TMP, "nonesuch.png")}, answers=("image", "error"), collect=seen) + check(m["t"] == "error" and m["message"].startswith("getImage: ") and "nonesuch.png" in m["message"], + f"and refuses a file there is not: {m}") + + since = len(seen) + c.send({"t": "pointer", "window": "box", "type": "press", "x": 5, "y": 3, "button": 1, "buttons": 1}) + c.send({"t": "pointer", "window": "box", "type": "release", "x": 5, "y": 3, "button": 1}) + c.send({"t": "runLua", "code": 'echo("after the click\\n")'}) + seen_or_next(c, shown("after the click"), seen, since) + # A line, as the echo after it ends it + clicks = [m["text"] for m in lines_of(seen[since:]) if m["text"].startswith("clicked")] + errors = [m for m in seen[since:] if m["t"] == "error"] + check(clicks == ["clicked LeftButton 5,3"] and not errors, + f"a pointer press runs the label's click callback, with the event table, and the release does not: {clicks} {errors}") + since = len(seen) + c.send({"t": "pointer", "window": "nonesuch", "type": "press", "x": 5, "y": 3, "button": 1, "buttons": 1}) + m = seen_or_next(c, lambda m: m["t"] == "error", seen, since) + check(m["message"].startswith("pointer: ") and "nonesuch" in m["message"], f"pointer on a window there is not is refused: {m}") + since = len(seen) + c.send({"t": "pointer", "window": "box", "type": "move", "x": 500, "y": 3}) + m = seen_or_next(c, lambda m: m["t"] == "error", seen, since) + check(m["message"].startswith("pointer: "), f"and outside the window: {m}") + + since = len(seen) + c.send({"t": "runLua", "code": 'box:echo("new words")'}) + m = seen_or_next(c, lambda m: m["t"] == "label" and m["name"] == "box" and "new words" in m["text"], seen, since) + check(m["accessibleName"] == "Hit me" and not any(windows_with("box", "label")(w) for w in seen[since:]), + f"a label whose content alone changed is sent again, with no windows message for it: {m}") + + since = len(seen) + c.send({"t": "runLua", "code": "\n".join([ + 'hp = Geyser.Gauge:new({name = "hp", x = 10, y = 600, width = 200, height = 20})', + 'hp:setValue(30, 60, "30/60")', + ])}) + seen_or_next(c, windows_with("hp_front", "label", lambda w: w["w"] == 100), seen, since) + m = seen_or_next(c, lambda m: m["t"] == "label" and m["name"] == "hp_text" and "30/60" in m["text"], seen, since) + back, front, text = (window_of(seen, f"hp_{part}", "label") for part in ("back", "front", "text")) + check(back is not None and (back["parent"], back["x"], back["y"], back["w"], back["h"]) == ("", 10, 600, 200, 20) + and back.get("bgAlpha") == 100 and "bg" in back, f"a Geyser.Gauge's back label is the gauge, half see-through: {back}") + check((front["x"], front["y"], front["w"], front["h"]) == (10, 600, 100, 20) and "bgAlpha" not in front, + f"its front label's width against the back's is the value: {front}") + check(text is not None and (text["x"], text["w"]) == (10, 200) and text.get("bgAlpha") == 0 + and back["stack"] < front["stack"] < text["stack"], f"and its text label is transparent, on top: {text}") + since = len(seen) + c.send({"t": "runLua", "code": "hp:setValue(15, 60)"}) + seen_or_next(c, windows_with("hp_front", "label", lambda w: w["w"] == 50), seen, since) + c.send({"t": "runLua", "code": 'echo("gauge sent\\n")'}) + seen_or_next(c, shown("gauge sent"), seen, since) + front = window_of(seen, "hp_front", "label") + resent = [m["name"] for m in seen[since:] if m["t"] == "label"] + check((front["x"], front["w"]) == (10, 50) and not resent, + f"a new value resizes the front label, and sends no label whose content stayed: {front} {resent}") + + c.close() + c = Client(SOCK) + msgs = [] + c.wait_for(lambda m: m["t"] == "synced", collect=msgs) + seen += msgs + full = next((i for i, m in enumerate(msgs) if m["t"] == "windows" and m.get("full")), None) + names = {m["name"]: i for i, m in enumerate(msgs) if m["t"] == "label"} + check(full is not None and {"box", "pic", "ico", "hp_back", "hp_front", "hp_text"} <= set(names) + and all(i > full for i in names.values()), f"a UI that attaches gets every label after the windows: {sorted(names)}") + box = label_of(msgs, "box") + check(box is not None and box["accessibleName"] == "Hit me" and "new words" in box["text"], f"as it was: {box}") + + since = len(seen) + c.send({"t": "runLua", "code": 'deleteLabel("pic")'}) + m = seen_or_next(c, lambda m: m["t"] == "windows" and m.get("removed"), seen, since) + check({"name": "pic", "kind": "label"} in m["removed"] and label_of(seen[since:], "pic") is None, + f"a deleted label is removed, with no label message: {m}") + code = stop_engine(proc) + check(code == 0, f"the engine exits cleanly on SIGTERM: {code}") + + if __name__ == "__main__": main() diff --git a/protocol/src/main/kotlin/org/mudlet/android/protocol/Protocol.kt b/protocol/src/main/kotlin/org/mudlet/android/protocol/Protocol.kt index bb2bff6..97a0e12 100644 --- a/protocol/src/main/kotlin/org/mudlet/android/protocol/Protocol.kt +++ b/protocol/src/main/kotlin/org/mudlet/android/protocol/Protocol.kt @@ -6,10 +6,11 @@ import java.io.DataInputStream import java.io.EOFException import java.io.InputStream import java.io.OutputStream +import java.util.Base64 // Wire format shared with native/enginehost/SocketBridge.cpp, specified in docs/protocol.md: a 4-byte // big-endian length, then one UTF-8 JSON object whose "t" field is the message type. -const val PROTOCOL_VERSION = 3 +const val PROTOCOL_VERSION = 8 private const val MAX_FRAME_BYTES = 16 * 1024 * 1024 // A run's colours: -1 default, 0-15 ANSI palette, RUN_RGB | 0xRRGGBB exact @@ -26,9 +27,124 @@ const val RUN_BLINK = 64 const val RUN_FAST_BLINK = 128 const val RUN_CONCEALED = 256 +// Qt's mouse buttons and keyboard modifiers, as Requests.pointer takes them +const val POINTER_LEFT = 1 +const val POINTER_RIGHT = 2 +const val POINTER_MIDDLE = 4 +const val MODIFIER_SHIFT = 0x02000000 +const val MODIFIER_CONTROL = 0x04000000 +const val MODIFIER_ALT = 0x08000000 +const val MODIFIER_META = 0x10000000 + /** A profile as the engine lists it; the password is never part of one. */ data class Profile(val name: String, val host: String, val port: Int, val tls: Boolean, val login: String, val autoReconnect: Boolean) +/** A script window's kind, as upstream names it on the wire. */ +enum class WindowKind(val wire: String) { + MINI_CONSOLE("miniConsole"), + USER_WINDOW("userWindow"), + LABEL("label"), + COMMAND_LINE("commandLine"), + TEXT_BOX("textBox"), + SCROLL_BOX("scrollBox"), + MAPPER("mapper"), + MXP_FRAME("mxpFrame"), + ; + + companion object { + /** Null for a kind this app does not know, whose windows it leaves out. */ + fun fromWire(wire: String): WindowKind? = entries.firstOrNull { it.wire == wire } + } +} + +/** A script window is known by its name and kind, as a console's own command line has its console's name. */ +data class WindowKey(val name: String, val kind: WindowKind) + +/** A window a profile's scripts made, its geometry in Mudlet's pixels within its parent's (docs/protocol.md, `windows`). */ +data class ScriptWindow( + val name: String, + val kind: WindowKind, + /** Empty for the main window's area, a dock or an external MXP frame, which have no [parentKind]. */ + val parent: String, + val parentKind: WindowKind?, + val x: Int, + val y: Int, + val w: Int, + val h: Int, + val visible: Boolean, + val shown: Boolean, + val stack: Int, + /** As a run's colour; -1 for none. */ + val bg: Int, + /** [bg]'s alpha, from 0 (transparent) to 255, which a label's may be below. */ + val bgAlpha: Int, + val fontFamily: String?, + /** In points; 0 for none. */ + val fontSize: Double, + val inDock: Boolean, + /** A user window's. */ + val dock: Dock?, + /** A mini console's, user window's or MXP frame's. */ + val console: Console?, +) { + val key get() = WindowKey(name, kind) + + /** [area] is left, right, top or bottom. */ + data class Dock(val autoDock: Boolean, val area: String, val floating: Boolean, val title: String) + + /** [scrolledBackTo] is the row it is scrolled back to, null while it follows new lines. */ + data class Console(val scrolledBackTo: Int?, val columns: Int, val rows: Int, val scrollBar: Boolean, val timeStamps: Boolean) +} + +/** What a label shows, as THeadlessSession::labelContent() gives it (docs/protocol.md, `label`). */ +data class LabelContent( + val name: String, + /** Rich text, as echo() or setLabelText() gave it; empty while an image or movie shows in its place. */ + val text: String, + /** The file setBackgroundImage() gave it, a raster image or, with [svg], an SVG drawn under the text. */ + val image: String, + val svg: Boolean, + /** As a run's colour, -1 for none, with its alpha from 0 to 255. */ + val svgTint: Int, + val svgTintAlpha: Int, + /** In degrees. */ + val svgRotation: Double, + val svgShearX: Double, + val svgShearY: Double, + /** The file setMovie() gave it, shown in place of the text. */ + val movie: String, + val moviePaused: Boolean, + /** The frame a paused movie shows; -1 while it plays. */ + val movieFrame: Int, + /** In percent of the file's own speed. */ + val movieSpeed: Int, + /** The size scaleMovie() gave it; null for the file's own. */ + val movieW: Int?, + val movieH: Int?, + val toolTip: String, + /** The pointer passes through it to what is under it. */ + val clickThrough: Boolean, + val accessibleName: String, + /** It has a click or release callback, so a screen reader is told it is a button. */ + val accessibleButton: Boolean, + /** As setLabelStyleSheet() gave it, which the engine does not interpret. */ + val styleSheet: String, +) + +/** What a pointer did, as upstream's WindowPointerEvent names it on the wire. */ +enum class PointerType(val wire: String) { + PRESS("press"), + RELEASE("release"), + DOUBLE_CLICK("doubleClick"), + MOVE("move"), + ENTER("enter"), + LEAVE("leave"), + WHEEL("wheel"), +} + +/** A row of a script window's console: [runs] as [EngineMessage.Line]'s, but a bg of -1 is the window's bg. */ +data class WindowRow(val text: String, val runs: IntArray, val prompt: Boolean, val wrapped: Boolean) + sealed interface EngineMessage { data class Hello(val version: Int, val pid: Int, val instance: String) : EngineMessage data class Line(val seq: Long, val text: String, val runs: IntArray, val prompt: Boolean, val replay: Boolean) : EngineMessage @@ -54,6 +170,24 @@ sealed interface EngineMessage { /** A script set the whole input line to [text]. */ data class CommandLine(val text: String) : EngineMessage + + /** The script windows: with [full], all of them; otherwise those that changed, and the keys of those [removed]. */ + data class Windows(val full: Boolean, val windows: List, val removed: List) : EngineMessage + + /** Rows of a script window's console from row [first] of its [lineCount]; [replyTo] is the getWindowText id. */ + data class WindowText(val window: String, val lineCount: Int, val first: Int, val lines: List, val replyTo: Any?) : EngineMessage + + /** The main window's size the UI gave, the main console's grid, and where its text goes, in Mudlet's pixels. */ + data class MainArea(val w: Int, val h: Int, val columns: Int, val rows: Int, val textX: Int, val textY: Int, val textW: Int, val textH: Int) : EngineMessage + + /** A script set the command line [name] of a script window to [text]. */ + data class WindowCommandLine(val name: String, val text: String) : EngineMessage + + /** A label's content, new or changed; it follows the [Windows] that has the label. */ + data class Label(val label: LabelContent) : EngineMessage + + /** The answer to getImage: [png] is a PNG of [w] by [h] pixels, empty if it did not decode. */ + data class Image(val path: String, val png: ByteArray, val w: Int, val h: Int, val replyTo: Any?) : EngineMessage data class Unknown(val type: String) : EngineMessage } @@ -89,6 +223,43 @@ object Requests { fun openProfile(name: String, password: String?, connect: Boolean, id: Long): JSONObject = JSONObject() .put("t", "openProfile").put("name", name).put("connect", connect).put("id", id) .apply { if (password != null) put("password", password) } + + // The script windows'. A refusal is an Error naming the request. + fun setMainWindowSize(w: Int, h: Int): JSONObject = JSONObject().put("t", "setMainWindowSize").put("w", w).put("h", h) + + /** Answered by [EngineMessage.WindowText] with [id] as its replyTo; a negative [first] counts back from the end. */ + fun getWindowText(window: String, first: Int, count: Int, id: Long): JSONObject = JSONObject() + .put("t", "getWindowText").put("window", window).put("first", first).put("count", count).put("id", id) + + fun placeUserWindow(name: String, x: Int, y: Int, w: Int, h: Int, floating: Boolean): JSONObject = JSONObject() + .put("t", "placeUserWindow").put("name", name).put("x", x).put("y", y).put("w", w).put("h", h).put("floating", floating) + + fun closeUserWindow(name: String): JSONObject = JSONObject().put("t", "closeUserWindow").put("name", name) + fun sendWindowCommandLine(name: String, text: String): JSONObject = JSONObject().put("t", "sendWindowCommandLine").put("name", name).put("text", text) + + /** + * What the pointer did at [x], [y] in [window]'s pixels ("main" for the main console's area): [button] that + * changed and [buttons] held after it as POINTER_*, [modifiers] as MODIFIER_*, [wheelY] in eighths of a degree. + * Each of those is sent only when not 0. Answered only by an Error, when refused. + */ + fun pointer(window: String, type: PointerType, x: Double, y: Double, button: Int, buttons: Int, modifiers: Int, wheelY: Int): JSONObject = JSONObject() + .put("t", "pointer").put("window", window).put("type", type.wire).put("x", x).put("y", y) + .apply { + if (button != 0) put("button", button) + if (buttons != 0) put("buttons", buttons) + if (modifiers != 0) put("modifiers", modifiers) + if (wheelY != 0) put("wheelY", wheelY) + } + + /** Answered by [EngineMessage.Image] with [id] as its replyTo: the file at [path], scaled to [w] by [h], given both, or its [frame]. */ + fun getImage(path: String, w: Int?, h: Int?, frame: Int?, id: Long): JSONObject { + require((w == null) == (h == null)) { "getImage takes w and h together" } + return JSONObject().put("t", "getImage").put("path", path).put("id", id) + .apply { + if (w != null && h != null) put("w", w).put("h", h) + if (frame != null) put("frame", frame) + } + } } /** @@ -159,6 +330,37 @@ class FrameReader(input: InputStream) { "commandLine" -> EngineMessage.CommandLine(o.optString("text")) + "windows" -> EngineMessage.Windows( + o.optBoolean("full"), + objects(o.optJSONArray("windows")).mapNotNull(::parseWindow), + objects(o.optJSONArray("removed")).mapNotNull { r -> WindowKind.fromWire(r.optString("kind"))?.let { WindowKey(r.optString("name"), it) } }, + ) + + "windowText" -> EngineMessage.WindowText( + o.optString("window"), + o.optInt("lineCount"), + o.optInt("first"), + objects(o.optJSONArray("lines")).map { WindowRow(it.optString("text"), flattenRuns(it.optJSONArray("runs")), it.optBoolean("prompt"), it.optBoolean("wrapped")) }, + o.opt("re"), + ) + + "mainArea" -> EngineMessage.MainArea( + o.optInt("w"), + o.optInt("h"), + o.optInt("columns"), + o.optInt("rows"), + o.optInt("textX"), + o.optInt("textY"), + o.optInt("textW"), + o.optInt("textH"), + ) + + "windowCommandLine" -> EngineMessage.WindowCommandLine(o.optString("name"), o.optString("text")) + + "label" -> EngineMessage.Label(parseLabel(o)) + + "image" -> EngineMessage.Image(o.optString("path"), decodeBase64(o.optString("png")), o.optInt("w"), o.optInt("h"), o.opt("re")) + else -> EngineMessage.Unknown(t) } @@ -170,6 +372,72 @@ class FrameReader(input: InputStream) { } } + private fun objects(array: JSONArray?): List = if (array == null) emptyList() else List(array.length()) { array.optJSONObject(it) }.filterNotNull() + + // Null for a window of a kind this app does not know + private fun parseWindow(w: JSONObject): ScriptWindow? { + val kind = WindowKind.fromWire(w.optString("kind")) ?: return null + return ScriptWindow( + name = w.optString("name"), + kind = kind, + parent = w.optString("parent"), + parentKind = WindowKind.fromWire(w.optString("parentKind")), + x = w.optInt("x"), + y = w.optInt("y"), + w = w.optInt("w"), + h = w.optInt("h"), + visible = w.optBoolean("visible"), + shown = w.optBoolean("shown"), + stack = w.optInt("stack"), + bg = w.optInt("bg", -1), + bgAlpha = w.optInt("bgAlpha", 255), + fontFamily = w.optString("fontFamily").ifEmpty { null }, + fontSize = w.optDouble("fontSize", 0.0), + inDock = w.optBoolean("inDock"), + dock = if (w.has("dockArea")) { + ScriptWindow.Dock(w.optBoolean("autoDock", true), w.optString("dockArea"), w.optBoolean("dockFloating"), w.optString("dockTitle")) + } else { + null + }, + console = if (w.has("columns")) { + val scrolledBackTo = if (w.has("scrolledBackTo")) w.optInt("scrolledBackTo") else null + ScriptWindow.Console(scrolledBackTo, w.optInt("columns"), w.optInt("rows"), w.optBoolean("scrollBar"), w.optBoolean("timeStamps")) + } else { + null + }, + ) + } + + private fun parseLabel(l: JSONObject) = LabelContent( + name = l.optString("name"), + text = l.optString("text"), + image = l.optString("image"), + svg = l.optBoolean("svg"), + svgTint = l.optInt("svgTint", -1), + svgTintAlpha = l.optInt("svgTintAlpha", 255), + svgRotation = l.optDouble("svgRotation", 0.0), + svgShearX = l.optDouble("svgShearX", 0.0), + svgShearY = l.optDouble("svgShearY", 0.0), + movie = l.optString("movie"), + moviePaused = l.optBoolean("moviePaused"), + movieFrame = l.optInt("movieFrame", -1), + movieSpeed = l.optInt("movieSpeed", 100), + movieW = if (l.has("movieW")) l.optInt("movieW") else null, + movieH = if (l.has("movieH")) l.optInt("movieH") else null, + toolTip = l.optString("toolTip"), + clickThrough = l.optBoolean("clickThrough"), + accessibleName = l.optString("accessibleName"), + accessibleButton = l.optBoolean("accessibleButton"), + styleSheet = l.optString("styleSheet"), + ) + + // Empty for what is not base64, as an engine's bad frame should not end the link + private fun decodeBase64(text: String): ByteArray = try { + Base64.getDecoder().decode(text) + } catch (_: IllegalArgumentException) { + ByteArray(0) + } + private fun parseStrings(strings: JSONArray?): List = if (strings == null) emptyList() else List(strings.length()) { strings.optString(it) } // [[start, length, fg(, bg, attributes(, link))], ...] -> [start0, length0, fg0, bg0, attributes0, link0, start1, ...] diff --git a/protocol/src/test/kotlin/org/mudlet/android/protocol/EngineLinkJvmTest.kt b/protocol/src/test/kotlin/org/mudlet/android/protocol/EngineLinkJvmTest.kt index 3536da4..6c5e638 100644 --- a/protocol/src/test/kotlin/org/mudlet/android/protocol/EngineLinkJvmTest.kt +++ b/protocol/src/test/kotlin/org/mudlet/android/protocol/EngineLinkJvmTest.kt @@ -115,6 +115,28 @@ class EngineLinkJvmTest { private fun percentile(sorted: List, p: Int) = sorted[(sorted.size * p / 100).coerceAtMost(sorted.size - 1)] + @Test + fun theStandInHasNoScriptWindows() { + val link = newLink() + link.start() + next() + assertEquals(EngineMessage.Windows(true, emptyList(), emptyList()), next()) + next() + link.send(Requests.getWindowText("chat", -300, 300, 5)) + val refused = next() + assertEquals("getWindowText: there is no console named \"chat\"", refused.message) + assertEquals(5, (refused.replyTo as Number).toInt()) + link.send(Requests.placeUserWindow("side", 0, 0, 300, 200, false)) + assertEquals("placeUserWindow: the stand-in engine has no script windows", next().message) + link.send(Requests.pointer("box", PointerType.PRESS, 1.5, 2.0, POINTER_LEFT, POINTER_LEFT, 0, 0)) + assertEquals("pointer: the stand-in engine has no script windows", next().message) + link.send(Requests.getImage("/no/such.png", 16, 16, null, 6)) + val noImage = next() + assertEquals("getImage: the stand-in engine has no script windows", noImage.message) + assertEquals(6, (noImage.replyTo as Number).toInt()) + link.stop() + } + @Test fun protocolLatencyThroughputAndEngineDeath() { val link = newLink() diff --git a/protocol/src/test/kotlin/org/mudlet/android/protocol/ProtocolSpecTest.kt b/protocol/src/test/kotlin/org/mudlet/android/protocol/ProtocolSpecTest.kt index cd0d46f..38b9008 100644 --- a/protocol/src/test/kotlin/org/mudlet/android/protocol/ProtocolSpecTest.kt +++ b/protocol/src/test/kotlin/org/mudlet/android/protocol/ProtocolSpecTest.kt @@ -75,10 +75,12 @@ class ProtocolSpecTest { val args: List = method.parameterTypes.map { type -> when (type) { String::class.java -> "x" - Int::class.javaPrimitiveType -> 1 + Int::class.javaPrimitiveType, Int::class.javaObjectType -> 1 + Double::class.javaPrimitiveType -> 1.0 Long::class.javaPrimitiveType -> 1L Boolean::class.javaPrimitiveType -> true Profile::class.java -> Profile("x", "x", 1, true, "x", true) + PointerType::class.java -> PointerType.PRESS else -> error("Requests.${method.name} takes a ${type.name}: teach ProtocolSpecTest about it") } } diff --git a/protocol/src/test/kotlin/org/mudlet/android/protocol/ProtocolTest.kt b/protocol/src/test/kotlin/org/mudlet/android/protocol/ProtocolTest.kt index bdf646b..840172c 100644 --- a/protocol/src/test/kotlin/org/mudlet/android/protocol/ProtocolTest.kt +++ b/protocol/src/test/kotlin/org/mudlet/android/protocol/ProtocolTest.kt @@ -3,9 +3,11 @@ package org.mudlet.android.protocol import org.json.JSONObject import org.junit.Assert.assertArrayEquals import org.junit.Assert.assertEquals +import org.junit.Assert.assertThrows import org.junit.Assert.assertTrue import org.junit.Test import java.io.OutputStream +import java.util.Base64 /** The run encoding and message types of native/enginehost/SocketBridge.cpp, without an engine. */ class ProtocolTest { @@ -90,6 +92,139 @@ class ProtocolTest { assertEquals(true, Requests.complete("say", true, 7).getBoolean("fromHistory")) } + @Test + fun scriptWindowsParseWithTheirOptionalParts() { + val message = FrameReader.parse( + JSONObject( + """{"t":"windows","full":true,"windows":[""" + + """{"name":"side","kind":"userWindow","parent":"","x":0,"y":0,"w":300,"h":200,"visible":true,"shown":true,"stack":0,""" + + """"bg":${RUN_RGB or 0x102030},"fontFamily":"Mono","fontSize":10.5,"inDock":true,"autoDock":false,"dockArea":"left",""" + + """"dockFloating":true,"dockTitle":"Side","scrolledBackTo":12,"columns":40,"rows":10,"scrollBar":true,"timeStamps":false},""" + + """{"name":"side","kind":"commandLine","parent":"side","parentKind":"userWindow","x":0,"y":0,"w":0,"h":0,""" + + """"visible":true,"shown":false,"stack":1,"inDock":false},""" + + """{"name":"hp_text","kind":"label","parent":"","x":10,"y":600,"w":200,"h":20,"visible":true,"shown":true,"stack":3,""" + + """"bg":$RUN_RGB,"bgAlpha":0,"inDock":false},""" + + """{"name":"later","kind":"hologram","parent":"","x":0,"y":0,"w":1,"h":1,"visible":true,"shown":true,"stack":2,"inDock":false}],""" + + """"removed":[{"name":"chat","kind":"miniConsole"},{"name":"x","kind":"hologram"}]}""", + ), + ) as EngineMessage.Windows + assertTrue(message.full) + val side = ScriptWindow( + "side", WindowKind.USER_WINDOW, "", null, 0, 0, 300, 200, true, true, 0, RUN_RGB or 0x102030, 255, "Mono", 10.5, true, + ScriptWindow.Dock(false, "left", true, "Side"), ScriptWindow.Console(12, 40, 10, true, false), + ) + val line = ScriptWindow("side", WindowKind.COMMAND_LINE, "side", WindowKind.USER_WINDOW, 0, 0, 0, 0, true, false, 1, -1, 255, null, 0.0, false, null, null) + // A transparent background, as a gauge's text label has + val text = ScriptWindow("hp_text", WindowKind.LABEL, "", null, 10, 600, 200, 20, true, true, 3, RUN_RGB, 0, null, 0.0, false, null, null) + // A kind the app does not know is left out + assertEquals(listOf(side, line, text), message.windows) + assertEquals(WindowKey("side", WindowKind.COMMAND_LINE), line.key) + assertEquals(listOf(WindowKey("chat", WindowKind.MINI_CONSOLE)), message.removed) + val update = FrameReader.parse(JSONObject("""{"t":"windows","windows":[]}""")) as EngineMessage.Windows + assertEquals(EngineMessage.Windows(false, emptyList(), emptyList()), update) + } + + @Test + fun windowTextMainAreaAndCommandLinesParse() { + val text = FrameReader.parse( + JSONObject( + """{"t":"windowText","window":"chat","lineCount":3,"first":1,"re":"w1","lines":[""" + + """{"text":"hello","runs":[[0,5,${RUN_RGB or 0xff0000}]],"prompt":false,"wrapped":false},""" + + """{"text":"HP>","runs":[],"prompt":true,"wrapped":true}]}""", + ), + ) as EngineMessage.WindowText + assertEquals("chat", text.window) + assertEquals(3, text.lineCount) + assertEquals(1, text.first) + assertEquals("w1", text.replyTo) + assertEquals(listOf("hello", "HP>"), text.lines.map { it.text }) + assertArrayEquals(intArrayOf(0, 5, RUN_RGB or 0xff0000, -1, 0, 0), text.lines[0].runs) + assertTrue(text.lines[1].prompt && text.lines[1].wrapped) + assertEquals( + EngineMessage.MainArea(800, 600, 90, 30, 10, 0, 780, 600), + FrameReader.parse(JSONObject("""{"t":"mainArea","w":800,"h":600,"columns":90,"rows":30,"textX":10,"textY":0,"textW":780,"textH":600}""")), + ) + assertEquals(EngineMessage.WindowCommandLine("chat", "say hi"), FrameReader.parse(JSONObject("""{"t":"windowCommandLine","name":"chat","text":"say hi"}"""))) + } + + @Test + fun theScriptWindowRequestsCarryWhatTheEngineReads() { + assertEquals(mapOf("t" to "setMainWindowSize", "w" to 800, "h" to 600), Requests.setMainWindowSize(800, 600).toMap()) + assertEquals( + mapOf("t" to "getWindowText", "window" to "chat", "first" to -300, "count" to 300, "id" to 8L), + Requests.getWindowText("chat", -300, 300, 8).toMap(), + ) + assertEquals( + mapOf("t" to "placeUserWindow", "name" to "side", "x" to 1, "y" to 2, "w" to 3, "h" to 4, "floating" to true), + Requests.placeUserWindow("side", 1, 2, 3, 4, true).toMap(), + ) + assertEquals(mapOf("t" to "closeUserWindow", "name" to "side"), Requests.closeUserWindow("side").toMap()) + assertEquals(mapOf("t" to "sendWindowCommandLine", "name" to "chat", "text" to "look"), Requests.sendWindowCommandLine("chat", "look").toMap()) + } + + @Test + fun labelsParseWithTheirOptionalParts() { + val full = FrameReader.parse( + JSONObject( + """{"t":"label","name":"box","text":"hi","image":"/data/a.svg","svg":true,"svgTint":${RUN_RGB or 0xff0000},""" + + """"svgTintAlpha":128,"svgRotation":90.5,"svgShearX":0.25,"svgShearY":-0.5,"movie":"/data/b.gif","moviePaused":true,""" + + """"movieFrame":3,"movieSpeed":50,"movieW":64,"movieH":48,"toolTip":"tip","clickThrough":true,"accessibleName":"Box",""" + + """"accessibleButton":true,"styleSheet":"QLabel { color: red; }"}""", + ), + ) + assertEquals( + EngineMessage.Label( + LabelContent( + "box", "hi", "/data/a.svg", true, RUN_RGB or 0xff0000, 128, 90.5, 0.25, -0.5, "/data/b.gif", true, 3, 50, 64, 48, + "tip", true, "Box", true, "QLabel { color: red; }", + ), + ), + full, + ) + // The engine leaves out the parts at their defaults + val plain = FrameReader.parse( + JSONObject( + """{"t":"label","name":"box","text":"","image":"","svg":false,"movie":"","moviePaused":false,"movieFrame":-1,""" + + """"movieSpeed":100,"toolTip":"","clickThrough":false,"accessibleName":"","accessibleButton":false,"styleSheet":""}""", + ), + ) as EngineMessage.Label + assertEquals(LabelContent("box", "", "", false, -1, 255, 0.0, 0.0, 0.0, "", false, -1, 100, null, null, "", false, "", false, ""), plain.label) + } + + @Test + fun anImageDecodesItsPng() { + val png = byteArrayOf(0x89.toByte(), 'P'.code.toByte(), 'N'.code.toByte(), 'G'.code.toByte(), 0, -1) + val image = FrameReader.parse( + JSONObject().put("t", "image").put("path", "/data/a.png").put("png", Base64.getEncoder().encodeToString(png)).put("w", 4).put("h", 3).put("re", 9), + ) as EngineMessage.Image + assertEquals("/data/a.png", image.path) + assertArrayEquals(png, image.png) + assertEquals(4 to 3, image.w to image.h) + assertEquals(9, image.replyTo) + // Not base64: empty, and the link goes on + val broken = FrameReader.parse(JSONObject("""{"t":"image","path":"x","png":"not base64!","w":1,"h":1}""")) as EngineMessage.Image + assertEquals(0, broken.png.size) + } + + @Test + fun thePointerAndImageRequestsCarryWhatTheEngineReads() { + assertEquals( + mapOf("t" to "pointer", "window" to "box", "type" to "press", "x" to 1.5, "y" to 2.0, "button" to POINTER_LEFT, "buttons" to POINTER_LEFT), + Requests.pointer("box", PointerType.PRESS, 1.5, 2.0, POINTER_LEFT, POINTER_LEFT, 0, 0).toMap(), + ) + assertEquals( + mapOf("t" to "pointer", "window" to "main", "type" to "wheel", "x" to 0.0, "y" to 0.0, "modifiers" to (MODIFIER_SHIFT or MODIFIER_CONTROL), "wheelY" to -120), + Requests.pointer("main", PointerType.WHEEL, 0.0, 0.0, 0, 0, MODIFIER_SHIFT or MODIFIER_CONTROL, -120).toMap(), + ) + assertEquals("doubleClick", PointerType.DOUBLE_CLICK.wire) + assertEquals(mapOf("t" to "getImage", "path" to "a.png", "id" to 3L), Requests.getImage("a.png", null, null, null, 3).toMap()) + assertEquals( + mapOf("t" to "getImage", "path" to "a.gif", "w" to 16, "h" to 8, "frame" to 2, "id" to 4L), + Requests.getImage("a.gif", 16, 8, 2, 4).toMap(), + ) + assertThrows(IllegalArgumentException::class.java) { Requests.getImage("a.png", 16, null, null, 5) } + } + @Test fun aFrameIsOneWriteAndNeverFlushed() { val writes = ArrayList()