0.5: extras, GPU and offline env install; packaging pipeline; env-setup/updater/report UI

README.md

@@ -63,13 +63,23 @@ it into the project at a fixed path, then alias and `paths` through the link:
 python -m de_shell.js        # prints …/de_shell/js
 ```
 
-Autopilot's `electron/scripts/shell-link.mjs` is the reference: it makes
-`electron/shell` a junction (a symlink off Windows) to that folder, runs from
-npm's postinstall and from the vite config on every build, and re-points a
-stale link rather than trusting it. With it in place:
+The shell makes the link itself:
+
+```
+uv run python -m de_shell.js link electron          # electron/shell -> …/de_shell/js
+```
+
+That makes `electron/shell` a junction (a symlink off Windows) to that
+folder. It re-points a stale link rather than trusting it, and refuses to
+replace a real directory; `--soft` turns the refusal into a warning, which is
+what npm's postinstall wants. The same logic ships as
+`shell/packaging/shell-link.mjs` for the vite config to call on every build,
+once the first link exists. The copies the apps carried as
+`electron/scripts/shell-link.mjs` can go. With the link in place:
 
 ```ts
 // electron.vite.config.ts
+import { ensureShellLink } from './shell/packaging/shell-link.mjs'
 const shell = ensureShellLink(__dirname)
 resolve: { alias: { '@de/shell-main': resolve(shell, 'main', 'index.ts') },
            dedupe: ['react', 'react-dom'] }
@@ -92,6 +102,92 @@ All three apps are wired this way: Autopilot, SpyDE
 and Ground Crew ([directelectron/de_ground_crew#183](https://github.com/directelectron/de_ground_crew/pull/183)),
 pinned to the `0.2` line.
 
+## Shipping an app
+
+An app ships as a per-user NSIS installer (an AppImage on Linux, a dmg on
+macOS). The installer is about 150 MB because the Python environment is **not**
+inside it. On first launch the shell builds the environment into `userData`,
+and an update re-syncs it only when the lock changed. Everything the pipeline
+needs lives in `shell/packaging/`.
+
+**The app provides:**
+
+* **A `deShell` block in `electron/package.json`.** This is read by
+  `bundle-python.mjs` and `build-wheelhouse.mjs`:
+
+  ```json
+  "deShell": { "appId": "groundcrew", "pythonDist": "de-ground-crew",
+               "pythonVersion": "3.13", "wheelhouseExtras": ["gpu"] }
+  ```
+
+* **`electron/electron-builder.yml`.** It extends the shell's base and adds
+  only what is the app's own:
+
+  ```yaml
+  extends: shell/packaging/electron-builder.base.yml
+  appId: com.directelectron.groundcrew
+  productName: Ground Crew
+  publish: { provider: github, owner: directelectron, repo: de_ground_crew }
+  ```
+
+* **`.github/workflows/release.yml`.** It calls the shared pipeline on a tag.
+  The pipeline runs in the app's repo and publishes to the app's releases:
+
+  ```yaml
+  on: { push: { tags: ["v*"] } }
+  jobs:
+    release:
+      permissions: { contents: write }
+      uses: directelectron/de-shell/.github/workflows/electron-release.yml@v0.5.0
+      with: { app-name: Ground Crew, wheelhouse-extras: gpu }
+      secrets: inherit
+  ```
+
+  The workflow links the shell with `uv run --frozen --no-dev`, so it syncs the
+  app's locked environment on the runner. It also attaches a
+  `<App>-<ver>-wheels-win_amd64.zip` offline wheelhouse to the release, and
+  signs Windows builds with Azure Trusted Signing when the `AZURE_*` secrets
+  and the `azure-*` inputs are set.
+
+* **Main-process wiring:**
+
+  ```ts
+  configureShell({ appId: 'groundcrew', appName: 'Ground Crew',
+                   pythonModule: 'de_groundcrew', pythonDist: 'de-ground-crew',
+                   gpuExtras: ['gpu'] })
+  registerShellDialogs(); registerUpdaterIpc(); registerProblemReportIpc()
+  const py = app.isPackaged
+    ? await runEnvSetup(win, { isPackaged: true, resourcesPath: process.resourcesPath,
+                               projectRoot, userData: app.getPath('userData') })
+    : await resolvePythonEnv({ isPackaged: false, resourcesPath: '', projectRoot, userData: '' })
+  ```
+
+  The renderer shows `EnvSetupOverlay` while `state.envSetup` is set, and
+  `UpdateCard`/`UpdateDialog` and `ReportProblemDialog` where the app wants
+  them.
+
+**Extras and the GPU.** `pythonExtras` are always installed. `gpuExtras` are
+installed when `nvidia-smi` finds a device. `<APPID>_GPU=0|1` overrides the
+probe either way.
+
+**Offline sites.** Unzip the release's wheelhouse into the install
+folder (next to the app's `.exe`) or into `userData/wheelhouse`, or let the
+user choose it with "Install from a folder…" when the online install fails. The first launch then installs with no network:
+
+* the interpreter comes from the wheelhouse's python-build-standalone mirror;
+* the packages are the exported lock, installed with `uv pip install --offline
+  --no-index --require-hashes`.
+
+A frozen `uv sync --offline --find-links` cannot do this, because uv looks for
+registry-locked wheels only in its cache. A GPU machine given a wheelhouse
+built without the GPU extra installs CPU-only and remembers that. It does not
+fail.
+
+**No residue.** The env, the uv cache and uv's managed interpreters all live
+under `userData`. The uninstaller offers to remove them. The answer defaults to
+No, and is always No for silent runs and updates, so an update never
+re-downloads the environment.
+
 ## Developing it
 
 ```bash

de_shell/js/__init__.py

@@ -1,6 +1,6 @@
 """The TypeScript half of the shell, shipped inside the wheel.
 
-Four folders, one per Electron target, each a self-contained tree of
+Four folders, one per Electron target (plus ``packaging/``, below), each a self-contained tree of
 TypeScript that the consuming app's bundler compiles:
 
 * ``main/``     — the Electron main process: the window and menus, the Python
@@ -18,14 +18,20 @@
 (``python -m de_shell.js``) and links it into its Electron project — see the
 README — which also makes an editable install live-editable.
 
+``packaging/`` is not compiled into the app: it is the build-time half — the
+Node scripts and config an app runs from its ``electron/`` directory to ship an
+installer (``bundle-python.mjs``, ``build-wheelhouse.mjs``, ``shell-link.mjs``,
+``installer.nsh``, ``electron-builder.base.yml``). ``python -m de_shell.js link
+<electronDir>`` creates the ``shell`` link without needing any of them.
+
 The peer dependencies (react, electron, electron-updater, @playwright/test) are
 the app's to declare; every app already does.
 """
 from __future__ import annotations
 
 from pathlib import Path
 
-TARGETS = ("main", "preload", "renderer", "testing")
+TARGETS = ("main", "preload", "renderer", "testing", "packaging")
 
 
 def path(target: str | None = None) -> Path:

de_shell/js/__main__.py

@@ -1,4 +1,115 @@
-"""``python -m de_shell.js`` prints where the TypeScript is, for build scripts."""
+"""``python -m de_shell.js`` — where the TypeScript is, and how to link it.
+
+    python -m de_shell.js                          print …/de_shell/js (build scripts rely on this)
+    python -m de_shell.js link <electronDir> [--soft]
+                                                    make <electronDir>/shell point at it
+
+``link`` is the Python twin of ``packaging/shell-link.mjs``: a junction on
+Windows (no admin or developer mode needed, unlike a symlink), a symlink
+elsewhere. CI and ``postinstall`` use it so an app does not need to carry its
+own copy of the script. Semantics, identical to the mjs:
+
+* an up-to-date link is left alone;
+* a stale link (a venv that moved, an editable install that was re-pointed) is
+  re-pointed, never trusted;
+* a REAL directory at the link path is refused, not deleted — someone put it
+  there, and a build script has no business removing it;
+* ``--soft`` turns every failure into a warning and exit 0, for an ``npm
+  install`` that runs before the first ``uv sync``.
+"""
+from __future__ import annotations
+
+import os
+import stat
+import subprocess
+import sys
+from pathlib import Path
+
 from . import path
 
-print(path())
+_REPARSE_POINT = getattr(stat, "FILE_ATTRIBUTE_REPARSE_POINT", 0x400)
+
+
+class LinkError(RuntimeError):
+    pass
+
+
+def _is_link(p: Path) -> bool:
+    """A symlink OR a Windows junction (``Path.is_symlink`` misses junctions
+    before 3.12, and a junction is exactly what we create there)."""
+    try:
+        st = os.lstat(p)
+    except OSError:
+        return False
+    if stat.S_ISLNK(st.st_mode):
+        return True
+    return bool(getattr(st, "st_file_attributes", 0) & _REPARSE_POINT)
+
+
+def _unlink(p: Path) -> None:
+    # rmdir removes a junction / directory symlink without touching its target
+    # on Windows; unlink is the POSIX spelling. Try both.
+    try:
+        os.unlink(p)
+    except OSError:
+        os.rmdir(p)
+
+
+def _make_link(link: Path, target: Path) -> None:
+    if os.name != "nt":
+        os.symlink(target, link, target_is_directory=True)
+        return
+    try:
+        import _winapi
+        _winapi.CreateJunction(str(target), str(link))
+    except (ImportError, AttributeError, OSError):
+        r = subprocess.run(["cmd", "/c", "mklink", "/J", str(link), str(target)],
+                           capture_output=True, text=True)
+        if r.returncode != 0:
+            raise LinkError(f"could not create a junction at {link}: {(r.stderr or r.stdout).strip()}")
+
+
+def link(electron_dir: str | os.PathLike, *, quiet: bool = False) -> Path:
+    """Make ``<electron_dir>/shell`` point at this install's ``de_shell/js``."""
+    electron_dir = Path(electron_dir)
+    if not electron_dir.is_dir():
+        raise LinkError(f"{electron_dir} is not a directory")
+    target = path().resolve()
+    lnk = electron_dir / "shell"
+    if _is_link(lnk):
+        if lnk.resolve() == target:
+            return lnk
+        _unlink(lnk)
+    elif lnk.exists():
+        raise LinkError(f"{lnk} exists and is not a link; refusing to replace it. Remove it by hand.")
+    _make_link(lnk, target)
+    if not quiet:
+        print(f"[shell-link] {lnk} -> {target}")
+    return lnk
+
+
+def main(argv: list[str] | None = None) -> int:
+    argv = sys.argv[1:] if argv is None else argv
+    if not argv:
+        print(path())
+        return 0
+    if argv[0] != "link":
+        print(f"usage: python -m de_shell.js [link <electronDir> [--soft]]", file=sys.stderr)
+        return 2
+    soft = "--soft" in argv
+    rest = [a for a in argv[1:] if a != "--soft"]
+    if len(rest) != 1:
+        print("usage: python -m de_shell.js link <electronDir> [--soft]", file=sys.stderr)
+        return 2
+    try:
+        link(rest[0])
+    except (LinkError, OSError) as err:
+        if not soft:
+            print(f"[shell-link] {err}", file=sys.stderr)
+            return 1
+        print(f"[shell-link] not linked yet: {err}", file=sys.stderr)
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

de_shell/js/main/config.ts

@@ -24,6 +24,16 @@ export interface ShellConfig {
    *  (e.g. module `spyde` / dist `spyde`). Used for the pre-built wheel prefix
    *  and the SETUPTOOLS_SCM_PRETEND_VERSION_FOR_<NAME> env var. */
   pythonDist?: string
+  /** Optional-dependency groups (`[project.optional-dependencies]`) the packaged
+   *  first-run install ALWAYS adds, each as `uv sync --extra <name>`. Changing
+   *  this list changes the lock stamp, so existing installs re-sync once. */
+  pythonExtras?: string[]
+  /** Extras added ONLY when an NVIDIA GPU is detected (`nvidia-smi -L`), e.g.
+   *  Ground Crew's CuPy stack behind `gpu`. The probe is skipped entirely when
+   *  this is empty. Overridable per run with `<APPID>_GPU=0|1` or
+   *  `EnsureOptions.gpu`. A machine that gains a GPU later re-syncs, because the
+   *  effective extras are part of the lock stamp. */
+  gpuExtras?: string[]
 }
 
 let _config: ShellConfig | null = null

de_shell/js/main/envSetup.ts

@@ -0,0 +1,79 @@
+/**
+ * envSetup.ts — `runEnvSetup`: what a packaged app calls INSTEAD of
+ * `resolvePythonEnv` directly.
+ *
+ * It does the same resolve, but narrates it to the renderer as `env_setup`
+ * shell messages (so the first run shows a real progress overlay rather than a
+ * blank window) and survives failure by asking the user what to do — retry,
+ * install from a local wheelhouse folder, or quit. The state machine itself is
+ * envSetupMachine.ts; this file is only the Electron glue: the IPC answer
+ * channel, the native folder dialog, and app.quit.
+ *
+ * Messages ride the window's normal message pipe (`sendToRenderer`), so they
+ * are buffered until the renderer listens — a progress line sent before React
+ * mounted is not lost.
+ */
+import { app, dialog, ipcMain } from 'electron'
+
+import { channel, shellConfig } from './config'
+import { runEnvSetupMachine } from './envSetupMachine'
+import type { EnvSetupAnswer } from './envSetupMachine'
+import { resolvePythonEnv } from './pythonEnv'
+import type { EnsureOptions, ResolvedPython } from './pythonEnv'
+import type { ShellWindow } from './window'
+
+export type { EnvSetupAnswer, EnvSetupWireMessage } from './envSetupMachine'
+export { EnvSetupQuitError } from './envSetupMachine'
+
+const ANSWERS: readonly EnvSetupAnswer[] = ['retry', 'use-wheelhouse', 'quit']
+
+/** The answer the machine is currently waiting on, if any. An answer arriving
+ *  while nothing waits (a double click, a stale renderer) is dropped. */
+let waiter: ((a: EnvSetupAnswer) => void) | null = null
+let listening = false
+
+function listenForAnswers(): void {
+  if (listening) return
+  listening = true
+  ipcMain.on(channel('env-setup-answer'), (_e, answer: unknown) => {
+    if (!ANSWERS.includes(answer as EnvSetupAnswer)) return
+    const w = waiter
+    waiter = null
+    w?.(answer as EnvSetupAnswer)
+  })
+}
+
+/**
+ * Resolve the Python environment with progress and a recovery path.
+ *
+ * `opts.wheelhouse`, if given, is used for the first attempt; later attempts use
+ * whatever folder the user picks. Rejects only with EnvSetupQuitError.
+ */
+export function runEnvSetup(
+  win: Pick<ShellWindow, 'sendToRenderer' | 'get'>,
+  opts: EnsureOptions,
+): Promise<ResolvedPython> {
+  listenForAnswers()
+  return runEnvSetupMachine<ResolvedPython>({
+    initialWheelhouse: opts.wheelhouse ?? null,
+    run: (wheelhouse, onProgress) => resolvePythonEnv({
+      ...opts,
+      wheelhouse,
+      // Keep the app's own sink working alongside ours.
+      onProgress: (line: string) => { opts.onProgress?.(line); onProgress(line) },
+    }),
+    emit: (msg) => win.sendToRenderer({ ...msg }),
+    waitForAnswer: () => new Promise<EnvSetupAnswer>((resolve) => { waiter = resolve }),
+    pickFolder: async () => {
+      const parent = win.get()
+      const options = {
+        title: `Choose the folder of packages to install ${shellConfig().appName} from`,
+        properties: ['openDirectory' as const],
+      }
+      const r = parent ? await dialog.showOpenDialog(parent, options)
+        : await dialog.showOpenDialog(options)
+      return r.canceled || !r.filePaths.length ? null : r.filePaths[0]
+    },
+    quit: () => app.quit(),
+  })
+}