diff --git a/CHANGELOG.md b/CHANGELOG.md index 7c4902e..3c9f3b7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -29,9 +29,14 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). extracted folder; like the portable exe, it does not update itself. The Setup installer is still the recommended download. - The AI engine remembers the GPU backend that worked (`gpu-backend-cache.json` in the app's data - folder) and tries it first on the next launch. An app update, a different GPU, a change in the GPU - runtimes node-llama-cpp finds (such as installing a CUDA toolkit), or 14 days make it run the full - backend search again. + folder) and tries it first on the next launch. An app update, a different GPU or graphics driver + version, a change in the GPU runtimes node-llama-cpp finds (such as installing a CUDA toolkit), or + 14 days make it run the full backend search again. The optional embedding worker + (`STRATOSORT_ENABLE_EMBEDDING_WORKER`) uses the same record and supported-backend filter. +- The Windows x64 build no longer ships node-llama-cpp's arm64 prebuilt (about 11 MB). +- On Windows and macOS the app no longer forces Chromium's `enable-gpu-rasterization` and + `enable-zero-copy` switches, so Chromium's per-driver GPU blocklist applies. Linux keeps forcing + them, as before. - After three GPU process exits in one session, software rendering now stays on for up to 3 days (`.gpu-fallback` in the app's data folder) instead of switching back on the next launch, which made launches alternate between slow and fast on a failing driver. Delete the file while the app diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 62a45cf..44a70f4 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -171,8 +171,9 @@ Orchestrates the boot sequence with progress reporting to the splash screen: on read. `simple-main.js` calls `clearGpuFallbackAfterStableRun()` `TIMEOUTS.GPU_FALLBACK_STABLE_RUN` (30 s) after creating the window, next to the `.crash-timestamps` reset; it deletes the flag only for a hardware-accelerated session with no GPU - process exit and a visible window, so a software-rendered session cannot clear it. Chromium's GPU - rasterization and zero-copy switches are left at their per-driver defaults. + process exit and a visible window, so a software-rendered session cannot clear it. On Windows and + macOS Chromium's GPU rasterization and zero-copy switches are left at their per-driver defaults; + Linux keeps forcing both, as earlier releases did (`getForcedRasterSwitches()`). - **First-paint gate** (`core/firstPaintGate.js`): coordinated startup and `runBackgroundSetup()` wait until the window has been shown or its document has finished loading, capped at `TIMEOUTS.FIRST_PAINT_GATE` (3 s), so the synchronous stretches of AI bring-up (node-llama-cpp @@ -181,15 +182,15 @@ Orchestrates the boot sequence with progress reporting to the splash screen: - **Last-known-good AI backend**: `LlamaService` hands `initLlamaWithBackend()` (`utils/llamaBackendSelector.js`) a record from `utils/llamaBackendCache.js`, stored as `userData/gpu-backend-cache.json`. When the record's fingerprint (platform, arch, app version, - node-llama-cpp version, GPU type and name; `driverVersion` stays null until `GPUMonitor` reports - one) and the GPU backends `getLlamaGpuTypes('supported')` reports both match what was saved, and - it is under 14 days old, the remembered GPU backend moves to the front of the candidate list - before the node-llama-cpp support filter runs. The supported set is probed before the record is - read, so a Vulkan record saved while the CUDA runtime was missing is not used once CUDA becomes - available. It only reorders: on Windows node-llama-cpp still test-loads the binary, a failed - attempt of the remembered backend deletes the record, and a GPU backend that differs from the - record is saved (CPU never is). `STRATOSORT_LLM_GPU_BACKEND` and software or safe mode bypass the - record entirely. + node-llama-cpp version, GPU type, name and driver version, which `GPUMonitor` reads from + `nvidia-smi` or the Windows video controller) and the GPU backends `getLlamaGpuTypes('supported')` + reports both match what was saved, and it is under 14 days old, the remembered GPU backend moves + to the front of the candidate list before the node-llama-cpp support filter runs. The supported + set is probed before the record is read, so a Vulkan record saved while the CUDA runtime was + missing is not used once CUDA becomes available. It only reorders: on Windows node-llama-cpp still + test-loads the binary, a failed attempt of the remembered backend deletes the record, and a GPU + backend that differs from the record is saved (CPU never is). `STRATOSORT_LLM_GPU_BACKEND` and + software or safe mode bypass the record entirely. - **Phases** (progress reported to splash): Starting (0%) -> Preflight checks (10%) -> Migration (12%) -> Services -> Models -> App Services -> Ready (100%). See `reportProgress(...)` calls in `StartupManagerCore.js` for the authoritative sequence. diff --git a/docs/CONFIG.md b/docs/CONFIG.md index 72667d3..289fe0e 100644 --- a/docs/CONFIG.md +++ b/docs/CONFIG.md @@ -120,7 +120,10 @@ concurrency and VRAM is accounted for by the model memory manager. Setting `STRATOSORT_ENABLE_EMBEDDING_WORKER=true` starts a separate `embeddingWorker` thread that loads its own `node-llama-cpp` stack. That pool runs with `maxThreads: 1`, which caps parallel embedding throughput, and it can hold a second copy of the embedding model in VRAM that the -main-process memory manager does not see. Enable it only for isolation experiments or debugging. +main-process memory manager does not see. Enable it only for isolation experiments or debugging. The +worker selects its GPU backend the same way as the main process (supported-backend filter and the +remembered backend from `gpu-backend-cache.json`, read-only), so it does not re-test GPU binaries +the main process has already ruled out. The two queue waits are off by default for the same throughput reason. `STRATOSORT_ANALYSIS_EMBEDDING_QUEUE_WAIT=true` restores the per-file `waitForAnalysisQueueCapacity` @@ -160,8 +163,11 @@ These settings are configurable via **Settings > Performance > Processing limits | `STRATOSORT_DISABLE_BACKGROUND_THROTTLING` | `0` | Set `1` to disable Chromium background-page throttling so analysis keeps running at full speed when the window is hidden. Increases idle CPU. | `STRATOSORT_FORCE_SOFTWARE_GPU`, `ELECTRON_FORCE_SOFTWARE` and `--safe-mode` also run the AI engine -on the CPU. The app no longer appends Chromium's `enable-gpu-rasterization` or `enable-zero-copy` -switches, so GPU rasterization follows Chromium's own per-driver decision. +on the CPU. On Windows and macOS the app no longer appends Chromium's `enable-gpu-rasterization` or +`enable-zero-copy` switches, so GPU rasterization follows Chromium's own per-driver decision. Linux +still appends both, as every earlier release did on all platforms, because Chromium's Linux default +varies with version and driver and the AppImage has not been measured without them. Software mode +appends neither. **Automatic software fallback (`.gpu-fallback`).** After three GPU process exits in one session, each within an hour of the one before (the count restarts after an hour with no exit), the app @@ -441,21 +447,25 @@ They are listed here for support engineers and contributors who need to debug ed **Last-known-good GPU backend (`gpu-backend-cache.json`).** After the text and embedding engine settles on a GPU backend (`cuda`, `vulkan` or `metal`), the app saves it to `gpu-backend-cache.json` in its data directory together with a fingerprint: platform, architecture, app version, -node-llama-cpp version, and the detected GPU type and name. (It has a driver version field, but the -GPU probe does not report a driver version yet, so a driver update alone does not reset the record.) -The record also keeps the GPU backends node-llama-cpp reported as supported at the time, so -installing a CUDA toolkit (or anything else that changes that set) makes the next launch run the -full ladder again instead of keeping an earlier Vulkan result first. On later launches a record -whose fingerprint and supported set match and that is less than 14 days old puts that backend first -in the candidate list (`[LlamaService] Llama backend candidates ordered by cached selection`), so -candidates that failed before are not tested again. It only changes the order: on Windows -node-llama-cpp still test-loads the binary, and if the remembered backend fails, the record is -deleted (`[LlamaService] Cleared cached Llama backend after it failed`) and the normal ladder -continues. A CPU result is never saved, and an unchanged record is not rewritten; a new or changed -GPU result logs `[LlamaService] Saved Llama backend selection for the next launch`. +node-llama-cpp version, and the detected GPU type, name and driver version (from `nvidia-smi` on +NVIDIA GPUs, from Windows' video controller entry otherwise; Linux without `nvidia-smi` and macOS +report none). A driver update therefore makes the next launch run the full ladder. The record also +keeps the GPU backends node-llama-cpp reported as supported at the time, so installing a CUDA +toolkit (or anything else that changes that set) makes the next launch run the full ladder again +instead of keeping an earlier Vulkan result first. On later launches a record whose fingerprint and +supported set match and that is less than 14 days old puts that backend first in the candidate list +(`[LlamaService] Llama backend candidates ordered by cached selection`), so candidates that failed +before are not tested again. It only changes the order: on Windows node-llama-cpp still test-loads +the binary, and if the remembered backend fails, the record is deleted +(`[LlamaService] Cleared cached Llama backend after it failed`) and the normal ladder continues. A +CPU result is never saved, and an unchanged record is not rewritten; a new or changed GPU result +logs `[LlamaService] Saved Llama backend selection for the next launch`. `STRATOSORT_LLM_GPU_BACKEND`, `STRATOSORT_FORCE_SOFTWARE_GPU`, `ELECTRON_FORCE_SOFTWARE` and `--safe-mode` bypass the record: it is neither read nor changed while they are set. Deleting the -file while the app is closed makes the next launch run the full ladder. +file while the app is closed makes the next launch run the full ladder. With +`STRATOSORT_ENABLE_EMBEDDING_WORKER=true` the worker thread reads the same record and applies the +same supported-backend filter, but only the main process saves or deletes the record; a failed +attempt there logs `[EmbeddingWorker] Cached Llama backend failed; record left to its owner`. ### Storage and persistence diff --git a/docs/DEBUGGING_GUIDE.md b/docs/DEBUGGING_GUIDE.md index 7637394..58c33cd 100644 --- a/docs/DEBUGGING_GUIDE.md +++ b/docs/DEBUGGING_GUIDE.md @@ -261,16 +261,16 @@ result and the GPU fallback decision at module load) are not in the file. `[LlamaService] Llama backend init failed` lines once the record exists. On Windows node-llama-cpp still test-loads the chosen binary in a child process, so the record saves the cost of candidates that fail, not of the one that works. The record is not used on the very - first launch (there is none yet), after an app update, when the detected GPU name or type - changes, when the GPU backends node-llama-cpp finds runtimes for change (for example after a - CUDA toolkit is installed), or once it is 14 days old; the full ladder then runs and a GPU - result is saved again. While `STRATOSORT_LLM_GPU_BACKEND`, `STRATOSORT_FORCE_SOFTWARE_GPU`, - `ELECTRON_FORCE_SOFTWARE` or `--safe-mode` chooses the backend, the record is neither read nor - changed. A CPU result is never saved. Deleting the file while the app is closed makes the next - launch run the full ladder. Why a record was not used is logged as - `[LlamaBackendCache] Ignoring cached backend record` with `reason` (`missing`, `unreadable`, - `corrupt`, `invalid`, `fingerprint-mismatch`, `supported-backends-changed` or `expired`) at - debug level, which packaged builds do not write by default. + first launch (there is none yet), after an app update, when the detected GPU name, type or + driver version changes, when the GPU backends node-llama-cpp finds runtimes for change (for + example after a CUDA toolkit is installed), or once it is 14 days old; the full ladder then + runs and a GPU result is saved again. While `STRATOSORT_LLM_GPU_BACKEND`, + `STRATOSORT_FORCE_SOFTWARE_GPU`, `ELECTRON_FORCE_SOFTWARE` or `--safe-mode` chooses the + backend, the record is neither read nor changed. A CPU result is never saved. Deleting the file + while the app is closed makes the next launch run the full ladder. Why a record was not used is + logged as `[LlamaBackendCache] Ignoring cached backend record` with `reason` (`missing`, + `unreadable`, `corrupt`, `invalid`, `fingerprint-mismatch`, `supported-backends-changed` or + `expired`) at debug level, which packaged builds do not write by default. 4. **Several `StratoSort Stack.exe` processes are normal.** In Task Manager, Details tab, add the Command line column (right-click a column header, Select columns). During a launch expect: one `StratoSort Stack.exe` without `--type`, one with `--type=gpu-process`, one `--type=renderer`, diff --git a/docs/RELEASING.md b/docs/RELEASING.md index 9c73c5a..4885ef0 100644 --- a/docs/RELEASING.md +++ b/docs/RELEASING.md @@ -423,10 +423,12 @@ the on-disk sizes in `node_modules`; the download shrinks less, by an estimated the installer compresses the source files well). Update the download size in [INSTALL_GUIDE.md](INSTALL_GUIDE.md) after the next packaged build. `verify-packaged-artifacts.js` checks that `llama/binariesGithubRelease.json` (read by node-llama-cpp on import) still ships and -that `llama/gitRelease.bundle` does not. `@node-llama-cpp/win-arm64` (11 MB) also lands in the x64 -build, because its package declares `cpu: ["arm64", "x64"]` and npm installs it on x64 runners; it -cannot be excluded from the shared `files` list without also removing it from the arm64 build, which -inherits that list through `extends`. +that `llama/gitRelease.bundle` does not. `@node-llama-cpp/win-arm64` (11 MB) declares +`cpu: ["arm64", "x64"]`, so npm installs it on x64 runners too. `win.files` excludes the other +arch's llama prebuilts with `!**/node_modules/@node-llama-cpp/win-!(${arch}|${arch}-*)/**`: +electron-builder expands `${arch}` per packed arch, so the x64 build drops `win-arm64` and the arm64 +build (which inherits `win.files` through `extends`) drops any `win-x64*` package. +`verify-packaged-artifacts.js` fails if a package for the other arch still holds files. ## Signing And Notarization Requirements diff --git a/electron-builder.json b/electron-builder.json index 722df97..cb0bcdf 100644 --- a/electron-builder.json +++ b/electron-builder.json @@ -89,7 +89,8 @@ "icon": "assets/icons/win/icon.ico", "target": ["nsis", "zip"], "files": [ - "!**/node_modules/better-sqlite3/prebuilds/{darwin,linux,linuxmusl}-*.node" + "!**/node_modules/better-sqlite3/prebuilds/{darwin,linux,linuxmusl}-*.node", + "!**/node_modules/@node-llama-cpp/win-!(${arch}|${arch}-*)/**" ] }, "nsis": { diff --git a/scripts/verify-packaged-artifacts.js b/scripts/verify-packaged-artifacts.js index 74e6376..678cd33 100644 --- a/scripts/verify-packaged-artifacts.js +++ b/scripts/verify-packaged-artifacts.js @@ -105,6 +105,27 @@ function getRequiredLlamaPrebuiltPackages(platform, arch) { ]; } +/** + * node-llama-cpp prebuilt packages for the OTHER Windows arch, which `win.files` + * excludes through `win-!(${arch}|${arch}-*)`. @node-llama-cpp/win-arm64 + * declares `cpu: ["arm64", "x64"]`, so npm installs it on x64 runners too. + */ +function getForeignArchLlamaPrebuiltPackages(platform, arch) { + const platformKey = String(platform || '').toLowerCase(); + if (platformKey !== 'win' && platformKey !== 'win32') return []; + return String(arch || '').toLowerCase() === 'arm64' + ? getRequiredLlamaPrebuiltPackages(platform, 'x64') + : getRequiredLlamaPrebuiltPackages(platform, 'arm64'); +} + +/** True when `dir` holds at least one file at any depth. */ +function containsAnyFile(dir) { + if (!fs.existsSync(dir)) return false; + return fs + .readdirSync(dir, { withFileTypes: true }) + .some((entry) => (entry.isDirectory() ? containsAnyFile(path.join(dir, entry.name)) : true)); +} + /** * The Windows x64 no-install artifact (electron-builder `zip` target, named by * the top-level artifactName). arm64 ships the installer only. @@ -256,6 +277,15 @@ function verifyLlamaPrebuiltPackages(unpackedNodeModulesRoot, platform, arch) { `node-llama-cpp prebuilt package ${packageName}` ); }); + getForeignArchLlamaPrebuiltPackages(platform, arch).forEach((packageName) => { + const packageDir = path.join(unpackedNodeModulesRoot, ...packageName.split('/')); + if (containsAnyFile(packageDir)) { + throw new Error( + `node-llama-cpp prebuilt package ${packageName} for another arch was packaged: ` + + `${packageDir}. Check the @node-llama-cpp exclusion in electron-builder.json "win.files".` + ); + } + }); } function verifyWhisperVariantFolders(unpackedNodeModulesRoot, platform, arch) { @@ -397,6 +427,7 @@ if (require.main === module) { module.exports = { getRequiredLlamaPrebuiltPackages, + getForeignArchLlamaPrebuiltPackages, getRequiredMacElectronRuntimeFiles, getRequiredWhisperVariantPackages, getWindowsZipArtifactName, diff --git a/src/main/core/gpuConfig.js b/src/main/core/gpuConfig.js index 536dd5a..2c1af0e 100644 --- a/src/main/core/gpuConfig.js +++ b/src/main/core/gpuConfig.js @@ -116,6 +116,16 @@ function shouldForceInferenceCpuMode() { ); } +/** + * Raster switches forced on this platform: none on Windows and macOS, the + * historical pair on Linux (see initializeGpuConfig). + * + * @returns {string[]} + */ +function getForcedRasterSwitches() { + return process.platform === 'linux' ? ['enable-gpu-rasterization', 'enable-zero-copy'] : []; +} + /** * Initialize GPU configuration * Must be called before app.ready @@ -163,13 +173,20 @@ function initializeGpuConfig({ getMainWindow, isFirstShowPending } = {}) { logger.warn('[GPU] GPU blocklist ignored via STRATOSORT_IGNORE_GPU_BLOCKLIST'); } - // No enable-gpu-rasterization / enable-zero-copy: both override - // Chromium's per-driver decisions. The first bypasses the GPU - // rasterization blocklist (GPU raster is already on by default where - // the driver is not blocklisted); the second forces native-buffer tile - // uploads that Chromium leaves off outside macOS. Either only changes - // behaviour on the drivers Chromium already distrusts. - logger.info(`[GPU] Flags set: ANGLE=${angleBackend}`); + // Windows and macOS: no enable-gpu-rasterization / enable-zero-copy. + // Both override Chromium's per-driver decisions; the first bypasses the + // GPU rasterization blocklist (GPU raster is already on by default + // there where the driver is not blocklisted), the second forces + // native-buffer tile uploads. Either only changes behaviour on drivers + // Chromium already distrusts. + // Linux keeps both, as every earlier release set them on all platforms: + // Chromium's Linux raster default varies with version and driver, and + // the AppImage has not been measured without them. + const rasterSwitches = getForcedRasterSwitches(); + for (const rasterSwitch of rasterSwitches) { + app.commandLine.appendSwitch(rasterSwitch); + } + logger.info(`[GPU] Flags set: ANGLE=${angleBackend}`, { rasterSwitches }); } } catch (e) { logger.warn('[GPU] Failed to apply GPU flags:', e?.message || 'Unknown error'); diff --git a/src/main/services/GPUMonitor.js b/src/main/services/GPUMonitor.js index a37d072..7babe85 100644 --- a/src/main/services/GPUMonitor.js +++ b/src/main/services/GPUMonitor.js @@ -46,6 +46,9 @@ async function runCommand(/** @type {any} */ cmd, /** @type {any} */ args = [], } } +// name, total VRAM (MiB) and driver version of the first NVIDIA GPU. +const NVIDIA_SMI_GPU_QUERY = '--query-gpu=name,memory.total,driver_version'; + function parseNvidiaSmiGpuInfo(/** @type {any} */ output) { if (!output || typeof output !== 'string') return null; const firstLine = output @@ -54,19 +57,26 @@ function parseNvidiaSmiGpuInfo(/** @type {any} */ output) { .find((line) => line.length > 0); if (!firstLine) return null; - const [rawName, rawVram] = firstLine.split(','); + const [rawName, rawVram, rawDriver] = firstLine.split(','); const name = (rawName || '').trim() || 'NVIDIA GPU'; const vramMB = Number.parseInt((rawVram || '').trim(), 10); if (!Number.isFinite(vramMB) || vramMB < 0) { return null; } - return { + /** @type {Record} */ + const info = { type: 'cuda', name, vramBytes: vramMB * 1024 * 1024, vramMB }; + // The driver version lets the GPU backend cache notice a driver update. + const driverVersion = (rawDriver || '').trim(); + if (driverVersion && !/^\[n\/a\]$/i.test(driverVersion)) { + info.driverVersion = driverVersion; + } + return info; } const logger = createLogger('GPUMonitor'); @@ -167,7 +177,7 @@ class GPUMonitor { // Try nvidia-smi first (use platform-specific executable for Windows) const nvidiaSmiCmd = getNvidiaSmiCommand(); const nvidiaSmi = await runCommand(nvidiaSmiCmd, [ - '--query-gpu=name,memory.total', + NVIDIA_SMI_GPU_QUERY, '--format=csv,noheader,nounits' ]); if (nvidiaSmi) { @@ -181,7 +191,7 @@ class GPUMonitor { const ps = await runCommand('powershell', [ '-NoProfile', '-Command', - 'Get-CimInstance Win32_VideoController | Select-Object Name,AdapterRAM | ConvertTo-Json' + 'Get-CimInstance Win32_VideoController | Select-Object Name,AdapterRAM,DriverVersion | ConvertTo-Json' ]); if (ps) { @@ -228,7 +238,10 @@ class GPUMonitor { type: inferWindowsBackendFromName(gpuName), name: gpuName, vramBytes: ramBytes, - vramMB: Math.round(ramBytes / 1024 / 1024) + vramMB: Math.round(ramBytes / 1024 / 1024), + ...(typeof gpu?.DriverVersion === 'string' && gpu.DriverVersion.trim() + ? { driverVersion: gpu.DriverVersion.trim() } + : {}) }; } } @@ -301,7 +314,7 @@ class GPUMonitor { // Try nvidia-smi (use platform-specific executable) const nvidiaSmiCmd = getNvidiaSmiCommand(); const nvidiaSmi = await runCommand(nvidiaSmiCmd, [ - '--query-gpu=name,memory.total', + NVIDIA_SMI_GPU_QUERY, '--format=csv,noheader,nounits' ]); if (nvidiaSmi) { diff --git a/src/main/simple-main.js b/src/main/simple-main.js index f4fb70c..f5531b5 100644 --- a/src/main/simple-main.js +++ b/src/main/simple-main.js @@ -1089,8 +1089,8 @@ if (gotTheLock || process.env.STRATOSORT_FORCE_LAUNCH === '1') { // the rest of the shutdown. It lives until the process exits. app.on('second-instance', secondInstanceHandler); - // enable-gpu-rasterization and enable-zero-copy are no longer appended: - // both override Chromium's per-driver GPU decisions (see initializeGpuConfig). + // enable-gpu-rasterization and enable-zero-copy are appended on Linux only + // (see initializeGpuConfig); elsewhere Chromium decides per driver. logger.info( `[${isDev ? 'DEVELOPMENT' : 'PRODUCTION'}] GPU acceleration ${forceSoftwareRenderer ? 'disabled (software renderer)' : 'enabled'}` ); diff --git a/src/main/utils/llamaBackendCache.js b/src/main/utils/llamaBackendCache.js index 05445b6..1563932 100644 --- a/src/main/utils/llamaBackendCache.js +++ b/src/main/utils/llamaBackendCache.js @@ -291,8 +291,43 @@ function createLlamaBackendCache({ return { filePath, fingerprint, read, write, clear }; } +/** + * Read-only view of the record for a second node-llama-cpp stack (the + * embedding worker thread). It tries the remembered backend first, but saving + * and clearing stay with the main process's LlamaService, so the two stacks + * never race on the file. Resolves undefined when no userData directory is + * given; never throws. + * + * @param {{ + * userDataPath?: string|null, + * appVersion?: string|null, + * nodeLlama?: any, + * gpuInfo?: any + * }} options + * @returns {Promise<{ read: LlamaBackendCache['read'] } | undefined>} + */ +async function createReadOnlyLlamaBackendCache({ + userDataPath, + appVersion = null, + nodeLlama, + gpuInfo = null +}) { + if (!asText(userDataPath)) return undefined; + let nodeLlamaVersion = null; + if (typeof nodeLlama?.getModuleVersion === 'function') { + try { + nodeLlamaVersion = await nodeLlama.getModuleVersion(); + } catch { + // The fingerprint carries null; the record just misses. + } + } + const cache = createLlamaBackendCache({ userDataPath, appVersion, nodeLlamaVersion, gpuInfo }); + return cache ? { read: cache.read } : undefined; +} + module.exports = { BACKEND_CACHE_FILENAME, BACKEND_CACHE_MAX_AGE_MS, - createLlamaBackendCache + createLlamaBackendCache, + createReadOnlyLlamaBackendCache }; diff --git a/src/main/utils/llamaBackendSelector.js b/src/main/utils/llamaBackendSelector.js index 6e6a23a..c6467d3 100644 --- a/src/main/utils/llamaBackendSelector.js +++ b/src/main/utils/llamaBackendSelector.js @@ -408,8 +408,16 @@ async function initLlamaWithBackend({ const forgetCachedBackend = async (/** @type {any} */ backend, /** @type {any} */ error) => { if (!cachedRecordValid || backend !== cachedBackend) return; cachedRecordValid = false; + // A read-only record (the embedding worker's) is left for its owner. + if (typeof backendCache?.clear !== 'function') { + log.info(`[${context}] Cached Llama backend failed; record left to its owner`, { + backend, + error: error?.message || String(error || 'Unknown error') + }); + return; + } try { - await backendCache?.clear?.(); + await backendCache.clear(); } catch { // The record is a hint; a stale one costs one failed attempt next time. } diff --git a/src/main/utils/workerPools.js b/src/main/utils/workerPools.js index 08cdbe8..c40b6bb 100644 --- a/src/main/utils/workerPools.js +++ b/src/main/utils/workerPools.js @@ -41,6 +41,25 @@ let _warnedEmbeddingWorkerVramBudget = false; function isEmbeddingWorkerEnabled() { return String(process.env.STRATOSORT_ENABLE_EMBEDDING_WORKER || 'false').toLowerCase() === 'true'; } + +/** + * What the embedding worker needs from electron's app, which a worker thread + * cannot reach: the userData folder (for the GPU backend record) and the app + * version (part of that record's key). Empty outside Electron. + * + * @returns {{ userDataPath?: string, appVersion?: string }} + */ +function getEmbeddingWorkerData() { + /** @type {{ userDataPath?: string, appVersion?: string }} */ + const data = {}; + try { + if (typeof app?.getPath === 'function') data.userDataPath = app.getPath('userData'); + if (typeof app?.getVersion === 'function') data.appVersion = app.getVersion(); + } catch { + // Not ready or not Electron: the worker runs its plain backend ladder. + } + return data; +} const DEFAULT_CPU_POOL_MAX_THREADS = 4; const ABSOLUTE_CPU_POOL_MAX_THREADS = 8; @@ -295,6 +314,9 @@ function getEmbeddingPool() { } embeddingPool = new Piscina({ filename, + // The worker cannot reach electron's app; it reads the GPU backend record + // LlamaService keeps under userData (see utils/llamaBackendCache). + workerData: getEmbeddingWorkerData(), maxThreads: 1, minThreads: 1, // 10-minute idle timeout: keeps the GPU-resident embedding model warm @@ -557,5 +579,6 @@ module.exports = { destroyImagePreprocessPool, getWorkerPoolHealth, shouldUsePiscina, - resolveWorkerPath // Export for testing + resolveWorkerPath, // Export for testing + getEmbeddingWorkerData // Export for testing }; diff --git a/src/main/workers/embeddingWorker.js b/src/main/workers/embeddingWorker.js index 9058ea7..5d84b9a 100644 --- a/src/main/workers/embeddingWorker.js +++ b/src/main/workers/embeddingWorker.js @@ -1,5 +1,6 @@ // @ts-check const path = require('path'); +const { workerData } = require('worker_threads'); const { createLogger } = require('../../shared/logger'); const { ERROR_CODES } = require('../../shared/errorCodes'); @@ -45,9 +46,10 @@ async function getLlamaInstance() { if (llamaInstance) return llamaInstance; const nodeLlama = await import(/* webpackIgnore: true */ 'node-llama-cpp'); - const { getLlama } = nodeLlama; + const { getLlama, getLlamaGpuTypes } = nodeLlama; const { GPUMonitor } = require('../services/GPUMonitor'); const { initLlamaWithBackend } = require('../utils/llamaBackendSelector'); + const { createReadOnlyLlamaBackendCache } = require('../utils/llamaBackendCache'); const { createFilteredLlamaLogger } = require('../utils/llamaLogFilter'); // Embedding models are exactly the case that trips the benign tokenizer // round-trip warning; suppress it here too. @@ -61,13 +63,32 @@ async function getLlamaInstance() { // Ignore detection error } + // Same supported-backend filter and remembered backend as LlamaService, + // so this second stack does not re-test GPU binaries the main process has + // already ruled out. userData and the app version come from the pool + // (workerPools passes them as workerData); the record is read-only here. + /** @type {any} */ + let backendCache; + try { + backendCache = await createReadOnlyLlamaBackendCache({ + userDataPath: workerData?.userDataPath, + appVersion: workerData?.appVersion, + nodeLlama, + gpuInfo + }); + } catch { + backendCache = undefined; + } + try { const selection = await initLlamaWithBackend({ getLlama, gpuInfo, logger, context: 'EmbeddingWorker', - llamaLogger + llamaLogger, + getLlamaGpuTypes, + backendCache }); llamaInstance = selection.llama; } catch (error) { diff --git a/test/GPUMonitor.test.js b/test/GPUMonitor.test.js index 4cd222e..a3f9726 100644 --- a/test/GPUMonitor.test.js +++ b/test/GPUMonitor.test.js @@ -41,6 +41,58 @@ describe('GPUMonitor', () => { expect(result.vramMB).toBe(24576); }); + test('nvidia-smi detection records the driver version and asks for it', async () => { + const monitor = new GPUMonitor(); + monitor._platform = 'win32'; + mockExecFileAsync.mockResolvedValueOnce({ stdout: 'RTX 4090, 24576, 581.57\n', stderr: '' }); + + const result = await monitor.detectGPU(); + + expect(result).toEqual( + expect.objectContaining({ type: 'cuda', name: 'RTX 4090', driverVersion: '581.57' }) + ); + // The mocked command name follows the real platform running the suite. + expect(mockExecFileAsync).toHaveBeenCalledWith( + expect.stringMatching(/^nvidia-smi(\.exe)?$/), + expect.arrayContaining(['--query-gpu=name,memory.total,driver_version']), + expect.anything() + ); + }); + + test('nvidia-smi detection leaves the driver version out when it is missing or [N/A]', async () => { + for (const stdout of ['RTX 4090, 24576', 'RTX 4090, 24576, [N/A]']) { + const monitor = new GPUMonitor(); + monitor._platform = 'linux'; + mockExecFileAsync.mockReset(); + mockExecFileAsync.mockResolvedValueOnce({ stdout, stderr: '' }); + + const result = await monitor.detectGPU(); + + expect(result.type).toBe('cuda'); + expect(result).not.toHaveProperty('driverVersion'); + } + }); + + test('windows WMI detection records the adapter driver version', async () => { + const monitor = new GPUMonitor(); + monitor._platform = 'win32'; + mockExecFileAsync + .mockRejectedValueOnce(new Error('no nvidia')) // nvidia-smi + .mockResolvedValueOnce({ + stdout: JSON.stringify({ + Name: 'AMD Radeon RX 7800 XT', + AdapterRAM: 2147483648, + DriverVersion: '32.0.12011.1036' + }), + stderr: '' + }); + + const result = await monitor.detectGPU(); + + expect(result.name).toBe('AMD Radeon RX 7800 XT'); + expect(result.driverVersion).toBe('32.0.12011.1036'); + }); + test('windows detection falls back to wmic', async () => { const monitor = new GPUMonitor(); monitor._platform = 'win32'; diff --git a/test/electron-builder-config.test.js b/test/electron-builder-config.test.js index 81e86fd..3c03100 100644 --- a/test/electron-builder-config.test.js +++ b/test/electron-builder-config.test.js @@ -52,9 +52,10 @@ describe('electron-builder.json Windows targets', () => { describe('electron-builder.json node_modules exclusions', () => { const { getNodeModuleFileMatcher, FileMatcher } = require('app-builder-lib/out/fileMatcher'); - function isPackaged(platformKey, moduleRelativePath) { + function isPackaged(platformKey, moduleRelativePath, arch = 'x64') { const packager = { config: builderConfig, debugLogger: { isEnabled: false } }; - const macroExpander = (pattern) => pattern; + // app-builder-lib expands ${arch} in `files` patterns per packed arch. + const macroExpander = (pattern) => pattern.replace(/\$\{arch\}/g, arch); const nodeModuleMatcher = getNodeModuleFileMatcher( '/project', '/out/app', @@ -107,6 +108,22 @@ describe('electron-builder.json node_modules exclusions', () => { expect(isPackaged('win', moduleRelativePath)).toBe(true); }); + test.each([ + ['x64', 'node_modules/@node-llama-cpp/win-arm64/bins/win-arm64/llama-addon.node', false], + ['x64', 'node_modules/@node-llama-cpp/win-arm64/package.json', false], + ['x64', 'node_modules/@node-llama-cpp/win-x64/bins/win-x64/llama-addon.node', true], + [ + 'x64', + 'node_modules/@node-llama-cpp/win-x64-vulkan/bins/win-x64-vulkan/ggml-vulkan.dll', + true + ], + ['arm64', 'node_modules/@node-llama-cpp/win-arm64/bins/win-arm64/llama-addon.node', true], + ['arm64', 'node_modules/@node-llama-cpp/win-x64-cuda/bins/win-x64-cuda/ggml-cuda.dll', false], + ['arm64', 'node_modules/node-llama-cpp/dist/bindings/getLlama.js', true] + ])('Windows %s build packages %s: %s', (arch, moduleRelativePath, expected) => { + expect(isPackaged('win', moduleRelativePath, arch)).toBe(expected); + }); + test('the Windows-only prebuild exclusion does not leak into macOS or Linux builds', () => { expect(isPackaged('mac', 'node_modules/better-sqlite3/prebuilds/darwin-arm64.node')).toBe(true); expect(isPackaged('linux', 'node_modules/better-sqlite3/prebuilds/linux-x64.node')).toBe(true); diff --git a/test/gpuConfig.test.js b/test/gpuConfig.test.js index 7f9513b..a8558b2 100644 --- a/test/gpuConfig.test.js +++ b/test/gpuConfig.test.js @@ -449,14 +449,46 @@ describe('GPU Configuration', () => { mockApp.commandLine.appendSwitch.mockImplementation(() => {}); }); - test('never appends enable-gpu-rasterization or enable-zero-copy (Chromium decides per driver)', () => { + const realPlatform = process.platform; + afterEach(() => { + Object.defineProperty(process, 'platform', { value: realPlatform, configurable: true }); + }); + + test.each(['win32', 'darwin'])( + 'never appends enable-gpu-rasterization or enable-zero-copy on %s (Chromium decides per driver)', + (platform) => { + Object.defineProperty(process, 'platform', { value: platform, configurable: true }); + gpuConfig = require('../src/main/core/gpuConfig'); + + gpuConfig.initializeGpuConfig(); + + expect(mockApp.commandLine.appendSwitch).not.toHaveBeenCalledWith( + 'enable-gpu-rasterization' + ); + expect(mockApp.commandLine.appendSwitch).not.toHaveBeenCalledWith('enable-zero-copy'); + expect(gpuConfig.applyProductionOptimizations).toBeUndefined(); + } + ); + + test('keeps enable-gpu-rasterization and enable-zero-copy on Linux, as earlier releases did', () => { + Object.defineProperty(process, 'platform', { value: 'linux', configurable: true }); + gpuConfig = require('../src/main/core/gpuConfig'); + + gpuConfig.initializeGpuConfig(); + + expect(mockApp.commandLine.appendSwitch).toHaveBeenCalledWith('enable-gpu-rasterization'); + expect(mockApp.commandLine.appendSwitch).toHaveBeenCalledWith('enable-zero-copy'); + }); + + test('appends no raster switch in software mode, Linux included', () => { + Object.defineProperty(process, 'platform', { value: 'linux', configurable: true }); + process.env.STRATOSORT_FORCE_SOFTWARE_GPU = '1'; gpuConfig = require('../src/main/core/gpuConfig'); gpuConfig.initializeGpuConfig(); expect(mockApp.commandLine.appendSwitch).not.toHaveBeenCalledWith('enable-gpu-rasterization'); expect(mockApp.commandLine.appendSwitch).not.toHaveBeenCalledWith('enable-zero-copy'); - expect(gpuConfig.applyProductionOptimizations).toBeUndefined(); }); }); diff --git a/test/healthCheckUtils.test.js b/test/healthCheckUtils.test.js index 9e30ab9..61ff083 100644 --- a/test/healthCheckUtils.test.js +++ b/test/healthCheckUtils.test.js @@ -41,26 +41,34 @@ describe('healthCheckUtils', () => { checker.stop(); }); + // Exact failure counts depend on how many interval ticks have run, so + // these tests drive the clock with fake timers: on a loaded CI runner a + // real 20 ms sleep can outlast a 30 ms interval. test('calls onUnhealthy when check fails', async () => { - const checkFn = jest.fn().mockRejectedValue(new Error('Service down')); - const onUnhealthy = jest.fn(); - - const checker = createHealthCheckInterval({ - checkFn, - intervalMs: 100, - timeoutMs: 50, - name: 'Test', - onUnhealthy - }); - - // Wait for initial check - await new Promise((r) => setTimeout(r, 30)); - - expect(onUnhealthy).toHaveBeenCalled(); - expect(checker.state.isHealthy).toBe(false); - expect(checker.state.consecutiveFailures).toBe(1); - - checker.stop(); + jest.useFakeTimers(); + try { + const checkFn = jest.fn().mockRejectedValue(new Error('Service down')); + const onUnhealthy = jest.fn(); + + const checker = createHealthCheckInterval({ + checkFn, + intervalMs: 100, + timeoutMs: 50, + name: 'Test', + onUnhealthy + }); + + // Settle the initial check without reaching the first interval tick. + await jest.advanceTimersByTimeAsync(0); + + expect(onUnhealthy).toHaveBeenCalledTimes(1); + expect(checker.state.isHealthy).toBe(false); + expect(checker.state.consecutiveFailures).toBe(1); + + checker.stop(); + } finally { + jest.useRealTimers(); + } }); test('tracks consecutive failures', async () => { @@ -92,25 +100,31 @@ describe('healthCheckUtils', () => { return Promise.resolve(true); }); - const checker = createHealthCheckInterval({ - checkFn, - intervalMs: 30, - timeoutMs: 100, - name: 'Test' - }); - - // Wait for initial failure - await new Promise((r) => setTimeout(r, 20)); - expect(checker.state.consecutiveFailures).toBe(1); - - // Make next check succeed - shouldFail = false; - await new Promise((r) => setTimeout(r, 50)); - - expect(checker.state.consecutiveFailures).toBe(0); - expect(checker.state.isHealthy).toBe(true); - - checker.stop(); + jest.useFakeTimers(); + try { + const checker = createHealthCheckInterval({ + checkFn, + intervalMs: 30, + timeoutMs: 100, + name: 'Test' + }); + + // Initial check only: the first interval tick is 30 ms away. + await jest.advanceTimersByTimeAsync(0); + expect(checker.state.consecutiveFailures).toBe(1); + + // Make the next check (the first interval tick) succeed + shouldFail = false; + await jest.advanceTimersByTimeAsync(30); + + expect(checkFn).toHaveBeenCalledTimes(2); + expect(checker.state.consecutiveFailures).toBe(0); + expect(checker.state.isHealthy).toBe(true); + + checker.stop(); + } finally { + jest.useRealTimers(); + } }); test('stop() clears interval', async () => { diff --git a/test/llamaBackendCache.test.js b/test/llamaBackendCache.test.js index 867a8f1..6ef6b1a 100644 --- a/test/llamaBackendCache.test.js +++ b/test/llamaBackendCache.test.js @@ -15,7 +15,8 @@ const { logger: mockLogger } = require('./mocks/logger'); const { BACKEND_CACHE_FILENAME, BACKEND_CACHE_MAX_AGE_MS, - createLlamaBackendCache + createLlamaBackendCache, + createReadOnlyLlamaBackendCache } = require('../src/main/utils/llamaBackendCache'); const NOW = Date.parse('2026-10-31T12:00:00.000Z'); @@ -320,4 +321,46 @@ describe('llamaBackendCache', () => { expect.objectContaining({ error: expect.any(String) }) ); }); + + describe('createReadOnlyLlamaBackendCache', () => { + const nodeLlama = { getModuleVersion: jest.fn(async () => '3.22.2') }; + + test('resolves undefined without a userData directory', async () => { + await expect( + createReadOnlyLlamaBackendCache({ userDataPath: undefined, nodeLlama, gpuInfo: GPU }) + ).resolves.toBeUndefined(); + }); + + test('reads the record the main process saved, and exposes nothing that writes', async () => { + await makeCache({ platform: process.platform, arch: process.arch, now: Date.now }).write( + 'vulkan' + ); + + const readOnly = await createReadOnlyLlamaBackendCache({ + userDataPath, + appVersion: '4.1.0', + nodeLlama, + gpuInfo: GPU + }); + + expect(Object.keys(readOnly)).toEqual(['read']); + await expect(readOnly.read()).resolves.toBe('vulkan'); + }); + + test('a failing module-version lookup only makes the record miss', async () => { + await makeCache({ platform: process.platform, arch: process.arch, now: Date.now }).write( + 'vulkan' + ); + + const readOnly = await createReadOnlyLlamaBackendCache({ + userDataPath, + appVersion: '4.1.0', + nodeLlama: { getModuleVersion: jest.fn().mockRejectedValue(new Error('no version')) }, + gpuInfo: GPU + }); + + await expect(readOnly.read()).resolves.toBeNull(); + expect(ignoredReasons()).toContain('fingerprint-mismatch'); + }); + }); }); diff --git a/test/llamaBackendSelector.test.js b/test/llamaBackendSelector.test.js index 2bc8714..72a158f 100644 --- a/test/llamaBackendSelector.test.js +++ b/test/llamaBackendSelector.test.js @@ -1243,6 +1243,37 @@ describe('llamaBackendSelector', () => { expect(backendCache.write).toHaveBeenCalledWith('cuda', { supportedGpuTypes: null }); }); + test('a read-only record (the embedding worker) is used but never cleared or written', async () => { + const read = jest.fn().mockResolvedValue('vulkan'); + const getLlama = jest + .fn() + .mockRejectedValueOnce(new Error('Vulkan binary test failed')) + .mockResolvedValueOnce({ gpu: 'cuda' }); + + const result = await initLlamaWithBackend({ + getLlama, + gpuInfo: { type: 'cuda' }, + logger: mockLogger, + context: 'test', + backendCache: { read } + }); + + expect(result.backend).toBe('cuda'); + expect(requestedGpus(getLlama)).toEqual(['vulkan', 'cuda']); + expect(mockLogger.info).toHaveBeenCalledWith( + '[test] Cached Llama backend failed; record left to its owner', + { backend: 'vulkan', error: 'Vulkan binary test failed' } + ); + expect(mockLogger.info).not.toHaveBeenCalledWith( + '[test] Cleared cached Llama backend after it failed', + expect.anything() + ); + expect(mockLogger.info).not.toHaveBeenCalledWith( + '[test] Saved Llama backend selection for the next launch', + expect.anything() + ); + }); + test('clears the record and saves nothing when the ladder ends on CPU', async () => { const backendCache = makeBackendCache('cuda'); const getLlama = jest diff --git a/test/verifyPackagedArtifacts.test.js b/test/verifyPackagedArtifacts.test.js index 862aab0..52e1cc1 100644 --- a/test/verifyPackagedArtifacts.test.js +++ b/test/verifyPackagedArtifacts.test.js @@ -8,6 +8,7 @@ const { NODE_LLAMA_CPP_RUNTIME_FILES, REQUIRED_ELECTRON_RUNTIME_FILES, REQUIRED_NATIVE_MODULE_PATHS, + getForeignArchLlamaPrebuiltPackages, getRequiredLlamaPrebuiltPackages, getRequiredWhisperVariantPackages, getWindowsZipArtifactName, @@ -287,6 +288,35 @@ describe('verify-packaged-artifacts', () => { ); }); + test('rejects Windows x64 builds that packaged the arm64 llama prebuilt', () => { + const buildRoot = createWindowsBuildRoot(); + tempDirs.push(buildRoot); + writeFile( + path.join( + windowsUnpackedModules(buildRoot), + '@node-llama-cpp', + 'win-arm64', + 'bins', + 'win-arm64', + 'llama-addon.node' + ) + ); + + expect(() => verifyWindowsArtifacts(buildRoot, 'x64', VERSION)).toThrow( + /prebuilt package @node-llama-cpp\/win-arm64 for another arch was packaged/ + ); + }); + + test('accepts an empty directory left by the excluded arm64 llama prebuilt', () => { + const buildRoot = createWindowsBuildRoot(); + tempDirs.push(buildRoot); + fs.mkdirSync(path.join(windowsUnpackedModules(buildRoot), '@node-llama-cpp', 'win-arm64'), { + recursive: true + }); + + expect(() => verifyWindowsArtifacts(buildRoot, 'x64', VERSION)).not.toThrow(); + }); + test('rejects builds that packaged the node-llama-cpp git bundle', () => { const buildRoot = createWindowsBuildRoot(); tempDirs.push(buildRoot); @@ -342,6 +372,17 @@ describe('verify-packaged-artifacts', () => { expect(getRequiredLlamaPrebuiltPackages('darwin', 'arm64')).toEqual([]); expect(getRequiredLlamaPrebuiltPackages('linux', 'x64')).toEqual([]); }); + + test("each Windows arch must not ship the other arch's packages", () => { + expect(getForeignArchLlamaPrebuiltPackages('win32', 'x64')).toEqual([ + '@node-llama-cpp/win-arm64' + ]); + expect(getForeignArchLlamaPrebuiltPackages('win', '')).toEqual(['@node-llama-cpp/win-arm64']); + expect(getForeignArchLlamaPrebuiltPackages('win', 'arm64')).toEqual( + getRequiredLlamaPrebuiltPackages('win', 'x64') + ); + expect(getForeignArchLlamaPrebuiltPackages('darwin', 'arm64')).toEqual([]); + }); }); describe('getRequiredWhisperVariantPackages policy', () => { diff --git a/test/workerPools.threadConfig.test.js b/test/workerPools.threadConfig.test.js index 02b1545..767ee21 100644 --- a/test/workerPools.threadConfig.test.js +++ b/test/workerPools.threadConfig.test.js @@ -45,6 +45,19 @@ describe('workerPools thread sizing', () => { jest.clearAllMocks(); }); + test('hands the embedding worker the userData folder and app version', () => { + process.env.STRATOSORT_ENABLE_EMBEDDING_WORKER = 'true'; + const { app } = require('electron'); + const workerPools = require('../src/main/utils/workerPools'); + + const pool = workerPools.getEmbeddingPool(); + + expect(pool.workerData).toEqual({ + userDataPath: app.getPath('userData'), + appVersion: app.getVersion() + }); + }); + test('scales OCR pool above two threads on capable CPUs', () => { const workerPools = require('../src/main/utils/workerPools');