feat(extensions): opt-in locked Python provisioning with isolated dependency reuse
What problem does this solve?
Extension-private Python environments protect against incompatible dependencies, but repeated installations may retain overlapping package content. Modly already shares pip download artifacts; this proposal adds a new installed-package materialization and lifecycle contract, not another shared download cache. It complements #223 (download/retry reuse) and #310 (requirements installation). PR #348 concerns model weights, separately. Nodes within one extension already reuse its venv, not necessarily one worker or loaded model.
LOCAL SNAPSHOT, 2026-09-14: an inventory of 45 physical extensions/44 conventional venvs found SD15 and SDXL declaring Python 3.12.3; Torch 2.11.0+cu130; torchvision 0.26.0+cu130; torchaudio ABSENT; diffusers 0.35.1; Transformers 4.57.6; NumPy 2.4.4. Five present core packages match complete metadata/normalized RECORD signatures, and all 67 package/version entries match. These are not exact locks or identical wheel bytes; RECORD/METADATA does not prove unchanged installed payload. Built-in publication provenance and transitive artifact identity remain unresolved. FLUX instead declares Torch 2.7.0+cu128/vision 0.22.0.
Dependency differences are real. Pinned published snapshots include SenseNova Transformers 4.49.0/NumPy 1.26.4; Trellis GGUF Transformers==5.2.0; Pixal3D Transformers==4.57.3; and ACE-Step diffusers 0.37.1/NumPy 2.3.5. These are source snapshots, not universal upstream facts or verified local install provenance. ComfyUI documents analogous conflicts; that is precedent, not a shared-environment solution.
Proposed solution
Provide author-declared, versioned managed provisioning with explicit user opt-in. Install full hash-locked selected wheels into each extension's private venv, reusing verified store content via verified copy-on-write cloning, otherwise explicit copying. Never implicitly fall back to hardlinks/symlinks. Keep generated scripts, metadata and mutable state private; do not promise every environment shares every byte or Windows clone support.
The selected lock must cover complete transitive dependencies, extras/markers, exact local build suffixes, wheel tags/filenames/hashes, approved origins/indexes and host-runner dependencies. Identify environments by full lock, Python implementation/ABI/build, platform/runtime floor, host contract and accelerator/native builds—not extension names or Torch alone. Record immutable extension revision/archive digest, logical venv interpreter, installed digests, materialization mode and validation tier. Hashes prove byte identity, not benign code; this provides no hostile-code sandbox guarantees.
Managed mode never executes legacy setup, even if setup.py coexists. Legacy remains unchanged independently. Reject unsupported contracts, incomplete closure, source distributions, editables, build/mutation hooks and unapproved sources; do not silently relax pins, select host Python or enter legacy setup.
Host prerequisites and acceptance gates:
- Disable unpinned automatic Pillow installation for managed workers; missing verified model/process environments must fail closed across all setup/runtime routes. Preserve the logical venv executable, without resolving to base Python. Host Python build changes invalidate receipts.
- Block launches through per-extension leases; track and quiesce/reap active Python-process children and every sibling model worker before mutation. Python-process termination currently does nothing; failed quiescence must block lifecycle changes.
- Define final-path transactional install, Repair, update and rollback with durable recovery/quarantine. Venvs cannot be arbitrarily moved after creation. Uninstall removes private ownership only after quiescence and preserves independently retained artifacts/weights.
- Host-own verified atomic publication, corruption quarantine/exact refetch, consumer pins/leases and explicit prune/bypass. Define download/extraction/retention/concurrency budgets and insufficient-space handling. Prove cross-device copy isolation; patch only private files or separately identified derived artifacts, never backend cache internals.
First execution slice: one CPU wheel-only Python-process extension, concrete selection unresolved. Validate two private consumers, private-write/store isolation, forbidden fallback, cold/warm/offline/cancelled/corrupted/concurrent installs, prune safety, real install/Repair/update/crash rollback/uninstall and real process output. Then investigate SD15/SDXL only after provenance/full-artifact/closure/inference proof. It is not supported today. Broader GPU/native research remains in scope: exact companion builds, driver/kernel targets, native symbols and real inference/UI/cancel/unload/restart require staged proof, not metadata acceptance.
Viability: bounded design viable; runtime UNTESTED. Measure physical allocated storage, retained cache, write divergence and latency after correctness. Copy fallback does not deduplicate installed bytes; retention may increase disk use. No RAM/VRAM benefit or disk percentage is promised.
Alternatives considered
Existing pip cache plus private venvs remains useful but lacks this materialization/receipt contract. Shared mutable venvs or hardlinked/symlinked packages permit cross-consumer mutation or cache-deletion breakage. Immutable whole-lock specifications may be reusable later without physically sharing an executable environment.
uv is a candidate backend, not an approved/installed choice. Its cache is thread-safe/append-only, forbids direct edits and favors same-filesystem placement; cross-device installs copy. Explicitly enforce clone/copy policy, since defaults differ; select the private interpreter with --python. pip differences rule out blind setup substitution; PyTorch guidance does not prove accelerator compatibility.
Confirmation
feat(extensions): opt-in locked Python provisioning with isolated dependency reuse
What problem does this solve?
Extension-private Python environments protect against incompatible dependencies, but repeated installations may retain overlapping package content. Modly already shares pip download artifacts; this proposal adds a new installed-package materialization and lifecycle contract, not another shared download cache. It complements #223 (download/retry reuse) and #310 (requirements installation). PR #348 concerns model weights, separately. Nodes within one extension already reuse its venv, not necessarily one worker or loaded model.
LOCAL SNAPSHOT, 2026-09-14: an inventory of 45 physical extensions/44 conventional venvs found SD15 and SDXL declaring Python 3.12.3; Torch 2.11.0+cu130; torchvision 0.26.0+cu130; torchaudio ABSENT; diffusers 0.35.1; Transformers 4.57.6; NumPy 2.4.4. Five present core packages match complete metadata/normalized RECORD signatures, and all 67 package/version entries match. These are not exact locks or identical wheel bytes; RECORD/METADATA does not prove unchanged installed payload. Built-in publication provenance and transitive artifact identity remain unresolved. FLUX instead declares Torch 2.7.0+cu128/vision 0.22.0.
Dependency differences are real. Pinned published snapshots include SenseNova Transformers 4.49.0/NumPy 1.26.4; Trellis GGUF Transformers==5.2.0; Pixal3D Transformers==4.57.3; and ACE-Step diffusers 0.37.1/NumPy 2.3.5. These are source snapshots, not universal upstream facts or verified local install provenance. ComfyUI documents analogous conflicts; that is precedent, not a shared-environment solution.
Proposed solution
Provide author-declared, versioned managed provisioning with explicit user opt-in. Install full hash-locked selected wheels into each extension's private venv, reusing verified store content via verified copy-on-write cloning, otherwise explicit copying. Never implicitly fall back to hardlinks/symlinks. Keep generated scripts, metadata and mutable state private; do not promise every environment shares every byte or Windows clone support.
The selected lock must cover complete transitive dependencies, extras/markers, exact local build suffixes, wheel tags/filenames/hashes, approved origins/indexes and host-runner dependencies. Identify environments by full lock, Python implementation/ABI/build, platform/runtime floor, host contract and accelerator/native builds—not extension names or Torch alone. Record immutable extension revision/archive digest, logical venv interpreter, installed digests, materialization mode and validation tier. Hashes prove byte identity, not benign code; this provides no hostile-code sandbox guarantees.
Managed mode never executes legacy setup, even if
setup.pycoexists. Legacy remains unchanged independently. Reject unsupported contracts, incomplete closure, source distributions, editables, build/mutation hooks and unapproved sources; do not silently relax pins, select host Python or enter legacy setup.Host prerequisites and acceptance gates:
First execution slice: one CPU wheel-only Python-process extension, concrete selection unresolved. Validate two private consumers, private-write/store isolation, forbidden fallback, cold/warm/offline/cancelled/corrupted/concurrent installs, prune safety, real install/Repair/update/crash rollback/uninstall and real process output. Then investigate SD15/SDXL only after provenance/full-artifact/closure/inference proof. It is not supported today. Broader GPU/native research remains in scope: exact companion builds, driver/kernel targets, native symbols and real inference/UI/cancel/unload/restart require staged proof, not metadata acceptance.
Viability: bounded design viable; runtime UNTESTED. Measure physical allocated storage, retained cache, write divergence and latency after correctness. Copy fallback does not deduplicate installed bytes; retention may increase disk use. No RAM/VRAM benefit or disk percentage is promised.
Alternatives considered
Existing pip cache plus private venvs remains useful but lacks this materialization/receipt contract. Shared mutable venvs or hardlinked/symlinked packages permit cross-consumer mutation or cache-deletion breakage. Immutable whole-lock specifications may be reusable later without physically sharing an executable environment.
uv is a candidate backend, not an approved/installed choice. Its cache is thread-safe/append-only, forbids direct edits and favors same-filesystem placement; cross-device installs copy. Explicitly enforce clone/copy policy, since defaults differ; select the private interpreter with --python. pip differences rule out blind setup substitution; PyTorch guidance does not prove accelerator compatibility.
Confirmation