docs: env-var configuration of stores (DJ_STORES, DJ_IGNORE_CONFIG_FILE) + Storage Adapter API spec (2.3)#172
Open
dimitri-yatsenko wants to merge 2 commits into
Open
docs: env-var configuration of stores (DJ_STORES, DJ_IGNORE_CONFIG_FILE) + Storage Adapter API spec (2.3)#172dimitri-yatsenko wants to merge 2 commits into
dimitri-yatsenko wants to merge 2 commits into
Conversation
5 tasks
dimitri-yatsenko
added a commit
to datajoint/datajoint-python
that referenced
this pull request
May 20, 2026
… 2.3 Adds env-var configuration for object stores so the DataJoint platform — and any env-var-only deployment — can configure plugin-registered storage adapters (Databricks Unity Catalog Volumes, custom HTTP stores, lab archive systems) without files on disk. - DJ_STORES (JSON-encoded) carries the entire `stores` dict in the same shape used in `datajoint.json`. Replaces the file's `stores` block when set. - DJ_IGNORE_CONFIG_FILE (default false) skips `datajoint.json`, the project `.secrets/`, and `/run/secrets/datajoint/` entirely. Hard guarantee that no file on disk leaks into config. Implementation notes: - New `Config.ignore_config_file` field (validation_alias DJ_IGNORE_CONFIG_FILE) auto-bound by pydantic-settings. - The `stores` field receives a `validation_alias` placeholder so pydantic-settings does NOT auto-bind DJ_STORES at Config() construction. Otherwise its built-in JSON parser intercepts before precedence logic runs and reports SettingsError instead of a clean ValueError. - New `Config._apply_stores_env()` parses DJ_STORES JSON, replaces self.stores wholesale, raises ValueError on bad JSON or non-object payloads. - `_create_config()` restructured to: skip file + secrets when ignore_config_file is set; apply DJ_STORES between file load and secrets fill so env wins over file and secrets only fill gaps. Functional precedence (high to low): programmatic > DJ_STORES > config file > `.secrets/stores.<name>.<attr>` (fills missing attrs only). Tests: new TestStoreEnv class with 8 tests; existing TestStoreSecrets and test_storage_adapter tests still pass. Companion docs: datajoint/datajoint-docs#172
…LE) + Storage Adapter API spec — new in 2.3 Companion to datajoint/datajoint-python#1452. - src/about/whats-new-23.md (new) — 2.3 release notes - src/reference/specs/storage-adapter-api.md (new) — public spec for the datajoint.storage entry-point group used by built-in adapters and plugins - src/reference/configuration.md — DJ_STORES, DJ_IGNORE_CONFIG_FILE in env-var tables and Top-Level Settings; arbitrary-attr secrets example - src/how-to/manage-secrets.md — replace the never-implemented DJ_STORES_<NAME>_<ATTR> pattern with the real DJ_STORES JSON env var; add Env-var-only deployments section; fix DJ_TLS → DJ_USE_TLS - src/how-to/configure-storage.md — Configuring stores via environment variables - src/reference/specs/object-store-configuration.md — precedence table and DJ_STORES override semantics - mkdocs.yaml — register whats-new-23.md and storage-adapter-api.md
dimitri-yatsenko
force-pushed
the
docs/dj-stores-env-vars-2-3
branch
from
64b3c89 to
6508d8b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Documents the env-var configuration of object stores added in datajoint-python 2.3, and publishes the Storage Adapter API plugin contract as a formal spec.
Status: WIP / draft — paired with the code PR; both should land together. Marked draft so reviewers can comment as the code PR firms up.
Companion PR: datajoint/datajoint-python#1452
New pages
src/about/whats-new-23.md— 2.3 release notes:DJ_STORES,DJ_IGNORE_CONFIG_FILE, arbitrary-attr secrets, Storage Adapter API.src/reference/specs/storage-adapter-api.md— Plugin contract for third-party storage protocols. Documents thedatajoint.storageentry-point group, theStorageAdapterbase class, packaging conventions, and the four built-in adapters (file,s3,gcs,azure). The entry-point group has existed in code since 2.0 but had no public spec.Updated pages
src/reference/configuration.md—DJ_STORESandDJ_IGNORE_CONFIG_FILErows in the Top-Level Settings table; env-var examples; arbitrary-attr.secrets/example. Removed the now-incorrect note "environment variable overrides are not supported for nested store configurations".src/how-to/manage-secrets.md— Replaced the (never-implemented)DJ_STORES_<NAME>_<ATTR>pattern with the realDJ_STORESJSON pattern; new Env-var-only deployments subsection; arbitrary-attr secrets note; correctedDJ_TLS→DJ_USE_TLS.src/how-to/configure-storage.md— New Configuring stores via environment variables section, cross-linking manage-secrets and the Storage Adapter API spec.src/reference/specs/object-store-configuration.md— Configuration sources & precedence table coveringDJ_STORESandDJ_IGNORE_CONFIG_FILE.mkdocs.yaml— Registeredwhats-new-23.mdandstorage-adapter-api.mdin the nav.What's accurate now that wasn't before
The previous docs claimed
DJ_STORES_MAIN_ACCESS_KEY/DJ_STORES_MAIN_SECRET_KEYenv vars existed (in `manage-secrets.md`). They never did — there was no code path for per-key store env vars. This PR removes the misleading text and documents the real `DJ_STORES` mechanism added in 2.3.Test plan
mkdocs build --strictsucceeds; no new broken anchors (pre-existing warnings unrelated to these changes)storage-adapter-api.md,whats-new-23.md,#env-var-only-deploymentsanchor)whats-new-23.mdrendering (admonition style, code blocks)Related
datajoint-company/platform-roadmap