Version 4.5.0rc1 (master)

.ai/ARCHITECTURE.md

@@ -752,6 +752,8 @@ Special handling for Colab:
 | `DASH_PRUNE_ERRORS` | Simplify tracebacks |
 | `HOST` | Server host |
 | `PORT` | Server port |
+| `DASH_SECRET_KEY` | Signing secret for page, background and stream tokens when `server.secret_key` is unset |
+| `DASH_SHARED_STORAGE` | Shared-storage backend when `shared_storage=` is not passed (see Shared Storage) |
 
 ## Stores and Client-Side State
 
@@ -850,7 +852,7 @@ outputs. Messages are encoded with `msgspec` (msgpack), a hard dependency.
 | `get(key, default=None)` | Read a value |
 | `set(key, value, ttl=None)` | Write a value; `ttl` = optional lifetime in seconds |
 | `delete(key)` | Remove a key |
-| `publish(topic, message)` | Append a message to a topic |
+| `publish(topic, message, ttl=None)` | Append a message to a topic; `ttl` = release the topic after that long idle |
 | `subscribe(topic, replay_from=None)` | Return a `Subscription` |
 
 **Key expiry (TTL).** `set(key, value, ttl=<seconds>)` gives a key a bounded
@@ -925,6 +927,27 @@ deployment is its own owner and pays no socket overhead. Knobs: `namespace`
 purpose, since each topic retains that many arbitrary payloads; raise it for a
 wider reconnect window).
 
+**Topic lifetime.** `publish(topic, message, ttl=None)` takes an optional ttl
+(seconds, at least `MIN_TOPIC_TTL` = 1s, so a reconnecting reader is not
+outrun). A topic with a ttl is released, buffer and sequence both, once nobody
+has published to or read from it for that long; a later publish starts it over
+at 1, so a consumer returning with an old cursor gets `SharedStorageGap`.
+Without a ttl a topic lives as long as the store. The latest publish's ttl
+wins. Streaming publishes with `STREAM_TOPIC_TTL` (300s); user topics default
+to no ttl. Every backend expires a topic as a whole, never message by message
+while it is read (`tests/shared_storage/test_topic_ttl.py` runs the same cases
+on all three). Local: the engine records the ttl on the topic and sweeps idle
+ones (no publish, head or poll, and no call holding it) at most once a second,
+on the next pub/sub call. The sweep also drops empty, unheld topics, such as
+the one a returning reader's poll recreates after a release. Redis: the ttl sits in a third key; the publish
+script `PEXPIRE`s all three to 1.25 ttl, and a poll script renews them. A poll
+on such a topic blocks in `XREAD` for at most a third of that lifetime, not the
+usual 5s, so a waiting reader renews it in time. Diskcache: the ttl
+sits in a key; on publish or poll, once less than one ttl is left, every key of
+the topic (counter, ttl, buffered messages) is renewed to 1.25 ttl, and at
+once when a publish changes the ttl. A poll on such a topic waits at most half
+a ttl before it renews again.
+
 *Durability* is controlled by `mode` (the key/value store only — pub/sub is
 always transient):
 
@@ -959,7 +982,13 @@ the replay window; a subscriber past the trimmed floor gets `SharedStorageGap`).
 Redis is the single source of truth, so no owner election. Pass a `url`
 (defaults to `$REDIS_URL`) or an existing `client` to reuse a connection pool.
 This is the only backend correct for **horizontally-scaled, multi-pod**
-deployments.
+deployments. On an event loop it uses `redis.asyncio` (one client per loop),
+and one reader task per loop serves every async subscription with a single
+multi-stream `XREAD`; a new subscription interrupts it through a private wake
+stream. Never park a blocking read in the executor per subscription: at a dozen
+open downlinks that starved every other store call in the worker (25-50% failed
+streams at 25 browsers on uvicorn `--workers 4`). Diskcache's async
+subscriptions likewise poll without blocking and sleep on the loop.
 
 ### Deployment Topology and Shared Storage
 
@@ -976,13 +1005,31 @@ election only reaches processes in the same network + filesystem namespace.
   This works at 1 pod and fragments silently once it scales — use
   `RedisSharedStorage` (one Redis shared by all pods) instead.
 
+A hosting platform can switch the backend without editing the app through
+`DASH_SHARED_STORAGE` (`_shared_storage/_env.py`), read only when the app did
+not pass `shared_storage=` (the default is a sentinel, so an explicit argument,
+`None` included, always wins): `local`, `none`, `diskcache:///abs/path`, or a
+`redis://` / `rediss://` URL. A `key_prefix` query parameter on the Redis URL
+(`redis://host/0?key_prefix=myapp`) sets `RedisSharedStorage(key_prefix=)` and
+is stripped before the URL reaches redis-py; the diskcache path is
+percent-decoded. `cluster://` is reserved and raises; anything else raises
+`InvalidConfig` at construction, without echoing the value (it can hold
+credentials). The value becomes a zero-argument
+factory, so nothing is built or connected until `app.shared_storage` is first
+read, but a missing extra (`dash[redis]`, `dash[diskcache]`) fails at
+construction with the backend's own `ImportError`.
+
 ### Custom and Out-of-Tree Backends
 
 `BaseSharedStorage` is the stable, public extension point. A backend — shipped
 in-tree or as a **separate package** — implements:
 
 - `get(key, default)` / `set(key, value)` / `delete(key)` — JSON-compatible values
-- `publish(topic, message)` and `subscribe(topic, replay_from=None) -> Subscription`
+- `publish(topic, message, ttl=None)` and `subscribe(topic, replay_from=None) -> Subscription`
+- topic expiry: with a `ttl`, release the topic as a whole once nobody has
+  published to or read from it for that long, never message by message while
+  it is read; reject a ttl under `MIN_TOPIC_TTL` (run `test_topic_ttl.py`
+  against a new backend)
 - optional `start()` / `close()` (idempotent, called once per worker)
 
 and returns a `Subscription` (`__iter__` / `__aiter__` / `close`) that raises
@@ -1010,6 +1057,7 @@ same way as a built-in: `Dash(shared_storage=PostgresSharedStorage(...))`.
 | `_shared_storage/_polling.py` | `PollingSubscription`: shared poll-loop subscription for the diskcache/Redis backends |
 | `_shared_storage/_transport.py` | Length-prefixed, token-gated socket transport (local backend) |
 | `_shared_storage/_codec.py` | msgspec msgpack codec (data-only) |
+| `_shared_storage/_env.py` | `DASH_SHARED_STORAGE` parsing |
 | `dash.py` | `shared_storage` constructor arg + lazy `app.shared_storage` property |
 | `_callback_context.py` | `dash.ctx.shared_storage` accessor |
 
@@ -1380,8 +1428,24 @@ The connection id is never chosen by the client: every stream request rides on
 `?endId=`, the server-signed per-page-load token, and the backend derives the
 id from it (`get_stream_connection_id`), answering 403 when it is missing or
 forged -- otherwise a client could read or inject into another page's topic.
-Across worker processes every worker must resolve the same signing secret
-(`secret_key`).
+Across worker processes every worker must resolve the same signing secret.
+`_get_signing_secret` resolves, in order: `server.secret_key`, then
+`DASH_SECRET_KEY` (for Dash's signing only, never copied onto
+`server.secret_key`, so Flask sessions are untouched), then a secret persisted
+in the background-callback store, then a per-process random one. With the last,
+tokens only verify on the worker that issued them: stream requests 403 on the
+other workers, and the first failure in each process logs a warning pointing at
+`DASH_SECRET_KEY` (`_warn_unverified_stream_token`). A request with no token at
+all is not logged.
+
+Each run of streams (from the first stream after idle until none is in flight)
+also carries `&downlinkId=`, picked fresh by the client, and the connection id
+is `<end_id>:<downlinkId>`: every run gets its own topic and the client's
+cursor restarts at 0. Without it a page that sat idle past `STREAM_TOPIC_TTL` would
+resume a cursor into a topic the store released, get `{reset: true}`, and fail
+its new stream. The id only partitions the page's own space, so it is not
+signed (just checked against `[A-Za-z0-9_-]{1,64}`). The downlink lifecycle
+record (`connection_key`) stays keyed on the `end_id` alone.
 
 The downlink is hosted in a SharedWorker (`dash-stream-worker.js`, served like
 the WebSocket worker; `config.stream.worker_url`) so **one connection per
@@ -1399,7 +1463,10 @@ back to a downlink of its own.
 - `stream` (ASGI: Quart, FastAPI): one long-lived NDJSON response per
   browser. It costs no thread -- the subscription parks the task on a future
   the store resolves (`StoreEngine.apoll`; asyncio streams to the owner from
-  other workers) -- so a single uvicorn worker holds thousands.
+  other workers; the per-loop Redis reader) -- so a single uvicorn worker
+  holds thousands. The downlink's connection record is written with the
+  store's async calls, and its closed record from a task of its own, since a
+  client disconnect cancels the response.
 - `poll` (WSGI: Flask): a WSGI response holds a worker thread for its whole
   life, so an open downlink per browser exhausts a thread pool at a few dozen
   browsers (gunicorn `--threads 2`: the second browser hung everything).
@@ -1478,6 +1545,7 @@ the right tool: constant latency and a fraction of the CPU.
 - `dash/_streaming.py` - `StreamedCallbackResponse` marker, context-safe iteration, NDJSON helpers, keepalives, shutdown flag
 - `dash/_stream_hub.py` - multiplexed transport: `Downlink`, `poll_downlink`, pumps (`pump_to_storage`/`apump_to_storage`), `cancel_stream`, `shutdown_active_streams`/`install_stream_shutdown_handler`
 - `dash/_shared_storage/_engine.py`, `local.py` - `poll`/`apoll`, loop-native `aget`/`aset`/`apublish`, async client connection
+- `dash/_shared_storage/redis.py` - `redis.asyncio` path, `_StreamReader` (one multi-stream `XREAD` per loop)
 - `dash/backends/_flask.py`, `_quart.py`, `_fastapi.py` - streaming dispatch branches
 - `dash/backends/ws.py` - `make_stream_frame_emitter`, `consume_stream_frames`/`aconsume_stream_frames`
 - `dash/dash-renderer/src/actions/callbacks.ts` - `applyStreamFrame`, NDJSON reader, WS frame handling

.github/dependabot.yml

@@ -4,12 +4,12 @@ updates:
   # NOTE: Dependabot's npm updater recurses from "/" into every nested
   # package.json, and neither the singular `directory` nor plural `directories`
   # with `!` exclusion globs stops that recursion. So the ONLY reliable way to
-  # keep this entry from bumping the components/renderer (which carry their own
-  # narrow allow/ignore lists, e.g. react-docgen is pinned) is the `allow` list
-  # below: it names only packages that exist SOLELY in the root package.json, so
-  # even while recursing, Dependabot opens root-only PRs. npm-run-all, rimraf and
-  # @types/jest are intentionally omitted — they also live in the components and
-  # would produce duplicate PRs.
+  # keep this entry from bumping the components/renderer/@plotly packages (which
+  # each carry their own narrow allow/ignore lists, e.g. react-docgen is pinned)
+  # is the `allow` list below: it names only packages that exist SOLELY in the
+  # root package.json, so even while recursing, Dependabot opens root-only PRs.
+  # npm-run-all, rimraf and @types/jest are intentionally omitted: they also live
+  # in the components and would produce duplicate PRs.
   - package-ecosystem: "npm"
     directory: "/"
     schedule:
@@ -194,6 +194,48 @@ updates:
       - dependency-name: "babel-loader"
         versions: [">=10.0.0"]
 
+  # @plotly workspace packages (build helpers, jupyterlab, and the generator /
+  # test-component fixtures). These have no per-package config, so without this
+  # entry Dependabot's root "/" recursion opens an individual, ungrouped PR for
+  # each one. The `directories` glob claims every /@plotly/* manifest and folds
+  # them into a single grouped PR.
+  - package-ecosystem: "npm"
+    directories:
+      - "/@plotly/*"
+    schedule:
+      interval: "weekly"
+      day: "monday"
+    cooldown:
+      default-days: 14
+    groups:
+      npm-dependencies:
+        applies-to: version-updates
+        patterns:
+          - "*"
+      npm-dependencies-security:
+        applies-to: security-updates
+        patterns:
+          - "*"
+    ignore:
+      # react-docgen is pinned for legacy metadata extraction (extract-meta.js uses
+      # the react-docgen 5 API) and must never be updated.
+      - dependency-name: "react-docgen"
+      # The test/generator fixtures pin React 16 (react-jsx-parser peer-deps on
+      # 16/17); a major bump breaks them. Stay on 16.x.
+      - dependency-name: "react"
+        versions: [">=17.0.0"]
+      - dependency-name: "react-dom"
+        versions: [">=17.0.0"]
+      # typescript 6+ (and the native tsc 7) break the type-check. Stay on 5.x.
+      - dependency-name: "typescript"
+        versions: [">=6.0.0"]
+      # Babel 8 / babel-loader 10 break the build (the class-properties plugin
+      # peer-deps on @babel/core ^7). Stay on the 7.x / 9.x line.
+      - dependency-name: "@babel/*"
+        versions: [">=8.0.0"]
+      - dependency-name: "babel-loader"
+        versions: [">=10.0.0"]
+
   # Python dependencies
   - package-ecosystem: "pip"
     directory: "/"

.github/workflows/testing.yml

@@ -974,6 +974,7 @@ jobs:
   build-windows:
     name: Build on Windows
     runs-on: windows-latest
+    needs: build
     timeout-minutes: 30
 
     steps:
@@ -1012,6 +1013,15 @@ jobs:
           npm ci
           npm run build
 
+      - name: Download built component folders
+        uses: actions/download-artifact@v4
+        with:
+          name: dash-components
+          path: ${{ github.workspace }}/dash
+
+      - name: Run Windows-specific unit tests
+        run: python -m pytest tests/unit/test_resources.py
+
   dcc-lint:
     name: DCC Lint Tests (Python ${{ matrix.python-version }})
     needs: [build, changes_filter]