Add load tests, a browser swarm and a public benchmark site

.ai/PERFORMANCE.md

@@ -95,6 +95,54 @@ production bundle, runs the harness against `baseline.json`, and:
 Thresholds live per-scenario in `scenarios.py` (`warn_ms` / `fail_ms`, keyed by
 metric). Keep them generous: this is a smoke alarm, not a microbenchmark.
 
+## Load tests and published numbers
+
+`benchmarks/streaming/` measures streaming-callback capacity per server setup
+(see `benchmarks/README.md` for how to run it). Simulated browsers follow the
+renderer's `StreamClient`: one downlink per browser, a long NDJSON response on
+ASGI and polls with the same backoff on WSGI. Frames carry the server's
+`time.time()`, so latency is measured end to end on one clock.
+
+Pitfalls the harness guards against:
+
+- One Python client process tops out at a few hundred browsers and then
+  reports its own lag as server latency. Clients run 100 browsers per process,
+  on CPUs separate from the server, and a point is flagged `client_bound` when
+  a client hits 85% CPU.
+- Client output goes to temp files, not pipes: a full pipe blocks the client
+  forever because the runner only reads after the clients exit.
+- Multi-worker setups need shared storage (Redis); with a per-process secret
+  the workers also reject each other's stream tokens, so the runner sets
+  a shared secret (`LOAD_SECRET`). The server imports dash from the checkout
+  under test (`PYTHONPATH`), not whatever is installed.
+
+`benchmarks/callbacks/` runs the same way for plain callbacks, HTTP vs
+websocket per backend; both load tests share `benchmarks/loadkit.py`.
+
+`benchmarks/swarm/` drives real headless Chromium (Playwright) against a
+deployed app, on as many machines as needed. Pitfalls it guards against:
+
+- Timings are taken in the page (`performance.now()` from the click to the DOM
+  update), so machines never need agreeing clocks; only the start is shared,
+  and an agent that got its browsers up after it is flagged `late`.
+- A failed reload restarts the page's click counts; the expected output text
+  is derived from them, so a stale count would time out every later click.
+- Steps stop when the swarm stops delivering its load (failed agents, users
+  that never loaded, agents over 85% CPU), not only when the server slows.
+- `page.on("websocket")` does not see sockets opened by a SharedWorker; to
+  confirm a click went over the websocket, check that it made no HTTP POST.
+
+`benchmarks/serialization/` times `dash._utils.to_json` (plotly's
+`to_json_plotly`) on one large figure, alone and next to a small value orjson
+cannot serialize, plus the figure inside a layout. Each engine runs in its own
+process; the stdlib one blocks the orjson import, so it is a real install
+without orjson, not just a config switch. See finding 6.
+
+`benchmarks/publish.py` commits results to `benchmarks/results/` (read by the
+docs) and the table in `README.md`. `.github/workflows/benchmarks-publish.yml`
+runs weekly and opens a pull request only when the numbers moved
+significantly (`--only-significant`).
+
 ## Profiling a slow scenario
 
 The runner can capture a **Chrome DevTools CPU profile** of a scenario and print
@@ -198,3 +246,48 @@ exists for growing containers; it is kept as a contrast with loose thresholds.
 limit) for a component tree nested deeper than ~254. The `deep_nesting`
 scenario is capped at 120 to stay clear. Worth remembering before recommending
 deeply-recursive layouts.
+
+### 6. One non-native value sends the whole response down plotly's slow path (#4065)
+
+With orjson, `to_json_plotly` tries `orjson.dumps` on the whole response. If
+any value is not orjson-native (a component, a `go.Figure` object, a
+`pd.Timestamp`), it discards the attempt, walks everything in Python with
+`clean_to_json_compatible`, then dumps again. A `go.Figure` also pays a
+`deepcopy` of its data in `to_dict()`. Layouts always take this path, since
+they are made of components. The cost scales with the number of Python-level
+values in the response: numpy arrays pass through the walk whole, lists of
+floats or datetimes are visited one value at a time. 10 traces x 20k points,
+Python 3.13, plotly 6.7:
+
+| figure data | alone | + html.Div | as go.Figure | in a layout | no orjson, alone |
+|--|--:|--:|--:|--:|--:|
+| numpy array | 3 ms | 3 ms | 5 ms | 5 ms | 10 ms |
+| list of float | 6 ms | 40 ms | 75 ms | 75 ms | 52 ms |
+| pd.DatetimeIndex | 14 ms | 50 ms | 42 ms | 42 ms | 75 ms |
+| list of datetime | 8 ms | 171 ms | 375 ms | 373 ms | 354 ms |
+
+The fix belongs in plotly.py, not Dash:
+
+- pass `default=` to `orjson.dumps` so only the non-native values are
+  converted, keeping the old path as a fallback on `TypeError`;
+- give figures and traces a `_to_json_dict()` that skips the `deepcopy` and
+  copies only the containers the base64 conversion changes; skip lists of
+  scalars with one `set(map(type, lst))` check instead of a Python loop;
+- in `PlotlyJSONEncoder.default`, return `isoformat()` first for exact
+  `datetime`/`date` (it now tries five encoders that raise before that one).
+
+A local patch to plotly 6.7 doing this brought every orjson case within ~3 ms
+of the figure alone (list of datetime as `go.Figure` or in a layout: 375 ->
+11 ms), and the stdlib list of datetime from 354 to 109 ms. Its JSON was
+identical except numpy `datetime64`, which then always uses orjson's short
+form (`2024-01-01T00:00:00`), as the fast path already did.
+
+Without orjson, what is left is the stdlib encoder: 200k floats take ~50 ms,
+and 200k datetimes ~70 ms just to make their `isoformat()` strings, before
+encoding. Bulk conversion does not help: numpy, pandas and pyarrow all pay
+per object to read a list of datetimes (77-149 ms), and pyarrow casts a
+timestamp column to strings in 5 ms but has no JSON writer for a nested
+response, so the strings come back out as Python objects at the same cost.
+Native encoders write bytes without Python objects: orjson and msgspec do
+200k floats in ~5 ms and 200k datetimes in 2-4 ms. Making orjson a Dash
+dependency is the change that helps every case.

.github/workflows/benchmarks-publish.yml

@@ -0,0 +1,236 @@
+name: Publish Benchmarks
+
+# Measures the benchmarks on the dev branch weekly (and on demand). Only when
+# the numbers moved enough to matter (a capacity step in the load tests, a
+# renderer median by over 25% and 5 ms, or a serialization median by over 25%
+# and 5 ms) does it open a pull request that
+# commits them to benchmarks/results/ (read by the docs) and the README table.
+# Per-PR renderer regressions are benchmarks.yml's job, not this one.
+#
+# Needs "Allow GitHub Actions to create and approve pull requests" in the
+# repo settings. A pull request opened with GITHUB_TOKEN starts no other
+# workflows: close and reopen it to run the checks before merging.
+
+on:
+  schedule:
+    - cron: '0 6 * * 1'
+  workflow_dispatch:
+    inputs:
+      streaming:
+        description: Run the streaming load sweep too
+        type: boolean
+        default: true
+      callbacks:
+        description: Run the HTTP vs websocket callback sweep too
+        type: boolean
+        default: true
+
+permissions:
+  contents: read
+
+concurrency:
+  group: benchmarks-publish
+  cancel-in-progress: false
+
+jobs:
+  renderer:
+    name: Renderer benchmarks
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: '24'
+          cache: npm
+      - run: npm ci
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+          cache: pip
+      - name: Install Dash
+        run: |
+          python -m pip install --upgrade pip
+          python -m pip install "setuptools<80.0.0"
+          python -m pip install -e .[ci,dev,testing]
+      - run: npm run build
+      - uses: browser-actions/setup-chrome@c785b87e244131f27c9f19c1a33e2ead956ab7ce # v1
+        with:
+          chrome-version: stable
+      - name: Set up virtual display
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y xvfb
+          sudo Xvfb :99 -ac -screen 0 1400x1000x24 &
+          echo "DISPLAY=:99" >> $GITHUB_ENV
+      - run: python -m benchmarks.run --out benchmarks/results.json
+      - uses: actions/upload-artifact@v4
+        with:
+          name: renderer-results
+          path: benchmarks/results.json
+
+  streaming:
+    name: Streaming load sweep
+    if: github.event_name == 'schedule' || (github.event_name == 'workflow_dispatch' && inputs.streaming)
+    runs-on: ubuntu-latest
+    timeout-minutes: 120
+    services:
+      redis:
+        image: redis:7-alpine
+        ports:
+          - 6379:6379
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: '24'
+          cache: npm
+      - run: npm ci
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+          cache: pip
+      - name: Install Dash
+        run: |
+          python -m pip install --upgrade pip
+          python -m pip install "setuptools<80.0.0"
+          python -m pip install -e .[ci,dev,testing,async,fastapi,quart,redis]
+          python -m pip install gunicorn uvicorn aiohttp psutil
+      - run: npm run build
+      - name: Run the sweep
+        env:
+          BENCH_RUNNER: GitHub Actions ubuntu-latest
+        run: |
+          ulimit -n 65536
+          python -m benchmarks.streaming.load \
+            --redis-url redis://127.0.0.1:6379/0 \
+            --out benchmarks/streaming/results.json
+      - uses: actions/upload-artifact@v4
+        with:
+          name: streaming-results
+          path: benchmarks/streaming/results.json
+
+  callbacks:
+    name: Callback load sweep
+    if: github.event_name == 'schedule' || (github.event_name == 'workflow_dispatch' && inputs.callbacks)
+    runs-on: ubuntu-latest
+    timeout-minutes: 120
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: '24'
+          cache: npm
+      - run: npm ci
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+          cache: pip
+      - name: Install Dash
+        run: |
+          python -m pip install --upgrade pip
+          python -m pip install "setuptools<80.0.0"
+          python -m pip install -e .[ci,dev,testing,async,fastapi,quart]
+          python -m pip install gunicorn uvicorn aiohttp psutil
+      - run: npm run build
+      - name: Run the sweep
+        env:
+          BENCH_RUNNER: GitHub Actions ubuntu-latest
+        run: |
+          ulimit -n 65536
+          python -m benchmarks.callbacks.load --out benchmarks/callbacks/results.json
+      - uses: actions/upload-artifact@v4
+        with:
+          name: callbacks-results
+          path: benchmarks/callbacks/results.json
+
+  serialization:
+    name: Serialization benchmarks
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: '24'
+          cache: npm
+      - run: npm ci
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+          cache: pip
+      - name: Install Dash
+        run: |
+          python -m pip install --upgrade pip
+          python -m pip install "setuptools<80.0.0"
+          python -m pip install -e .[ci,dev,testing]
+      - run: npm run build
+      # Runs with orjson and again with its import blocked.
+      - run: python -m benchmarks.serialization.bench --out benchmarks/serialization/results.json
+      - uses: actions/upload-artifact@v4
+        with:
+          name: serialization-results
+          path: benchmarks/serialization/results.json
+
+  publish:
+    name: Open the results pull request
+    needs: [renderer, streaming, callbacks, serialization]
+    # Only dev's numbers are committed, even when a dispatch runs elsewhere.
+    if: >-
+      always() && github.ref == 'refs/heads/dev' &&
+      (needs.renderer.result == 'success' || needs.streaming.result == 'success' ||
+       needs.callbacks.result == 'success' || needs.serialization.result == 'success')
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write
+      pull-requests: write
+    env:
+      BRANCH: benchmarks/update-results
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+      - uses: actions/download-artifact@v4
+        with:
+          path: results
+      - name: Update the results
+        env:
+          BENCH_RUNNER: GitHub Actions ubuntu-latest
+        run: |
+          args=""
+          [ -f results/renderer-results/results.json ] && args="$args --renderer results/renderer-results/results.json"
+          [ -f results/streaming-results/results.json ] && args="$args --streaming results/streaming-results/results.json"
+          [ -f results/callbacks-results/results.json ] && args="$args --callbacks results/callbacks-results/results.json"
+          [ -f results/serialization-results/results.json ] && args="$args --serialization results/serialization-results/results.json"
+          python -m pip install -e .
+          python -m benchmarks.publish --commit "${GITHUB_SHA::9}" \
+            --only-significant --changes "$RUNNER_TEMP/changes.md" $args
+      - name: Open or update the pull request
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: |
+          git add benchmarks/results README.md
+          if git diff --cached --quiet; then
+            echo "No significant change."
+            exit 0
+          fi
+          {
+            echo "Benchmark numbers from ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID} moved enough to publish:"
+            echo
+            cat "$RUNNER_TEMP/changes.md"
+            echo
+            echo "Merge to update the README table and the docs."
+          } > "$RUNNER_TEMP/body.md"
+          git config user.name "github-actions[bot]"
+          git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
+          git switch -c "$BRANCH"
+          git commit -m "Update benchmark results for ${GITHUB_SHA::9}"
+          # Each run starts from dev, so an unmerged run is replaced, not stacked.
+          git push --force origin "$BRANCH"
+          if [ -z "$(gh pr list --head "$BRANCH" --state open --json number --jq '.[].number')" ]; then
+            gh pr create --base dev --head "$BRANCH" \
+              --title "Update benchmark results" --body-file "$RUNNER_TEMP/body.md"
+          else
+            gh pr edit "$BRANCH" --body-file "$RUNNER_TEMP/body.md"
+          fi

README.md

@@ -32,6 +32,17 @@ Built on top of [Plotly.js](https://github.com/plotly/plotly.js), [React](https:
 
 To learn more about Dash, read the [extensive announcement letter](https://medium.com/@plotlygraphs/introducing-dash-5ecf7191b503) or [jump in with the user guide](https://plotly.com/dash).
 
+### Performance
+
+How many people can use one Dash server at once? CI measures it weekly, from
+simulated browsers clicking callbacks and streaming against each server setup.
+
+<!-- benchmarks:start -->
+No published run yet.
+<!-- benchmarks:end -->
+
+Charts, the full numbers and how they are measured: [benchmarks](https://github.com/plotly/dash/tree/dev/benchmarks).
+
 ### Dash OSS & Dash Enterprise
 
 With Dash Open Source, Dash apps run on your local laptop or workstation, but cannot be easily accessed by others in your organization.

benchmarks/.gitignore

@@ -1,5 +1,7 @@
-# Generated benchmark artifacts (baseline.json is committed on purpose)
+# Generated benchmark artifacts (baseline.json and results/ are committed on purpose)
 results.json
 summary.md
 *.cpuprofile
 __pycache__/
+streaming/results.json
+callbacks/results.json