Tango performance comparison harness

.github/workflows/perf-tango-compare.yml

@@ -0,0 +1,126 @@
+name: "Perf: bazel-diff vs Tango"
+
+# Nightly performance comparison of bazel-diff against Uber's Tango (https://github.com/uber/tango)
+# on identical workloads. Runs tools/perf/tango_compare.py, which builds the Tango example
+# server/client (go build, pinned commit), generates synthetic genrule workspaces of several sizes,
+# and for each one times both tools end to end (checkout -> bazel query -> hash -> diff) cold (fresh
+# Bazel servers), warm (servers alive) and diff-only, cross-checking that the two tools report the
+# same impacted targets. See tools/perf/README.md for the methodology.
+#
+# Every run writes a markdown step summary and a 90-day metrics artifact, so trends can be read
+# across nights, and a Tango commit can be compared on demand via the tango_ref input. Scheduled
+# runs pin Tango to the commit in the harness so a Tango change never silently moves the baseline.
+# Note: scheduled + dispatched workflows run from the default branch, so this must land on master
+# before the cron fires.
+on:
+  workflow_dispatch:
+    inputs:
+      workloads:
+        description: "Comma-separated synthetic presets (tiny, small, medium, large, xlarge)"
+        type: string
+        default: "small,medium,large,xlarge"
+      iterations:
+        description: "Timed iterations per workload"
+        type: string
+        default: "3"
+      tango_ref:
+        description: "uber/tango commit/branch/tag to build (empty = the harness's pinned commit)"
+        type: string
+        default: ""
+      tango_pool_size:
+        description: "Tango worker pool size (1 = sequential like bazel-diff; 2 = parallel revisions)"
+        type: string
+        default: "1"
+      quick:
+        description: "Smoke profile: tiny workload, one iteration (overrides workloads/iterations)"
+        type: boolean
+        default: false
+  schedule:
+    # Nightly at 03:45 UTC, offset from the serve crons (30 6, 30 8, 0 */12) and the format/badge
+    # crons (12:00-13:00) so the jobs never contend for runners.
+    - cron: "45 3 * * *"
+
+permissions:
+  contents: read
+
+jobs:
+  TangoCompare:
+    runs-on: ubuntu-latest
+    # The default legs x 3 iterations x (cold + warm) x 2 tools, plus the bazel-diff Rust build and
+    # the Tango Go build, fit well inside an hour on a 4-core runner (the cold numbers are dominated
+    # by Bazel server start-up, not graph size). The timeout is generous so a slow build or a runner
+    # hiccup yields data, not a kill.
+    timeout-minutes: 120
+    steps:
+      - name: Setup Go environment
+        # Tango's go.mod declares go 1.25; the harness also sets GOTOOLCHAIN=auto as a fallback.
+        uses: actions/setup-go@v5
+        with:
+          go-version: '>=1.25.0'
+          cache: false
+        id: go
+      - name: Setup Bazelisk
+        # Retries guard against transient proxy.golang.org/sum.golang.org module-fetch errors.
+        run: |
+          for i in 1 2 3; do
+            go install github.com/bazelbuild/bazelisk@latest && break
+            echo "bazelisk install failed (attempt $i); retrying in 10s..."
+            sleep 10
+          done
+      - uses: actions/checkout@v4
+      - name: Build bazel-diff
+        # Built here rather than by the harness so the Rust build has its own step and log. The
+        # .bazelrc lint aspects are skipped: this is a measurement binary, CI lints it elsewhere.
+        run: >
+          ~/go/bin/bazelisk build //src:bazel-diff -c opt
+          --output_groups=-clippy_checks,-rustfmt_checks
+          --enable_bzlmod=true --enable_workspace=false
+      - name: Run comparison harness
+        # Dispatch inputs reach the harness through the environment and a bash array, never by
+        # interpolating them into the command line. Scheduled runs see empty inputs and take the
+        # defaults. --bazel points both tools (and the harness's own `bazel shutdown`s) at bazelisk;
+        # each workspace's .bazelversion selects the Bazel release, so both tools run the same one.
+        env:
+          IN_WORKLOADS: ${{ inputs.workloads || 'small,medium,large,xlarge' }}
+          IN_ITERATIONS: ${{ inputs.iterations || '3' }}
+          IN_TANGO_REF: ${{ inputs.tango_ref }}
+          IN_POOL: ${{ inputs.tango_pool_size || '1' }}
+          IN_QUICK: ${{ inputs.quick && 'true' || 'false' }}
+        run: |
+          args=()
+          if [ "$IN_QUICK" = "true" ]; then
+            args+=(--quick)
+          else
+            IFS=',' read -ra WLS <<< "$IN_WORKLOADS"
+            for w in "${WLS[@]}"; do
+              w="${w//[[:space:]]/}"
+              [ -n "$w" ] && args+=(--workload "$w")
+            done
+            args+=(--iterations "$IN_ITERATIONS")
+          fi
+          [ -n "$IN_TANGO_REF" ] && args+=(--tango-ref "$IN_TANGO_REF")
+          args+=(--tango-pool-size "$IN_POOL")
+          echo "harness args: ${args[*]}"
+          python3 tools/perf/tango_compare.py \
+            --bazel "$HOME/go/bin/bazelisk" \
+            --bazel-diff bazel-bin/src/bazel-diff \
+            --skip-build \
+            --metrics-out tango-compare-metrics.json \
+            --summary-out tango-compare-summary.md \
+            "${args[@]}"
+      - name: Publish step summary
+        if: always()
+        run: |
+          if [ -f tango-compare-summary.md ]; then
+            cat tango-compare-summary.md >> "$GITHUB_STEP_SUMMARY"
+          fi
+      - name: Upload metrics artifact
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: tango-compare-${{ github.run_id }}
+          path: |
+            tango-compare-metrics.json
+            tango-compare-summary.md
+          if-no-files-found: warn
+          retention-days: 90

tools/perf/BUILD

@@ -0,0 +1,22 @@
+load("@rules_python//python:defs.bzl", "py_library", "py_test")
+
+# tango_compare.py is intentionally NOT a py_binary, for the same reason as the serve harnesses in
+# //tools: by default it shells out to `bazel build //src:bazel-diff`, which would deadlock on the
+# output-base lock if the harness were itself launched via `bazel run`. Run it directly:
+# `python3 tools/perf/tango_compare.py` (see its module docstring and README.md). Python stdlib only.
+#
+# Its pure helpers (output decoding, timing math, summary rendering, CLI parsing) never shell out, so
+# a py_test over them is safe and runs in CI with the rest of //tools/....
+py_library(
+    name = "tango_compare_lib",
+    srcs = ["tango_compare.py"],
+    imports = ["."],
+    visibility = ["//visibility:private"],
+)
+
+py_test(
+    name = "tango_compare_test",
+    srcs = ["tango_compare_test.py"],
+    python_version = "PY3",
+    deps = [":tango_compare_lib"],
+)

tools/perf/README.md

@@ -0,0 +1,103 @@
+# bazel-diff vs Tango performance harness
+
+[`tango_compare.py`](tango_compare.py) benchmarks bazel-diff against
+[Uber's Tango](https://github.com/uber/tango) (Target Analyzer in Go) on the same workspaces and
+commit pairs, and cross-checks that both tools report the same impacted targets. The
+[`perf-tango-compare` workflow](../../.github/workflows/perf-tango-compare.yml) runs it nightly
+and publishes the comparison as a step summary plus a 90-day metrics artifact.
+
+## What is compared
+
+Both tools compute impacted targets the same way: check out each revision, `bazel query` the whole
+graph (`deps(//...:all-targets)`, streamed proto), hash every target, diff the two hash maps. The
+harness drives each tool through its own natural end-to-end flow on one workspace and one
+`base..head` pair:
+
+| | bazel-diff | Tango |
+|---|---|---|
+| Flow | `git checkout base` → `generate-hashes` → `git checkout head` → `generate-hashes --depEdgesFile` → `get-impacted-targets` (the README recipe) | one `get-changed-targets` RPC to Tango's example server, which leases a worker clone, checks out and queries each revision, hashes, and compares |
+| Timed as | wall time of the five stages (each also reported) | wall time seen by Tango's example client; the server's own `total_duration` is pulled from its log alongside |
+| Includes | its two checkouts | its two checkouts |
+
+Each flow's one-off setup (Tango cloning its origin and worker, both Bazel servers fetching
+`bazel_tools` & co.) happens in an untimed warm-up. Then, per iteration:
+
+- **cold** -- `bazel shutdown` for both tools' servers, then run each. Fresh server start plus full
+  graph load: what a CI job pays per PR today.
+- **warm** -- run each again with the servers alive (incremental re-analysis after the checkouts).
+  Tango runs with `-bypass-cache` so it recomputes rather than serving its cached result.
+- **diff only** -- no Bazel on either side: Tango with its result cache enabled (a treehash lookup)
+  next to bazel-diff's `get-impacted-targets` over the hash files already on disk. Informational.
+
+Tool order alternates every iteration so neither tool consistently runs into a page cache the other
+just warmed. Medians are reported; the ratio is Tango / bazel-diff, so above `1.00x` means
+bazel-diff was faster.
+
+The impacted sets from the last iteration are compared label by label. The tools' semantics are
+close enough that the synthetic workloads agree exactly; on real repositories differences (external
+repository handling, generated files) are reported, not failed, unless `--require-match` is given.
+
+## Workloads
+
+Synthetic workspaces come from [`tools/firecracker/bench/gen_project.py`](../firecracker/bench/gen_project.py):
+layered genrule DAGs with real source files and no external toolchains, so query time is graph
+load, not fetching. Presets (`--workload`):
+
+| Preset | Packages | genrules / package | Rule targets |
+|---|---:|---:|---:|
+| `tiny` | 60 | 3 | 240 (smoke tests only) |
+| `small` | 400 | 4 | 2,000 |
+| `medium` | 1,500 | 4 | 7,500 |
+| `large` | 4,000 | 4 | 20,000 |
+| `xlarge` | 10,000 | 4 | 50,000 |
+
+"Rule targets" counts genrules plus one filegroup per package; the query also sees
+`sources_per_package` (3) source files and the genrule outputs per package, plus `bazel_tools`,
+so `xlarge` is a ~130k-target `bazel query`. With no workload flags the harness runs the four
+nightly presets.
+Custom sizes: `--synthetic name:packages:targets_per_package[:layers]`. A real repository:
+`--repo-url URL --repo-from HEAD~1 --repo-to HEAD [--repo-clone-args --depth=50]`.
+
+## Running locally
+
+```bash
+# Build bazel-diff, clone+build Tango at the pinned commit, run the four presets x 3 iterations.
+python3 tools/perf/tango_compare.py --bazel ~/go/bin/bazelisk
+
+# Smoke profile: tiny workload, one iteration, reusing an existing binary and Tango checkout.
+python3 tools/perf/tango_compare.py --quick --skip-build --bazel-diff bazel-bin/src/bazel-diff \
+    --tango-dir ~/src/tango
+
+# Keep everything on disk and write the report files the workflow publishes.
+python3 tools/perf/tango_compare.py --workload medium --iterations 5 \
+    --workdir /tmp/tc --keep-workdir --metrics-out metrics.json --summary-out summary.md
+```
+
+Requirements: `git`, `go` 1.25+ (or any Go with `GOTOOLCHAIN=auto`, the default the harness sets,
+so the right toolchain is fetched), a `bazel`/`bazelisk` binary, Python 3.9+. No third-party
+Python. Tango's example server hard-codes `127.0.0.1:8081`; the port must be free.
+
+Tango is pinned to `TANGO_PINNED_REF` in the script so a Tango change never silently moves the
+baseline. Compare against another Tango with `--tango-ref <sha|branch|tag>` (the workflow exposes
+this as a dispatch input). `--tango-pool-size 2` lets Tango analyse the two revisions in parallel on
+two workers -- an architectural advantage it has over bazel-diff's sequential flow -- at the cost of
+twice the Bazel servers; the default of 1 keeps the comparison sequential-vs-sequential.
+
+## Output
+
+`--summary-out` is GitHub-flavored markdown (the workflow appends it to `$GITHUB_STEP_SUMMARY`):
+the per-workload timing table, bazel-diff's cold stage breakdown, the impacted-set agreement table,
+and any failures. `--metrics-out` is JSON with the environment (versions, SHAs of both tools, CPU and
+memory), per-phase medians/min/max, every raw sample (bazel-diff stages, Tango server-side
+durations), and the full label diff when the tools disagree.
+
+Exit status is non-zero when any run fails, and with `--require-match` also when the tools disagree.
+
+## Unit tests
+
+```bash
+bazel test //tools/perf:tango_compare_test
+```
+
+The harness is not a `py_binary` (it builds bazel-diff with `bazel build`, which would deadlock under
+`bazel run`); the test covers the pure helpers between raw tool output and the summary.