feat: ABI-aware incremental rebuild to skip unnecessary downstream rebuilds

[LalatenduMohanty]

Problem

When a dependency updates (e.g., numpy 1.26 to 1.27), all downstream packages are rebuilt. Most updates don't change the ABI — downstream wheels would be identical. For large dependency trees across multiple variants and architectures, this wastes significant compute.

Proposed Solution

Three-layer system to determine which packages actually need rebuilding:

Layer 1 — ABI Fingerprinting: Fingerprint each wheel's binary interface after build. Store this as a compact JSON digest alongside the wheel. Pure Python wheels marked as is_purelib: true.

Layer 2 — Build Environment Diff: Compare build inputs (version, build tag, build-dep versions, env vars, patches) against previous build. If nothing changed, skip rebuild.

Layer 3 — Graph Pruning: Walk the dependency graph in topological order. For each package whose dependency updated:

• Pure Python downstream — always skip
• Native, no linkage to changed dependency — skip
• Native, links to changed dependency, ABI unchanged — skip
• Native, links to changed dependency, ABI changed — rebuild

Design decisions

• Build-time dependency version changes always trigger rebuild (catches header-level dependencies missed by ELF analysis)
• Conservative defaults: missing fingerprint, tool errors, unknown linkage all trigger rebuild
• Opt-in via --incremental-rebuild, with --dry-run-incremental for validation
• --force overrides everything (existing behavior preserved)

ABI tooling (open question)

elfdeps (already integrated, soname/symbol-version level) vs. libabigail (type-level ABI comparison via abidiff/abidw, new dependency). Architecture is tool-agnostic.

[1] https://sourceware.org/libabigail/

Rollout

1. Layer 1 only — fingerprinting, no behavioral change
2. Layers 2+3 behind --incremental-rebuild flag
3. Production enablement after --dry-run-incremental validation

Future opportunity

PyTorch stable C ABI for extensions would make this even more impactful — torch version bumps could skip rebuilding most downstream extensions.

[LalatenduMohanty]

After I get some feedback will send the proposal.

[LalatenduMohanty]

Related:

• trigger rebuilds of packages when one of their build time dependency is rebuilt #478 — ABI fingerprinting would trigger downstream rebuilds when a dependency's ABI changes, while naturally skipping pure Python dependencies like setuptools.
• feat: Unique wheel file names with a new wheel_build_tag hook #1059 — enriched build tags encode the build environment (OS, accelerator, torch version) into wheel filenames. If feat: Unique wheel file names with a new wheel_build_tag hook #1059 lands first, Layer 2 gets simpler since build tag mismatches already capture most environment changes.

[LalatenduMohanty]

End-to-end walkthrough

Build N (baseline):

1. Fromager builds numpy 1.26

2. add_extra_metadata_to_wheels() runs elfdeps (existing) to record which .so files the wheel provides and requires

3. New step alongside elfdeps: run abidw --exported-interfaces-only on each .so, hash the output to produce a composite ABI fingerprint. The result is stored as fromager-abi-fingerprint.json in the wheel's .dist-info/:

   {
     "schema_version": 1,
     "is_purelib": false,
     "tool": "abidw",
     "tool_version": "2.6",
     "so_files": {
       "numpy/core/_multiarray_umath.cpython-312-x86_64-linux-gnu.so": {
         "abi_hash": "sha256:a1b2c3..."
       },
       "numpy/linalg/_umath_linalg.cpython-312-x86_64-linux-gnu.so": {
         "abi_hash": "sha256:d4e5f6..."
       }
     },
     "composite_hash": "sha256:a1b2c3d4..."
   }

   Each .so gets its own hash from abidw output. The composite_hash combines all per-file hashes into a single deterministic value. For pure Python wheels the fingerprint is simply {"schema_version": 1, "is_purelib": true}.

4. Store the composite_hash in DependencyNode, serialized to graph.json

At the end of Build N, graph.json contains every package's version, dependency edges, and ABI fingerprint. Today a node in graph.json looks like this:

"numpy==1.26.4": {
  "download_url": "https://files.pythonhosted.org/.../numpy-1.26.4.tar.gz#sha256=...",
  "pre_built": false,
  "version": "1.26.4",
  "canonicalized_name": "numpy",
  "edges": [
    {"key": "setuptools==70.0.0", "req_type": "build-system", "req": "setuptools"},
    {"key": "cython==3.1.1", "req_type": "build-system", "req": "Cython>=3.0"}
  ]
}

With ABI fingerprinting, each node gains an abi_fingerprint field:

"numpy==1.26.4": {
  "download_url": "https://files.pythonhosted.org/.../numpy-1.26.4.tar.gz#sha256=...",
  "pre_built": false,
  "version": "1.26.4",
  "canonicalized_name": "numpy",
  "abi_fingerprint": "sha256:a1b2c3d4...",
  "edges": [
    {"key": "setuptools==70.0.0", "req_type": "build-system", "req": "setuptools"},
    {"key": "cython==3.1.1", "req_type": "build-system", "req": "Cython>=3.0"}
  ]
}

Pure Python packages get "abi_fingerprint": null. The field is optional for backward compatibility with older graph files -- a missing fingerprint is treated as "unknown, rebuild to be safe".

Build N+1 (numpy bumps to 1.27):

The previous graph.json is passed via --previous-bootstrap-file. For each package in build order:

1. Check own build inputs. numpy's version changed, so rebuild it.
2. Fingerprint the new wheel. Run abidw on numpy 1.27's .so files to produce a new composite hash.
3. Compare with previous fingerprint. numpy 1.26 hash from the previous graph vs. numpy 1.27 hash just computed.
4. Walk downstream. For each package that depends on numpy:
   • scipy is native and links numpy (elfdeps confirms linkage). Fingerprints match -- skip.
   • pandas is native and links numpy. Fingerprints match -- skip.
   • requests is pure Python -- skip (always).
   • pillow is native but does not link numpy (elfdeps shows no linkage) -- skip.

If the fingerprints had differed, scipy and pandas would rebuild. After rebuilding scipy, its own fingerprint would be compared against the previous build to decide whether scipy's downstream dependents need rebuilding. The decision propagates through the graph, stopping at the first unchanged ABI boundary.

Decision logic

For each package, the decision tree is:

Is this package's version or build environment changed?
    |
    NO --> reuse cached wheel (existing behavior)
    |
    YES --> rebuild this package
            |
            v
        Fingerprint the new wheel with abidw
        Compare fingerprint with previous build
            |
            v
        Did the ABI change?
            |
            NO --> downstream can reuse their cached wheels
            |
            YES --> downstream that links against this
                    package must rebuild

When dependency X updates and is rebuilt, downstream packages fall into four categories:

1. Pure Python -- no native code. Never needs ABI-driven rebuild.
2. Native, no linkage to X -- has .so files but none link against X (checked via elf-requires vs. elf-provides). Safe to skip.
3. Native, links to X, ABI unchanged -- links against X but fingerprint is identical. Safe to skip.
4. Native, links to X, ABI changed -- must rebuild.

[tiran]

Which Python packages link to shared libraries from other Python wheels? AFAIK it's only packages that depend on libtorch and NVIDIA CUDA wheels.

[rd4398]

This is a great idea and looks really promising. It will help us significantly downstream as well