do not verify the chunk id on every read from an AEAD repo, fixes #9994

[ThomasWaldmann]

Fixes #9994.

What this does

With the AEAD ciphersuites, the chunk id is part of the AAD (and, for repo objects v2, the header AAD + slot tag), so a successful decryption already proves that a holder of the repo key deliberately stored this exact ciphertext for this exact chunk id. A malicious or buggy repository therefore cannot swap, splice, truncate or substitute objects, whether or not we additionally recompute the id hash over the plaintext.

What assert_id() adds on top of that is only the detection of chunks whose content does not match their id — which only a malicious or compromised borg client that had the borg key could have written (see #7362, which made the check mandatory in borg2 back then).

Whether that extra full-plaintext hash pass is worth it depends on where borg reads, so this PR adds BORG_ASSERT_ID, a comma-separated list of the places that shall verify the chunk id:

place	what it covers
read	the general read path: borg extract, borg mount, borg export-tar, borg diff, ...
repair	borg check --repair (rebuilds archives by re-packing the item metadata stream into new chunks with fresh ids, recreates manifest / archives directory entries from what it reads)
transfer	borg transfer, for everything it reads from the source repository (a trust boundary)
rechunk	borg recreate --chunker-params ... (new ids are computed from the content read, so a violation would not be noticeable afterwards)

Default (variable not set): repair,transfer,rechunk — every place that re-anchors content, but not the hot read path. Unknown place names are an error. BORG_ASSERT_ID=read,repair,transfer,rechunk restores the "always verify" behaviour of borg 2.0.0b22 and before.

The chunk id is verified no matter what the variable says:

• in borg check --verify-data. That audit re-certifies the id/content invariant for all chunks of the repository and is what makes not verifying elsewhere defensible, so it is not configurable - there is no verify_data place name, and giving one is an error.
• for authenticated and none mode repositories and for borg 1.x repo objects: there is no AEAD there, so assert_id() is the read path's authentication and switching it off would silently demote authenticated to none. This is a key class property, KeyBase.id_check_is_authentication (default True, False only in AEADKeyBase), so any other/future key class is safe by default.

Measurement

Apple M3 Pro, 18 GB RAM (hw accelerated sha256), one 20 GiB incompressible file, --encryption=aes256-ocb --compression none, read back with borg export-tar arch - > /dev/null. 4 runs per setting, alternating between the two settings so that any drift hits both equally:

id hash	BORG_ASSERT_ID	run times [s]	median	throughput
sha256	default	20.43 21.13 20.79 20.93	20.86	982 MiB/s
sha256	...,read	29.42 28.91 28.82 28.05*	28.87	709 MiB/s
blake3	default	19.99 19.97 20.00 19.93	19.98	1025 MiB/s
blake3	...,read	24.70 24.95* 24.72 24.60	24.71	829 MiB/s

*) repeated run: the original one (31.64 s resp. 28.37 s) was disturbed by unrelated background load on the test machine.

So 28% faster for hmac-sha256 id hashes and 19% faster for blake3 ones. The file is much bigger than the RAM, so every run also reads 20 GiB from the SSD - for a repo whose data is in the page cache (or on faster storage), the relative difference is bigger.

With the default, running borg check --verify-data periodically is the recommended audit: it re-certifies the invariant for all chunks of the repository in the background, instead of on every read.

Implementation

RepoObj.parse() is the single choke point (nothing else calls key.assert_id()). It verifies if the key class says the id check is the read path authentication, if the place of this read is mandatory (ASSERT_ID_PLACES_MANDATORY, currently verify_data), or if it is in the configured set. Reads are attributed to read unless a command sets its own place for a repository (RepoObj.set_assert_id_place("transfer"|"rechunk"|"repair")) - which also covers the reads that go through DownloadPipeline.fetch_many(), e.g. --chunker-params re-chunking - or unless a single call names its place (parse(..., assert_id_place="verify_data")).

Tests

New unit tests in repoobj_test.py for the place logic: the default set, parsing of the env var (including whitespace, the empty value, an invalid name and the rejected verify_data), that verify_data verifies whatever the variable says, and that non-AEAD keys verify at every place no matter what is configured. Plus end-to-end tests that plant a chunk whose content does not match its id — i.e. what an evil client could write, which the AEAD layer authenticates happily — and check that borg extract only notices it with read in the list, while check --verify-data, check --repair, transfer (with and without --chunker-params) and recreate --chunker-params notice it with the default.

Full test suite passes locally (excluding remote/binary variants).

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.97%. Comparing base (a293a67) to head (87b8d61).
⚠️ Report is 1 commits behind head on master.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #9995      +/-   ##
==========================================
+ Coverage   85.88%   85.97%   +0.09%     
==========================================
  Files          95       95              
  Lines       17088    17124      +36     
  Branches     2616     2621       +5     
==========================================
+ Hits        14676    14723      +47     
+ Misses       1674     1667       -7     
+ Partials      738      734       -4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.