fsmonitor: preserve proofs and avoid repeated recovery scans - #59
Open
ttaylorr-oai wants to merge 44 commits into
Open
fsmonitor: preserve proofs and avoid repeated recovery scans#59ttaylorr-oai wants to merge 44 commits into
ttaylorr-oai wants to merge 44 commits into
Conversation
A cache-tree miss makes write-tree rewrite the physical index. Unlike other proof-preserving writers, it never initialized the clean-status configuration digest or restored authenticated external history. Consequently, an otherwise valid FSCF extension was rewritten without its token and stat bindings while FSMN and FSUC remained valid. Subsequent read-only status calls reject the weakened proof and rebuild the entire attribute manifest. Snapshot commands copying that index repeat the same scan, and no proof-loss event is emitted because FSUC survived the rewrite. Initialize the configuration digest and authenticated history before write-tree reads the index, using the existing default configuration callback. This preserves complete proofs for both primary and linked worktrees without weakening alternate-index, filter, or attribute checks. Cover physical cache-tree rewrites, paired provider tokens, read-only follow-up status, and hostile staged attribute changes under both object formats.
Temporary snapshot indexes can legitimately lose clean-proof bindings when they stage new paths. Do not initialize physical-index clean-status history for an explicit GIT_INDEX_FILE; preserve the previous configuration-only behavior and avoid scanning the complete attribute manifest. Handle write-tree help before inspecting repository hash settings so -h and --help-all continue to work outside a repository. Extend the physical-index regression with a newly staged temporary path, verify its intentionally weakened proof remains isolated, and require write-tree to avoid a full manifest scan.
Ordinary index writers can discard authenticated FSUC/FSCF history even when their changes are semantically safe. This leaves status, snapshot generation, hooks, and pull/rebase chains rebuilding the complete attribute manifest on subsequent commands. Carry clean-status configuration and history through physical write-tree calls, rm/mv, mixed reset, fast-forward merge, rebase, and autostash. Authenticate every affected index entry, and do not preserve proofs for attributes, ignore rules, active filters, conflicts, sparse indexes, or genuine alternate indexes. Keep untracked snapshots explicitly pending when diff closes a restarted provider epoch, and refuse to certify stale untracked data from tracked- only status. Restore private extensions stripped by legacy writers and recognize the authenticated update-index invocation used by doctor. Treat truly distinct history-free snapshot indexes as a strongly invalidated stat-only fallback instead of rebuilding the whole manifest. Reject canonical primary-index, lockfile, symlink, and hardlink aliases. Exercise primary and linked worktrees under both object formats, native provider restarts, offline files and attributes, real pre-commit hooks, mixed writers, pull/rebase/autostash chains, doctor recovery, hostile filters, scratch indexes, and read-only index immutability.
A mixed reset now authenticates every changed index entry before keeping its clean-status history. Updating an ordinary tracked file is safe, so require a coherent proof and no full attribute-manifest scan while still verifying that the worktree modification is reported.
A configured LFS filter is not necessarily active for any tracked path. Bind the normal clean-status configuration before diff, but retain the existing rule against loading sidecar history for configured filters. Treat Git's actual primary index.lock as a physical index when a pre-commit hook runs write-tree. Keep arbitrary temporary indexes, symlinks, and hardlink aliases on their existing isolated paths. Initialize authenticated history for native-fsmonitor cherry-pick and revert operations, while leaving hook providers and unsupported systems untouched. After a daemon reset, preserve tracked-only status directory snapshots only as pending candidates. Require an authenticated complete tracked proof, exclude scoped, sparse, and alternate cases, and validate every directory before its untracked entries can be trusted. Cover inactive LFS, active required filters, generating commit-a hooks, replay conflicts, daemon restarts, offline untracked files, and repeated immutable read-only statuses in primary and linked SHA-1/SHA-256 trees.
An intervening status may refresh the external checkpoint after a legacy writer stages a new directory. A later legacy unstaging can then restore the same logical index directly or recover both its authenticated semantic and untracked history across the changed index. Accept either complete authenticated restore path while retaining the exact status oracle and the existing no-bulk-traversal assertions.
A configured but inactive filter must not prevent diff commands from restoring authenticated external clean history. Otherwise linked indexes rewritten by older Git repeatedly rescan every tracked path, even when their matching checkpoint remains intact. Always enable external history after binding the repository configuration. Checkpoint restoration already validates configuration, attributes, provider tokens, worktree identity, and active filter scope. Exercise diff, diff-files, and diff-index against a linked worktree with an inactive required LFS filter and missing physical proof extensions. Also verify that activating the filter invalidates the checkpoint and fails safely.
External-history restoration currently computes a logical digest of every index entry before checking whether an authenticated checkpoint was actually loaded. Missing, malformed, or differently namespaced sidecars therefore add an unnecessary full-index pass to every read-only diff. Return before the digest when optional locks are disabled and no checkpoint exists. Preserve the writable path because its source digest is required to issue the first authenticated checkpoint, and retain missing-provider invalidation telemetry. Cover all three unusable checkpoint cases, subsequent authenticated restoration, writable checkpoint issuance, and active required filters in the existing linked-worktree regression.
Optional locks describe what a command may do, not whether it will publish external clean history. Ordinary diff, diff-files, and diff-index can therefore still hash every index entry after failing to find a checkpoint, even though none of them can create one. Explicitly mark writable status as the sole command that requires an external-history source digest. Missing or unusable checkpoints then return immediately for both ordinary and read-only diff commands, while status retains the pre-refresh digest needed to publish its first authenticated checkpoint. Exercise missing, malformed, and wrongly namespaced checkpoints with optional locks enabled and disabled, preserving writable checkpoint issuance and active-filter failure coverage.
After the builtin fsmonitor daemon loses its history, diff previously advanced the tracked token while leaving the untracked proof pending. Every subsequent full or read-only status then had to revalidate every cached directory again. Perform one complete authenticated worktree and exclude validation before updating the index. Pin the original physical index, keep its lock unheld during directory traversal, and verify that the same index still exists after acquiring the lock. Publish the repaired index only after tracked, untracked, configuration, and provider-token proofs all agree. Preserve fail-closed behavior for concurrent writers, held locks, sparse or split indexes, conflicts, filters, unreadable directories, and lost provider events. Extend scripted and real-daemon regressions to require fully paired proofs, bounded follow-up directory work, and correct offline changes.
60077b9 (status: preserve semantic history across scoped and index changes, 2026-08-11) lets a second closing query preserve verified sibling subtrees. A directory event cannot use that path while recovery is still establishing its semantic proof, so even a local change rebuilds the complete attribute manifest and rescans every directory. Permit a bounded directory check only around the second closing query, after the tracked semantic proof has closed. Require a continuous provider delta, the same pinned index and namespace, and an unchanged configuration. Reopen each affected attribute candidate beneath the worktree root and compare its contents, indexed fallback, or absence with the checked manifest. Keep the existing full fallback for lost or global events, hardlinks, unsupported indexes, and changed sources. The directory still dirties every tracked entry and its untracked cone. Directory timestamps never certify tracked contents. When this new shortcut is used, revalidate the completed scan's exclude-source proof before retaining sibling results: a directory event can also conceal a changed .gitignore. Do not add that extra work to ordinary file events. In a local 2,050-directory fixture, the second walk visits two directories instead of 2,051, and only one attribute manifest is built. Three alternating runs reduced median recovery time from 604 ms to 480 ms; the existing file-event path remained within 2% of baseline. Retain both paths in the regression test and synchronize attribute and ignore changes between the two scans to exercise the full fallback.
dfc55f9 (diff: repair provider-reset history before publishing its index, 2026-08-15) repairs tracked and untracked history together after the provider loses its token. The repair rereads the physical index, however, discarding the pending provider token and attribute manifest that diff has already obtained. The reread then repeats both operations. Keep the existing index state when its pinned physical source still matches, its logical changes are limited to acceleration metadata, and its configuration, attributes, and pending proof epoch remain valid. Clear every tracked entry's up-to-date and fsmonitor-valid bits, along with any bulk result: observations made before the repair epoch cannot certify tracked contents. The ordinary complete tracked and untracked validation still runs before the single index write. Retain the reread fallback, late optional index lock, and final pinned path check. A concurrent writer must win even with index.skipHash set. The regression test now requires one index read and manifest scan, and pauses a repair while another process stages a file. Across three local real-daemon runs with 2,050 sibling directories, index reads and complete manifest scans fell from two to one. Median repair time fell from 138 ms to 55 ms. Each run published a fully paired proof and kept read-only followers correct without another full directory scan.
cae38fd (status: close fsmonitor tokens around complete status scans, 2026-07-29) rechecks cached ignore sources before pruning an fsmonitor-valid untracked tree. Older UNTR writers hash the newline appended for parsing, however, so their cached OIDs do not match the indexed blobs. A read-only status cannot persist the canonical OIDs and repeats those content checks on every invocation. Group eligible repeated index OIDs before starting the existing workers. When a worker has read a source coherently and proved both its raw and newline-appended hashes, publish that content relation for the group. Other members still require their own complete, non-racy singleton-file stat match and the normal conversion check. Unique blobs, failed mutex initialization, hardlinks, weak stat settings, and conversions retain the existing content-verification path. No extra reads or object-store lookups are needed to populate the cache. Keep both historical newline forms in a repeated read-only regression. In a 1,024-directory fixture with identical 32 KiB ignore files, three alternating runs reduced median status time from 71.2 ms to 35.9 ms and avoided 1,020 content checks. An all-unique control stayed within 1.3% of baseline. The preserved large-worktree index avoided about 883 repeated reads, with unchanged status output and physical index bytes.
e4f0e54 (status: issue sidecars after a verified full scan, 2026-07-28) refuses to issue a clean-status sidecar when any clean filter is configured. The reader imposes the same restriction. Even after semantic verification proves that no tracked path uses a filter, a repository with a global Git LFS configuration must read its index on every clean status. Allow issuance in the existing configured-filter proof domain only when the current configuration and semantic hashes match the supplied digest, the filter scope is authenticated and inactive, and the usual complete history and closed-token checks pass. Readers retain the exact configuration, provider, attribute, exclude, index, and hardlink checks. No sidecar format change is needed: previous issuers could not write a sidecar in this proof domain. A complete command-line filter-disable override intentionally shares the underlying configuration digest. Record that normalization in the in-memory digest and reject such invocations for both issuance and reuse, so a temporarily disabled filter cannot prime a clean proof. Cover consecutive read-only hits without index reads, configuration changes, active filters, external attributes, and the normalized override boundary. On a 100,000-file repository with an unused required LFS filter, five alternating read-only pairs against the same physical index reduce the median from 24.4 ms to 20.8 ms. Each new reader avoids the 8.8 MB index entirely, without changing the index or sidecar.
Scoped directory-delta recovery must revalidate its exclude sources before retaining already scanned siblings. Removing that check would miss a changed ignore rule, but reopening each tiny source repeatedly is expensive. A 2,050-directory fixture spends about 155 ms validating only 20 KiB of ignore-file contents. After reading a regular source, use anchored fstatat() for the first pathname identity check. Preserve the follow policy and compare the same complete identity. The final check still reopens the source for reading, and the complete content hash and parent checks are unchanged. Nonregular sources and unsupported platforms retain their old path. Five alternating runs against the same initial index reduce the median exclude-validation phase from 155 ms to 129 ms and complete recovery from 476 ms to 447 ms. Both builds produce identical repaired indexes. Extend the unit tests to validate followed regular symlinks and reject same-content symlink replacement under a no-follow policy.
4bc13b1 (status: ignore command-scoped preload tuning in proofs, 2026-08-13) excludes both preload settings from authenticated status configuration hashes. Legacy FSCF proofs lack an explicit tracked-policy hash, so migrating them across harmless configuration drift also checks that configuration sources predate the index. That separate check recognizes core.preloadIndexBulk but not core.preloadIndex. Consequently, adding core.preloadIndex to an otherwise safe migration discards the authenticated tracked state. In the existing 258-path legacy fixture, enabling preload sends all 258 entries through bulk preload; disabling it performs 258 individual stats. The unmodified control refreshes only the reported path. Use the existing command-scoped acceleration predicate for legacy migration too. The proof-neutral key set and all source-epoch checks remain unchanged. Extend the migration regression to require preserved tracked state and no full preload, refresh, or manifest scan for both values.
7c19bf4 (fsmonitor: retain checked manifests across scoped directory deltas, 2026-08-15) limits both the tracked entries and the distinct attribute sources examined for a directory event. The entry limit makes a flat directory's 65th tracked file trigger a second full manifest and a whole-worktree retry even when the directory has one attribute source. Bound the expensive attribute-source checks, not the in-memory entry walk. Keep every entry eligibility check and the independent 64-source limit. The provider callback already has to invalidate each affected entry, and none of those content checks is removed. Configuration, namespace, index, attribute, and exclude proofs remain unchanged. In five alternating runs with 2,048 unaffected sibling directories, median recovery fell from 278 ms to 155 ms for a 65-file cone, and from 316 ms to 206 ms for a 1,024-file cone. Both cases build one manifest and retry three directories instead of rebuilding the manifest and visiting 2,050 directories. Extend the regression to require the exact number of refreshed entries, retain the distinct-source limit, and exercise attribute and ignore changes while a large directory is being closed.
f5d69f4 (status: admit clean sidecars for inactive configured filters, 2026-08-15) lets the first clean status in the preload test issue a sidecar. A later status can then return without reading the index, so the test's required config/coherent event is absent despite a valid clean-proof hit. Keep the existing test on the physical-index path by disabling optional locks for both the tuned and untuned readers. Require the index to stay byte-identical and the sidecar to remain absent, while retaining the strict coherence, no-manifest, and no-refresh assertions. Exercise the same five preload configurations separately against an issued inactive-filter sidecar in t7530. Require an exact independent oracle, a clean-proof hit with no index or directory scan, and unchanged index and sidecar bytes. This covers both paths without weakening the original proof-admission test. Production behavior is unchanged.
dreynaud-oai
approved these changes
Aug 16, 2026
b4bd792 (dir: share normalized exclude hashes across identical blobs, 2026-08-15) initializes its sharing map only after creating a mutex. With NO_PTHREADS, the dummy mutex initializer returns ENOSYS, so identical ignore files are read and hashed separately even though the sole preload worker runs synchronously. Initialize the map directly when threads are unavailable. The dummy lock and destroy operations are no-ops, and the map is still released after the worker finishes. A failed real mutex initialization continues to disable sharing in threaded builds. The existing historical-normalized-excludes regression failed with no shared object or reused index stat. It and the neighboring hardlink and conversion checks pass with NO_PTHREADS for both object hashes.
28dac0a (stash: preserve authenticated worktree proofs during creation, 2026-08-14) and 9b689df (diff: close fsmonitor tokens after complete tracked refresh, 2026-08-14) check exact index-preload counts. With NO_PTHREADS, preload_index() returns before entering that trace region, so the tests reject the serial refresh even when its output and authenticated worktree proofs are correct. Keep the exact preload counts in threaded builds and require the whole preload region to be absent otherwise. Continue checking the ordinary refresh count, index immutability, provider-token pairing, and status output in both modes.
ef7d150 (status: preload full recovery for scoped pathspecs, 2026-08-13) verifies recovery with 6, 8, 12, and 16 preload workers. Those requests are intentionally ignored in NO_PTHREADS builds, where the cache is checked synchronously by one worker. The test nevertheless requires the requested parallel count and fails before checking the remaining recovery cases. Retain the existing worker counts when threads are available and test one worker otherwise. Both modes still require exact status output, accepted token closure, and correct handling of outside-directory, ignore-file, and global changes.
b76a6c5 (status: show delayed progress during semantic refresh, 2026-08-10) protects progress updates with a mutex. The provider-reset repair added in dfc55f9 (diff: repair provider-reset history before publishing its index, 2026-08-15) similarly protects its warning flag. Both require pthread_mutex_init() to succeed, but the NO_PTHREADS stub returns ENOSYS. Interactive status can therefore abort when progress starts, and diff silently declines an otherwise valid durable repair. Initialize these mutexes only when threads are available. Serial builds run the callbacks on the main thread and use no-op lock and destroy operations. Threaded builds retain the existing failure handling, and the recovery proof, warning, snapshot, and index-lock checks are unchanged. Add a progress unit suite that checks repository selection and the start/update/stop lifecycle without installing signal handlers. It reproduces the abort before the fix. The existing provider-reset diff test also exercises the missing serial repair. Both object-hash variants pass the complete t7519 suite with NO_PTHREADS.
7d5f592 (status: reuse complete APFS untracked preload results, 2026-07-21) lets normal status reuse a complete bulk directory scan when no untracked cache exists. A native provider can discard an unauthenticated legacy cache while reading the index, but leaves an empty replacement behind. Even with bulk preload enabled, that replacement forces status to enumerate the entire worktree again. Record the discard on the replacement cache as process-local state. For a read-only, whole-worktree status, remove that empty cache only after the existing bulk-provider admission checks succeed. The normal bulk result can then supply untracked paths. Keep usable caches, writable recovery, explicit opt-outs, and ineligible query shapes on their existing paths. A failed scan or closing query still falls back to ordinary traversal; no untracked-cache proof or index is written. A 1.16-million-entry worktree spent 24.6 seconds in the duplicate traversal. The existing no-cache control reduced the same read-only command from 34.1 to 12.6 seconds. Cover the historical cache encoding, repeated immutable reads, independent status output, and the paired, writable, disabled, and unsupported-backend cases with both hashes.
e36890c (status: reuse bulk results after discarding legacy caches, 2026-08-15) avoids a second directory traversal when bulk preload is already enabled. Ordinary read-only status still takes the serial path because core.preloadIndexBulk defaults to false, even after losing a populated legacy cache and invalidating enough tracked entries to make the complete bulk scan useful. Let that exact recovery request supply the default while status closes its provider token. An explicit test override or configured boolean still wins, including false. Keep the existing useful-entry threshold, backend, index-shape, provider, attribute, exclusion, and closing-query checks. Clear the process-local request after closure so it cannot change later preload decisions. This does not enable bulk scans for healthy caches, ordinary sparse provider deltas, writable recovery, or unrelated commands. Document the narrow default exception and extend the legacy-cache regression to check automatic selection and each explicit opt-out. The automatic case previously performed a second walk even though the explicit bulk case already completed it correctly.
In e36890c (status: reuse bulk results after discarding legacy caches, 2026-08-15), we remembered a discarded legacy cache only if it had a directory tree. A foreign index writer can leave a valid legacy UNTR extension with zero directory nodes and no FSUC or FSCF proof. The 1,160,465-entry checkout which exposed the duplicate walk has exactly that state, so the populated-cache restriction missed the real workload. Remember every actual replacement of a matching legacy cache. Newly initialized and current-format empty caches still do not qualify. Keep the existing read-only, provider, configuration, filter, and bulk-scan admission checks unchanged. Extend the regression with an empty legacy cache and a clean provider response, checking both explicit and automatic recovery. Also verify that a current-format empty cache retains ordinary traversal. On the real checkout, the complete recovery path reduces an explicitly enabled bulk status from 26.2 to 8.6 seconds by removing its second directory walk, with identical output and no index or sidecar writes.
ttaylorr-oai
changed the title
The environment-override case added in 5abe1ce (preload-index: select bulk for read-only cache recovery, 2026-08-16) selects test_env through "$@" and prefixes it with read-only and fake-provider variables. Assignments before a shell function need not have the same lifetime as an external command's environment. Linux CI exposes both a supposedly read-only control that rewrites the index and a final writable control that unexpectedly enters read-only bulk recovery. Use the external env command for the bulk override, and explicitly enable optional locks for the writable control. This keeps the two test modes independent without changing production behavior.
In 8abeda5 (fsmonitor: preserve authenticated proofs across ordinary commands, 2026-08-15), the conservative temporary-index path required both a complete proof and partial manifest history to be absent. A sparse add to a copied index can retain only the manifest-complete and full-index flags. That history cannot authenticate the index, but it still excludes the conservative path. The next sparse add or write-tree therefore rebuilds the complete worktree manifest. Use the existing strong invalidation path whenever a genuine private index lacks a complete proof. Keep the physical-index, index-lock, device/inode, provider, filesystem, and split-index guards unchanged. This does not promote partial history into a valid proof: tracked entries and the untracked cache are invalidated before normal processing. Extend the primary and linked-worktree regression with repeated sparse staging through one copied index. Compare its tree with independently staged, fsmonitor-disabled controls, and retain alias, attribute, and required-filter failure checks. Both hash formats reproduce the unwanted scan without this change and pass with it.
The query-barrier tests added by ef6cdb5 (diff: reuse pinned observations during provider-reset repair, 2026-08-15) and 7c19bf4 (fsmonitor: retain checked manifests across scoped directory deltas, 2026-08-15) leave their subshell EXIT traps armed after successfully waiting for the background Git process. Under dash with --verbose-log -x, those now-empty cleanups can emit xtrace lines outside the test body's log redirection. The linux32 and linux-TEST-vars jobs pass all 97 assertions, but prove rejects the six extra lines as invalid TAP. Disarm each trap after a successful wait and clearing the saved PID. Keep it armed while the child is running, so a failed assertion still terminates and reaps the process. No production behavior changes.
In e36890c (status: reuse bulk results after discarding legacy caches, 2026-08-15), writable status deliberately kept the ordinary directory walk so that it could rebuild durable untracked metadata. The regression verifies the resulting paired proof, but not whether the next process can reuse it. Follow that repair with a read-only status. Require the same independent oracle output and a coherent paired proof, with no new manifest scan, physical directory opens, tracked-file stats, bulk walk, or index write. Also compare the physical index and all existing history sidecars before and after the command. Cached untracked-directory nodes may still be visited to produce the requested output.
The physical-index recovery added in 2181cb2 (fsmonitor: seed missing-history baselines from legacy tokens, 2026-07-24) can rebuild the complete attribute manifest before an index-only or path-limited reader starts its actual work. Repeating such commands against an incomplete proof repeats the same repository-wide scan. Let a narrowly classified reader request conservative bootstrap for its first index read. Expire the tracked-entry semantics and untracked cache instead of reconstructing a proof, so the requested paths still receive ordinary content checks. Restrict this to the canonical, non-split, non-sparse index with a reliable native provider. Cached diffs, bounded regular-file diffs, explicit check-attr requests, and stage-only ls-files requests opt in; other command forms keep the existing recovery path. Check the parsed index representation before skipping external-history restoration, which would otherwise hash a stale checkpoint before the fallback runs. Suppress diff's opportunistic index refresh only when this noncertifying fallback was actually used. Exercise main and linked worktrees in both hash formats. Compare each reader with a provider-disabled oracle, including same-stat content changes, changed attributes, and a failing required filter. Require no manifest reconstruction, checkpoint digest, or physical index write, and retain the original path for an on-disk split index and an unbounded worktree diff.
In d067582 (status: reuse closed proofs for scoped queries, 2026-08-11), a changed file can leave its cached directory valid while queuing a targeted refresh in memory. The UNTR format records the valid bit and cached names, but not that queue. If another command writes the index before consuming the event, a later status can trust the advanced provider token and omit a newly created untracked file. Mark each still-dirty directory invalid before serializing it. The existing invalid-directory path discards its stale names and omits the valid bitmap bit. Its validated descendants remain available, but a new reader must inspect the affected directory instead of treating the lost queue as completed work. No index-format change is needed. Add a regression in which update-index receives the event and rewrites the paired index, then a fresh read-only status receives no new events. Require the same untracked output as an independent provider-disabled reader, and verify that the reader does not write the index. The old code reports a clean tree in both object formats.
The provider-reset recovery in 49d03b6 (fsmonitor: retain untracked candidates across expired provider tokens, 2026-08-14) can replace a semantically unchanged index entry without invalidating its untracked directory. An ordinary indexed patch needs the same operation, but must not borrow the reset-revalidation flag to obtain it. Add an explicit add_index_entry() option for this case. Require the canonical strong-stat index, native fsmonitor, a complete semantic proof, equal authenticated tracked and untracked tokens, and a safe same-name regular-file replacement. Existing callers keep their old behavior. The replacement still clears CE_FSMONITOR_VALID. Keep the untracked cache's existing invalid and dirty state intact. In 04abe79 (dir: persist unresolved fsmonitor events as invalidation, 2026-08-16), serialization was made to expire unresolved dirty nodes, so preserving this state cannot turn pending work into a valid cache.
A path-specific stash removes and re-adds an existing index entry while applying its reverse patch. That unnecessarily invalidates the rooted untracked cache even when the path, type, attributes, and provider proof are unchanged. The next indexed stash apply then has to reconstruct the missing history. Enable authenticated history before the first canonical indexed-apply read. Preflight the complete patch batch and use the explicit safe replacement option only for ordinary same-path regular-file edits. An unsafe member retains the existing two-phase removal behavior for the whole batch. Attribute and ignore changes, active filters, renames, conflicts, alternate indexes, and weak stat configurations remain conservative. Give scoped stash inspection its own history guard, without changing the deliberate invalidation for dirty whole-worktree stashes. Reuse the existing digest suspension while writing the synthetic stash tree, and permit the final physical indexed restore to use unpack-trees' guarded semantic transfer. The synthetic index never receives that permission. Compare the complete stash cycle with an independent disabled-provider repository, including both stash trees, staged entries, and worktree bytes. Cover an unrelated staged sibling, mixed attribute/ignore patch orders, required-filter failure, alternate indexes, and pending untracked changes. Retained proofs must stay paired without losing untracked output or requiring a complete attribute-manifest scan.
In 062e636 (status: preserve proofs across guarded Codex invocations, 2026-08-14), harmless command-scoped overrides were excluded from the clean-status configuration fingerprint. A redundant submodule.recurse=false override still creates a separate proof domain. After a guarded checkout rewrites a linked worktree's index, ordinary diffs can repeatedly hash the stale checkpoint from the original domain, especially when index.skipHash leaves no usable physical checksum. Track the last known submodule.recurse value while reading config. Ignore a command-scoped false value only when recursion already defaults to false or the preceding effective value is known to be false. Preserve true overrides, effective changes, invalid values, and unknown scopes in the fingerprint. This does not broaden the configuration-epoch shortcut or change the semantic and tracked-policy hashes. Cover precedence and boolean spellings with both hash algorithms. Add a linked-worktree regression with a zero index checksum and authenticated sidecars: the guarded checkout must leave a directly reusable proof, and two ordinary diffs must match an independent oracle without hashing the old checkpoint. A persistent true setting remains a counterexample.
The new reader and stash suites run directly, but Meson's explicit test list does not include them. Its configure-time inventory check rejects the source tree before any tests can run. Register both scripts next to the existing status and preload coverage, and keep the existing history unit-test entries in the order required by check-meson. The nested stash regressions also negate test_grep at the shell level. Use the helper's negative-match form so a grep error cannot be mistaken for an expected absence. This satisfies test-greplint without changing the assertions or product code. The old source reproduces both inventory omissions and all four lint errors. The corrected inventory matches the test files, check-meson and the complete test-lint target pass, and the nine stash cases pass with SHA-1 and SHA-256.
The same-stat check added by d079f66 (fsmonitor: avoid repository-wide bootstrap for bounded readers, 2026-08-16) compares the bounded reader with an fsmonitor-disabled control. It rewrites a file without changing its size and restores the original mtime. Without USE_NSEC, the control compares ctime at second resolution. An edit in the same second can therefore leave its cached stat data unchanged. The control reports a clean file while the bounded reader correctly detects the changed contents, making the test fail spuriously. For this control only, copy the index and reinsert the tracked entry with its original mode and object ID using update-index --cacheinfo. The new entry has no cached stat data, so the ordinary diff must read the file contents. The staged contents and physical index are unchanged, and the existing index and sidecar immutability checks still apply.
bc0e68c (t7534: force the same-stat oracle to read file contents, 2026-08-16) gives the same-stat comparison an index whose tracked entry has no cached stat data. The later required-filter comparison still uses the physical index, even though the same deliberately hidden edit remains in the worktree. On a filesystem whose cached timestamps match, that control can report a clean file without invoking the required filter. The candidate reads the file and correctly reports the filter failure, so the comparison fails for the wrong reason. Reuse the content-backed scratch index for the required-filter control. The candidate continues to read the physical index, and the existing filter-error and index-immutability assertions remain unchanged.
428d829 (stash: retain paired history for scoped regular-file changes, 2026-08-16) adds a full-proof test for an unrelated staged sibling. Its setup helper clears GIT_TEST_SPLIT_INDEX inside a subshell, but the test starts another subshell after that helper returns. The linux-TEST-vars job therefore restores the inherited split-index setting. The two ordinary adds enable split index and downgrade the proof before the stash operation under test. The full-proof assertion fails for a fixture that no longer exercises the intended index form. Clear GIT_TEST_SPLIT_INDEX in the subshell that runs the scenario, as the other full-proof tests do. Keep the proof and independent-control assertions intact. The original fails and the corrected test passes under the inherited CI setting with both object formats.
e3c9a5b (fsmonitor: preserve proofs through filters, replay, and index locks, 2026-08-15) initializes authenticated history for cherry-pick and revert. But am invokes the apply machinery directly and never supplies the current clean-status configuration before its pre-apply refresh writes the index. The existing FSCF remains eligible for serialization, but the writer cannot bind it to the current configuration. It therefore clears the token and stat bindings even when the provider token has not changed. Each subsequent read-only status rebuilds the worktree manifest. Initialize the same guarded native-provider configuration and history before am first reads the index. Keep the existing checks for temporary three-way indexes, attribute changes, filters, and unsafe replacements. Cover primary and linked worktrees with an unchanged provider token, immediate physical-proof checks, independent commit and content oracles, and two immutable read-only followers. A patch that changes attribute semantics must still invalidate its proof.
5f2a52e (t7519: cover stash creation from linked external history, 2026-08-14) ages the initial tracked files before creating its linked history checkpoint, but later adds a new file with a fresh timestamp. On builds without USE_NSEC, the final writable status can reach the checkpoint writer in the same second as that new index entry. The writer correctly rejects the racy index, so the test intermittently misses its expected history/external-stored marker. Set the new file's mtime to 120 seconds before the current time before adding it. This keeps the fixture eligible without weakening the checkpoint guard. A future-mtime control reproduces the rejection on both the current and parent-equivalent native builds in SHA-1 and SHA-256. The aged fixture publishes the checkpoint in both hash formats.
d9b6634 (stash: be careful what we store, 2023-10-11) added a stash-like check before updating refs/stash. It formats the candidate object ID in a GIT_MAX_HEXSZ-byte stack buffer, but oid_to_hex_r() also writes a terminating NUL. SHA-1 fits in the buffer; SHA-256 writes one byte past its end when storing an otherwise valid stash. Reserve the extra byte required by the formatting API. Add an explicit SHA-256 repository test for stash store and ordinary push so the usual SHA-1 sanitizer jobs also exercise a maximum-width object ID. The old code aborts with a stack-buffer-overflow in hash_to_hex_algop_r(); the corrected buffer preserves the existing stash validation and updates.
e4f0e54 (status: issue sidecars after a verified full scan, 2026-07-28) added a prerequisite which runs status and expects a builtin fsmonitor token in the physical index. With a non-racy index, status can instead save external history and leave that index alone. The daemon works, but dump-fsmonitor sees no physical FSMN extension and the prerequisite skips every sidecar test. Query the running daemon with an explicit token, as t7527 already does, instead of requiring status to write the index. Keep the status smoke test and make its index non-racy so the probe covers this no-write case. The sidecar tests continue to check proof publication and reuse; no production behavior changes.
The FIFO tests added by 61f7705 (status: answer exact clean status before index deserialization, 2026-07-27) put their configured core.excludesFile directly in the shared temporary directory. Exclude proofs compare the parent directory's full stat data while reading a source. An unrelated process creating or removing a temporary file can therefore invalidate the initial capture. In the raced test, status can then take the conservative fast-excludes fallback and exit successfully before reaching the synchronization barrier. The test fails without having replaced the excludes file. Give each FIFO fixture a private temporary directory outside its worktree. The intended regular-file-to-FIFO replacement still changes the authenticated source and its parent, and all existing race and fallback assertions remain intact. Leave the production proof checks unchanged.
dreynaud-oai
approved these changes
Aug 17, 2026
5684c45 (status: refresh external history before exact proofs, 2026-08-10) captures the original logical index before refreshing its entries. That lets a later checkpoint authenticate the pre-repair source without losing a required physical stat update. On Linux, where the physical index has no durable identity, an ordinary tracked-only scoped status repeats that full-index hash even when it needs no repair. Two calls on an 858,001-entry dirty index spend about 350 ms each doing so. Defer the capture for explicit --untracked-files=no queries over a bounded set of literal, semantically safe tracked files. Require a full physical index, authenticated current provider and filter proofs, paired recursively valid untracked history, and a pinned original index. Keep the ordinary content checks, collection, and optional index lock. Omit the checkpoint only after those checks leave no persistent entry change, the complete proof still matches, and the entire index is non-racy. If refresh repairs an entry, read the original index through a duplicate of its pinned descriptor and recover the original logical hash. Then use the existing checkpoint and physical-index writers. Recheck the source under index.lock before publication and immediately before writing; if the source changed, roll back only our lock. This also handles a skipped index checksum without treating zero as a durable identity or allowing one zero-checksum writer to overwrite another. On a separately re-primed, 1,111,063-entry Linux index with a scripted provider, balanced old/new runs drop from 828-836 ms to 216 ms. The old logical digest takes 456-460 ms; the new path does not compute it. Both versions match an independent provider-disabled status and leave the physical index and existing checkpoints unchanged. The index read alone still takes 134-138 ms, and the whole-index racy check remains. Cover genuinely dirty and clean zero-stat entries, forwarded provider tokens, main and linked worktrees, both object formats, racy entries, active filters, and a competing zero-checksum writer. Other platforms, query shapes, and ordinary checkpoint publication keep their existing behavior.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Ordinary index writers and provider resets can discard authenticated
clean-status history even when the tracked changes are safe. For example,
a cache-tree miss in
write-treecan retainFSMNandFSUCwhileweakening a fully bound
FSCFproof. Subsequent status, snapshot, andhook commands then rebuild the complete attribute manifest.
Recovery can also duplicate work: after discarding an unauthenticated
legacy untracked cache, read-only status can scan the worktree in bulk
and then walk the same directories again to collect untracked files.
An index writer must also preserve pending untracked-directory events:
serializing the advanced provider token without those invalidations can
make the next status omit a newly created untracked file.
Changes
physical
write-tree, ordinary index writers, reset, merge, rebase,replay, stash, and real primary-index lockfile hooks. Keep temporary
indexes, active filters, attribute changes, conflicts, and unsafe index
aliases on their existing fail-closed paths.
the existing conservative stat fallback. Strongly invalidate tracked
and untracked hints without rebuilding a full worktree manifest; keep
canonical index, lockfile, inode-alias, attribute, and filter guards.
bootstrap path. Invalidate hints instead of certifying incomplete
metadata, avoid the full manifest and stale-checkpoint digest, and
suppress optional index writes only when that fallback is used.
diffpublishes its index. Reusepinned observations, keep the index lock unheld during traversal, and
recheck the physical index and all proof bindings before publishing.
deltas. Bound reuse by the affected attribute sources, and fall back for
changed ignore rules, lost events, races, or incomplete proofs.
unnecessary reopen of anchored regular sources, and admit clean
sidecars when configured filters are proven inactive. Preserve the same
correctness and progress behavior without pthreads.
empty legacy untracked caches using one complete provider-authenticated
bulk scan. Select bulk preload automatically only for this recovery;
explicit
core.preloadIndexBulk=falsestill opts out. Writable statuscontinues to rebuild durable authenticated untracked metadata.
Preserve paired history across authenticated same-path regular-file
replacements and scoped stash push/apply, without resurrecting dirty
cache entries. Mixed or unsafe patch batches remain conservative.
amfirst reads or refreshesthe index. Preserve fully bound proofs at an unchanged provider token;
attribute-changing patches and temporary three-way indexes retain the
existing fail-closed behavior.
submodule.recurse=falseonly when theeffective setting is already false. A guarded checkout can then leave
directly reusable linked-worktree proofs without making ordinary diffs
repeatedly hash an old checkpoint.
do_store_stash(). Giveoid_to_hex_r()room for its terminating NUL,and exercise an explicit SHA-256 repository in the ordinary stash suite
so the default SHA-1 sanitizer jobs cover the failing path.
daemon query. A successful status may deliberately preserve the physical
index and publish external history instead, so the prerequisite must not
require a physical
FSMNextension.Unrelated temporary-file churn must not invalidate their initial exclude
proof before the tests reach the intended race barrier. Keep the
production proof checks and all FIFO assertions unchanged.
These paths do not treat directory mtimes as proof that tracked-file
contents are unchanged.
Verification
Exact source
e94acd7cc2c7087a6e097e2b87251b04c374e59dis the currentreview candidate. Its clean build passes 386 unit tests,
check-mesonand
test-lint, and nine focused status, copied-index, race, dirty-UNTR,linked-precommit, mail-application, and native linked-checkpoint
regressions under both SHA-1 and SHA-256. The new ordinary stash test also
passes under both default hash modes; it creates a SHA-256 repository in
either mode. The corrected native sidecar prerequisite and six selected
sidecar cases, including both external-exclude FIFO tests, also pass under
both hashes, with actual provider-query evidence. The preceding
284a8ae410head also passed all nine scoped-stashcases and the bounded-reader matrix in both object formats with the
complete
linux-TEST-varsenvironment.The exact stash correction has paired ASan/UBSan evidence. An immutable
fec9824447build passes ordinary SHA-1 stash operations, but the newexplicit SHA-256 test reproduces the stack-buffer-overflow under both
default hash modes. With only the one-byte buffer correction and that
regression, the precommit build passes 386 units and the complete ordinary
and scoped stash suites under both hashes: 308 successes, four existing
known breakages, two expected skips, and no unexpected failures. This
precommit result is not a substitute for the clean successor's final
sanitizer qualification.
The clean
687de436f1no-pthreads gate passed 386 units and 206assertions. Its sanitizer build passed 386 units, then the selected
sidecar suite exposed an inherited prerequisite false negative. Matched
native controls on immutable predecessor and candidate builds reproduce
the failure under both hashes: non-racy status stores external history,
leaves the physical index unchanged, and the old prerequisite incorrectly
reports no fsmonitor. Checking the real daemon response passes all four
controls while preserving that no-write behavior. The corrected
prerequisite also passes all eight selected sidecar sanitizer cases.
ef3af6c978changes only this test setup; production is unchanged from687de436f1.The clean
ef3af6c978no-pthreads gate passes 386 units and 206assertions. Its ASan/UBSan build also passes 386 units and all 82 selected
assertions under both hashes, including the ordinary explicit-SHA-256
stash regression and all eight native sidecar cases. Its hosted sanitizer
job also passes the full ordinary stash suite. These results remain tied
to
ef3af6c978.The clean
e94acd7cc2no-pthreads gate passes 386 unit tests and 206assertions. Its ASan/UBSan build passes 386 units and all 86 selected
assertions across 12 suite/hash runs. Both default hashes execute the
explicit SHA-256 stash regression and all six native sidecar cases,
including both FIFO tests. The authoritative counts, exact sanitizer
binary and helper hashes, raw logs, and build provenance are retained.
The full native and compatibility gates remain pending.
The external-exclude fixture correction has a matched causal control.
Unrelated sibling churn reproduces the exact pre-barrier
fast-excludesfallback on immutable parent and candidate builds under both hashes. The
exclude file itself remains unchanged. All four controls pass when the
fixture uses its own parent directory, and still perform the real FIFO
replacement and require
fast-excludes-raced. The original quiet controlalso passes. The proposed test-only change then passes all 12 selected
native sidecar assertions under the pinned
ef3af6c978sanitizer build.The specific parent mutation in the hosted failure was not recorded; the
controlled comparison establishes the mechanism without weakening it.
The new
amfix passed 100 affected suite/hash runs before commit:1,296 successes, ten existing known breakages, and no unexpected failures.
The same-token positive and changed-attributes negative pass under both
object formats. The clean
810ab59f8dcommit also passed four real-daemonprimary/linked SHA-1/SHA-256 cases: immediate fully paired physical proofs,
correct commit and worktree contents, and eight immutable read-only
followers with no manifest scans or index writes. The subsequent
fec9824447commit changes only test setup;687de436f1adds the separatestash buffer correction described above.
ef3af6c978corrects thesidecar prerequisite, and
e94acd7cc2isolates the two FIFO fixtures.The test-only follow-ups register the new suites with Meson, correct
negative grep assertions, and give the same-stat and required-filter
controls an unchanged-staged-content scratch index that forces content
inspection. The staged-sibling fixture also clears inherited split-index
mode in the subshell that actually runs the scenario. Its original
failure and the one-line correction were reproduced under both object
formats; the full-proof assertions remain unchanged. The linked-history
fixture also ages its newly staged file before adding it. A native
future-mtime control reproduces the original
racy-indexcheckpointrejection on current and parent-equivalent builds in both hashes; the
corrected fixture preserves every checkpoint and proof assertion.
The exact
284a8ae410source also passed all 154 compatibility suite/hashruns: 4,388 successes, 36 skips, and 212 existing known breakages, with
no unexpected failures. Its complete native matrix passed all 12
suite/hash runs: 721 successes, nine expected skips, and no failures.
An earlier restricted run stopped before establishing its initial
provider-bound proof. The unchanged regression passes on both immutable
baseline and candidate builds, under both object formats, with normal
platform permissions; that earlier restricted-run trigger remains unknown.
The exact
fec9824447head subsequently passed all 154 compatibilitysuite/hash runs (4,388 successes), all 12 native runs (725 successes), and
the no-pthreads gate (204 successes). Its sanitizer gate then exposed the
pre-existing SHA-256 stash overflow, so it is not release-qualified.
Final qualification of
e94acd7cc2remains open.The scoped-stash source at
428d829240passed 1,832 assertions across120 suite/hash runs, including all nine new independent-oracle stash and
indexed-apply cases, the apply and stash compatibility suites, and
required-filter, mixed-patch, alternate-index, and pending-untracked-event
controls. The subsequent config-normalization source passed the same
nine stash cases, the bounded-reader matrix, and six focused status
regressions in both object formats.
The earlier complete recovery qualification remains specific to
3fcb4c5a12b6c699f1a85cc3f5601d6c6578d530:skips, no failures.
breakages, no unexpected failures.
coverage: 50 successes.
object formats and primary/linked worktrees, with independent
offline-change oracles.
read-only recovery, identical status output, and unchanged physical
index, sidecars, and daemon.
Review and release scope
Hosted CI on
116bc02570found missing Meson registrations andtest-greplint errors. The next head reached substantive tests and exposed
the stale-stat required-filter control and inherited split-index fixture.
The corrected
284a8ae410head passed all 92 executed hosted checks,with seven intentional skips. Both complete CI workflows passed, including
all previously failing configurations, leaks, and sanitizers. The next
810ab59f8dhead added the separately testedaminitialization fix.Its complete PR workflow passed; the parallel push workflow exposed the
pre-existing linked-checkpoint timestamp race described above.
fec9824447fixes that fixture; its completePR CI run and
push CI run each
passed 45 jobs with two expected skips. Those workflows did not combine
SHA-256 with sanitizers. The
687de436f1head adds the buffer correctionand an explicit SHA-256 regression that closes that gap. Its complete
push CI run
passed 45 jobs with two expected skips; the sanitizer job executed the
new ordinary stash test. The test-only
ef3af6c978prerequisite fix thenpassed its complete PR CI run.
Its parallel push CI run
passed 44 jobs and failed only the existing raced-exclude-FIFO fixture on
macOS GCC, with two expected skips. The same exact head and compiler
passed in the PR workflow; both Meson jobs passed all 50 sidecar tests.
The retained failure is a successful conservative
fast-excludesfallback before the intended barrier, not a blocked FIFO read. The
matched parent-churn controls above reproduce that failure and validate
the fixture-only correction in
e94acd7cc2. Its freshPR CI run and
push CI run are
in progress. The production checks remain unchanged.
The preceding
b208bc4f2298bdd37279f3b8e8835a07b82254fbhead passedall 92 hosted checks, with seven intentional skips. That includes both
workflows' AlmaLinux, linux32, and Linux test-variable jobs, which confirm
the optional-lock isolation and successful-cleanup-trap fixes.
The prior approval covers only
dfc55f903ca1dc355fd916fcae91974a37891017. Fresh independent approvaland platform CI must cover the current exact head. Earlier local or
hosted results are not substitutes for that review.
Release qualification still includes temporary-index commands, scoped
readers, stash push/apply, coherent packaged writers, and linked worktrees
on macOS and Linux. Full-size and native-daemon results must identify the
exact source tested. The release remains held until those gates and the
protected controller's reviewed integration are complete.
Writable tracked-only scoped status on platforms without durable index
identity can still compute a full logical-index digest. Avoiding that
cost without stranding zero-stat entry repairs needs a separate change;
it is not included here. An experimental writable directory-stat reuse
change was also dropped after its controlled performance comparison.