Incremental indexing: reuse committed generations and bound write amplification across worktrees - #806
Open
zzet wants to merge 185 commits into
Open
Incremental indexing: reuse committed generations and bound write amplification across worktrees#806zzet wants to merge 185 commits into
zzet wants to merge 185 commits into
Conversation
Preserve source-group cardinality and resolver provenance in point and batch updates. Fall back to structural indexing for ambiguous duplicate targets and ignore foreign-file rows when matching owned output. Validation: 37 focused tests, repeated race runs, and before/after presentation, structural, and ambiguous row-mutation benchmarks.
Do not advise restarting another daemon for a database newer than this binary. Close constructor resources when backend initialization fails, allowing a later safe retry to acquire the store lock. Validation: full serverstack suite, new regression tests under race detection three times, vet and private future-schema refusal benchmark.
Separate tree/path/mode/blob identity from staging and timestamp observations. Rehash uncertain files and fence root/path replacement, recent writes, and clock discontinuities; retain only bounded digest metadata. Validate with full unskipped gitstate normal/race tests, Linux and Windows compilation, and scoped lint. Benchmarks document the extra Git-status cost and exclude subprocess reads from sampler byte counts. Git-clean-filter admission remains a documented limitation.
Add an optional bounded regular-file capability with confined filesystem reads, bounded Git batch headers, and explicit non-content inventory. Preserve legacy ContentSource behavior and selected-layer ownership. Validated with source tests and race tests, lint, Windows amd64/386 and Linux amd64 compile checks, 1595 indexer tests (2 opt-in skips), and corrected 100x3 reader/inventory benchmarks. Native mutation publication is fresh. Post-change graph analysis timed out even for a single 14-line staged file; those graph checks are unavailable, not passed. Committed with user approval on the completed isolated validation. This is a prerequisite for safe manifest discovery, not completion of immutable-base publication or write-amplification repair.
Never compose a dedicated full root over mutable generation zero. Validate graph, checkout, repository and full-root identity for standalone and inherited dedicated ancestry. Preserve legacy commit composition and existing lease release behavior. Validated on actual files: full graphview tests, full race suite, scoped lint, formatting, and diff checks. Regression cases cover mutable-root leakage, malformed roots, masks, ancestry and public-ref lease cleanup. Paired 100x3 benchmarks show about 11 microseconds of inherited-root validation overhead; no indexing-throughput claim. All native edits published fresh. Staged native detect timed out after 59 seconds without symbol IDs; downstream native checks remain unavailable, not passed. User approved isolated tests/race/lint/benchmark fallback for unavailable native post-analysis. This is a prerequisite, not completion of immutable-base runtime or disk-I/O work.
Preserve the probe variable scope while clearing ineffassign. Validate 15 normal and 15 race executions, vet, and full store-package lint; record the isolated probe benchmark without attributing I/O variance to this declaration-only change.
Share the physical generation runner while preserving leader ownership and ordinary sparse behavior. Add guarded initial-snapshot construction with empty-commit and ready-reuse coverage; runtime activation remains deferred. Validation: full indexer suite 1608 pass, 2 opt-in skips; final claimed tests 13 normal and 39 race executions; vet and lint clean. Native staged analysis unavailable while the exact view refreshes; isolated validation used under the approved post-analysis fallback.
Normalize store-emitted sealed payload refusals at the panic boundary while preserving legacy nonfatal precedence and unrelated panic behavior. Add regression coverage and document validation and benchmark evidence. Validation: complete normal package coverage; split race coverage covers 780 passing and two skipped tests after the original run hit its 15-minute global timeout. Focused physical-source tests, vet, and lint passed. Native post-change graph analysis was unavailable; no native green result is claimed.
Serialize reference admission with retirement, qualify managed writers, and recheck physical flights and reader ownership before deleting payload. Preserve guarded terminal transitions and compatibility error priorities. Add deterministic Store, reader, and physical-builder regressions, repair fixtures to use valid catalog identities, and document measured transaction cost plus remaining runtime and disk-I/O gates.
The staging-graph fence lists every non-test file permitted to build an in-memory graph that no restart can recover. The dedicated claimed builder legitimately needs one: an initial claimed snapshot composes against an empty lower operand, and that operand never receives the indexed corpus. The reserved positive SQLite generation stays the durable payload target and the workers stream their output into it, so nothing recoverable is held in memory. Record the caller with that reasoning so the fence keeps failing for any other new entry.
…rded it Twenty-one evidence-log rows all read 2026-09-10 because that date was stamped by default, but the work they describe was actually recorded between 2026-09-11 and 2026-09-15. Each row now carries the date of the ledger commit that first wrote it, which also puts the date column back in order; the genuine 2026-09-10 rows and the later ones are untouched.
Two files needed a hand resolution, and both conflicts had the same cause. Main gave blame.EnrichGraph a third argument that scopes the pass to one repository, so a walk over one repo's root can no longer stamp another repo's identically-pathed nodes. This branch had rerouted the same call to read the store and root handed out by the enrichment output authority. Each side rewrote the same call for a different reason, so neither rewrite could simply win; both were kept. cmd/gortex/daemon_controller.go: kept the branch's admission of an output generation before the enricher runs, and gave blame.EnrichGraph the target's repository prefix that main now requires. internal/mcp/tools_enhancements.go: kept the branch's per-target output admission together with its generation and superseded reporting in the blame enrichment handler, and gave blame.EnrichGraph the loop's repository prefix. Main's ownership data-state counting elsewhere in the same file merged without conflict and is unchanged. Everything else composed cleanly: main's hybrid vector channel in daemon status, the durable vector corpus restore on warm reconcile routes, the C# using-static call binding, and the dependabot module bumps.
Adds the merge evidence row (conflict resolutions, suites, race subsets, lint, both end-to-end matrices on the merged daemon, verifier findings), declares that the paired write measurement was not re-run after the merge, and points the reproducible-evidence gate at the merged-source row. Also corrects an orphaned commit reference and two stale file:line pointers.
The limitation cited the doc comment line; the constant itself is declared nine lines later.
The private git fixtures set GIT_CONFIG_GLOBAL and GIT_CONFIG_NOSYSTEM so a developer's configuration cannot change what they commit. That also hides every configured user.name and user.email, leaving git to auto-detect an identity from the passwd entry and the hostname. A developer machine has a hostname git accepts, so the guess works and the fixture looks hermetic. A CI runner has an unqualified hostname, git marks the guess bogus, and every commit the fixture makes dies with "Author identity unknown" - 22 failures on the Linux leg and 18 on Windows, all of them through the dedicated-advance fixture's git helper. Name the identity in the helper's environment instead. Putting it there rather than in -c user.name= arguments covers every git subcommand that writes an object, so a later merge or am in one of these fixtures cannot reintroduce the failure by forgetting the flags. Reproduced first by injecting user.useConfigOnly through GIT_CONFIG_COUNT, the one config channel these helpers do not override: red with the same "Author identity unknown" and the same file:line as CI, green afterwards.
(&url.URL{Scheme: "file", Path: path}).String() writes the "//" that introduces
an authority whenever a scheme and a path are both present. A POSIX path already
starts with "/", so the result is file:///tmp/... and is correct. A Windows path
does not, so the drive letter lands in the authority position and SQLite refuses
the DSN with "invalid uri authority: C:%5CUsers%5C...". That is ~25 Windows test
failures, nearly all of them through the two write-audit probes in the indexer
tests.
Move the rules to one place. internal/testdsn mirrors the production builder's
sqliteDSN: absolute path, separators folded to "/", a leading "/" prefixed when
the result lacks one so the drive letter sits in the path, and url.URL for the
escaping. Its own test asserts the exact DSN strings for Windows-style and POSIX
inputs, which is the only Windows evidence a POSIX host can produce, and is why
the rendering half folds backslashes on every platform instead of delegating to
filepath.ToSlash.
Every hand-rolled DSN in the branch's new tests now goes through it, including
the "file:" + ToSlash sites that survive a drive letter but escape nothing.
The isolated tracking child is this test binary re-executed with an environment built from nothing but the isolation variables. That is not a loadable process environment on Windows: the binary is CGo-linked, and the loader resolves its own runtime DLL imports from PATH before main runs. With PATH rebuilt as git's directory plus System32, the toolchain directories are unreachable and the child dies at load time with STATUS_DLL_NOT_FOUND (0xc0000135), which the parent reports as "isolated tracking child failed" with no child output at all. Start from the parent's environment and drop only what the fixture supplies itself - HOME, the temp and AppData names, and anything prefixed GORTEX_, XDG_ or GIT_ - then append the isolation variables as before; exec dedupes to the last value per key, so the overrides still win. The parent's PATH goes behind the directories the fixture names, so the resolved git stays first. SystemRoot, COMSPEC, PATHEXT and the rest of a startable Windows environment now come through on their own. The child's own git helper inherits this environment rather than building one, so it gets an explicit author and committer identity here too: the isolation hides every configured user.name, and a runner's unqualified hostname turns git's fallback guess into a fatal "Author identity unknown". Isolation is unchanged and the child proves it: it still asserts HOME is unset, the XDG and temp roots are the private ones, no external provider or telemetry endpoint is inherited, the four Git guards hold exactly, and every platform directory resolves inside the isolation root.
The lifecycle matrix probed free disk space with syscall.Statfs, which does not exist on Windows. That single helper made the whole cmd/gortex test package fail to build there, so none of its tests ran, while Ubuntu and macOS built it fine. Move the helper into two build-tagged files. The POSIX file keeps the statfs implementation unchanged. The Windows file reports the space available to the calling user via golang.org/x/sys/windows.GetDiskFreeSpaceEx, the same figure internal/platform/disk_windows.go already uses; like statfs f_bavail it excludes space the caller cannot actually write, so the filler loop still stops when the volume is full for the user running it. golang.org/x/sys is already a direct requirement, so this adds no dependency. The disk-full row itself stays Darwin-only, since it builds its volume with hdiutil, and it records a named skip elsewhere rather than a pass. Both call sites already treated a failed probe as "cannot measure" - a recorded note and the string "unknown" - and that is unchanged.
zzet
force-pushed
the
fix/incremental-index-write-amplification
branch
from
September 15, 2026 06:40
303cbfe to
aff1d47
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Indexing a repository with more than one checkout stops rewriting storage it does not need to change.
main 56a1c29d: a main advance with ten dependents writes 0.28× the bytes, untrack/retrack 0.51×, a cold index 1.04×; one phase regresses at 1.28× and is reported, not rounded away.Problem
On the measured baseline
main 56a1c29d, everything a sibling worktree reads sits on the primary's mutable generation zero. Four consequences the measurements name directly:checkouts: 1, checkout_routes: 0).Underneath that, a payload write paid per-row B-tree maintenance across 19 secondary indexes, and a read-only tool call wrote a durable ~37 KB sidecar transaction.
What changes
Committed generations are published on demand and are immutable in effect. Sealing a generation is separate from making it active, and publication waits for a consumer: a no dependent checkout skip plus an on-demand schedule from the first reader's claim replace the unconditional one. Adoption is fenced on owner, incarnation, lifecycle, the previous active pointer, the desired tree and the complete build identity; head and active pointer move together or not at all. Generation zero is never relabelled.
A dependent pins the base it was built against. A committed-base advance costs a dependent whose own tree did not move zero commit-layer builds, zero working-tree builds and zero catalog or payload writes. Old coherent routes stay available with truthful freshness while a new one composes, and a new base is never spliced into an old delta.
A matching tree is copied, not re-parsed, inside a generation-scoped bulk write window whose bracket closes on every exit path. The earlier candidate measured 0.35× the WAL bytes of the re-parse route on a 1,500-file-sized payload, before counting avoided parsing; that figure predates the main merge and subsequent production correction.
Safely separable context is withdrawn from persisted output. The in-memory route removes context from payload, FTS, inventory and ancillary output before publication. Contract-bearing context paths are conservatively retained as explicit output because canonical contracts can still have surviving owners after file eviction. The hard check against payload on a withdrawn path remains intact, and ordinary context still withdraws. This can retain extra unchanged file payload; its write cost has not been remeasured. An earlier 120-file/6-changed audit measured WAL −32.1 %, db −17.4 %, free-list 73,728 → 0 before this correction; where the in-memory route cannot be taken the build falls back to write-then-withdraw and reports it.
A committed build reads the tree it advertises, and every rider tells the truth. Side channels can no longer reach dirty working-copy bytes on an immutable view; text search and file bytes come from the selected snapshot or are refused. A degraded capability, a drifted route, a labelled base fallback and a refused answer each say so in their own words.
Freshness is a contract the caller can state, and the tool list did not grow.
require_fresh, an absolute RFC3339wait_deadlineandrequire_exactare accepted.fresh:truesays which route answered, never that a wait returned;require_exactrefuses every non-fresh outcome rather than a list of known reasons, so a new reason cannot bypass it; fallbacks stay read-only. A wait ends on a bound the request itself set — a 60-second deadline used to end at ~5 s, about 12× early, whenever agit statuswas slow. The three published profile bodies got smaller while absorbing it: 4,605 → 4,360 bytes against an unchanged 4,608-byte ceiling, 3,677 → 3,552, 7,518 → 7,356.Storage stays shared and the migration is additive. One logical store, not a database per worktree or generation; schema v25 adds a generation axis to the analysis cache, legacy rows are copied at generation 0, and the newer-schema refusal still works. Maintenance moved into a serialized, pre-emptible lane, and the savings ledger commits once per window instead of once per read-only call — the historical measurement retained a sidecar database of 2,516,768 bytes against 4,695,192 (0.54×), before the later main merge and production correction.
Upgrade cost
index.affected_by_reresolve_max) now reaches the committed-base arm at all, and the closure cap there is sized by the change rather than raised. No new required configuration; the matrices and the measurement are opt-in, behind variables with no default.Validation
Portability and maintenance repairs
Committed head:
5b18ea3d.CI run 35037425925 passed all 12 test jobs: Store, indexer, MCP and the remaining packages on Linux, macOS and Windows. Linux and macOS ran with race detection and coverage; Windows ran the normal suite. All seven supporting jobs also passed, including security and skill drift. The tested merge
acc97a2a4a109289eda7cb889f95c7b586114d02has the same tree as this committed head.These are package elapsed times, not whole-job times or controlled performance comparisons. Some unchanged smaller packages used Go's test cache.
Windows executed and passed all three repaired cases:
TestIncrementalMultiFileBatchKeepsFailedFileAndCommitsSiblings,TestIncrementalReindex_FailedFileSurfacedAndRetriedandTestWatcherStormBatchDeletionFailureIsolationAndOneResolve. The precondition diagnostic observed a real lock violation on content read while the file remained stale, admitted and stat-able through the default filesystem path. Their original failed-file reporting, sibling progress, retained retry watermark and recovery assertions passed. The ownership-source rebind test and all seven confined-reader cases also passed without skips.Skip identities are unchanged from the preceding
ccc02e22CI on every partition. Windows recorded 13 indexer, one Store, 42 MCP and 105 remaining-package skip events, including subtests; the logs retain their individual names and reasons. These include platform-specific and opt-in checks. The original acceptance-gate statuses and all 31 declared limitations remain as recorded below; this unit-test matrix does not supply new paired-write measurements or rerun the opt-in end-to-end matrices.The follow-up local checks executed four common indexer tests on Darwin/arm64 in both normal and race/coverage modes: four passes in each mode, with no failures or skips. Formatting, whole-module build, vet, all-package test compilation, two test-name inventories, and lint also passed. The subsequent Windows-only helper change leaves all 4,898 declared repository inputs to those checks unchanged. Twenty-six earlier Go results are reused only for unchanged declared inputs; the earlier full local indexer normal and seven race runs below remain attached to their recorded source.
The Windows read-denial helper separately passed formatting, Windows/amd64 vet, and standalone test-file cross-compilation with CGO disabled. These checks compiled the helper without executing a Windows binary or cross-building the full indexer package. The helper preserves the default filesystem path, holds an exclusive byte-range lock, checks that metadata access succeeds while a second-handle content read fails, and explicitly unlocks and closes during restoration. The original failed-file, sibling-progress, retry-watermark and recovery assertions remain intact.
The preceding CI at
ccc02e22passed 11 of 12 test jobs. It proved all seven Windows regular-file cases and the ownership-source rebind case. The three remaining Windows cases failed because their read-denial fixture still allowed a direct read before indexing; stale state, admission and metadata access were correct. The replacement lock fixture's runtime result is reported in the current CI record above. The exact reason the prior sharing-mode handle was ineffective on that runner remains unexplained.Local validation recorded at
6e7b2db8The historical local checks in this table ran against merge
fc13d2a03b1fcb08b7fdd926bfe53f094bc1428cplus the repairs subsequently committed at6e7b2db8. Their 27-path fingerprint isd57b208eccf742d5f8a8a9263b6a18dc185a828b896f03a720acdc17dccb3c4b. They used Go 1.27 on Darwin/arm64, CGO enabled,GOMAXPROCS=2,GOMEMLIMIT=2GiB, isolated home/Git/temporary directories, and serialized test processes. Per-run manifests bracket Go-declared source/dependency/embed inputs and selected fixture/configuration inputs; those inputs remained unchanged. These checks do not make system libraries, SDK headers, other runtime-discovered inputs or daemon background load hermetic.The unlimited-error indexer compilation and both focused Linux audit regressions pass; their race-and-coverage run has two top-level/four total pass events, no skips or failures. All 37 audit-helper calls pass the actual fixture Store. The scoped independent reviews found no weakened workload or assertions. Graph-excluded tests require this actual compiler and test evidence; a graph zero-count is not treated as proof of absence. The new cancellation test covers an already-canceled context, not a timed post-join cancellation race.
The earlier local timings below used different source and, for the large race suites, no coverage instrumentation. They remain historical evidence and are not controlled speedup comparisons.
The previous CI run at
4ef554463b1d0099c9bfced0cdd2cdbd333937e4failed all three platform test jobs. The repairs close notebook sidecars after shared-server cleanup, make command fixtures and compile-database paths portable, and add a confined Windows regular-file reader. Dedicated-write audits now join already queued maintenance before installing their external SQLite triggers and taking the baseline. Their trigger corpus and logical-write, WAL-size and WAL-timestamp assertions remain intact. Store conformance tests reuse private copies of a pristine schema while preserving the conformance corpus and caller-owned cleanup.CI selects every package exactly once across four partitions on each platform: Store, indexer, MCP and the remaining packages. Linux and macOS retain race detection and coverage; Windows runs the normal suite. The 45-minute package timeout and 60-minute job limit are unchanged. Workflow lint and selection dry runs pass; the current CI result above records execution on all three platforms.
The branch also integrates
main fdf7fd02d3f0db59b5a7ca834423850f9da567bain mergefc13d2a03b1fcb08b7fdd926bfe53f094bc1428c. The merge was conflict-free and preserved the pending repair-file bytes. Main's resolver change belongs to the Store package's compiled dependencies, so the pre-merge Store race-and-coverage pass is retained as scoped evidence and the merged source is validated separately.The Windows reader's parent-directory acquisition can synchronously await a directory oplock; cancellation cannot interrupt that OS open. Confinement, final-reparse rejection, regular-file checks and handle cleanup remain enforced. The seven confined-reader cases and the ownership-source rebind case passed in Windows CI at
ccc02e22; the latest CI record above covers the current revision. See limitation 31.Historical local validation at
4ef55446Local checks ran against the saved Go inputs subsequently committed at
4ef554463b1d0099c9bfced0cdd2cdbd333937e4. They used Go 1.27 on Darwin/arm64,GOMAXPROCS=2,GOMEMLIMIT=2GiB, and isolated home, Git and temporary directories. Before/after manifests record unchanged declared inputs. Their scope differs: indexer manifests cover known/changed Go and module inputs plus the unchanged oracle files; root runners additionally cover Go-declared dependencies and selected fixture/configuration files. They do not establish hermetic execution or control daemon background load.4ef55446The Store’s 290 added tests pass in 209.80s, missing the 180s target by 29.80s; the slowest takes 14.91s, below 20s. The indexer’s 491 added tests pass in 702.40s, missing the 240s target by 462.40s. Its only added test above 20s is the pinned-dependent case at 23.42s in chunk order. The isolated pinned-dependent race repetitions passed at 19.87s and 20.10s; the latter also exceeds 20s. These runtime targets are test-cost measurements, separate from product write-volume measurements.
The subsequent CI run at this source failed all three platform test jobs. Its local results remain historical evidence; they do not describe the repaired source or establish current platform readiness.
Validation manifests retain exact commands and named skips; the execution ledger records their scope and results. End-to-end matrices and paired write measurements were not rerun at this source. Historical E2E failures and all acceptance-gate boundaries below remain in force; these package results do not close them.
The current changes give private Git fixtures their own identity, make SQLite test URIs portable while preserving literal POSIX backslashes, preserve the Windows child environment, and use a portable free-space check. Test fixtures reuse private pristine stores and fresh Go-only registries only where the corpus is proven to contain Go exclusively. Reopen/isolation regressions protect against accidentally replacing populated databases. The checkout-removal test uses a separate terminal-error control and retains all four operations against a live lifecycle.
The publication correction preserves canonical contracts when withdrawing resolution context. Its regressions reproduce the previous failure and check ownership edges, masks, and unchanged lower layers. A second correction makes composed graph totals use the resolved base node’s file when accounting for replaced or removed identities. This prevents opaque IDs from being subtracted twice and relocated IDs from being missed. Its eight-case regression compares NodeCount, Stats and repository totals with expected enumeration, preserves the base graph, and rejects a direct AllNodes fallback from aggregation. These checks do not close an end-to-end or performance acceptance gate.
Historical validation provenance
Historical commit IDs below identify the source recorded when each binary was built. A later message-only history rewrite preserved every commit tree and merge-parent order; it did not rerun those binaries. For orientation, the recorded merge
0dd501bcis nowa69e441a, and measured candidates2fd5db82and271a9e9fcorrespond to0caa20a6and19df574e. Historical numbers remain attached to their original recorded identities.Merged with
mainorigin/mainata4b5c4df— 25 commits past the base56a1c29d, 45 changed files — is merged into the branch as0dd501bc, whose parents are85d315c6(the branch head before the merge) anda4b5c4df. Exactly two files conflicted, both on the sameblame.EnrichGraphcall each side had rewritten: the branch's store/root routing is kept verbatim andmain's new repo-scoping argument is applied to the rerouted call, so neither side's change is dropped. Every other path in the merged tree is byte-identical to git's own clean-merge result. The merge was validated for correctness; its write cost was not remeasured. See limitation 28.0dd501bcgo build ./...,go vet ./..., andgo test -count=1 -run '^$' ./...(every test package compiles)cmd,internal/mcp(six first-letter chunks),internal/indexer(seven),internal/resolver,internal/daemon,graphview,storecmdcoordinator/checkout/dedicated/lease/handoff/status,internal/mcp^Test[C-D],internal/indexerincremental/vector/warm/reconcileDATA RACEgolangci-lintv2.13.1 (the CI pin), whole modulegofmt -l ./cmd ./internalgortex-merged, sha256c8b941c5…)main 56a1c29dStatic checks and package suites
9fc2e7ceDATA RACE, no new skipgolangci-lintv2.13.1 (the CI pin), whole module;gofmt -lover the diff;go vet ./...;go build ./...9fc2e7cegofmtempty; vet and build exit 0internal/graph/store_sqliteunder-race9fc2e7ce-timeout 30mis not enough: it times out at 1800.6 s and is green at 90m in 2256.7 sea4aad93, commits through55b5f5bf-racearm — renames and comments onlySkips are reported with their printed reasons. The historical rows retain their original platform and opt-in gates. Current platform-specific regressions are identified separately in the final validation record; a platform skip is not a runtime pass on that platform.
End-to-end matrices
Seven opt-in matrices drive a private daemon through the public CLI. Below is the re-run at
9fc2e7ce(daemongortex-e2esha25628624cd5…), except the two rows that moved later, atfcc9ddd7/15b00d4fand again at the merged source0dd501bc.15b00d4f, with a dependent checkout and a live calibration (dirty halfseq +0, catalog 0; commit halfseq +1, catalog 3). At9fc2e7cenine of its ten cases failed on one shared instrument error, now gone15b00d4f; the SKIPs are the declared semantic-metadata shortfall. Both FAILs reproduce byte-identically on a daemon built frommain 56a1c29d, so they are pre-existing producer-side path-admission defects, not regressions9fc2e7ce, 20 rows: 13 pass, 2 not_exercised, 5 FAIL. Its own round scored 12 pass · 5 FAIL · 3 not_exercised against amain 56a1c29darm of 9 pass · 8 FAIL · 3 not_exercised: five fail on both arms, three fail on the baseline and pass hereDo not read "end-to-end validated" from this: five of the seven matrices ran at
9fc2e7ce; the no-op family and the edit taxonomy re-ran at15b00d4f, a descendant of it, and both ran a third time at the merged source0dd501bc. The other five have not run since9fc2e7ce. Both measurement verdicts ran against ancestor candidates (2fd5db82,271a9e9f) and neither was re-run at any later source.Paired measurement
Baseline
main 56a1c29dagainst candidate271a9e9f; three repetitions, medians with min/max; 1,500 generated files / 60 packages / 10 dependents / 20 commits / 10 edits / 60 s idles; judged against a budget file frozen from the baseline arm, never rewritten, digest-checked before judging.P0_cold_indexP1_idle_coldP2_small_editsP3_touch_stage_unstageP4_amend_same_treeP5_main_advanceP6_dependent_editsP7_dependent_untrack_retrackP8_idle_warmSeries: logical writes, checkpoint-excluded for
P2–P4andP8(an upper bound on non-checkpoint work).over budget: none · incomparable: none · regressions preserved: P4_amend_same_tree 1.28×. All nine phases completed in all three repetitions. Retained bytes after teardown 375,751,391 against the baseline's 434,887,204 (0.86×), under a frozen ceiling; store at run end 0.83×; at the end of the cold index 1.006×, with the generation table empty and one copy of the corpus innodes(14,989 rows on both arms, against 29,978 before).The one regression, plainly. Amending a commit without changing the tree writes 1.28× the baseline's bytes (112,432 median bytes) on a ~0.4 MB base, and no mechanism has been identified. The counters say what it is not: no publication (6 advances dispatched, 6 skipped, 0 published), no generation movement, not the amend alone.
What the headline is. The candidate's own counters put the main-advance saving in what a publish writes, not in publishes avoided: ~20 delta-shaped publishes, 20 dependent recompositions, zero reused base claims.
Declared limitations
All of them, as behaviour — what the branch does not deliver.
Text search on a committed identity is not served completely — no generation-scoped text corpus exists (the trigram index has no serialization path), and a generation built before this change keeps its explicit "complete" row until rebuilt.
No real compaction or reseed. Chain growth is bounded only by retention and the forced-full-root policy, as a measured number, not a proven bound: 48 advances gave
superseded=1, retired=0, swept=0.Non-Git repositories have no immutable source image; correctness and lifetime closure are scoped to Git-backed ones. A raw roster member is witnessed, not frozen — read under a 2 s budget, a timeout refusing the cohort.
Dirty-layer reuse does not survive a restart: a restart between two identical dirty states pays a full rebuild, and the dependent's base pin is in-process too.
The primary's own working route stays on legacy generation 0 — its uncommitted edits remain the standing skew, stated at
internal/indexer/checkout_coordinator.go:1759; publication is not activation.The forward closure is as wide as today: the forward-frontier delta gate (
builder_closure.go:872) is not implemented, and the import-placement harness widens the relative arm the way the closure may err.gortex reposreads freshness at generation 0 by construction — the base view-generation constant is declared atinternal/graph/store_sqlite/read_index_state.go:21and bound into theWHERE view_gen = ?query at:77— so it shows neither worktree nor derived generations; the door declares this in three places rather than changing it.Two identity changes each invalidate every cached generation once on first deploy — one deliberate deploy, as the upgrade cost states.
Producer incompletenesses survive the context/output split: clone and similarity symmetry stays incomplete for a sparse generation, the reference-fact sidecar is not generation-inherited, and enrichment on the in-memory route sees the change set rather than the closure — untested, its hook being nil on every fixture in the package.
In-memory context mode is an optimisation, not a guarantee: an oversized closure, a backend with no bulk path or a shadow slot not granted inside 2 s falls back to write-then-withdraw, where the orphan sidecars (
file_mtimes,file_index_failures,clone_shingles,vectors,constant_values) remain. Both routes publish the same generation.A downgraded reader has nothing to refuse on: the context ownership mode is a plain TEXT vocabulary extension with no version gate, so an older binary serves such a path empty until it is rebuilt. The new-binary direction fails closed.
A pin held by a dormant checkout cannot be asked back: its base is offered, refused and re-offered every sweep pass until the checkout is activated — bounded in payload, not in sweep work. The release path holds one slot; a second request overwrites rather than queues.
A recomposition cannot reuse the dependent's working-tree payload — the survive-restart half is not implemented; what is bounded, and measured, is the build.
The bulk window's second-copy payload ceiling ("≤ 260 MB, from 841 MB") is unverified: WAL bytes for the same payload are byte-identical across a 4000×
cache_sizerange, the dense secondary indexes stay live, andsynchronousis untouched.The change-bounded delta's re-measurement misses both targets — store delta 7.91 MB against ≤ 1 MB, a burst of 83.0 MB against ≤ 20 MB — attributed to its own withholding comparison, not to the cap. Its contribution to bounded costs is blocked.
The 6,000-file scale axis has no post-fix counterpart: cold-idle 7.44× and amend 674× were measured once, before the fixes, and that arm did not finish.
The same-tree amend regression survives at 1.28× (112,432 median bytes) with no identified mechanism, preserved rather than rounded away.
The copy route carries residuals: generation zero's post-ready enrichment sidecars are copied though a re-parse never ran (a superset, never less), the four ownership masks unscoped, and a resolver-version bump between that pass and the reservation stays unprovable; containment is proved, not equality.
Five product defects found by the resolution matrix reproduce on
main 56a1c29dand have no owner: a withdrawn definition rebinding to an unrelated same-named module withorigin=text_matched; a byte-identical restore not restoring the binding; a fresh-index oracle of the same tree binding correctly; a comment-only re-parse dropping provenance-bearingvalue_flowrows; a dynamically reached bind indistinguishable from a proof. With them, the user-levelconfig.yamlindex:block never reaching a repository's indexer (internal/config/manager.go:281-286).Two snapshot-correctness divergences found by the edit taxonomy are open and unowned: a withdrawn path's name is still served, and an excluded package's name is still served out of that file.
Some diagnostics no live run has exercised: the view-lifecycle counters the quiescence verdict reads never appeared live, so it guards nothing it can see. The rendering half is closed (
renderDaemonViews).X-Gortex-Cwdhas no hop or loop guard. The class is pre-existing — the bodycwdargument was always relayed verbatim — but the header widens the affected population; a mutually-claiming roster would cycle.The
internal/persistencesidecar database has no generation axis — notes, memories, scopes and notebooks in a separate file, exactly what the storage criterion excludes. Neither planned nor rejected.Measurement honesty. Process-accounted writes are not NAND writes; a WAL size is not cumulative writes; the checkpoint counter is a lower bound; a 1,500-file replay is not sustained daemon behaviour. n = 3, darwin only, host load uncontrolled, janitor at 5 s rather than 1 h. One named phase improved by a measured factor on one fixture under stated accelerations; that is not "the problem is fixed".
No acceptance gate is closed — the branch's evidence closes none of the ten, and bounded costs in particular is not claimable.
The raw-repository owner surface ships with no production registrar: its seven exported entry points have zero non-test callers, so a raw roster member cannot exist in a running daemon and limitation 3 describes a path production never takes.
Two mutable package globals exist only as test seams —
indexer.installCppIncludeSearchPath(indexer.go:1518) andstore_sqlite.maintenanceQuiesceTimeout(store_compact.go:118), each overwritten by one test file and unguarded against a parallel test.The paired measurement predates the merge of
main. Both paired write-budget verdicts were taken on candidate binaries built beforemainwas merged in; none of the paired write-volume ratios was produced by the merged or current source.mainbrought its own changes to the write paths — the durable vector-corpus restore on all warm reconcile routes (ee34a3f7), the hybrid vector channel surfaced indaemon status(8555824e), C# using-static incremental binding and its narrowed reuse-tag carry (13633a09,023bf139), and the Pydantic validator/serializer entry points (4990233c). That delta was validated for correctness at the merged source (build, vet, the full test-package compile, 18 normal suite runs, 3 race subsets, both matrices, lint at 0 issues) and its write cost was not re-measured: not one phase of the paired protocol ran after the merge, so every paired write ratio and paired write-budget verdict above belongs to the pre-merge candidate and none is claimed for the merged or current source.Contract-bearing context paths are retained conservatively. A path carrying a canonical contract is explicit output even if no surviving owner ultimately requires every node on that path. This preserves coherent publication and can add unchanged payload. The paired write protocol has not run after this production change, so the historical ratios do not establish its write cost.
Correct aggregate counts can require additional base-node lookups. The count correction resolves deduplicated layer candidate IDs before checking their actual base-file ownership. It adds no full-corpus enumeration, but can issue more bounded lookups than the former ID-prefix shortcut. That additional lookup cost has not been benchmarked.
Windows parent-directory opens remain synchronous. On Windows, acquiring the confined parent directory uses synchronous os.Root operations. A directory oplock can delay that open, and context cancellation cannot interrupt it. The final metadata-file open rejects an incomplete oplock break, reparse entries, and nonregular descriptors. This change does not bound parent-directory open latency.
Open findings
.gortex.yamlexclude:does not withdraw content the incremental path already holds. Both reproduce byte-identically against a daemon built frommain 56a1c29d, on the same test binary, fixture and oracle, three samples a minute apart — so neither arm is convergence lag.main 56a1c29d: 9 pass · 8 FAIL · 3 not_exercised; this branch: 12 pass · 5 FAIL · 3 not_exercised); three fail on the baseline and pass here. They are listed in limitation 19.9fc2e7ce, where the whole production delta is a lint/format commit and "no measured quantity can move" is an argument rather than a measurement; and not after the merge, wheremain's own changes to the write paths make even that argument unavailable. See limitations 28–30.coretool-list byte budget was 20 bytes from its hard ceiling. The assertion is a strict< 97500; the recorded merged source measured 97480, against 97439 before the merge, the +41 bytes beingmain's newsearch_textlimit description. That recorded assertion passed with only 20 bytes of margin; current package-suite results are reported in the validation section; the exact byte figures here remain historical. Neither side is at fault alone — it is a composition effect, and a compaction pass is worth doing before the next description lands. At that recorded source, the two narrower published surfaces measured 29,968 / 30,000 and 21,525 / 21,600 bytes.gofmt -l ./cmd ./internalis not empty at the merged source — 99 files — and the merge caused none of it. Every one of the 99 is byte-identical to a parent (98 to the branch head before the merge, 1 tomain, which is itself unformatted under go1.27.0's gofmt), and neither conflicted file is among them. It is an inherited repository-wide condition, almost certainly a toolchain formatting-rule change, recorded rather than fixed here;golangci-lintreports 0 issues over the same tree.How to reproduce
Use a checkout of this repository, Git, the Go 1.27 toolchain, and the C toolchain required for CGO builds. The shell examples use a POSIX shell. Each opt-in test starts its own daemon under a private root: it inherits your
HOMEbut drops everyGORTEX_*,XDG_*andGIT_*variable from the environment and replaces them with privateXDG_CONFIG_HOME,XDG_DATA_HOMEandXDG_CACHE_HOMEdirectories, an explicit store path, its own Git fixture, and a privateGIT_CONFIG_GLOBALwithGIT_CONFIG_NOSYSTEM=1. The daemon's socket, PID file and logs are derived from that private cache root, so the socket is private too; none of it addresses a running daemon, store or configuration.The commands below cover selected package checks and the historical opt-in experiment. The execution ledger records local validation and its environmental boundaries; the CI record above supplies the completed current-head platform results.
The matrices and the measurement are opt-in: with the variables below unset they skip by name, so a plain
go test ./...never runs them. Each takes a daemon binary you build yourself —BINis any path you choose.Without
GX_SUSTAINED_IO_BASELINE_BINARYonly the candidate arm runs. The fixture defaults are the measured ones (1,500 files / 60 packages / seed 767 / 10 worktrees / 20 commits / 10 edits / 60 s idles / 1 s sampling), each overridable within bounds. The measured arms accelerate the janitor to 5 s from its 1 h default; a confirmatory arm at that default is recorded separately.The execution ledger records current state and validation, the measurement record records the historical write-volume results, and the design document explains the design.