Record changed code entities in checkpoint metadata (gated, off by default) - #2041
Open
suhaanthayyil wants to merge 5 commits into
Open
Record changed code entities in checkpoint metadata (gated, off by default)#2041suhaanthayyil wants to merge 5 commits into
suhaanthayyil wants to merge 5 commits into
Conversation
Adds a fifth member to the persistent-store write union: a session-level backfill that writes an opaque entity-delta document into one session's directory inside an already-written checkpoint. It follows CheckpointAttribution exactly — an applier on treeWriter, a dispatch case on each backend (git-branch, git-refs, the fsstore reference backend), and a backfillTarget case so store routing sends it down the read order and falls through only on ErrCheckpointNotFound. The one behavioural difference from the other session-level backfill: the target session is resolved from the session ID rather than assumed to be the latest. A checkpoint can hold several sessions and each has its own deltas, so writing into "the latest" would attribute one session's entities to another. A session that is not present is a hard error, not the not-found sentinel: there is nothing to attach the document to and no slot to append. The v1 advance is a compare-and-swap, not a force-set. This backfill is the only checkpoint write that runs in a detached child, sharing no lock with condensation, so the tip can move between reading it and writing: a force-set would then commit onto a stale parent and silently orphan whatever landed in between. On a lost race the document is rebuilt on the winner's tip (bounded retries, then dropped with a log line) so both survive. Same on the git-refs backend, where the ref that moves is the checkpoint's own.
…oints (COR-782) Records which functions, classes and types changed between a session's attribution base and the committed HEAD, as entity_deltas.json inside the session's checkpoint directory. The deltas come from the entire-graph plugin binary, resolved on $PATH under the CLI's kubectl-style `entire-<name>` convention. Off by default behind the entity_deltas setting. The work does not run on the hook. Condensation writes the checkpoint, then forks a detached `entire __entity_deltas` child (execx.SpawnDetached, as telemetry and the trails-enablement refresh already do) and returns; the child runs the producer and attaches the result with a SessionEntityDeltas backfill. The hook's only added cost is a small job file and a fork. Running the producer inline would cost seconds per session per commit, serialised across sessions, which no post-commit path can afford and which the 2s session-end budget could never fit; a staging failure could also propagate out of CondenseSession and fail the whole checkpoint. Both are structurally impossible here: the scheduler returns nothing, and the backfill is a separate process writing a later commit onto an already-committed checkpoint, so every failure — producer absent, wedged, oversized, unparseable, or a store write that errors — is logged and dropped. With nobody waiting, the producer deadline is 120s. Children of one commit serialise on a per-repo lock so two sessions' backfills cannot clobber each other. Subprocess hygiene: the producer runs in its own process group and the whole group is SIGKILLed at the deadline (the checkpoint/remote pattern), so a forked worker cannot outlive it. Output is capped at 8MB — these documents live on the checkpoint branch forever, and a 200-commit range was measured at 2.1MB — and an over-cap run is dropped with an Info line. Document mapping, all verified against real producer output: - read the producer's actual top-level producer_version key - fall back to before_start_line, the only line a removal carries, instead of recording start_line 0 for every deleted entity - carry old_name whenever it differs from the resolved name, not only on type=="renamed": the producer also sets it on a move that renamed - carry the file's old_path down to its entities, so a git-renamed file's contents are not reported as having always lived at the new path - scope entities to the session's own files (the same set attribution scopes by). The producer diffs the raw base..head range, so without this every session's document would contain the other sessions' and the human's entities. A session that touched no files skips the producer entirely. - the producer's finer-grained in-place change types (signature_changed, body_changed) fold into the schema's single "modified" Observability and gating: one Info line per condense when the gate is on but the producer is unavailable, so an enabled feature with no binary installed is not silent at the default log level; per-entity noise stays at Debug. ENTIRE_ENTITY_DELTAS is tri-state, so 0/false/no/off is a kill switch over a repo setting that enables the feature, and an unset or unrecognised value falls through to settings. The frozen-schema fixture is captured verbatim from the real producer binary rather than hand-written, and covers every change shape including moved+renamed and a renamed file. The hang test asserts against the deadline rather than a slack bound that would pass with the timeout removed. The child must also not create push failures it did not have to. The pre-push OPF rewrite rebuilds v1 and compare-and-swaps the ref; a backfill landing between its tip read and that swap makes the swap fail and aborts the user's push. The rewrite therefore takes the backfill lock for its duration, so the child parks instead — nobody waits on a backfill, and its critical section is one small tree write. Taking the lock is fail-soft: a rewrite that cannot get it proceeds exactly as before, because the compare-and-swap, not the lock, is what makes the write correct. Other v1 movers are untouched. The job file carries a format version, since an upgrade between the fork and the exec makes writer and reader different builds; an unknown version is refused with a log line and an absent one reads as v1. A spawn that fails takes the job file with it rather than leaving one nobody will read.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds an opt-in, asynchronous “entity deltas” backfill so each persistent checkpoint can include a per-session entity_deltas.json describing which code entities (functions/types/methods/etc.) changed across the session’s base→head range. The feature is gated and off by default, and is designed to avoid adding meaningful latency to hook paths by computing and writing deltas in a detached child process.
Changes:
- Introduces the detached backfill pipeline (
entire __entity_deltas) that runsentire-graph diff --json, filters to the session’s touched files, and writesentity_deltas.jsoninto the checkpoint tree. - Extends checkpoint storage backends to support a new backfill write request (
SessionEntityDeltas), including CAS + bounded retry to avoid orphaning concurrent writes. - Adds settings + env tri-state gate (
entity_deltas/ENTIRE_ENTITY_DELTAS) and integrates the backfill lock with the OPF pre-push rewrite to prevent user-visible push aborts.
Reviewed changes
Copilot reviewed 25 out of 25 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| cmd/entire/cli/strategy/testdata/entity_deltas_producer_diff.json | Captured real producer fixture for diff→schema mapping tests. |
| cmd/entire/cli/strategy/manual_commit_opf_rewrite.go | Takes entity-deltas backfill lock during OPF rewrite (fail-soft). |
| cmd/entire/cli/strategy/manual_commit_condensation.go | Schedules detached entity-deltas job after checkpoint write. |
| cmd/entire/cli/strategy/entity_deltas.go | Core entity-deltas job, producer invocation, filtering, locking, and backfill write. |
| cmd/entire/cli/strategy/entity_deltas_test.go | End-to-end + mapping + concurrency tests for entity-deltas backfill. |
| cmd/entire/cli/strategy/entity_deltas_cancel_windows.go | Windows cancellation behavior for producer subprocess (no-op tree kill). |
| cmd/entire/cli/strategy/entity_deltas_cancel_unix.go | Unix process-group cancellation for producer subprocess. |
| cmd/entire/cli/settings/settings.go | Adds entity_deltas setting + ENTIRE_ENTITY_DELTAS tri-state gate. |
| cmd/entire/cli/settings/settings_entity_deltas_test.go | Tests for new tri-state gating behavior. |
| cmd/entire/cli/root.go | Registers hidden __entity_deltas command. |
| cmd/entire/cli/paths/paths.go | Adds EntityDeltasFileName constant. |
| cmd/entire/cli/execx/spawn_detached.go | Adds error-returning SpawnDetachedErr for callers needing spawn failure detection. |
| cmd/entire/cli/entity_deltas_cmd.go | Implements hidden __entity_deltas command entry point + logging init. |
| cmd/entire/cli/checkpoint/store.go | Adds CAS helpers + bounded retry utility for ref-race resilience. |
| cmd/entire/cli/checkpoint/routing_store.go | Routes new SessionEntityDeltas backfill to correct backend. |
| cmd/entire/cli/checkpoint/refs_store.go | Adds entity-deltas backfill for git-refs backend with CAS. |
| cmd/entire/cli/checkpoint/persistent.go | Adds tree-writer apply + git-branch backend backfill with CAS + retry. |
| cmd/entire/cli/checkpoint/persistent_write.go | Dispatches SessionEntityDeltas in git-branch store writer. |
| cmd/entire/cli/checkpoint/persistent_write_test.go | Adds dispatch coverage for SessionEntityDeltas. |
| cmd/entire/cli/checkpoint/persistent_entity_deltas_test.go | Verifies correct session targeting + ref-race behavior across backends. |
| cmd/entire/cli/checkpoint/fsstore/fsstore.go | Adds SessionEntityDeltas handling to the test-only fsstore backend. |
| cmd/entire/cli/checkpoint/fsstore/fsstore_test.go | Tests fsstore handling of SessionEntityDeltas. |
| cmd/entire/cli/checkpoint/aliases.go | Exposes SessionEntityDeltas alias in CLI checkpoint package. |
| cmd/entire/cli/checkpoint_policy_warning.go | Excludes hidden __entity_deltas from checkpoint policy warning. |
| api/checkpoint/interfaces.go | Adds SessionEntityDeltas write request type to the API contract. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Copilot review: the comment claimed base/head/schema_version were part of the mapping; only producer_version and files are actually decoded. Co-authored-by: Cursor <cursoragent@cursor.com> Entire-Checkpoint: 01M0DMSPQ6HH74A8SQKRK6JM4W
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://entire.io/gh/entireio/cli/trails/1083
Implements the entity-deltas half of the VCS keystone (COR-782): each persistent checkpoint gains an
entity_deltas.jsonin its per-session tree dir recording which code entities (functions/types/methods) the session changed — schema_version 1.0, shared contract with entire-brain's entity→checkpoint index.Off by default. Gated on
entity_deltassettings key /ENTIRE_ENTITY_DELTASenv (tri-state: force-on, force-off, fall-through). With the gate off the only cost is the settings read.Design — nothing runs in the hook path:
SpawnDetacheds a hiddenentire __entity_deltaschild; the hook's added cost is spawn only.entire-graph(silent skip + one Info line if absent), runsdiff --base --head --repo --json(120s cap, process-group kill, 8MB output cap), filters entities to the session's own files, and applies a newSessionEntityDeltasbackfill write request (modelled onBackfillAttribution, implemented for all four store backends).Verification: race and rewrite interactions covered by deterministic tests (
TestBackfillEntityDeltas_TipMovedUnderTheWrite_KeepsBothCheckpoints,TestRewriteUnpushedV1WithOPF_BackfillCannotLandMidRewrite, both mutation-verified — reverting the CAS or the rewrite lock fails them with the real failure). Producer fixture captured verbatim from a realentire-graphbuild (commit noted in the fixture).mise run checkgreen (lint 0, unit+integration, canary e2e 4/4).