BE-815: atlas: serve more than one generation at a time - #9642
Draft
indietyp wants to merge 74 commits into
Draft
Conversation
indietyp
added this pull request to stack #9506
September 10, 2026 10:56
|
The latest updates on your projects. Learn more about Vercel for GitHub.
1 Skipped Deployment
|
github-actions
Bot
added
area/deps
indietyp
temporarily deployed
to
pull-request
September 10, 2026 10:57 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
indietyp
temporarily deployed
to
pull-request
September 10, 2026 10:57 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
![semgrep-code-hashintel[bot]](https://avatars.githubusercontent.com/u/22116112?s=60&v=4){kind=small}
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## bm/be-831-hashql-support-type-parameters-in-the-id-macro #9642 +/- ##
============================================================================================
- Coverage 65.89% 61.14% -4.76%
============================================================================================
Files 1887 1352 -535
Lines 198443 140468 -57975
Branches 8248 6365 -1883
============================================================================================
- Hits 130773 85885 -44888
+ Misses 66140 53513 -12627
+ Partials 1530 1070 -460 Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
clippy found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.
Merging this PR will not alter performance
|
| Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|
as_constant |
< 1 ns | < 1 ns | N/A | |
constant_equal |
< 1 ns | < 1 ns | N/A | |
constant_not_equal |
< 1 ns | < 1 ns | N/A | |
access |
< 1 ns | < 1 ns | N/A | |
runtime_equal |
< 1 ns | < 1 ns | N/A | |
runtime_not_equal |
< 1 ns | < 1 ns | N/A |
Comparing bm/be-815-atlas-serve-more-than-one-generation-at-a-time (37e33e2) with bm/be-831-hashql-support-type-parameters-in-the-id-macro (d52ac7c)
indietyp
force-pushed
the
bm/be-815-atlas-serve-more-than-one-generation-at-a-time
branch
from
September 11, 2026 14:50
b72821e to
0038630
Compare
|
|
||
| // SAFETY: `buffer` holds exactly `source.len()` slots of `u32` with `u32`'s alignment, and | ||
| // `as_mut_ptr` points at its first byte. | ||
| unsafe { source.clone_to_uninit(buffer.as_mut_ptr().cast::<u8>()) }; |
| fn boxed_clone_empty() { | ||
| let source: Box<IdSlice<TestId, Rc<u8>>> = IdVec::new().into_boxed_slice(); | ||
|
|
||
| let cloned = source.clone(); |
| let source: Box<IdSlice<TestId, Unit>> = | ||
| IdVec::from_raw(alloc::vec![Unit, Unit, Unit]).into_boxed_slice(); | ||
|
|
||
| let cloned = source.clone(); |
| let source: Box<IdSlice<TestId, Aligned>> = | ||
| IdVec::from_raw(alloc::vec![Aligned(1), Aligned(2)]).into_boxed_slice(); | ||
|
|
||
| let cloned = source.clone(); |
| assert_eq!(cloned.len(), 2); | ||
| assert_eq!(cloned[TestId::from_usize(0)].0, 1); | ||
| assert_eq!(cloned[TestId::from_usize(1)].0, 2); | ||
| assert_eq!(cloned.as_raw().as_ptr().addr() % 64, 0); |
- Extract error types to dedicated `error.rs` module - Move lock-related `RemoveError` and locking implementation to generation module root - Create `scratch.rs` and `staging.rs` modules for staging/publishing workflow - Add `ColumnWriter` for document codec serialization - Consolidate codec imports and remove duplication in edges/locate response encoding
- Implement `Serialize` for `ArchivedEntityId` to serialize as entity ID string - Add `Envelope::encode_json` method for JSON document encoding - Create `CurrentDocument` type for generation responses - Implement custom serialization for `TranslateDocument` and related types - Add comprehensive tests for JSON encoding with buffer reuse and partial values
Document trait's encode method now returns Result instead of panicking on serialization errors. CurrentDocument and TranslateDocument use `Report<serde_json::Error>`; other document types use the never type `!` since they cannot fail.
- Make `LocateRequest` fields public for external use - Destructure request body in handler signature for clarity - Clarify `LocateDocumentError` to `Problem` mapping documentation - Mark `offset_rule` as const-evaluable function
- Serve command now spawns a background maintenance task for generation updates - HTTP server starts before initial generation publication, returning 503 until ready - Updated visibility limits to production values (1GB bytes, 8m soft, 10m hard)
The `PrincipalLimitLayer` requires the request extension written by `AuthenticationLayer`. Moving `PrincipalLimitLayer` after `AuthenticationLayer` ensures the extension is available when needed. Add integration tests validating layer behavior.
- Remove unused `OwnedLegend::heap_bytes` method - Make error types `CurrentError` and `OpenError` crate-private - Add `#[must_use]` attributes to `Depth` methods - Make `Sha256Digest` and related APIs crate-private - Change `Zoom` to public visibility - Refactor `NaiveIdentityProvider::new` to use `from_ref` in tests - Add validation and error handling to `Encoding::open` - Add `WorldError::TooManyRows` variant with test coverage - Enable `const` impl for UUID conversion
- Updates tests to use `from_ref` instead of `new` for borrowed bases - Adds `from_mut` constructor for mutable references
indietyp
force-pushed
the
bm/be-815-atlas-serve-more-than-one-generation-at-a-time
branch
from
September 11, 2026 19:05
814850b to
37e33e2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🌟 What is the purpose of this PR?
Serve more than one fit generation per process and switch between them without a restart. Before this change, the server opened one generation at start and held it for the process lifetime. A swap meant a restart, and a restart dropped every in-flight session, since a token carried one epoch and the new epoch invalidated it. We also had no way to coordinate a switch with S3, since the logic would need to run while the server is still running, then shut itself down completely via AWS, and then recreate it for the new root. This has obvious downsides, as it couples the implementation to the chosen provider. Having generations be native allows for two things:
What made this project larger is that the prior implementation was heavily tied to a "god class", aka a single class does everything, and only after its inception delta-insertion was grafted on top of it (which made OOMs possible). This changes it so everything now revolves around delta insertion, and everything outside of two classes is oblivious to it. Every operation now operates over a
Scenewhich contains aWorld(the fact as we know it right now - aka space) and anEpoch, which tracks delta accumulated stores. When accessing a world, you must always specify the point in time (viaEpoch) that pins the revision; this allows us to stop advancing prematurely.As this is larger in quantity (as I also took care to clean up the previously messy code), the following things are to be considered during review:
🔍 What does this change?
This replaces the serving layer (
libs/@local/graph/atlas/src/serve/).flowchart LR C[current pointer] -->|poll| M[GenerationManager] M -->|open| W[World + Delta feed] M -->|promote / retire / unlink| R[UniverseRegistry] Q[request] -->|observe| R R -->|"Observation: present + requested"| S[Scene] S --> D[document: tile · edges · locate · translate · manifest · current] D -->|saltile / cbor / json| Qruntime/:GenerationManagerowns generation execution (open, promote, retire, unlink, shutdown) on a poll loop the host runs as its own task.UniverseRegistryis the read side:observe(requested)returns the present generation and the requested one together under one lock, with retention timed from the replacement's promotion.world/: the opened, immutable artifacts of one generation (layout, topology, node index, ontology), checked for a shared node domain at open.Universepairs aWorldwith the deltaEpochcaptured for the request.delta/: the live overlay on a world, versioned by revision and identified per lifetime by aDeltaIddrawn at open. A reopened generation gets a new id, so tokens from the previous lifetime refuse uniformly. Placement, projector, history and feed each have their own module.scene.rsanddocument/: one capturedScene(world, epoch, visibility mask, delivery schedule) feeds every document type, so every response comes from one snapshot. Documents encode throughdocument/codecinto the saltile envelope, which now records its own content type for the API to return.visibility/andapi/authorization/: one visibility-cache budget across all generations. Entries key on the requested generation and itsDeltaId, so a retired generation's misses cannot resolve a fresh scope. Authority tokens verify the actor, the requested generation and theDeltaId. Renewal on the same filter keeps its offset, and a changed filter rebinds.hydrate/: the store boundary behind resolver traits (OntologyResolver,TypeUrlResolver) with SQL statement snapshots, so the tree pins what the process asks Postgres.file/generation/: a generation lock with cooperative removal, plus staging and scratch directories. The original metadata bytes stay on disk so a generation's content-derived identity survives a reopen.hash-graph atlas servegains--generation-poll-interval,--unlink-expired-generationsand--delta-minimum-projection-interval, each with itsHASH_GRAPH_ATLAS_*variable.ServeCommand::runreturnsServing, whoseinto_partshands the host the router and the maintenance future as two values, andapps/hash-graphregisters that future on its lifecycle before binding the listener.apps/hash-frontend/…/atlas-decode): the intern-table check accepts unique entries in index order rather than requiring bytewise sort, matching the relaxed writer.--unlink-expired-generations) and off by default.🛡 What tests cover this?
The tests in the serve directory 🙂
❓ How to test this?
Run the server, create a new generation (either by copying or otherwise), then wait. You should see the changeover.