docs: record the design for a Kubernetes-ready, highly available Ingot - #9
Merged
Conversation
Running two instances against one database and one bucket is unsafe today, for reasons that are specific rather than architectural: schema DDL runs on every pod start, metadata writes are read-modify-write without a lock that crosses processes, and there is no readiness signal separate from liveness. The document records those findings, the fixes, and three decisions that constrain them. Locks go into the database because a lost lock is a lost update and Galera already provides the guarantee. NATS covers fan-out only, and Redis gets no niche. The eventual split runs through deployment targets in one image rather than separate services, so the drop-in contract survives. Four stages, of which only the first two are required for high availability. Each gets its own implementation plan; this is the shared context.
Six tasks covering the three correctness problems that block a second replica: a shared advisory lock built and proven against a real MariaDB, schema initialisation serialised around the plugin loader, metadata writes serialised per repository and directory, and the statistics buffer flushed on shutdown. Stage 2 gets its own plan. Stage 1 delivers working software on its own, and every cut made before the correctness problems are solved would be built on them. Two findings recorded in the self-review rather than silently fixed. The PreservedBuildsListener races the same metadata the lock now protects, but it runs after the lock is released, so covering it needs a decision about whether event listeners may block a deployment. And widening the Reposilite constructor breaks a third-party plugin that constructs it directly, which nothing in this repository does.
The plan had two tests that only proved the code does not crash, with their teeth checked by editing the implementation by hand and reverting. That is not reproducible, and an interrupted task leaves a disabled lock in the tree. Both now carry a negative control instead: the identical body run through the no-op strategy against the same database, asserting the violation does appear. The statistics task asserted behaviour that already worked, so it would have passed before the change it was meant to drive. It now emits the dispose event against a running instance and asserts the buffer is unwritten before and written after, which is the wiring that actually changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Running two Ingot instances against one database and one bucket is not safe
today. The reasons are specific rather than architectural, and this document
records what they are, what has to change, and in which order.
What the analysis found
Three correctness problems block a second replica:
SchemaUtils.createduring startup, so a rolling update issuesconcurrent DDL against the same tables. On Galera that is total order isolation,
which stalls the cluster at best and aborts a node at worst.
MetadataService.generatePomis a read-modify-write onmaven-metadata.xmlwith no lock that crosses processes. The filesystem provider holds one per
location, and its own comment says it is not truly respected; the S3 provider
holds none. Two parallel deploys of the same coordinate lose a version.
/api/status/healthreports only that Jetty is running, so a pod with anexhausted connection pool keeps receiving traffic. There is no draining signal
and no
stopTimeout, so in-flight uploads die on SIGTERM.Alongside those, the document answers how configuration reaches a pod, how the
dashboard scales separately from the server, and where a message bus belongs.
Decisions it records
provides the guarantee, and a Redis lock without a fencing token does not
protect against a paused process that resumes and writes anyway.
propagation, token revocation and console streaming are best effort, so core
publish and subscribe is enough. Redis gets no niche.
and Mimir use, rather than through separate services.
--target=allstays thedefault, so the drop-in contract survives untouched.
Scope
Four stages. Only the first two are required for high availability; stage three
is optional and inert without a NATS URL, and stage four is the cloud native and
enterprise structure. Each gets its own implementation plan, so this document is
shared context rather than a single work item.
Worth flagging from the "deliberately not done" section: the largest throughput
gain in the whole design is splitting the read path from the write path by HTTP
method, and it needs neither a target mechanism nor a message bus.
No code changes. Documentation only.
Also in this pull request
docs/superpowers/plans/2026-08-08-ha-stage-1-correctness.md, the implementation plan forstage 1. Six tasks: build and prove a shared advisory lock against a real MariaDB, serialise
schema initialisation around the plugin loader, serialise metadata writes per repository and
directory, flush the statistics buffer on shutdown, and document what running several
replicas requires.
Stage 2 gets its own plan, because stage 1 delivers working software on its own.
Two findings the plan records rather than quietly fixing.
PreservedBuildsListenerraces thesame metadata the new lock protects, but it runs on
DeployEventafter the lock is released,so covering it needs a decision about whether an event listener may block a deployment.
And widening the
Reposiliteconstructor would break a third-party plugin that constructs itdirectly, which nothing in this repository does.