Skip to content

docs: record the design for a Kubernetes-ready, highly available Ingot - #9

Merged
TheMeinerLP merged 3 commits into
mainfrom
docs/kubernetes-ha-design
Aug 8, 2026
Merged

docs: record the design for a Kubernetes-ready, highly available Ingot#9
TheMeinerLP merged 3 commits into
mainfrom
docs/kubernetes-ha-design

Conversation

@TheMeinerLP

@TheMeinerLP TheMeinerLP commented Aug 8, 2026

Copy link
Copy Markdown

Running two Ingot instances against one database and one bucket is not safe
today. The reasons are specific rather than architectural, and this document
records what they are, what has to change, and in which order.

What the analysis found

Three correctness problems block a second replica:

  • Every pod runs SchemaUtils.create during startup, so a rolling update issues
    concurrent DDL against the same tables. On Galera that is total order isolation,
    which stalls the cluster at best and aborts a node at worst.
  • MetadataService.generatePom is a read-modify-write on maven-metadata.xml
    with no lock that crosses processes. The filesystem provider holds one per
    location, and its own comment says it is not truly respected; the S3 provider
    holds none. Two parallel deploys of the same coordinate lose a version.
  • /api/status/health reports only that Jetty is running, so a pod with an
    exhausted connection pool keeps receiving traffic. There is no draining signal
    and no stopTimeout, so in-flight uploads die on SIGTERM.

Alongside those, the document answers how configuration reaches a pod, how the
dashboard scales separately from the server, and where a message bus belongs.

Decisions it records

  • Locks go into the database. A lost lock is a lost update. Galera already
    provides the guarantee, and a Redis lock without a fencing token does not
    protect against a paused process that resumes and writes anyway.
  • NATS covers fan-out and nothing else. Cache invalidation, configuration
    propagation, token revocation and console streaming are best effort, so core
    publish and subscribe is enough. Redis gets no niche.
  • The split runs through deployment targets in one image, in the shape Loki
    and Mimir use, rather than through separate services. --target=all stays the
    default, so the drop-in contract survives untouched.

Scope

Four stages. Only the first two are required for high availability; stage three
is optional and inert without a NATS URL, and stage four is the cloud native and
enterprise structure. Each gets its own implementation plan, so this document is
shared context rather than a single work item.

Worth flagging from the "deliberately not done" section: the largest throughput
gain in the whole design is splitting the read path from the write path by HTTP
method, and it needs neither a target mechanism nor a message bus.

No code changes. Documentation only.

Also in this pull request

docs/superpowers/plans/2026-08-08-ha-stage-1-correctness.md, the implementation plan for
stage 1. Six tasks: build and prove a shared advisory lock against a real MariaDB, serialise
schema initialisation around the plugin loader, serialise metadata writes per repository and
directory, flush the statistics buffer on shutdown, and document what running several
replicas requires.

Stage 2 gets its own plan, because stage 1 delivers working software on its own.

Two findings the plan records rather than quietly fixing. PreservedBuildsListener races the
same metadata the new lock protects, but it runs on DeployEvent after the lock is released,
so covering it needs a decision about whether an event listener may block a deployment.
And widening the Reposilite constructor would break a third-party plugin that constructs it
directly, which nothing in this repository does.

Running two instances against one database and one bucket is unsafe today,
for reasons that are specific rather than architectural: schema DDL runs on
every pod start, metadata writes are read-modify-write without a lock that
crosses processes, and there is no readiness signal separate from liveness.

The document records those findings, the fixes, and three decisions that
constrain them. Locks go into the database because a lost lock is a lost
update and Galera already provides the guarantee. NATS covers fan-out only,
and Redis gets no niche. The eventual split runs through deployment targets
in one image rather than separate services, so the drop-in contract survives.

Four stages, of which only the first two are required for high availability.
Each gets its own implementation plan; this is the shared context.
Six tasks covering the three correctness problems that block a second
replica: a shared advisory lock built and proven against a real MariaDB,
schema initialisation serialised around the plugin loader, metadata
writes serialised per repository and directory, and the statistics
buffer flushed on shutdown.

Stage 2 gets its own plan. Stage 1 delivers working software on its own,
and every cut made before the correctness problems are solved would be
built on them.

Two findings recorded in the self-review rather than silently fixed. The
PreservedBuildsListener races the same metadata the lock now protects,
but it runs after the lock is released, so covering it needs a decision
about whether event listeners may block a deployment. And widening the
Reposilite constructor breaks a third-party plugin that constructs it
directly, which nothing in this repository does.
The plan had two tests that only proved the code does not crash, with
their teeth checked by editing the implementation by hand and reverting.
That is not reproducible, and an interrupted task leaves a disabled lock
in the tree. Both now carry a negative control instead: the identical
body run through the no-op strategy against the same database, asserting
the violation does appear.

The statistics task asserted behaviour that already worked, so it would
have passed before the change it was meant to drive. It now emits the
dispose event against a running instance and asserts the buffer is
unwritten before and written after, which is the wiring that actually
changes.
@TheMeinerLP
TheMeinerLP merged commit 0ff43f9 into main Aug 8, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant