Skip to content

Publish the realm's signed line index, so the omission check can fire - #25

Merged
avrabe merged 1 commit into
mainfrom
feat/publish-the-line-index
Oct 1, 2026
Merged

avrabe merged 1 commit into
mainfrom
feat/publish-the-line-index

Conversation

@avrabe

@avrabe avrabe commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

Closes #6.

The check that could not fire

varve has read line-index-<line> since 2026-09-06 and no realm has ever
written one. Re-measured today: ghcr.io/pulseengine/layers serves 23 layer
tags and zero line-index-*
. REQ-INDEXAUTH-001's own notes name the gap —
"a fetch path with no publisher is a check that cannot fire" — so every
consumer-side guarantee resting on the index has been reasoning about a
document nobody had written, and varve outdated could only ever answer
"cannot answer".

The list is committed here, never read from the registry

Per varve DD-027, and it is the whole point. The index exists to catch a
registry hiding a layer, and every artifact a hiding registry does serve
still verifies. Derive the list from oras repo tags and the check is vacuous:
the hiding registry omits the layer from the listing and therefore from the
index
, the two agree perfectly, and varve reports nothing wrong. It would pass
its own tests, publish real signatures, and detect nothing.

So line-index/<line>.json is a reviewed file, and each deposit appends the
layer it just pushed.

What the deposit now does, after the push

After, never before — an index naming a layer the registry does not serve
is exactly what omission detection screams about, so publishing one before the
push succeeded would manufacture the alarm this mechanism exists to raise
honestly.

  1. re-reads the payload digest from the layout (not passed down, so the step
    cannot be handed a digest belonging to another layer);
  2. tools/append-index.py appends {layer, digest, channel, counter};
  3. commits the document and git pull --rebasees, because the scanner commits
    layer.toml on its own schedule;
  4. varve sign-index with the realm root, key via file descriptor not a
    file (REQ-NOKEYDISK-001);
  5. oras blob push + oras manifest push under line-index-<line>, with the
    manifest shape copied from varve docs attach-index rather than
    re-derived — a role annotation spelled differently reads as no index at all.

The step gates itself

It reads the index back the way a consumer does — fetch the manifest, fetch
the blob, decode the DSSE payload — and requires this layer to be named in it.
A signature over a document nobody can fetch is the failure this step exists to
end, and "oras said ok" is not "a consumer can read it".

The appender

Idempotent, because a deposit is re-runnable: the same layer twice is a
no-op that does not bump the document counter. A counter that climbed on a
re-dispatch would make the next legitimate publish look stale to a consumer
that refuses an older index.

One layer id with different bytes is a refusal, naming both digests — not
an update. Quietly rewriting the entry would hide exactly what varve's
immutability rule exists to catch.

Layer ids order numerically. A string sort puts 2026.09.12 before
2026.09.2 — a mistake made against this realm on 2026-10-01 that reported a
staleness figure wrong by eight layers. Swapping the sort key for the raw
string turns two tests red (verified).

9 new tests, 33 total green via the existing check.yml unittest discovery.

The 23 existing layers are deliberately not backfilled

line-index/README.md says why. The only source for their payload digests is
the registry, and a registry-derived entry protects nothing by the same
argument as above: a registry that had hidden one of them would have hidden it
from the backfill too. A document that looks complete and protects nothing is
worse than a short one honest about its reach.

varve treats the index as a floor — extra layers a source serves are never
an error — so detection covers every layer from here on. If you would rather
have the backfill anyway, say so and I will add it, labelled as
registry-provenance bootstrap.

signed-index = true is NOT set

A realm declaring it with no index published fails closed on every install. The
declaration is the last step, and it lives in varve's varve-realms.toml, not
here — so the order is: merge this, let one deposit run, confirm the
line-index-2026.10 tag exists, then declare it.

What I could not test here

The workflow step itself has not run — it needs the realm's signing secret and
a dispatch. What is verified locally: the appender's behaviour (9 tests, one
negative-controlled), that varve sign-index accepts exactly the document the
appender writes, and that the read-back jq assertion in the step works
against a real signed envelope. The same publish recipe is also now exercised
end to end against a live registry by varve's own tools/systest/line-index.sh
(pulseengine/varve#233), including controls for an index signed by the wrong
key and one naming a layer the registry will not serve.

🤖 Generated with Claude Code

https://claude.ai/code/session_019TNtfRjLNhEz82G2ggeeNu

varve has read `line-index-<line>` since 2026-09-06 and no realm has ever
written one. Re-measured 2026-10-01: ghcr.io/pulseengine/layers serves 23 layer
tags and zero `line-index-*`. REQ-INDEXAUTH-001's own notes name the gap — "a
fetch path with no publisher is a check that cannot fire" — so every
consumer-side guarantee resting on the index has been reasoning about a
document nobody had written, and `varve outdated` could only ever answer
"cannot answer".

THE LIST IS COMMITTED HERE AND NEVER READ FROM THE REGISTRY (varve DD-027).
The index exists to catch a registry HIDING a layer, and every artifact a
hiding registry does serve still verifies. Derive the list from `oras repo
tags` and the check is vacuous: the hiding registry omits the layer from the
listing and therefore from the index, the two agree, and varve reports nothing
wrong. It would pass its own tests, publish real signatures, and detect
nothing.

So `line-index/<line>.json` is a reviewed file, and each deposit appends the
layer it just pushed.

AFTER the push, never before. An index naming a layer the registry does not
serve is precisely what omission detection screams about, so publishing one
before the push succeeded would manufacture the alarm this mechanism exists to
raise honestly.

`tools/append-index.py` is idempotent, because a deposit is re-runnable: the
same layer twice is a no-op that does NOT bump the document counter, since a
counter that climbed on a re-dispatch would make the next legitimate publish
look stale to a consumer that refuses an older index. One layer id arriving
with DIFFERENT bytes is a refusal naming both digests, not an update — quietly
rewriting the entry would hide exactly what varve's immutability rule exists to
catch.

Layer ids are ordered NUMERICALLY. A string sort puts 2026.09.12 before
2026.09.2, and that mistake was made against this very realm on 2026-10-01 and
reported a staleness figure wrong by eight layers. It is the negative control
for the ordering test: swapping the key for the raw string turns two tests red.

THE STEP GATES ITSELF by reading the index back the way a CONSUMER does —
fetching the manifest, fetching the blob, decoding the DSSE payload — and
requiring this layer to be named in it. A signature over a document nobody can
fetch is the failure this step exists to end, and "oras said ok" is not "a
consumer can read it".

THE 23 EXISTING LAYERS ARE DELIBERATELY NOT BACKFILLED, and line-index/README.md
says why. The only source for their payload digests is the registry, and a
registry-derived entry protects nothing by the same argument as above: a
registry that had hidden one of them would have hidden it from the backfill.
A document that looks complete and protects nothing is worse than a short one
that is honest about its reach. varve treats the index as a FLOOR — extra
layers a source serves are never an error — so detection covers every layer
from here on.

`signed-index = true` is NOT set. A realm declaring it with no index published
fails closed on every install; the declaration is the last step, and it lives
in varve's varve-realms.toml rather than here.

Closes #6.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019TNtfRjLNhEz82G2ggeeNu
@avrabe
avrabe merged commit b9a6ab4 into main Oct 1, 2026
1 check passed
@avrabe
avrabe deleted the feat/publish-the-line-index branch October 1, 2026 20:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Publish the realm's signed line-index — the omission check has no publisher

1 participant