Skip to content

Fix four correctness defects on the dependency and fail-open paths - #5

Merged
christiangda merged 2 commits into
mainfrom
fix/cache-correctness
Aug 22, 2026
Merged

Fix four correctness defects on the dependency and fail-open paths#5
christiangda merged 2 commits into
mainfrom
fix/cache-correctness

Conversation

@christiangda

Copy link
Copy Markdown
Contributor

Found while reviewing c3e on the authorization path of a multi-tenant service, where it now backs every permission decision. All four were reproduced live against Valkey before being fixed, and every fix has a test that was confirmed to fail when the fix is reverted.

1. A shared dependency set could have its TTL shortened

Set issued an unconditional EXPIRE on each reverse-dependency set. Those sets are shared by every entry depending on the same thing, so the last writer won even when its TTL was shorter.

With jittered TTLs that's ordinary, not rare — at 10% jitter on a 12h hard TTL entries land in 10.8h–13.2h, so a dependency set can expire more than two hours before an entry still listed in it. An Invalidate landing in that window finds an empty set, cascades to nothing, and reports success.

after U1 cached (60s entry TTL):  TTL(dep:role:R) = 60s
after U2 cached ( 3s entry TTL):  TTL(dep:role:R) =  3s   <- shortened
... 4s later ...  dep:role:R exists=0   cache:authz:U1 exists=1
Invalidate(role:R) -> cache:authz:U1 still exists = 1

In the service that found this, a revoked role kept working until the dependent entry expired on its own.

The expiry may now only be raised: EXPIRE … NX sets it when the set has none, EXPIRE … GT raises it when a longer-lived dependent joins.

Both are needed. GT treats a key with no expiry as infinite and refuses to set one, so GT alone would leave these sets persistent forever — exactly the leak the TTL exists to prevent. That trap is worth knowing about.

2. A failed encode failed the request

fetchAndCache handled the two halves of a write inconsistently: a failing cache.Set was logged and swallowed — correctly, the value was already fetched — but a failing encode three lines earlier was returned to the caller.

A serialization problem, which is purely a caching concern, became user-visible even though the expensive round trip had succeeded. The library is fail-open everywhere else; this one branch was fail-closed, and it's what turned "caching silently doesn't work" into "the service is down".

Encode and wrapper-marshal failures now log and return the fetched value uncached, matching Set. fetchAndCache returns a small fetchResult so blockingFetch can serve the raw value when there's nothing to decode.

3. A payload that no longer decoded failed every read for the whole TTL

A corrupt wrapper was already treated as a miss and refetched. A corrupt payload inside a valid wrapper was a hard error returned to the caller — so changing EncoderType on a warm cache, or a cached type whose shape moves during a rolling deploy where two versions share one server, broke every read of that key until its hard TTL expired, with no self-healing.

Now treated as a miss.

4. The invalidation cascade was O(nodes) sequential round trips

The breadth-first walk issued one SMEMBERS per node plus one more per dependent found, each as its own round trip. A wide dependency graph became hundreds of serial commands on the write path — against a server the read path deliberately fast-fails on.

The walk now batches per level with DoMulti: one round trip for the level's dependency sets, one for their dependents' forward lists. Cost is proportional to graph depth rather than node count. Behaviour is unchanged, including the de-duplication of entries reachable from two sets in the same level.

Tests

New correctness_test.go, all against a real Valkey (skipping when none is reachable, matching the existing convention):

  • dependency TTL cannot be shortened by a later, shorter-lived dependent
  • dependency TTL is raised by a longer-lived one
  • a corrupt payload heals instead of failing
  • an unencodable value is still returned, and leaves no entry behind
  • the cascade still clears a graph that is both deep and wide (the batching groups by level, so a single chain wouldn't exercise it)

Each was reverted-and-rerun to confirm it fails without its fix:

dependency set TTL was shortened to 3s by a later, shorter-lived dependent
Get returned an error instead of refetching: invalid character 'h' ...
a value that cannot be cached must still be returned: json: unsupported type: chan int

Verification

Against Valkey 9.1.1: full suite green under -race with -tags=integration, 89.3% coverage, golangci-lint clean, go vet clean, gofmt clean.

docs.go updated for the new dependency-TTL rule, the decode-error handling, and the batched cascade.

🤖 Generated with Claude Code

All four were found by reviewing this library on the authorization path of a
multi-tenant service, and each was reproduced live against Valkey before
being fixed. Every fix has a test that was confirmed to fail when the fix is
reverted.

1. A shared dependency set could have its TTL shortened

Set issued an unconditional EXPIRE on each reverse-dependency set. Those sets
are shared by every entry that depends on the same thing, so the last writer
won even when its TTL was shorter. With jittered TTLs that is ordinary rather
than rare: at 10% jitter on a 12h hard TTL, entries land in 10.8h-13.2h, so a
dependency set can expire more than two hours before an entry still listed in
it. An Invalidate landing in that window finds an empty set, cascades to
nothing, and reports success — in the service that found this, a revoked role
kept working until the dependent entry expired on its own.

The expiry may now only be raised: EXPIRE NX sets it when the set has none,
EXPIRE GT raises it when a longer-lived dependent joins. Both are needed; GT
treats a key with no expiry as infinite and would refuse to set one, leaving
the leak the TTL exists to prevent.

2. A failed encode failed the request

fetchAndCache handled the two halves of a write inconsistently. A failing
cache.Set was logged and swallowed, correctly, because the value had already
been fetched — but a failing encode three lines earlier was returned to the
caller. A serialization problem, which is purely a caching concern, became
user-visible even though the expensive round trip had succeeded. The library
is fail-open everywhere else; this one branch was fail-closed, and it turned
"caching silently does not work" into "the service is down".

Encode and wrapper-marshal failures are now logged and the fetched value is
returned uncached, matching Set.

3. A payload that no longer decoded failed every read for the whole TTL

A corrupt wrapper was already treated as a miss and refetched. A corrupt
payload inside a valid wrapper was a hard error returned to the caller, so
changing EncoderType on a warm cache — or a cached type whose shape moves
during a rolling deploy where two versions share one server — broke every
read of that key until its hard TTL expired, with no self-healing.

A payload that cannot be decoded is now treated as a miss.

4. The invalidation cascade was O(nodes) sequential round trips

The breadth-first walk issued one SMEMBERS per node, plus one more per
dependent found, each as its own round trip. A wide dependency graph became
hundreds of serial commands on the write path, against a server the read path
deliberately fast-fails on.

The walk now batches per level with DoMulti: one round trip for the level's
dependency sets, one for their dependents' forward lists. Cost is now
proportional to graph depth rather than node count.

Verified against Valkey 9.1.1: full suite green with -race, 89.3% coverage,
golangci-lint clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@christiangda christiangda self-assigned this Aug 22, 2026
@christiangda
christiangda merged commit 6f39467 into main Aug 22, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant