fix: stop caching error responses

[joewiz]

[This PR was prompted by Joe, drafted by Claude Code, and reviewed by Joe.]

Background: the editorial date floor is doing its job

app:last-modified resolves a page's timestamp and then floors it at $config:EDITORIAL_DATE_TIME:

if ($last-modified-date-time > $config:EDITORIAL_DATE_TIME)
then $last-modified-date-time
else $config:EDITORIAL_DATE_TIME

That floor is deliberate and worth preserving. It guarantees that any copy a client cached before the editorial date is treated as stale, so raising the constant re-generates every page still being served from a cache. For a successful response this is exactly the behavior we want, and this PR does not change it.

The problem: the floor also reaches error responses

view.xql runs for error pages too. So a 404 is sent with Last-Modified set to that same constant, and because hsg-shell sets no Cache-Control, Expires, or ETag, nothing marks the response uncacheable. Caches fall back to heuristic freshness and store it.

From there the 404 is self-confirming. Every revalidation compares the client's If-Modified-Since against the constant and gets 304 Not Modified, so the cached 404 is renewed rather than re-checked.

The part that makes this consequential is that a 304 carries no body. The server cannot tell whether the client cached a 200 or a 404 — it only compares dates. So a client that once cached a 404 keeps re-serving it.

An error response describes a moment, not the editorial state of the site, so it has no business in a scheme designed to describe content.

When this actually bites

Not every 404 sticks. It depends on which timestamp the URL reports:

URL	Last-Modified	Revalidation by a client holding a cached copy
/	Tue, 14 Jul 2026 — its own	200, so a cached response is replaced
/countries/afghanistan	Mon, 29 Jun 2026 — the editorial floor	304, so a cached response is renewed
/historicaldocuments	Mon, 29 Jun 2026 — the editorial floor	304

A page whose content is newer than the editorial date reports its own timestamp, so a deployment moves it and any cached response — including a cached 404 — is replaced on the next revalidation. Those URLs are self-healing.

A page floored at the editorial date reports a constant. Redeploying rdcr does not change an unmodified article's commit time, and redeploying hsg-shell does not touch rdcr content at all, so the value never moves and revalidation returns 304 indefinitely. Since most content predates 29 June, that describes most of the site.

So the precondition is specific: a transient 404 on a page whose content predates the editorial date.

This is a development problem, not a production one. In production, instances are taken out of the load balancer while packages are installed, so no request is served during the window when a package's URLs would 404, and nothing is there to be cached. A local instance has no such protection, and requests do land during deploys and uploads. While preparing this PR I saw exactly that class of transient: /exist/apps/hsg-shell/ returned 400 and then 200 on retry, and a first request after a restart returned 400 before settling.

The fix is still worth having — a cached error that can only be cleared by a force-reload is a genuine defect, and the change is confined to error paths — but it should not be read as fixing a production problem.

Reproduction

$ curl -sD - -o /dev/null .../hsg-shell/countries/zzdoesnotexist
HTTP/1.1 404 Not Found
Last-Modified: Mon, 29 Jun 2026 00:00:00 +0000

$ curl -s -o /dev/null -H "If-Modified-Since: Mon, 29 Jun 2026 00:00:00 +0000" \
    -w "%{http_code}\n" .../hsg-shell/countries/zzdoesnotexist
304

No Cache-Control, and the 404 revalidates as 304.

The fix

Errors are raised two different ways, and both needed handling:

• Controller-level — local:serve-not-found-page and local:serve-bad-request-page, used at 25 call sites. These now pass an error-page parameter through the existing local:render-page parameter mechanism.
• Template-level — a publication route renders a normal page and signals failure during templating by setting the hsg-shell.errcode request attribute, which app:handle-error turns into a 4xx. Seven modules do this, and it is the path behind countries/… and historicaldocuments/…. view.xql now checks for that attribute after rendering.

In both cases the response is sent with Cache-Control: no-store and takes no part in Last-Modified negotiation.

Successful responses are untouched. They keep the editorial-date floor, keep sending Last-Modified, and keep revalidating to 304, so the sitewide invalidation lever still works exactly as before. The change removes error responses from that scheme and nothing else.

The duplicated templates:apply block is factored into a local:render() function rather than copy-pasted into the new branch.

Tests

tests/bats/cache-headers.bats — 7 tests, alongside the existing smoke-test.bats and using the same harness. It honors $HSG_BASE so it can run against a container or a local instance.

Three of the tests exist specifically to guard the behavior described at the top: that successful responses remain cacheable, still send Last-Modified, and still revalidate to 304. If a future change to error handling leaks into the content path, they should catch it.

I verified the tests catch the bug by stashing the fix, rebuilding, redeploying, and re-running:

test	without fix	with fix
controller-level 404 not cacheable	fail	pass
publication-route 404 not cacheable	fail	pass
over-long path 404 not cacheable	fail	pass
successful page still cacheable	pass	pass
successful page still sends Last-Modified	pass	pass
successful page still revalidates to 304	pass	pass
404 page still renders	pass	pass

The three new assertions fail without the fix; the four regression guards hold either way.

Knowingly left open

A conditional request for a publication-route 404 still returns 304. view.xql honors If-Modified-Since and short-circuits before rendering, so at that point it cannot yet know the request will not resolve to a document. This is a question of resolution order rather than of the Last-Modified value, and addressing it would mean determining that a request is an error before the short-circuit — a larger change than this one.

In practice it is now unreachable through normal browser behavior: with no-store the client never caches the 404, so it never sends the conditional request that would produce the stale 304.

Related, but not addressed here

The local-preview symptom that originally led here — editing a document, uploading it, and reloading to find the old page — is not fixed by this PR, because that is a successful response rather than an error. It is handled by a development caching mode in a separate PR, which makes our setup directions hold as written.

Verifying locally

After deploying the package, clear eXist's compiled query cache — otherwise the previously compiled controller keeps serving and the change appears to have no effect:

system:clear-xquery-cache()

Then:

HSG_BASE=http://127.0.0.1:8080/exist/apps/hsg-shell bats tests/bats/cache-headers.bats

[joewiz]

[This response was prompted by Joe, drafted by Claude Code, and reviewed by Joe.]

Flagging a revision to the description above, since the original went out by email.

Two corrections. First, the original implied that any cached 404 sticks indefinitely; in fact only pages floored at $config:EDITORIAL_DATE_TIME do. Pages whose content is newer report their own timestamp and are replaced on the next revalidation. Second, and more importantly, the original implied a production risk from the redeploy window. There isn't one: instances are taken out of the load balancer while packages are installed, so nothing is served that could be cached.

This is a development problem, not a production one. The description now says so, and adds the precondition — a transient 404 on a page whose content predates the editorial date — along with the evidence for both.

The scope of the change itself is unaffected.