fix: collapse a bulk invalidation's URL-map writes into one per domain

CHANGELOG.md

@@ -2,36 +2,7 @@
 
 All notable changes to this project will be documented in this file.
 
-## [Unreleased]
-
-### Fixed
-
-- A host app whose `statamic.static_caching.invalidation.rules` is the string
-  `all` no longer fatals the moment static caching is switched on. Statamic
-  documents that value and checks for exactly it in
-  `DefaultInvalidator::invalidate()`, so a stock config hit
-  `TypeError: GraphInvalidator::__construct(): Argument #2 ($rules) must be of
-  type ?array, string given` during boot. Any non-array value is now normalised
-  to `[]`: `all` means flush everything on any save, which is precisely the
-  behaviour the graph replaces, so it must not reach the parent.
-- Globals no longer come back from the cache as `__PHP_Incomplete_Class`. Laravel
-  unserializes cache payloads against an allow list when
-  `cache.serializable_classes` holds an array, and Statamic fills that array with
-  its own classes. Binding `TrackingVariables` over the `Variables` contract meant
-  the globals store cached items of a class nobody had allowed, so the next method
-  call on a global set fatalled — on every page, since layouts read globals. The
-  addon now adds its own cached classes to that allow list, leaving the host's
-  entries and the unrestricted `null`/`true` settings untouched.
-- Custom query scopes now work on the tracking entry and term query builders.
-  Statamic keys its scope registry on the exact builder class, so a scope a site
-  registered against `Stache\Query\EntryQueryBuilder` was invisible to the
-  subclass this addon substitutes, and calling it fatally threw
-  `BadMethodCallException: Call to undefined method ...::yourScope()`. The
-  ordering was not a fluke: addons boot inside `$app->booted()`, so a site's own
-  `boot()` always registers its scopes before this addon rebinds the builder. A
-  scope registered against any ancestor of a tracking builder now applies to it.
-
-## [2.0.0] - 2026-08-06
+## [2.0.0] - 2026-09-15
 
 Invalidation is now derived from what pages actually read, instead of from rules
 describing what they might read. There is no configuration to write.
@@ -113,6 +84,52 @@ renders; v2 observes it rather than restating it.
 
 ### Fixed
 
+- A save no longer leaves pages stale when it clears a large number of URLs.
+  Statamic keeps one URL map per domain and rewrites all of it for every URL it
+  clears, so invalidating n URLs out of a map of m cost O(n*m). On a site with a
+  few thousand cached URLs that is a few hundred kilobytes rewritten thousands of
+  times per save, each write taking an exclusive lock that live traffic is
+  contending for on every uncached render. The queued
+  `Statamic\StaticCaching\Invalidate` job ran past its timeout and was killed
+  part-way through the list, so every URL it had not reached yet went on serving
+  the old page. Nothing said so: the only trace was a `TimeoutExceededException`
+  in `failed_jobs`, while the editor saw a saved entry that never appeared. The
+  map is now read once and written once per domain actually touched, whatever the
+  size of the batch — Statamic's own matching, response deletion and events are
+  left exactly as they are. The `full` strategy still scans its cache directory
+  once per URL, which is Statamic's own `FileCacher` and unchanged here.
+- Invalidation no longer hands the cacher URLs that are not cached any more. The
+  graph outlives the cache by design — rows are pruned as they are invalidated,
+  not when a page falls out — so on a site that has been up for a while a tag
+  routinely resolves to two or three times as many URLs as the cache holds, and
+  every one of those costs a lookup that can only miss. They are now filtered out
+  before the cacher sees them. A cacher whose contents cannot be enumerated still
+  receives the full set: an empty list there means "unknown", not "nothing is
+  cached", and narrowing against it would clear nothing at all.
+- A host app whose `statamic.static_caching.invalidation.rules` is the string
+  `all` no longer fatals the moment static caching is switched on. Statamic
+  documents that value and checks for exactly it in
+  `DefaultInvalidator::invalidate()`, so a stock config hit
+  `TypeError: GraphInvalidator::__construct(): Argument #2 ($rules) must be of
+  type ?array, string given` during boot. Any non-array value is now normalised
+  to `[]`: `all` means flush everything on any save, which is precisely the
+  behaviour the graph replaces, so it must not reach the parent.
+- Globals no longer come back from the cache as `__PHP_Incomplete_Class`. Laravel
+  unserializes cache payloads against an allow list when
+  `cache.serializable_classes` holds an array, and Statamic fills that array with
+  its own classes. Binding `TrackingVariables` over the `Variables` contract meant
+  the globals store cached items of a class nobody had allowed, so the next method
+  call on a global set fatalled — on every page, since layouts read globals. The
+  addon now adds its own cached classes to that allow list, leaving the host's
+  entries and the unrestricted `null`/`true` settings untouched.
+- Custom query scopes now work on the tracking entry and term query builders.
+  Statamic keys its scope registry on the exact builder class, so a scope a site
+  registered against `Stache\Query\EntryQueryBuilder` was invisible to the
+  subclass this addon substitutes, and calling it fatally threw
+  `BadMethodCallException: Call to undefined method ...::yourScope()`. The
+  ordering was not a fluke: addons boot inside `$app->booted()`, so a site's own
+  `boot()` always registers its scopes before this addon rebinds the builder. A
+  scope registered against any ancestor of a tracking builder now applies to it.
 - `Invalidator::refresh()` is honoured. `DefaultInvalidator` flips its `$refreshing`
   flag before delegating to `invalidate()`, which 1.x overrode without checking, so
   `statamic.static_caching.background_recache` hard-purged instead of refreshing.

README.md

@@ -353,9 +353,10 @@ listed so you know where the edges are.
 
 ## Good to know
 
-- Invalidation is one indexed lookup plus the deletes. Nothing walks content, which
-  matters with a single queue worker or `QUEUE_CONNECTION=sync`, where it runs inside
-  the editor's save request.
+- Invalidation is one indexed lookup plus the deletes, and the URL map Statamic keeps
+  per domain is read and written once per pass rather than once per URL cleared.
+  Nothing walks content, which matters with a single queue worker or
+  `QUEUE_CONNECTION=sync`, where it runs inside the editor's save request.
 - A page with more than 2,000 dependencies is treated as depending on everything.
 - Globals are not scoped per site, so on a multisite install saving one clears the
   pages that read it across every site.

src/Cachers/BatchesInvalidation.php

@@ -0,0 +1,146 @@
+<?php
+
+declare(strict_types=1);
+
+namespace RoxDigital\CacheInvalidation\Cachers;
+
+use Illuminate\Support\Collection;
+
+/**
+ * Collapses a bulk invalidation's URL-map writes into one per domain.
+ *
+ * Statamic keeps every cached URL for a domain in a single cache entry, and
+ * AbstractCacher::invalidateUrls() walks the URLs one at a time. Each call reads
+ * that entry, and each match calls forgetUrl(), which reads it again and writes
+ * the whole thing back. Clearing n URLs from a map of m costs O(n*m): on a site
+ * with a few thousand cached URLs the entry is hundreds of kilobytes, every
+ * write takes an exclusive lock on one file, and live traffic is rewriting the
+ * same entry on each uncached render.
+ *
+ * That is what the graph makes expensive rather than cheap. Targeted
+ * invalidation hands the cacher a long list of individually correct URLs — the
+ * whole point — and the per-URL rewrite turns precision into work. A queued
+ * invalidation job then runs past its timeout, is killed part-way through the
+ * list, and every URL it had not reached yet stays stale. The failure is silent:
+ * the editor sees a saved entry and the site keeps serving the old page.
+ *
+ * Buffering getUrls() for the duration of the call leaves the parent's matching,
+ * response deletion and event dispatch exactly as they are — nothing here knows
+ * what a cached response looks like — while turning m reads and writes into one
+ * read and one write per domain actually touched.
+ */
+trait BatchesInvalidation
+{
+    /**
+     * Domain => URL map, filled lazily while a batch is in flight. Null whenever
+     * no batch is running, which is what every other caller sees.
+     *
+     * @var array<string, Collection>|null
+     */
+    private ?array $bufferedUrls = null;
+
+    /**
+     * Domains whose buffered map was actually modified. A pass that matches
+     * nothing must not write the map back.
+     *
+     * @var array<string, true>
+     */
+    private array $dirtyDomains = [];
+
+    /**
+     * @param  array  $urls
+     * @return void
+     */
+    public function invalidateUrls($urls)
+    {
+        $this->whileBuffering(fn () => parent::invalidateUrls($urls));
+    }
+
+    /**
+     * Refreshing never writes the map — it reads it to find URLs to warm — but it
+     * reads it once per URL, so the same buffer removes that too.
+     *
+     * @param  array  $urls
+     * @return void
+     */
+    public function refreshUrls($urls)
+    {
+        $this->whileBuffering(fn () => parent::refreshUrls($urls));
+    }
+
+    /**
+     * @param  string|null  $domain
+     * @return Collection
+     */
+    public function getUrls($domain = null)
+    {
+        if ($this->bufferedUrls === null) {
+            return parent::getUrls($domain);
+        }
+
+        return $this->bufferedUrls[$this->bufferKey($domain)] ??= parent::getUrls($domain);
+    }
+
+    /**
+     * @param  string  $key
+     * @param  string|null  $domain
+     * @return void
+     */
+    public function forgetUrl($key, $domain = null)
+    {
+        if ($this->bufferedUrls === null) {
+            parent::forgetUrl($key, $domain);
+
+            return;
+        }
+
+        // Collection::forget() mutates, so this updates the buffered map in place
+        // and the parent's next getUrls() in the same pass sees the removal.
+        $this->getUrls($domain)->forget($key);
+
+        $this->dirtyDomains[$this->bufferKey($domain)] = true;
+    }
+
+    private function whileBuffering(callable $callback): void
+    {
+        // A nested call is already inside a batch. Letting it run against the open
+        // buffer keeps one flush at the end instead of writing a half-finished map.
+        if ($this->bufferedUrls !== null) {
+            $callback();
+
+            return;
+        }
+
+        $this->bufferedUrls = [];
+        $this->dirtyDomains = [];
+
+        try {
+            $callback();
+        } finally {
+            $buffered = $this->bufferedUrls;
+            $dirty = $this->dirtyDomains;
+
+            // Closed before the writes so they, and anything reading the map
+            // afterwards, take the ordinary unbuffered path. In a finally block
+            // because a throw part-way through still has to persist the deletions
+            // already made — the alternative is forgetting them and serving pages
+            // whose responses are gone.
+            $this->bufferedUrls = null;
+            $this->dirtyDomains = [];
+
+            foreach (array_keys($dirty) as $domain) {
+                $this->cache->forever($this->getUrlsCacheKey($domain), $buffered[$domain]->all());
+            }
+        }
+    }
+
+    /**
+     * @param  string|null  $domain
+     */
+    private function bufferKey($domain): string
+    {
+        // Matches how getUrls() and getUrlsCacheKey() resolve a null domain, so a
+        // pass that mixes null and explicit domains still shares one buffer entry.
+        return $domain ?: $this->getBaseUrl();
+    }
+}

src/Cachers/TrackingApplicationCacher.php

@@ -11,5 +11,6 @@
  */
 final class TrackingApplicationCacher extends ApplicationCacher
 {
+    use BatchesInvalidation;
     use RecordsDependencies;
 }

src/Cachers/TrackingFileCacher.php

@@ -11,5 +11,6 @@
  */
 final class TrackingFileCacher extends FileCacher
 {
+    use BatchesInvalidation;
     use RecordsDependencies;
 }

src/Invalidation/GraphInvalidator.php

@@ -81,14 +81,45 @@ public function refresh($item): void
     public function invalidate($item): void
     {
         $tags = $this->tags->forItem($item);
+        $cached = $this->cached->all();
 
         $this->clear([
             ...$this->getItemUrls($item),
-            ...$tags === [] ? [] : $this->graph->urlsFor([...$tags, Tag::OVERFLOW]),
-            ...$this->graph->untracked($this->cached->all()),
+            ...$tags === [] ? [] : $this->stillCached($this->graph->urlsFor([...$tags, Tag::OVERFLOW]), $cached),
+            ...$this->graph->untracked($cached),
         ]);
     }
 
+    /**
+     * The graph outlives the cache: a URL keeps its row after its cached copy is
+     * gone, and rows are only pruned as they are invalidated. On a site that has
+     * been up for a while most of what a tag resolves to is no longer cached —
+     * often two thirds of it — and every one of those costs the cacher a lookup
+     * that can only miss. Narrowing here is what makes the remaining work match
+     * the number of pages that actually have to be cleared.
+     *
+     * Deliberately not applied to getItemUrls(): those are Statamic's own, may be
+     * wildcards, and are not graph rows to begin with.
+     *
+     * @param  list<string>  $urls
+     * @param  list<string>  $cached
+     * @return list<string>
+     */
+    private function stillCached(array $urls, array $cached): array
+    {
+        // An empty list is ambiguous. It means "nothing is cached" on a site that
+        // was just flushed, but it equally means "this cacher cannot be
+        // enumerated" — CachedUrls gives up on anything that is not an
+        // AbstractCacher, which includes a host app's own cacher. Narrowing
+        // against it would then clear nothing at all, the one failure this addon
+        // exists to prevent. Hand the full set over and let the cacher decide.
+        if ($cached === []) {
+            return $urls;
+        }
+
+        return array_values(array_intersect($urls, $cached));
+    }
+
     /**
      * @param  list<string>  $urls
      */