Bound the graph on disk, and let sites suppress their own navigation - #5
Merged
Merged
Conversation
…ation
Two problems found on a site running v2 since early September.
Graph growth
------------
The graph only learns about a URL by rendering it, so nothing ever removed a URL
that quietly fell out of the static cache. That site's graph described 53,713
URLs while only 5,880 were cached, and `untracked()` compared every save against
rows that could never match.
Adds `prune(array $keepUrls)` and a `cache-invalidation:prune` command that feeds
it the currently cached URLs. CachedUrls::supported() guards the case that would
otherwise empty the graph: a cacher that cannot enumerate returns an empty list,
which is indistinguishable from an empty cache.
Sqlite file size
----------------
`DELETE` moves pages onto sqlite's freelist and never shortens the file, so the
same site carried a 251 MB file whose freelist held 93% of its pages -- roughly
18 MB of live rows. Flushing on StaticCacheCleared made it worse, not better.
Adds `compact()` to the contract: a real VACUUM on sqlite, a no-op where the
database manages its own storage. Under WAL, VACUUM alone still leaves the main
file at its old length, so it is followed by a truncating checkpoint -- the test
asserts the file actually shrinks, which is what caught that.
Navigation built by hand
------------------------
TrackingNavTag already records `{{ nav }}` as the navigation rather than as every
entry in it. A site that assembles its menu in its own PHP gets no such
treatment, so each entry the menu reads becomes a dependency of every page
carrying it. On that site the seven top-level landing pages were each a
dependency of 77-99% of all tracked URLs, which is why saving one of them timed
the invalidation job out.
`suppressed()` already existed for Statamic's own resolution machinery but was
not reachable from outside. Exposes it as `CacheTags::withoutRecording()`, to be
paired with `add()` so one honest dependency stays behind.
156 tests, 266 assertions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two problems found on a site running v2 since early September.
Graph growth
The graph only learns about a URL by rendering it, so nothing ever removed a URL that quietly fell out of the static cache. That site's graph described 53,713 URLs while only 5,880 were cached, and
untracked()compared every save against rows that could never match.Adds
prune(array $keepUrls)and acache-invalidation:prunecommand that feeds it the currently cached URLs.CachedUrls::supported()guards the case that would otherwise empty the graph — a cacher that cannot enumerate returns an empty list, which is indistinguishable from an empty cache.Sqlite file size
DELETEmoves pages onto sqlite's freelist and never shortens the file. The same site carried a 251 MB file whose freelist held 93% of its pages — roughly 18 MB of live rows. Flushing onStaticCacheClearedmade it worse, not better.Adds
compact()to the contract: a realVACUUMon sqlite, a no-op where the database manages its own storage. Under WAL,VACUUMalone still leaves the main file at its old length, so it is followed by a truncating checkpoint. The test asserts the file actually shrinks, which is what caught that.Navigation built by hand
TrackingNavTagalready records{{ nav }}as the navigation rather than as every entry in it. A site that assembles its menu in its own PHP gets no such treatment, so each entry the menu reads becomes a dependency of every page carrying it.On that site the seven top-level landing pages were each a dependency of 77–99% of all tracked URLs, which is why saving one of them timed the invalidation job out — every
Invalidateentry infailed_jobsbetween 3 and 15 September was one of these.suppressed()already existed for Statamic's own resolution machinery but was not reachable from outside. Exposed asCacheTags::withoutRecording(), to be paired withadd()so one honest dependency stays behind:Applied on the reporting site, its navigation went from dozens of
entry:tags to exactly five coarse ones, and none of the landing pages.Notes
^2.0consumers get it either way.