Apply Query Loop post exclusions in PHP instead of SQL - #31
roborourke wants to merge 14 commits into
Conversation
WP_Query builds its `post-queries` cache key from the query vars and the generated SQL, so putting excluded IDs into `post__not_in` gives every URL a cache entry of its own. A "related posts" loop that excludes the post being viewed therefore never shares a cached result set with any other post, even though every one of those queries asks the same question. For non-inherited query loops, over-fetch by the number of exclusions and drop the unwanted posts on `the_posts` instead. Core writes the result to the object cache before `the_posts` runs, so the shareable superset is what gets cached and the filtering costs nothing in cache terms. Fetching `per_page + count(exclude)` rows guarantees a full page: at most one row can be dropped per excluded ID. This covers the plugin's own "exclude already displayed posts" setting, core's `excludeCurrent` block attribute, and anything added via the new `hm_query_loop_deferred_exclusions` filter. It falls back to SQL exclusion past `hm_query_loop_max_deferred_fetch`, when `hm_query_loop_defer_exclusions` is disabled, or when the query cannot reach `the_posts`. Alongside that: - Post templates now share one query. Each used to get a narrowed query of its own, with the preceding templates' posts in `post__not_in`; they now all issue the loop's own unmodified query and window the results in PHP, so N templates cost one query and one cache entry. - The plugin no longer leaks its own state into the cache key. Every custom query var is hashed into it, and `query_id` is derived from the post ID, so it was giving each loop a private key on every URL. Both it and the tracking flag are now stripped on `pre_get_posts`, before the key is generated. - `paged` is only set where `offset` is absent, since `offset` overrides it in the LIMIT clause and it was otherwise just noise in the key. - The exclusion set is snapshotted per loop, so a loop's post templates and its pagination query all exclude the same posts and share one entry. Adds unit tests for the planner that need only PHP, and docs/query-caching.md with the reasoning and what is still left to do. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fbx2jG1eYxRD4UzUB7gxab
Playwright test resultsDetails
Failed testschromium › exclude-with-post-in.spec.js › Exclude Displayed Posts with post__in › should exclude displayed posts even when post__in is set via Advanced Query Loop |
I reviewed and benchmarked this against a client project (28k posts, homepage + article pages). Short version: the approach works as intended, but I couldn't see a benefit with our content. The reason seems to be outside this PR. You mentioned this was untested, so I focused mainly on measuring it. The test suites are green on the current head ( The approach looks goodMoving everything into I instrumented Why it doesn't change anything for usI got zero difference on this project: 74 SQL / 45 cached, 37% hit rate, the same with and without the PR across 12 cold article loads. The reason is not this PR. Our Because I tested removing it at priority 11, after query-filter's transpose, and it took us from 68 → 60 queries across 12 page loads, and 41% → 49% hit rate. The output was byte-identical and filtering was still working correctly. This fix only works together with this PR though. Without this PR, Bucketing the fetch - tried it, but I don't recommend it
I tried rounding the fetch up to a whole page and measured it, but got no difference at all - 76 SQL / 68 cached / 47% hit, the same with and without it, tested twice. The variance I initially saw ( Based on that, I don't think the extra complexity is worth it. It can read up to an extra page of rows and also breaks three tests that expect the exact fetch size ( I'm parking this instead of pushing it. It could be useful if we have loops that are identical except for their exclusion count - same post type, same taxonomy, no Bucketing patch (not recommended, for reference)--- a/inc/deferred-exclusions.php
+++ b/inc/deferred-exclusions.php
@@ -262,12 +262,35 @@ function can_filter_results( array $query ): bool {
return ! in_array( $query['fields'] ?? '', [ 'ids', 'id=>parent' ], true );
}
+/**
+ * Granularity the over-fetch is rounded up to.
+ *
+ * A page of results is the natural unit: it is already the size the loop thinks
+ * in, and it keeps the worst-case waste to one page of rows.
+ *
+ * @param int $loop_per_page Posts the loop renders per page.
+ * @return int Bucket size, or 0 to round not at all.
+ */
+function bucket_size( int $loop_per_page ): int {
+ /**
+ * Filters the granularity the deferred-exclusion over-fetch rounds up to.
+ *
+ * Larger buckets share cache entries more widely and read more rows to do it.
+ * Return 0 to fetch exactly what the exclusions require.
+ *
+ * @param int $bucket Defaults to the loop's page size.
+ * @param int $loop_per_page Posts the loop renders per page.
+ */
+ return max( 0, (int) apply_filters( 'hm_query_loop_fetch_bucket', $loop_per_page, $loop_per_page ) );
+}
+
/**
* Work out what to fetch so the loop can be assembled in PHP afterwards.
*
* Over-fetching by `count( $exclude )` guarantees a full page after filtering:
* at most one fetched post can be dropped per excluded ID, so at least as many
- * survive as the loop asked for.
+ * survive as the loop asked for. The fetch is then rounded up — see
+ * bucket_size() — which only ever adds to that margin.
*
* @param array $query Query vars for the loop.
* @param int[] $exclude Post IDs to exclude.
@@ -317,6 +340,18 @@ function build_plan( array $query, array $exclude, int $page, array $context ):
$base_offset = $plan['fetch_offset'] - ( $loop_per_page * ( $page - 1 ) );
$fetch = ( $loop_per_page * $page ) + count( $exclude );
+ // The fetch size lands in the cache key, so letting it track the exclusion
+ // count exactly puts the variance straight back where it was taken from —
+ // a loop excluding five posts and the same loop excluding six issue
+ // different queries. Rounding up to a whole number of pages collapses that
+ // into one query per page of results. The extra rows are trimmed in PHP,
+ // so only the size of the fetch changes, never what the loop renders.
+ $bucket = bucket_size( $loop_per_page );
+
+ if ( $bucket > 0 ) {
+ $fetch = (int) ( ceil( $fetch / $bucket ) * $bucket );
+ }
+
// A negative base offset means the offset was not built by core's
// formula, and re-slicing would silently move the window. Past
// get_max_fetch() the over-fetch costs more than the shared cache entry |
Three conflicts, all from the sticky-posts feature landing in the same places this branch touches: - `hm-query-loop.php` require/init blocks: both modules load, sticky first so its `posts_orderby` filter is registered before the exclusion planner runs. - `modify_query_from_block_attrs()`: kept the sticky-posts stash and dropped main's `excludeDisplayedForCurrentLoop` block, which this branch replaced — post templates no longer exclude each other's posts, they window one shared result set. - README: both features kept, sticky renumbered to 8. The two compose as they stand. Sticky ordering is applied in SQL, so the over-fetched set arrives already ordered and the PHP filtering and windowing preserve that order. `StickyPosts\QUERY_VAR` does reach `WP_Query` — unlike this plugin's own bookkeeping, which is stripped on `pre_get_posts` — but it only repeats IDs the ORDER BY already puts in the SQL, so it adds no cache-key fragmentation of its own. Also documents `stickyPosts` in the CLAUDE.md context block, which the README already listed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fbx2jG1eYxRD4UzUB7gxab
Benchmarked the branch against b9f3925 on WP 6.9 / PHP 8.4 / MariaDB with 601 posts, rendering block fixtures across many URLs with the object cache persisting between them. The headline result is not the one the docs led with. A single query loop with no plugin settings at all — nothing for the over-fetching to do — still goes from 306 database queries to 108 across 100 URLs, because the gain there is entirely from no longer putting `query_id` in the query vars. Three distinct queries were being re-executed 201 times purely because their cache keys differed by post ID. Since `query_id` was set for every post-template query, every query loop the plugin touched had a private cache entry on every URL it rendered on. On a 12-loop page across 40 URLs: 1263 database queries to 72, and 50.4 to 34.5 ms per URL, with output identical post for post. A cold render against an empty cache — the worst case for over-fetching — is within noise at 74.9 vs 73.8 ms, and still drops 87 queries to 66. The admin editor is unaffected in every measurement, which is what the code predicts: the built editor bundle is byte-identical between the two builds and `query_loop_block_query_vars` does not fire in admin. Also corrects a claim these docs made: `excludeCurrent` is not in WordPress 6.9, only in core trunk. On 6.9 and earlier core ignores the attribute, so this plugin is not "taking it over" there — it is what makes the setting do anything at all. That is a behaviour change on those versions and is now called out as one; the benchmark shows it changing the rendered posts on 5 of 25 URLs, exactly those where the current post fell inside the window. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fbx2jG1eYxRD4UzUB7gxab
Same rig, second install on 7.1. The results hold: a plain query loop with no plugin settings goes from 306 database queries to 108 across 100 URLs, and the 12-loop page from 1261 to 70 across 40 URLs, with render time down roughly a third in both. The cold-render worst case stays within noise. Admin is unaffected on both versions. 7.1 also settles the `excludeCurrent` question. Core gained the attribute in 7.1 — not 6.9, and not 7.0, both of which ignore it — so on 7.1 this branch is output-identical on every fixture including the related-posts one, where on 6.9 it differs on the URLs whose current post falls inside the window. The version boundary in the docs was wrong in the other direction too and is now stated as measured rather than assumed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fbx2jG1eYxRD4UzUB7gxab
BenchmarksWordPress 6.9 and 7.1, PHP 8.4, MariaDB same host, Twenty Twenty-Five, 601 posts. Each "URL" is a full block render with the object cache persisting between renders, as a persistent object cache does between requests. Compared against A plain query loop, no plugin settings, 100 URLs
This loop has no exclusion settings, so none of the over-fetching does anything. The whole gain is that 12-loop page (two split across multiple post templates), 40 URLs
Cost: one cold render, empty cache (worst case for over-fetching)
Within noise over 15–20 interleaved runs per build. AdminMedians of 15 requests, two rounds, shown as a range across rounds.
Every gap is smaller than the spread between two rounds of the same build. Expected: the built editor bundle is byte-identical and Rendered output, compared post by post across 25 URLs
The one divergence is a core version boundary, measured rather than assumed: Full method and numbers in Generated by Claude Code |
An earlier edit spliced the corrected version boundary into the middle of a sentence and left it reading badly. Same facts, stated once. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fbx2jG1eYxRD4UzUB7gxab
The 2 Playwright failures are pre-existing, not from this PRBoth failures are in Control experiment: I re-ran
Unchanged code, same failure. So this is drift in the test environment, not a regression here. Why it's not this branch, independently of that: this PR ships no editor JavaScript — the built bundle is byte-identical to main's — and its PHP hooks either don't fire in admin ( Likely cause: I've not tried to fix it inside this PR, since it is unrelated to the change and would muddy the diff. Happy to take it as a separate piece of work — pinning core and AQL to fixed versions would stop the class of failure, not just this instance. Generated by Claude Code |
Playwright test results — WP 6.9Details
|
Playwright test results — WP 7.0Details
|
Playwright test results — WP 7.1Details
|
WordPress 7.0 moved the site editor onto path routes. The template editor used to be reached at `?postType=wp_template&postId=theme//slug`; it is now `?p=/wp_template/theme//slug`. The old URL does not error on 7.x — it quietly resolves to the dashboard route, so no editor header is drawn and every test that opened the settings sidebar timed out on a button that had never existed. That is all eight WP 7.0 failures and nine of the eleven on 7.1; the specs that drive the editor through `wp.data` rather than the chrome were unaffected, which is what pointed at navigation rather than at the panels. `visitSiteEditor` now tries the path route first and falls back to the query-string form, deciding which one worked by asking the editor store which entity it has open rather than by matching markup. The answer cannot change within a run, so it is remembered per worker. `openSettingsSidebar` no longer hunts for a header button at all. Where that button lives, and whether it is drawn, has moved between versions; `core/interface` and the two sidebar ids have not. Asking for the block sidebar directly also settles which tab opens, which the previous Block-tab click was working around. If it ever fails, it now reports the buttons that were on the page. The remaining two 7.1 failures were the grid Columns spinbutton, which 7.1 moves into a ToolsPanel. Those steps set up a layout the assertions never look at — the tests count posts on the front end — so they are gone rather than pinned to a version. With the suite passing, 7.0 and 7.1 move into the blocking matrix. Only trunk stays non-blocking, where a failure is news about WordPress rather than about the commit. Also restores the PHP unit tests job, which the merge of main dropped when it rewrote this workflow, and folds it into the `test` aggregate so the required check covers it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fbx2jG1eYxRD4UzUB7gxab
The route fix got the editor open; the modal over it was the next thing in
the way. Every remaining 7.0 and 7.1 failure reported the same page state —
"Buttons on the page: Get started" — which is the welcome modal and nothing
else.
Two reasons it was missed. The dismissal was gated on the modal's
surrounding copy ("Edit your site"), which is not what 7.x shows, and it ran
before the editor had booted, when there was nothing to find yet. It now
runs after the editor confirms the template is open, and drives off the
buttons rather than the copy.
Underneath both: `locator.isVisible()` ignores the timeout it is handed and
answers immediately, so these probes were racing whatever the editor
rendered a beat later. Replaced with a real wait here and in expandPanel,
where a panel left collapsed keeps its controls out of the DOM and the
failure lands later on whichever control the test wanted — which is how
"should apply query preset on frontend" failed on 7.1, in the post editor,
nowhere near the site editor routes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fbx2jG1eYxRD4UzUB7gxab
With the modal out of the way the sidebar opens, and the panel list it reported named the next problem: `[ 'Layout', 'Post Template Settings', 'Advanced' ]` on the first attempt and `[ 'Content' ]` on the retries. The first is the post template's inspector, the second is the Document tab — neither is the Query Loop block, so of course none of this plugin's panels were there. selectBlock.byName dispatched once into a store that had not parsed the template's blocks yet, found an empty list, and silently did nothing. In the post editor the content is already in the store when the helper runs, which is why this only ever showed up in the site editor. It now waits for the block to exist and then for the selection to take. expandPanel reports the panels that were present when it cannot find the one it wants, on the same reasoning as the sidebar diagnostic: this suite can only be watched through CI, so a failure has to explain itself in one round. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fbx2jG1eYxRD4UzUB7gxab
`Panels present: Content` on every 7.0 failure — the Document tab, which holds none of the panels this plugin adds. Block selection is right now; the tab was not. Enabling the block sidebar through the interface store is meant to settle which tab opens, and does not always: the sidebar mounts after the dispatch and picks its own default. Clicking the tab is the part that sticks, and that click was still guarded by the `isVisible()` probe this suite has been using as though it waits — so on a tab list that had not rendered yet, it was skipped. Wait for the tab, click it, then wait for it to take. The "Block" label itself is unchanged across 6.9, 7.0 and 7.1, so nothing here is version-specific; the race just lost more often on 7.x. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fbx2jG1eYxRD4UzUB7gxab
Three rounds of "Panels present: Content" have said what is not there without saying why. The inspector diagnostic now carries the active complementary area, the selected block's name, and the tab list with which tab is selected — enough to tell a wrong tab from a wrong block from a panel that genuinely is not rendered on this WordPress. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fbx2jG1eYxRD4UzUB7gxab
The last round settled the scaffolding: area=edit-post/block, block=core/query, Block tab selected — the right block, the right tab, and still no panel. So the block's edit component is not mounted, and the two candidates are this plugin's own viewport placeholder standing in for it, or a canvas that is not in edit mode. The placeholder carries a class of its own, so the diagnostic can just say which. It now also prints the visible tab panel's text, since allTextContents() does not filter hidden nodes and "panels=[Content]" may have been the hidden Template panel all along — in which case the Block tab is rendering nothing at all, which is the answer rather than a clue. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fbx2jG1eYxRD4UzUB7gxab
WordPress 7.0 and 7.1 content-lock template patterns by default — for example twentytwentyfive's "List of posts" pattern, which wraps the Query Loop block on the index template used throughout this suite. Selecting a block inside one still shows the pattern's own card and a single "Content" panel in the sidebar, none of this plugin's controls, until "Edit pattern" is clicked to enter editing mode for the blocks inside it. The a2c020a diagnostic probe (placeholders=0) had already ruled out the lazy viewport placeholder as the cause. Reproducing locally against WP 7.0.4 and dumping the sidebar's DOM showed the real block card: title "List of posts, 1 column" with a "Pattern" badge, plus an "Edit pattern" button — the content-locking UI, not a missing panel. openSettingsSidebar() now clicks that button (scoped to the "Editor settings" region, since an identical-looking button also lives in the block toolbar) once the Block tab is confirmed active, and waits for the inspector to re-render with the selected block's own controls. This runs unconditionally but is a no-op off pattern-locked content, so it costs nothing on 6.9 or on non-templated pages. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
WordPress 7.1's core/query edit component throws reading a post type label when query.postType is undefined, crashing the block's React tree with "Cannot read properties of undefined (reading 'singular_name')" — visible as zero inspector panels at all, not even core's own, rather than a missing plugin panel. query is a single object-shaped block attribute, so passing a partial object to editor.insertBlock() replaces block.json's whole default (which does set postType: 'post') instead of merging into it. Earlier WordPress releases papered over the resulting undefined postType with a defensive fallback in core's own edit.js; 7.1 no longer has one. Confirmed locally against WP 7.1 by reproducing the crash and fixing it with an explicit postType, which the block.json default was always supposed to provide. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
| # One core version is enough: this varies the plugin, and running | ||
| # every lane would only multiply the same AQL signal. | ||
| core_matrix: '[{ "label": "6.9", "core": "WordPress/WordPress#6.9.7", "comment": false }]' |
There was a problem hiding this comment.
For this specific one the default may as well be nightly or 7.1 rather than 6.9 here while we're bringing things up to date
The problem
WP_Querybuilds itspost-queriescache key frommd5( serialize( $args ) . $sql )— the query vars and the generated SQL. Two things onmainmake that key vary per URL:query_idinto the query vars for everycore/post-templatequery.query_idis derived from the post ID, so every query loop the plugin touches gets a private cache entry on every URL it renders on — whether or not any of the plugin's features are enabled.post__not_inputs the excluded IDs into the SQL, so a loop excluding the post being viewed can never share a cached result.Benchmarking says (1) is by far the bigger effect.
What it measures out at
Against
b9f3925, on WP 6.9 / PHP 8.4 / MariaDB, 601 posts, object cache persisting between renders (what a persistent object cache does between requests):A single query loop with no plugin settings at all, across 100 URLs:
SELECTs executedNothing in the over-fetching machinery is doing anything here. Three distinct queries were being re-executed 201 times purely because their cache keys differed by post ID.
A 12-loop magazine page (two loops split across multiple post templates), across 40 URLs:
SELECTs executedCost, where no cache benefit is available — one cold render against an empty cache, the worst case for over-fetching:
Within noise over 20 interleaved runs each.
The admin editor is unaffected. Medians of 15 requests, two rounds: block editor 218–235 ms before / 215–221 ms after; posts list 61–65 / 62–63; site editor 117–125 / 121–125; REST
wp/v2/posts28–30 / 28. Every gap is smaller than the spread between two rounds of the same build. That is what the code predicts — the built editor bundle is byte-identical between the two, andquery_loop_block_query_varsdoes not fire in admin.Method and full numbers are in
docs/query-caching.md.The approach
To show 5 posts excluding the current one, fetch 6 with no exclusion at all, drop the current post in PHP, render the first 5. The query — and the cache key — is then identical on every URL.
This works because of where core writes to the cache: in
WP_Query::get_posts()the object cache write happens beforeposts_resultsandthe_posts(WP 6.9: ~line 3455 vs ~3633). The unfiltered superset is what gets cached; anything removed inthe_postsis removed per request.Over-fetching by exactly
count( $exclude )guarantees a full page: at most one fetched row can be dropped per excluded ID. The fetch size depends on how many IDs are excluded, not which, so it stays stable across URLs.Scoped to non-inherited loops. Inherited loops run against the main query, whose key is per-URL regardless.
Changes
Nothing the plugin tracks reaches the cache key.
generate_cache_key()strips exactly seven query vars and serialises everything else — there is no allow-list.query_idandhm_query_loop_collect_idsnow travel in one var that is stripped onpre_get_posts, before the key is generated, and bound to theWP_Queryinstance instead.pagedis only set whereoffsetis absent, sinceoffsetoverrides it in the LIMIT clause.Deferred exclusions (
inc/deferred-exclusions.php). Collects the IDs a loop wants to exclude, plans an over-fetch, drops them onthe_postsat priority 9 — before post tracking at 10, so only posts that really render are recorded.Post templates share one query. Each used to get a narrowed query of its own — smaller
posts_per_page, preceding templates' IDs inpost__not_in. They now all issue the loop's own unmodified query and window the results in PHP.found_postsis corrected per request, not through thefound_postsfilter — that filter only runs on a cache miss and its result is baked into the shared entry.A behaviour change worth flagging
excludeCurrentis not in WordPress 6.9 — only in core trunk. On 6.9 and earlier core ignores the attribute entirely, so this plugin does not "take it over" there; it is what makes the setting do anything. A loop whose block attributes carryexcludeCurrentstarts excluding the current post where the attribute previously did nothing. The benchmark shows this changing the rendered posts on 5 of 25 URLs — exactly those where the current post fell inside the window. Intended behaviour for the setting, but a behaviour change, and the docs now say so.Falling back
Exclusion returns to SQL when the fetch would exceed
hm_query_loop_max_deferred_fetch(default 100 — deep pagination fetchesper_page * page + n), whenhm_query_loop_defer_exclusionsis filtered false, or when the query cannot reachthe_posts(fields => 'ids'andsuppress_filtersboth skip it in core).Testing
npm run test:php— 27 assertions on the planner, running on plain PHP with WordPress stubbed, added as a CI job. Playwright: 22/22 green. PHPCS (added tomainsince this branch opened) passes.Merged
mainin: the sticky-posts feature landed in the same places. Both compose — sticky ordering is applied in SQL, so the over-fetched set arrives already ordered and the PHP filtering and windowing preserve it.StickyPosts\QUERY_VARdoes reachWP_Query, but it only repeats IDs the ORDER BY already puts in the SQL, so it adds no fragmentation of its own.Still worth knowing
Any post meta write, on any post, bumps
postslast_changedand invalidates every cached query site-wide. On a site with view counters or similar per-request meta writes that swamps all of this.docs/query-caching.mdhas a snippet to measure it before investing further, plus a ranked list of what is left — inherited queries, sharing result sets between sibling loops, quantisingposts_per_page.🤖 Generated with Claude Code
https://claude.ai/code/session_01Fbx2jG1eYxRD4UzUB7gxab