From 1c8a5e9eb3551b6f8a223de4afb21c70c4b72125 Mon Sep 17 00:00:00 2001 From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com> Date: Wed, 2 Sep 2026 20:10:20 +0200 Subject: [PATCH 1/6] ci(indexnow): treat an edge 403 on the key file as deployed The readiness loop treated every non-200 as "not deployed yet"; a runner that Cloudflare's bot management answers with 403 would have slept the full eight minutes on every run. Mirrors the sister repo's review. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01SrKzcwZBnref1sWYtdXynu --- .github/workflows/indexnow-submit.yml | 13 +++++++++---- CHANGELOG.md | 5 +++++ 2 files changed, 14 insertions(+), 4 deletions(-) diff --git a/.github/workflows/indexnow-submit.yml b/.github/workflows/indexnow-submit.yml index 83ed76988a..308e26dd2c 100644 --- a/.github/workflows/indexnow-submit.yml +++ b/.github/workflows/indexnow-submit.yml @@ -123,11 +123,16 @@ jobs: # either way: a runner behind Cloudflare's bot management may see a # 403 that Bing's own fetch does not, and IndexNow verifies itself. for i in $(seq 1 16); do - if curl -fsS --max-time 15 -o /dev/null "https://${HOST}/${INDEXNOW_KEY}.txt"; then - echo "::notice::key file reachable"; break - fi + status=$(curl -sS --max-time 15 -o /dev/null -w '%{http_code}' \ + "https://${HOST}/${INDEXNOW_KEY}.txt" || echo 000) + case "$status" in + 200) echo "::notice::key file reachable"; break ;; + # Deployed, but this runner is blocked at the edge: no point in + # waiting — Bing's own fetch is not subject to that block. + 403) echo "::notice::key file answers 403 to this runner (edge bot management); treating as deployed"; break ;; + esac if [ "$i" -eq 16 ]; then - echo "::warning::key file not confirmed reachable after 8 min; submitting anyway" + echo "::warning::key file not confirmed reachable after 8 min (last status ${status}); submitting anyway" else sleep 30 fi diff --git a/CHANGELOG.md b/CHANGELOG.md index 53c76205a1..79116eeac5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -41,6 +41,11 @@ aggregate instead: an italic *Catalog* line at the end of the version section an ### Fixed +- **The IndexNow workflow no longer waits eight minutes behind an edge 403** — its + key-file readiness loop treated every non-200 as "not deployed yet"; a GitHub runner + that Cloudflare's bot management answers with 403 would have slept the full budget on + every run. A 403 now counts as deployed (Bing's own fetch is not subject to that + block), and the final warning names the last status seen. - **Implementation pages no longer share one meta description per spec** — the SEO proxy reused the spec description verbatim as the `` and OG description of every implementation page, so up to 15 library pages and their hub From 782a386adda8a4087996776943b49e827567b784 Mon Sep 17 00:00:00 2001 From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com> Date: Wed, 2 Sep 2026 20:16:04 +0200 Subject: [PATCH 2/6] ci(indexnow): send the body from a file, tolerate the first-use 403 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The first sitemap-scope run (33665718870) never reached IndexNow: a full-sitemap body is ~260 KB and a single command-line argument is capped at 128 KB on Linux, so curl failed to exec ("Argument list too long") and the run reported HTTP 000 for 4,593 URLs. The body now goes through `--data @body.json`. The retry by hand then answered 403 SiteVerificationNotCompleted: IndexNow verifies a new key file asynchronously. That is a warning, not an error — the next push resubmits its URLs and scope=sitemap is a re-run away. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01SrKzcwZBnref1sWYtdXynu --- .github/workflows/indexnow-submit.yml | 25 ++++++++++++++++++++----- 1 file changed, 20 insertions(+), 5 deletions(-) diff --git a/.github/workflows/indexnow-submit.yml b/.github/workflows/indexnow-submit.yml index 308e26dd2c..345817c739 100644 --- a/.github/workflows/indexnow-submit.yml +++ b/.github/workflows/indexnow-submit.yml @@ -141,24 +141,39 @@ jobs: # under that today (4.6k) but the split keeps this future-proof. split -l 10000 -d urls.txt batch_ for f in batch_*; do - body=$(jq -n --arg host "$HOST" --arg key "$INDEXNOW_KEY" \ + # The body goes through a file: a full-sitemap batch is ~250 KB, + # and a single command-line argument is capped at 128 KB on Linux + # — the first sitemap-scope run (33665718870) failed to exec curl + # at all and reported HTTP 000 for 4,593 URLs. + jq -n --arg host "$HOST" --arg key "$INDEXNOW_KEY" \ --arg loc "https://${HOST}/${INDEXNOW_KEY}.txt" \ --rawfile list "$f" \ '{host: $host, key: $key, keyLocation: $loc, - urlList: ($list | split("\n") | map(select(length > 0)))}') + urlList: ($list | split("\n") | map(select(length > 0)))}' > body.json # Bounded: a slow or flaky api.indexnow.org must not burn the job # timeout; three retries cover a transient error, and a transport # error after them yields code 000 for the branch below instead of # aborting under `set -e`. code=$(curl -sS -o response.txt -w '%{http_code}' \ - --max-time 30 --retry 3 --retry-delay 5 --retry-all-errors \ + --max-time 60 --retry 3 --retry-delay 5 --retry-all-errors \ -X POST "https://api.indexnow.org/indexnow" \ -H "Content-Type: application/json; charset=utf-8" \ - --data "$body") || code="000" + --data @body.json) || code="000" n=$(grep -c . "$f") case "$code" in 200|202) echo "::notice::IndexNow accepted ${n} URL(s) (HTTP ${code})" ;; - # 4xx is a protocol or key problem on our side; make it visible. + # First use of a key (rollout or rotation): IndexNow verifies the + # key file asynchronously and answers 403 SiteVerificationNotCompleted + # until it has — seen on the first sitemap-scope run, minutes + # after the file went live. Not our fault, so warn; the next + # push resubmits its own URLs, and scope=sitemap is a re-run away. + 403) + if grep -q SiteVerificationNotCompleted response.txt 2>/dev/null; then + echo "::warning::IndexNow has not finished verifying the key file yet (403 SiteVerificationNotCompleted); ${n} URL(s) not accepted — re-run with scope=sitemap later" + else + echo "::error::IndexNow rejected ${n} URL(s) (HTTP 403): $(head -c 300 response.txt 2>/dev/null)"; exit 1 + fi ;; + # Any other 4xx is a protocol or key problem on our side; make it visible. 4*) echo "::error::IndexNow rejected ${n} URL(s) (HTTP ${code}): $(head -c 300 response.txt 2>/dev/null)"; exit 1 ;; # 5xx / no response: their side. Every later push resubmits its # own URLs and `scope=sitemap` covers a longer gap, so warn. From 648009a99de9e73a03c48729e432309477f3f49b Mon Sep 17 00:00:00 2001 From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com> Date: Wed, 2 Sep 2026 20:17:02 +0200 Subject: [PATCH 3/6] ci(indexnow): assign the status fallback after the substitution; changelog ref Review feedback: curl prints 000 for %{http_code} on a transport failure, so '|| echo 000' inside the substitution recorded 000000. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01SrKzcwZBnref1sWYtdXynu --- .github/workflows/indexnow-submit.yml | 4 +++- CHANGELOG.md | 6 +++++- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/.github/workflows/indexnow-submit.yml b/.github/workflows/indexnow-submit.yml index 345817c739..12ea119ef5 100644 --- a/.github/workflows/indexnow-submit.yml +++ b/.github/workflows/indexnow-submit.yml @@ -123,8 +123,10 @@ jobs: # either way: a runner behind Cloudflare's bot management may see a # 403 that Bing's own fetch does not, and IndexNow verifies itself. for i in $(seq 1 16); do + # On a transport failure curl still prints 000 for %{http_code}; + # the fallback is assigned afterwards so nothing is appended. status=$(curl -sS --max-time 15 -o /dev/null -w '%{http_code}' \ - "https://${HOST}/${INDEXNOW_KEY}.txt" || echo 000) + "https://${HOST}/${INDEXNOW_KEY}.txt") || status=000 case "$status" in 200) echo "::notice::key file reachable"; break ;; # Deployed, but this runner is blocked at the edge: no point in diff --git a/CHANGELOG.md b/CHANGELOG.md index 79116eeac5..8550fac947 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -45,7 +45,11 @@ aggregate instead: an italic *Catalog* line at the end of the version section an key-file readiness loop treated every non-200 as "not deployed yet"; a GitHub runner that Cloudflare's bot management answers with 403 would have slept the full budget on every run. A 403 now counts as deployed (Bing's own fetch is not subject to that - block), and the final warning names the last status seen. + block), and the final warning names the last status seen. The submission body goes + through a file (a full-sitemap batch is ~260 KB, above the 128 KB command-line + argument cap that made the first sitemap-scope run report HTTP 000), and a first-use + `403 SiteVerificationNotCompleted` from IndexNow is a warning, not a failed run. + (#11204) - **Implementation pages no longer share one meta description per spec** — the SEO proxy reused the spec description verbatim as the `` and OG description of every implementation page, so up to 15 library pages and their hub From e94c0ca5c1febeed5466ab4d6ac21608adc7da92 Mon Sep 17 00:00:00 2001 From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com> Date: Wed, 2 Sep 2026 20:23:51 +0200 Subject: [PATCH 4/6] ci(indexnow): retry the first-use 403 for ten minutes, then fail; self-trigger Review feedback: a green warning on 403 SiteVerificationNotCompleted would drop a `changed` run's URLs for good, because later pushes submit only their own diffs, and the docs promise visible 4xx failures. The case is now retried once a minute for up to ten minutes and then fails the run, which keeps it re-runnable with the same URL set. A push that changes this workflow file triggers it with the full list, so a fix runs before the next plot merge. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01SrKzcwZBnref1sWYtdXynu --- .github/workflows/indexnow-submit.yml | 56 +++++++++++++++++---------- CHANGELOG.md | 4 +- docs/reference/seo.md | 7 +++- 3 files changed, 43 insertions(+), 24 deletions(-) diff --git a/.github/workflows/indexnow-submit.yml b/.github/workflows/indexnow-submit.yml index 12ea119ef5..5efa6a6ae3 100644 --- a/.github/workflows/indexnow-submit.yml +++ b/.github/workflows/indexnow-submit.yml @@ -24,6 +24,9 @@ on: branches: [main] paths: - 'plots/**' + # A change to this workflow exercises itself with the full list; a + # fix that never runs until the next plot merge is a fix nobody saw. + - '.github/workflows/indexnow-submit.yml' workflow_dispatch: inputs: scope: @@ -81,6 +84,11 @@ jobs: commits=$(jq 'length' <<<"$PUSH_COMMITS") if [ "$SCOPE" = "sitemap" ]; then full_list | sort -u > urls.txt + elif [ "$commits" -eq 1 ] && git diff --name-only HEAD~1 HEAD -- .github/workflows/indexnow-submit.yml | grep -q .; then + # The workflow itself changed (the push.paths entry above): run + # the real thing end to end rather than an empty diff. + echo "::notice::workflow file changed; submitting the full list" + full_list | sort -u > urls.txt elif [ "$commits" -gt 1 ]; then # A multi-commit push (rare on main) is not what fetch-depth 2 can # diff; submitting everything is cheap and always correct. @@ -152,30 +160,36 @@ jobs: --rawfile list "$f" \ '{host: $host, key: $key, keyLocation: $loc, urlList: ($list | split("\n") | map(select(length > 0)))}' > body.json - # Bounded: a slow or flaky api.indexnow.org must not burn the job - # timeout; three retries cover a transient error, and a transport - # error after them yields code 000 for the branch below instead of - # aborting under `set -e`. - code=$(curl -sS -o response.txt -w '%{http_code}' \ - --max-time 60 --retry 3 --retry-delay 5 --retry-all-errors \ - -X POST "https://api.indexnow.org/indexnow" \ - -H "Content-Type: application/json; charset=utf-8" \ - --data @body.json) || code="000" n=$(grep -c . "$f") + # First use of a key (rollout or rotation): IndexNow verifies the + # key file asynchronously and answers 403 SiteVerificationNotCompleted + # until it has — seen minutes after the file went live. Retry that + # case for up to ten minutes; if it persists, the run FAILS, so a + # `changed` run keeps its URL set and can be re-run — a green + # warning would have dropped those URLs for good, because later + # pushes submit only their own diffs. + for try in $(seq 1 10); do + # Bounded: a slow or flaky api.indexnow.org must not burn the job + # timeout; three retries cover a transient error, and a transport + # error after them yields code 000 for the branch below instead of + # aborting under `set -e`. + code=$(curl -sS -o response.txt -w '%{http_code}' \ + --max-time 60 --retry 3 --retry-delay 5 --retry-all-errors \ + -X POST "https://api.indexnow.org/indexnow" \ + -H "Content-Type: application/json; charset=utf-8" \ + --data @body.json) || code="000" + if [ "$code" = 403 ] && grep -q SiteVerificationNotCompleted response.txt 2>/dev/null; then + echo "::notice::IndexNow has not finished verifying the key file yet (try ${try}/10); waiting 60 s" + sleep 60 + continue + fi + break + done case "$code" in 200|202) echo "::notice::IndexNow accepted ${n} URL(s) (HTTP ${code})" ;; - # First use of a key (rollout or rotation): IndexNow verifies the - # key file asynchronously and answers 403 SiteVerificationNotCompleted - # until it has — seen on the first sitemap-scope run, minutes - # after the file went live. Not our fault, so warn; the next - # push resubmits its own URLs, and scope=sitemap is a re-run away. - 403) - if grep -q SiteVerificationNotCompleted response.txt 2>/dev/null; then - echo "::warning::IndexNow has not finished verifying the key file yet (403 SiteVerificationNotCompleted); ${n} URL(s) not accepted — re-run with scope=sitemap later" - else - echo "::error::IndexNow rejected ${n} URL(s) (HTTP 403): $(head -c 300 response.txt 2>/dev/null)"; exit 1 - fi ;; - # Any other 4xx is a protocol or key problem on our side; make it visible. + # 4xx is a protocol or key problem on our side (or the key still + # unverified after ten minutes); make it visible and keep the run + # re-runnable. 4*) echo "::error::IndexNow rejected ${n} URL(s) (HTTP ${code}): $(head -c 300 response.txt 2>/dev/null)"; exit 1 ;; # 5xx / no response: their side. Every later push resubmits its # own URLs and `scope=sitemap` covers a longer gap, so warn. diff --git a/CHANGELOG.md b/CHANGELOG.md index 8550fac947..3a3ecba8aa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -48,8 +48,8 @@ aggregate instead: an italic *Catalog* line at the end of the version section an block), and the final warning names the last status seen. The submission body goes through a file (a full-sitemap batch is ~260 KB, above the 128 KB command-line argument cap that made the first sitemap-scope run report HTTP 000), and a first-use - `403 SiteVerificationNotCompleted` from IndexNow is a warning, not a failed run. - (#11204) + `403 SiteVerificationNotCompleted` from IndexNow is retried for up to ten minutes + before the run fails, so a `changed` run keeps its URL set re-runnable. (#11204) - **Implementation pages no longer share one meta description per spec** — the SEO proxy reused the spec description verbatim as the `` and OG description of every implementation page, so up to 15 library pages and their hub diff --git a/docs/reference/seo.md b/docs/reference/seo.md index 99b1e90d3e..a260e3ab26 100644 --- a/docs/reference/seo.md +++ b/docs/reference/seo.md @@ -521,7 +521,12 @@ sitemap. Three pieces, kept in sync when the key is rotated: | Manual full load | `gh workflow run indexnow-submit.yml -f scope=sitemap` | Submits every URL of the live sitemap; used once at rollout and after a long outage of the workflow. | The engines answer `200` or `202` for an accepted batch; `4xx` means a key or -payload problem and fails the run so it is visible. Bing Webmaster Tools shows +payload problem and fails the run so it is visible. One `4xx` is transient by +nature: after a rollout or a key rotation IndexNow verifies the key file +asynchronously and answers `403 SiteVerificationNotCompleted` for a while, so +the workflow retries that case once a minute for up to ten minutes before it +fails — a `changed` run must not report success without its URLs accepted, +because later pushes submit only their own diffs. Bing Webmaster Tools shows the received submissions under *IndexNow*. ## Discoverability for assistants From bfd7d84495e2790ebd18c224f1bdf05653629858 Mon Sep 17 00:00:00 2001 From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com> Date: Wed, 2 Sep 2026 20:33:15 +0200 Subject: [PATCH 5/6] Cover both waits in the job timeout and document the self-trigger The job kept `timeout-minutes: 10` while the step can now wait 8 min for the key file and another 10 min for key verification, so GitHub would have cancelled the run before the final 403 error. The timeout covers both bounded waits, the verification loop no longer sleeps after its last try, and the workflow docs (seo.md, workflows/overview.md) now mention that a change to the workflow file submits the full list. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01SrKzcwZBnref1sWYtdXynu --- .github/workflows/indexnow-submit.yml | 8 +++++--- docs/reference/seo.md | 2 +- docs/workflows/overview.md | 2 +- 3 files changed, 7 insertions(+), 5 deletions(-) diff --git a/.github/workflows/indexnow-submit.yml b/.github/workflows/indexnow-submit.yml index 5efa6a6ae3..cee38c48e0 100644 --- a/.github/workflows/indexnow-submit.yml +++ b/.github/workflows/indexnow-submit.yml @@ -51,7 +51,9 @@ env: jobs: submit: runs-on: ubuntu-latest - timeout-minutes: 10 + # Worst case: 8 min waiting for the key file + 10 min of verification + # retries + the curl timeouts, with room to spare. + timeout-minutes: 25 steps: - name: Checkout repository uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 @@ -179,8 +181,8 @@ jobs: -H "Content-Type: application/json; charset=utf-8" \ --data @body.json) || code="000" if [ "$code" = 403 ] && grep -q SiteVerificationNotCompleted response.txt 2>/dev/null; then - echo "::notice::IndexNow has not finished verifying the key file yet (try ${try}/10); waiting 60 s" - sleep 60 + echo "::notice::IndexNow has not finished verifying the key file yet (try ${try}/10)" + [ "$try" -lt 10 ] && sleep 60 continue fi break diff --git a/docs/reference/seo.md b/docs/reference/seo.md index a260e3ab26..1803c900ac 100644 --- a/docs/reference/seo.md +++ b/docs/reference/seo.md @@ -517,7 +517,7 @@ sitemap. Three pieces, kept in sync when the key is rotated: | Piece | Where | Purpose | |-------|-------|---------| | Key file | `app/public/.txt` (served by nginx to every client, bots included — an explicit `location =` like `robots.txt`) | Proves the submitter controls `anyplot.ai`; the engines fetch it on every submission. The key is public by design. | -| Submission workflow | `.github/workflows/indexnow-submit.yml` | On every push to `main` that touches `plots/`, maps the diff to page URLs (`/{spec}` and `/{spec}/{language}/{library}`) and POSTs them to `https://api.indexnow.org/indexnow`, 10,000 per request. A deleted implementation is submitted too — the protocol means "this URL changed". | +| Submission workflow | `.github/workflows/indexnow-submit.yml` | On every push to `main` that touches `plots/`, maps the diff to page URLs (`/{spec}` and `/{spec}/{language}/{library}`; a changed specification covers the hub and all of its implementation pages) and POSTs them to `https://api.indexnow.org/indexnow`, 10,000 per request. A deleted implementation is submitted too — the protocol means "this URL changed". A push that changes the workflow file itself, or one carrying more than one commit, submits the full list instead of a diff. | | Manual full load | `gh workflow run indexnow-submit.yml -f scope=sitemap` | Submits every URL of the live sitemap; used once at rollout and after a long outage of the workflow. | The engines answer `200` or `202` for an accepted batch; `4xx` means a key or diff --git a/docs/workflows/overview.md b/docs/workflows/overview.md index 7bde8754bf..04c57d679e 100644 --- a/docs/workflows/overview.md +++ b/docs/workflows/overview.md @@ -172,7 +172,7 @@ Located in `.github/workflows/`: | `report-validate.yml` | Validates user-submitted issue reports | | `sync-postgres.yml` | Syncs `plots/` filesystem state to PostgreSQL on push to main | | `sync-labels.yml` | Auto-syncs spec/impl labels after manual PR merges | -| `indexnow-submit.yml` | Pushes changed page URLs to IndexNow (Bing, Yandex, Seznam, Naver, Yep) on every push to main that touches `plots/`; `workflow_dispatch` with `scope=sitemap` submits the whole sitemap | +| `indexnow-submit.yml` | Pushes changed page URLs to IndexNow (Bing, Yandex, Seznam, Naver, Yep) on every push to main that touches `plots/`; a push that changes the workflow file itself submits the full list, as does `workflow_dispatch` with `scope=sitemap` | | `codeql.yml` | CodeQL scanning (actions, JavaScript/TypeScript, Python) on pushes to main, PRs and a weekly cron; `plots/**` is excluded from triggers and analysis, so pipeline PRs never start a scan | | `ci-lint.yml` | Ruff lint check on PRs | | `ci-tests.yml` | Unit + integration tests on PRs | From f3508e38b2b1fc01a3c06588a7da287aa43c7b00 Mon Sep 17 00:00:00 2001 From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com> Date: Wed, 2 Sep 2026 20:41:16 +0200 Subject: [PATCH 6/6] Bound the verification retry by elapsed time; do not read a 403 as deployed Ten attempts with 60-second requests and 60-second gaps could take about 19 minutes, not the ten the docs promise; the loop now retries until a ten-minute elapsed deadline has passed and the job timeout covers the real worst case. An edge 403 on the key-file probe is answered before the origin is asked, so the notice, the comment and the changelog no longer call it "deployed": the wait is skipped and IndexNow's own fetch decides. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01SrKzcwZBnref1sWYtdXynu --- .github/workflows/indexnow-submit.yml | 37 +++++++++++++++------------ CHANGELOG.md | 6 +++-- docs/reference/seo.md | 7 ++--- 3 files changed, 29 insertions(+), 21 deletions(-) diff --git a/.github/workflows/indexnow-submit.yml b/.github/workflows/indexnow-submit.yml index cee38c48e0..7a7b12160e 100644 --- a/.github/workflows/indexnow-submit.yml +++ b/.github/workflows/indexnow-submit.yml @@ -51,9 +51,10 @@ env: jobs: submit: runs-on: ubuntu-latest - # Worst case: 8 min waiting for the key file + 10 min of verification - # retries + the curl timeouts, with room to spare. - timeout-minutes: 25 + # Worst case: ~12 min waiting for the key file (16 probes, 15 s timeout, + # 30 s apart) + the 10 min verification deadline + one last request with + # its retries (up to ~4 min), with room to spare. + timeout-minutes: 30 steps: - name: Checkout repository uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 @@ -130,8 +131,8 @@ jobs: # The engines validate a submission by fetching the key file. It is # served by the app deploy, which a rollout or a key rotation may # still have in flight — wait for it (up to ~8 min), but submit - # either way: a runner behind Cloudflare's bot management may see a - # 403 that Bing's own fetch does not, and IndexNow verifies itself. + # either way: this probe is a courtesy, IndexNow's own key fetch + # below is the authoritative check. for i in $(seq 1 16); do # On a transport failure curl still prints 000 for %{http_code}; # the fallback is assigned afterwards so nothing is appended. @@ -139,9 +140,10 @@ jobs: "https://${HOST}/${INDEXNOW_KEY}.txt") || status=000 case "$status" in 200) echo "::notice::key file reachable"; break ;; - # Deployed, but this runner is blocked at the edge: no point in - # waiting — Bing's own fetch is not subject to that block. - 403) echo "::notice::key file answers 403 to this runner (edge bot management); treating as deployed"; break ;; + # Cloudflare's bot management answers 403 before the origin is + # asked, so this says nothing about the deploy either way, and + # waiting cannot change it. Bing's fetch is not subject to it. + 403) echo "::notice::key file answers 403 to this runner (edge bot management), which cannot confirm the deploy; skipping the wait, IndexNow verifies the key itself"; break ;; esac if [ "$i" -eq 16 ]; then echo "::warning::key file not confirmed reachable after 8 min (last status ${status}); submitting anyway" @@ -166,11 +168,13 @@ jobs: # First use of a key (rollout or rotation): IndexNow verifies the # key file asynchronously and answers 403 SiteVerificationNotCompleted # until it has — seen minutes after the file went live. Retry that - # case for up to ten minutes; if it persists, the run FAILS, so a - # `changed` run keeps its URL set and can be re-run — a green - # warning would have dropped those URLs for good, because later - # pushes submit only their own diffs. - for try in $(seq 1 10); do + # case once a minute until a ten-minute deadline has passed (an + # elapsed deadline, so slow responses cannot stretch it); if it + # persists, the run FAILS, so a `changed` run keeps its URL set and + # can be re-run — a green warning would have dropped those URLs for + # good, because later pushes submit only their own diffs. + deadline=$(( $(date +%s) + 600 )) + while :; do # Bounded: a slow or flaky api.indexnow.org must not burn the job # timeout; three retries cover a transient error, and a transport # error after them yields code 000 for the branch below instead of @@ -180,9 +184,10 @@ jobs: -X POST "https://api.indexnow.org/indexnow" \ -H "Content-Type: application/json; charset=utf-8" \ --data @body.json) || code="000" - if [ "$code" = 403 ] && grep -q SiteVerificationNotCompleted response.txt 2>/dev/null; then - echo "::notice::IndexNow has not finished verifying the key file yet (try ${try}/10)" - [ "$try" -lt 10 ] && sleep 60 + if [ "$code" = 403 ] && grep -q SiteVerificationNotCompleted response.txt 2>/dev/null \ + && [ "$(date +%s)" -lt "$deadline" ]; then + echo "::notice::IndexNow has not finished verifying the key file yet; retrying in 60 s" + sleep 60 continue fi break diff --git a/CHANGELOG.md b/CHANGELOG.md index 3a3ecba8aa..7d2675e6f2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -44,8 +44,10 @@ aggregate instead: an italic *Catalog* line at the end of the version section an - **The IndexNow workflow no longer waits eight minutes behind an edge 403** — its key-file readiness loop treated every non-200 as "not deployed yet"; a GitHub runner that Cloudflare's bot management answers with 403 would have slept the full budget on - every run. A 403 now counts as deployed (Bing's own fetch is not subject to that - block), and the final warning names the last status seen. The submission body goes + every run. A 403 now ends the wait without claiming anything about the deploy (the + edge answers before the origin is asked, so it is inconclusive either way; IndexNow's + own key fetch is the authoritative check), and the final warning names the last + status seen. The submission body goes through a file (a full-sitemap batch is ~260 KB, above the 128 KB command-line argument cap that made the first sitemap-scope run report HTTP 000), and a first-use `403 SiteVerificationNotCompleted` from IndexNow is retried for up to ten minutes diff --git a/docs/reference/seo.md b/docs/reference/seo.md index 1803c900ac..63427d5d16 100644 --- a/docs/reference/seo.md +++ b/docs/reference/seo.md @@ -524,9 +524,10 @@ The engines answer `200` or `202` for an accepted batch; `4xx` means a key or payload problem and fails the run so it is visible. One `4xx` is transient by nature: after a rollout or a key rotation IndexNow verifies the key file asynchronously and answers `403 SiteVerificationNotCompleted` for a while, so -the workflow retries that case once a minute for up to ten minutes before it -fails — a `changed` run must not report success without its URLs accepted, -because later pushes submit only their own diffs. Bing Webmaster Tools shows +the workflow retries that case once a minute until a ten-minute deadline has +passed (elapsed time, so slow responses cannot stretch it) and then fails — a +`changed` run must not report success without its URLs accepted, because later +pushes submit only their own diffs. Bing Webmaster Tools shows the received submissions under *IndexNow*. ## Discoverability for assistants