feat(seo): IndexNow submissions for changed pages

[MarkusNeusinger]

Summary

• Key file app/public/anyplot-indexnow-ab738f04ea92446a.txt proves control of anyplot.ai to the IndexNow engines. It is public by design. app/nginx.conf gets an explicit location = for it (like robots.txt / llms.txt), because Bing verifies with a crawler UA that the bot map would otherwise proxy to /seo-proxy/<key>.txt and a 404.
• .github/workflows/indexnow-submit.yml runs on every push to main that touches plots/: the diff (HEAD~1..HEAD, one squash merge) is mapped to /{spec} and /{spec}/{language}/{library} URLs and POSTed to https://api.indexnow.org/indexnow in batches of 10,000 (the protocol limit). A deleted implementation is submitted too — IndexNow means "this URL changed". 200/202 is success; 4xx fails the run so a key or payload problem is visible.
• workflow_dispatch with scope=sitemap submits every URL of the live sitemap (4.6k today) — I will run it once after the app deploy has served the key file.
• Docs: new "IndexNow" section in docs/reference/seo.md, a row in docs/workflows/overview.md, CHANGELOG under Added.

Plan

N/A

Test plan

• Workflow: YAML parses, every run step passes bash -n, the diff→URL awk mapping was dry-run against a synthetic diff (spec page, metadata and implementation paths, an unrelated file).
• After the app deploy: curl https://anyplot.ai/anyplot-indexnow-ab738f04ea92446a.txt returns the key both with a browser UA and with -A bingbot.
• gh workflow run indexnow-submit.yml -f scope=sitemap answers 200/202; Bing Webmaster Tools lists the submission under IndexNow within a day.
• The next pipeline merge triggers a changed run with exactly the touched URLs.

🤖 Generated with Claude Code

https://claude.ai/code/session_01SrKzcwZBnref1sWYtdXynu

[Copilot]

🟢 Approval recommended

The changes are cohesive and low-risk, with only a minor misleading comment in the new workflow noted for cleanup.

Pull request overview

Adds IndexNow support so search engines that participate in IndexNow can be notified immediately when plot/spec pages change, improving crawl freshness for the plots/-driven site surface.

Changes:

• Add an IndexNow submission GitHub Actions workflow that submits either changed URLs (on main pushes touching plots/) or the full sitemap (manual dispatch).
• Serve the IndexNow key file directly from nginx (including for crawler user agents) and add documentation describing setup and operation.
• Document the new workflow and add a CHANGELOG entry under Added.

File summaries

File	Description
docs/workflows/overview.md	Documents the new indexnow-submit.yml workflow in the workflows overview table.
docs/reference/seo.md	Adds an “IndexNow” section describing the key file, workflow behavior, and manual sitemap submission.
CHANGELOG.md	Adds an Added entry describing the new IndexNow integration and operational details.
app/public/anyplot-indexnow-ab738f04ea92446a.txt	Adds the public IndexNow key file required for ownership verification.
app/nginx.conf	Adds an exact-match location = so the key file is served directly (not routed through the bot proxy path).
.github/workflows/indexnow-submit.yml	New workflow to collect URLs (diff-based or sitemap) and POST batches to the IndexNow API.

Review details

• Files reviewed: 6/6 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[Copilot]

🟡 Changes recommended

The new IndexNow workflow has reliability/correctness issues (diff range handling, shallow checkout assumptions, and missing network hardening) that can cause missed submissions or flaky failures.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (3)

.github/workflows/indexnow-submit.yml:98

• The IndexNow POST uses curl without retry/time limits; intermittent network issues or temporary 5xx responses can make pushes to main report a failure even though a retry would succeed. Consider adding the same --retry and --max-time safeguards used elsewhere in the repo's monitoring workflows.

            # Bounded: a slow or flaky api.indexnow.org must not burn the job
            # timeout; three retries cover a transient error.
            code=$(curl -sS -o response.txt -w '%{http_code}' \
              --max-time 30 --retry 3 --retry-delay 5 --retry-all-errors \

.github/workflows/indexnow-submit.yml:66

• With scope=sitemap, add retry/timeout and fetch the sitemap from the direct app origin (set via APP_ORIGIN) to reduce flaky failures from transient network issues or edge blocking.

            curl -sfS "https://${HOST}/sitemap.xml" \
              | grep -oE '<loc>[^<]+</loc>' | sed -E 's#</?loc>##g' | sort -u > urls.txt

.github/workflows/indexnow-submit.yml:72

• Diffing only HEAD~1..HEAD means a multi-commit push to main will submit URLs for just the last commit, and the all-zero before SHA (first push / history rewrite) is not handled. Using github.event.before/github.sha (like sync-labels.yml does) avoids missing submissions.

            git diff --name-only HEAD~1 HEAD -- plots/ | awk -F/ '
              $1 != "plots" || NF < 3 { next }
              $3 ~ /^specification\./ { print "https://'"$HOST"'/" $2; next }
              ($3 == "metadata" || $3 == "implementations") && NF == 5 {
                lib = $5; sub(/\.[^.]+$/, "", lib)

• Files reviewed: 6/6 changed files
• Comments generated: 3
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

The workflow’s URL count output can be space-padded and trigger empty submissions, and the new changelog bullet is missing the required PR reference.

Review details

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

.github/workflows/indexnow-submit.yml:77

• wc -l < urls.txt produces a space-padded count (e.g. " 0"), which then gets written to GITHUB_OUTPUT and makes if: steps.urls.outputs.count != '0' evaluate true even when there are 0 URLs. That can lead to submitting an empty urlList batch and failing the run with a 4xx from IndexNow.

CHANGELOG.md:39

• This new changelog bullet is missing the required PR reference (e.g. (#12345)) at the end. The file header explicitly states every [Unreleased] entry must include its PR reference, and the surrounding bullets follow that pattern.

  `.github/workflows/indexnow-submit.yml` maps every push to `main` that touches `plots/`
  onto the affected `/{spec}` and `/{spec}/{language}/{library}` URLs and POSTs them to
  `api.indexnow.org` (10,000 per request; a deleted implementation is submitted too).
  `workflow_dispatch` with `scope=sitemap` submits the whole live sitemap for the initial
  load. Google does not take part and keeps reading the sitemap. The protocol is free.

• Files reviewed: 6/6 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

The changed-URL mapping currently misses implementation URLs when only specification.* changes, so IndexNow won’t be notified for affected /{spec}/{language}/{library} pages.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 6/6 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🟢 Approval recommended

The implementation and wiring look coherent and low-risk, with only minor documentation wording/style nits identified.

Review details

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

CHANGELOG.md:32

• The changelog entry is missing the Oxford comma in the IndexNow engine list ("Naver and Yep"), which is inconsistent with the repo's documentation style baseline.
  docs/reference/seo.md:508
• The sentence ending with "Three pieces, kept in sync…" is a fragment and the list is missing the Oxford comma; this reads awkwardly in docs and diverges from the repo's Google-style prose baseline.

• Files reviewed: 6/6 changed files
• Comments generated: 0 new
• Review effort level: Lite