Skip to content

Clarify crawler protection, cache testing, and domain verification - #94

Draft
suatsulun wants to merge 3 commits into
mainfrom
suat/crawler-cache-domain-docs
Draft

suatsulun wants to merge 3 commits into
mainfrom
suat/crawler-cache-domain-docs

Conversation

@suatsulun

@suatsulun suatsulun commented Oct 29, 2026 •

Copy link
Copy Markdown

What was wrong

Customer items 25 and 28 exposed gaps in crawler and cache-testing guidance. The routing guide did not explain inspecting local header rules, and domains documentation treated every verification projection as DNS ownership proof.

Change

  • Document the observed crawler challenge without promising genuine AI crawler access.
  • Explain private preview cache policy, local header inspection, and testing public edge caching after promotion.
  • Separate inventory ownership proof, attachment HTTPS readiness, and hostname diagnostic observations.

These content-only changes apply directly to main and do not depend on #88, #89, or #90.

Verification

  • node scripts/verify-public-safety.mjs: passed (1342 files, 3 symlinks).
  • node scripts/verify-command-examples.mjs: passed (567 Markdown files).
  • node scripts/verify-routes.mjs: requires uninstalled @spacefast/routing; full dependency-backed checks will run in CI.

Not in this PR

No generated references, routing implementation, or changes from #88–#90.


View with [code]smith Show before/after
Need help on this PR? Tag @codesmith-bot with what you need.

Review corrections

Legacy attachment verification can still reflect ownership proof, so the guide now directs readers to certificate diagnostics before treating it as HTTPS-ready. Null TXT diagnostics are inconclusive and do not override ownership instructions. Public-safety and command-example checks passed after this correction; local prose verification could not run because Vale is not installed. Final-head docs CI provides the full prose/build checks.

Distinguish observed crawler challenges from confirmed crawler access, explain preview cache limits, and document verification response projections. Public-safety and command-example checks pass; full checks require CI dependencies.
@coderabbitai

coderabbitai Bot commented Oct 29, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Add the genuine Googlebot, Bingbot, and GPTBot proper names used by the crawler protection guide. The first docs CI run identified these three dictionary omissions.

This branch was successfully deployed

1 active deployment
preview/suat/crawler-cache-domain-docs — 2c49ee26 Deployed Oct 29, 2026 by spacefast[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant