Clarify crawler protection, cache testing, and domain verification

content/(serve)/access.mdx

@@ -75,6 +75,16 @@ sf share grant --to public --role viewer --path '/**' --target all-versions --sp
 
 Use `version:<id>` to open only one version. Branch previews need a `branch:<name>` grant; `all-versions` does not cover them or the live site. A team grant admits the owning team; it cannot name another team you belong to.
 
+## Crawler protection
+
+Crawler identification is not the same as Space access. Requests identifying as Googlebot, Bingbot, GPTBot, or ClaudeBot from IPs that do not belong to those crawlers have been observed receiving a **403** page saying **“Checking search engine crawler”**. Sending a crawler `User-Agent` with `curl` does not establish how a genuine crawler is treated.
+
+Whether genuine GPTBot, ClaudeBot, or PerplexityBot requests pass is not established. There is no documented guarantee that AI crawlers are allowed. Private and preview hosts also have access and indexing restrictions independent of this challenge.
+
+When reporting a challenge, include the URL, time, status, response body, user agent, and source IP. An absent edge-log row is not evidence that a request was allowed.
+
+WP Cloud's `ssl-social-crawler-redirect` endpoint is unrelated: its documented purpose is HTTPS-to-HTTP redirects for Facebook, Twitter, and LinkedIn to retain historic share counts, not AI-crawler access.
+
 ## Share links
 
 A new team-owned Space does not create a share link in the current CLI or dashboard. The owning team signs in at the live URL. Create a link when you want to share outside the team.

content/(serve)/caching.mdx

@@ -62,6 +62,22 @@ Publish warns with `header_cache_control_platform_managed`, because the header y
 
 The non-GET rule is absolute. The shared cache in front of a Space keys on host, path and query without reading the method, so a cacheable answer to a `POST` could be replayed to a `GET`.
 
+## Testing before promotion
+
+Previews are private by default. An authenticated preview response uses `private, no-store`, so it cannot demonstrate the cache behavior of a public live response.
+
+Before promotion, inspect the exact build output you intend to publish:
+
+```bash
+sf routing inspect --routing ./dist --url https://example.com/path
+```
+
+Replace `./dist` with that build's publish root. Repeat `--url` to match more URLs. This shows compiled custom header rules, not the full runtime response or managed edge cache policy. See [`_headers` syntax](/routing#_headers).
+
+`sf routing compute --space my-site` resolves routing with current variables for the ready pending version if there is one, otherwise the live version. It does not accept `--url` or select an arbitrary version. An authenticated request to a preview can check its actual response headers, but its private cache policy remains intentional.
+
+To observe live edge caching, run `sf promote <version>` and request the public live URL without private access credentials. Use `sf rollback <version>` if you need to return to an earlier version.
+
 ## Vary is not part of the cache key
 
 The edge keys a stored response on **host, path, and query**. It does not read `Vary`.

content/(serve)/domains.mdx

@@ -99,11 +99,11 @@ The apex is decided by the public suffix list, not by counting labels. `example.
 
 ## Verification states
 
-`verification.status` on the domain record answers one question: is DNS control proved right now?
+`verification.status` depends on the response projection. **Legacy inventory records** report DNS ownership proof (the table below); **native inventory records** report `not_required`. The default `/v1/domains` list and detail compatibility responses can overlay an attached hostname's HTTPS readiness onto the inventory identity, so a `verified` status is not universally a DNS-proof verdict.
 
 | Status | What it means | What to do |
 | --- | --- | --- |
-| `not_required` | The domain is in the team inventory but not assigned to a Space. | Assign it to a Space. |
+| `not_required` | A native inventory record, or a legacy record with no ownership verification required. | Inspect the Space attachment for HTTPS readiness. |
 | `pending` | Assigned, waiting for the records to show up in public DNS. | Add the records, then wait or run `sf domains check`. |
 | `verified` | A current positive signal proves control. | Nothing. |
 | `failed` | No current proof. Either the records were never observed, or the domain verified in the past and has stopped proving. | Run `sf domains diagnostics` to see which records are observed. |
@@ -112,6 +112,10 @@ The apex is decided by the public suffix list, not by counting labels. `example.
 
 **Verification failure never takes a live site down.** Compiled routes and certificates stay in place while the domain re-proves itself. Only sustained failure past the notice window blocks the attachment.
 
+**Native Space-attachment responses** (`/v1/spaces/{spaceId}/domains` and its domain detail route) use `verification.status` for provider HTTPS readiness instead. `pending` means DNS, provider attachment, or certificate provisioning is still settling; `verified` means the configured hostname is active; `failed` reports an attachment error. Legacy attachments can still report DNS ownership proof in this field, so check certificate diagnostics before treating them as HTTPS-ready. The Space's `liveUrl` changes to the selected primary custom hostname once its attachment becomes active.
+
+For separate progress steps, use hostname diagnostics: `dns.txtVerified` reports observed TXT ownership proof, `dns.pointsToProvider` reports routing, and `ssl.status` reports certificate state. A `null` TXT observation is inconclusive: diagnostics may not include a prefixed ownership record. Follow the domain's verification instructions rather than assuming no TXT proof is required. A pending certificate does not undo completed ownership proof.
+
 ## Certificates
 
 A certificate is issued once DNS points at Spacefast. The state is reported per hostname.

content/(serve)/routing.mdx

@@ -89,6 +89,14 @@ Header names must match `^[A-Za-z0-9-]+$`. Some names are owned by the platform
 
 `_headers` matchers cannot include a port. They match the raw pathname, so they are sensitive to a trailing slash where `_redirects` is not. Header rules also apply to the URL the visitor asked for, not to the path a rewrite landed on.
 
+Check a particular URL against the build's local rules before promotion:
+
+```bash
+sf routing inspect --routing ./dist --url https://example.com/downloads/file.zip
+```
+
+Use the exact publish root in place of `./dist`; repeat `--url` for more URLs. The result shows matching custom header rules, not the complete runtime headers or edge cache policy. Private previews use `private, no-store`; [test live edge caching after promotion](/caching#testing-before-promotion).
+
 ## Rules in `sf.jsonc`
 
 ```jsonc sf.jsonc

styles/Spacefast/spelling-exceptions.txt

@@ -16,6 +16,7 @@ Automattic
 Automattician
 Automatticians
 backoff
+Bingbot
 Blume
 Blume's
 boolean
@@ -72,6 +73,8 @@ frontmatter
 fsynced
 github
 globals
+Googlebot
+GPTBot
 Gravatar
 grayscale
 Gridsome