feat(webapp): serve robots.txt disallowing /api/ and /n5Data/

[jcschaff]

First half of the crawler defence for #2021. The ingress-side rule for crawlers that ignore
robots.txt is virtualcell/vcell-fluxcd#52.

Why

A crawler walking /api/v0/biomodel/{id} → /api/v0/biomodel/{id}/simulation/{id} links exhausted
the heap of two prod api pods on 2026-08-22. Those paths are machine interfaces returning JSON and
VCML — useless as search results, and not cheap: one request can parse a multi-megapixel geometry.

Disallow, not Crawl-delay

Crawl-delay is not part of the robots.txt standard and Google ignores it outright, so it would
read as a control while being none. And a slower crawl still pays the full parse cost per request —
the non-fatal repeat of the incident was a single request taking 4 seconds on its own.

User-agent: *
Disallow: /api/
Disallow: /n5Data/

/n5Data/ is the S3 proxy serving data blobs. The web application at / stays crawlable.

This is a request, not a control — it binds only compliant crawlers. PetalBot, the one observed,
advertises a webmaster control page in its user agent, so it is the compliant kind. That's why it
pairs with the ingress rule rather than replacing it.

One detail worth knowing

It's listed in angular.json assets rather than dropped in src/assets — the latter would
publish it at /assets/robots.txt, which no crawler reads. Both build configurations needed the
entry.

And there's a pre-existing wrinkle this fixes: nginx-custom.conf serves location / with
try_files $uri $uri/ /index.html, so without this file a request for /robots.txt returned
index.html with a 200 — crawlers were being handed the SPA and parsing it as robots.txt.

Verified rather than assumed

Ran a production build: dist/login-demo/robots.txt is emitted at the dist root, which
Dockerfile-webapp copies to /usr/share/nginx/html and nginx serves from location / via
try_files $uri.

Refs #2021, #2025

🤖 Generated with Claude Code

https://claude.ai/code/session_018kr8SbzXtwW3gMVUgMfDDt