[bhushan4work]

what does this change?

adds sitemap-independent website crawling to @chatcops/ingest, allowing ingestion directly from a homepage. supports breadth-first same-origin crawling with configurable depth and page limits, url normalization, deduplication, robots.txt, include/exclude rules, concurrency limits, and fetch timeouts.

adds --crawl <url> and crawl:<url> source support to the cli. reuses fetched html during extraction to avoid duplicate requests. updates tests, api snapshots, documentation, and changesets.

closes #586

how i tested this

• pnpm test:coverage passed (85/80 thresholds met).
• pnpm lint passed.
• pnpm api passed (all 10 package api snapshots verified).
• pnpm docs:check passed (205 typescript samples checked).
• added ingest tests covering traversal, depth and page limits, url filtering, robots.txt, exclude rules, and duplicate-fetch prevention.
• added cli tests covering --crawl and crawl: source support.

ai assistance

none

checklist

• this pr targets dev
• i was assigned the linked issue
• pnpm -r build, pnpm test and pnpm -r typecheck pass locally
• tests added or updated where it makes sense
• for a bug fix: the new test fails without my change
• changeset added (pnpm changeset) if this changes core, widget or server
• pnpm api run and api/ committed if an exported type or option changed
• docs updated if behavior or options changed