fix(crawler): stop the crawl looping on null repository nodes

[DerekRoberts]

The scheduled crawl hangs: it never gets past the first page of bcgov repositories.

Cause

• What happens: GitHub's GraphQL API returns node: null (with an error entry) for a repository it can't return. reposFromEdges read edge.node.name on that null node and threw. The page loop's catch only logged the error and didn't advance the cursor, so it asked for the same page again, about every 5 seconds, forever.
• Manual run 37978104944: logged 735 TypeError: Cannot read properties of null (reading 'name') and one GraphQL 502 over 61 minutes, then was cancelled. It never reached bulk-load.
• Scheduled run 37864259112: looped 25 minutes in the older code, then GraphQL stopped returning edges. It ended green with "No yaml files found", so nothing was crawled.
• What it isn't: the bulk-load change in fix(api): answer bulk load only after saving #662 isn't involved. The crawler never sent a bulk-load request.

Changes

• crawler/src/main.js
  • Skips null nodes and logs how many were skipped, with GitHub's error message.
  • The cursor always advances to the last edge of the page.
  • A GraphQL response with no repositories fails the run, with GitHub's errors, instead of ending quietly. The catch that retried the same page forever is removed.
  • If no readable repositories are found, the run fails. If GitHub returns every repository as null for the workflow token, the run now fails with that message instead of passing green.
  • GraphQL 5xx answers are retried with the existing backoff, like rate limits.
  • Every axios request times out after 30 seconds, matching the API route. Before, axios had no timeout at all.
• scheduled.yml: the crawl job gets timeout-minutes: 45, about twice the last full run, instead of the 6-hour default.
• Test: a new crawler test checks that null nodes are skipped. It fails on the old code.

---

Thanks for the PR!

Deployments, as required, will be available below:

• Frontend
• API

Please create PRs in draft mode. Mark as ready to enable:

• Analysis Workflow

After merge, new images are deployed in:

• Merge Workflow

[Copilot]

🟢 Approval recommended

The changes directly resolve the documented loop while adding appropriate operational safeguards and regression coverage.

0 open findings

What changed in this PR

Prevents scheduled crawler hangs caused by null GitHub repository nodes and improves failure handling.

Changes:

• Skips unreadable repository nodes and advances pagination safely.
• Adds request timeouts, 5xx retries, and explicit crawl failures.
• Limits scheduled crawler execution to 45 minutes.

File	Description
crawler/​src/​main.js	Hardens GraphQL crawling and request handling.
crawler/​test/​main.test.js	Tests null repository-node handling.
.github/​workflows/​scheduled.yml	Adds a crawler job timeout.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.