Skip to content

Fix four failing workflows: traffic, message center, catalog, PR sweeper - #502

Merged
soyalejolopez merged 4 commits into
microsoft:masterfrom
soyalejolopez:soyalejolopez-fictional-umbrella
Jul 27, 2026
Merged

Fix four failing workflows: traffic, message center, catalog, PR sweeper#502
soyalejolopez merged 4 commits into
microsoft:masterfrom
soyalejolopez:soyalejolopez-fictional-umbrella

Conversation

@soyalejolopez

Copy link
Copy Markdown
Member

Four workflows have been failing independently. Each had a distinct root cause.

Collect Traffic Data — 100% failure daily since ~7/6

The token is expired, but the run gave zero diagnostic to say so. Two bugs hid it:
gh api stderr went to /dev/null, and ::error:: was echoed to stdout from
inside a $(...) substitution, so the error text got captured into a variable
instead of reaching the log. Both fixed, plus an upfront empty-token guard.

Requires a rotated TRAFFIC_TOKEN to actually go green. The traffic API needs
push access on this repo; GITHUB_TOKEN cannot read traffic at all.

Update Message Center — weekly failure since 7/6

The branch push succeeds every time; only gh pr create fails, with
"GitHub Actions is not permitted to create or approve pull requests". That's an
org policy, not a bug in the run, and it shouldn't throw away a branch that
already landed. Now warns, writes a job summary with a compare URL, and exits 0.
Any other create failure still fails hard.

Also seeds the remote-tracking ref so --force-with-lease stops dying with
"stale info" on same-day reruns.

Build catalog — 5 of 10 recent runs

GH006: Protected branch update failed for refs/heads/master. The publish job
pushed generated files straight to master. Runs that appeared to pass only did so
through the "no changes" early exit, so the catalog has silently not been landing.

Now escalates: direct push, then rebase and retry, then a catalog-refresh/*
branch with a PR and auto-merge. No retrigger loop — all four generated files sit
outside this workflow's paths: filters.

PR Sweeper Report — 3 failures on 7/22

Fork PRs leave workflow_run.pull_requests[0].number empty, so the job falls back
to a SHA lookup that ran exactly once. PR #495 was created at 16:49:09Z and queried
at 16:49:37Z — 28 seconds, before GitHub's commit-to-PR index had propagated. The
same SHA resolves fine now.

Retries at 0/3/8/15s. If it still can't resolve it warns, still posts the commit
status
, and returns instead of exiting 1. No security change: setStatus only
needs the trusted head SHA and the deterministic guardrails remain the
authoritative merge gate. Only the sticky comment is lost, and a red X on a healthy
PR is worse than a missing comment.


Validated offline: YAML parse, bash -n on every run block, and stub harnesses
covering expired token, policy block, protected branch, failed auto-merge, and
same-day rerun.

Both PR-creating workflows now honor an optional AUTOMATION_PAT secret, which
bypasses the Actions PR restriction. They degrade gracefully without it, so this
is safe to merge before any secret or settings change lands.

AnkitaDudeja and others added 4 commits July 27, 2026 13:33
The Collect Traffic Data workflow has failed every day since 7/6 with no
usable diagnostic. Two bugs conspired to hide the cause:

- gh_api_retry sent gh's stderr to /dev/null, so the retry warnings said
  "attempt N/3 failed" and nothing else.
- fetch_required_json echoed its ::error:: to stdout while running inside
  a command substitution, so the error text was captured into the caller's
  variable instead of reaching the log.

Now the real gh stderr is preserved and included in the retry warning,
errors route to stderr and return instead of exiting from a subshell, and
an empty TRAFFIC_TOKEN fails immediately with a message naming the secret
and the administration:read scope it needs.

Also skips the pointless sleep after the final attempt.

The underlying failure is an expired TRAFFIC_TOKEN. This does not fix
that, but it makes the next failure legible in one glance.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 68fa07a3-7ce8-442f-a9e8-f6b1775d8a1f
Every weekly run since 7/20 has pushed its branch successfully and then
failed on gh pr create with:

  GitHub Actions is not permitted to create or approve pull requests

That is a repo setting, not a bug in the run, and it should not throw away
a branch that already landed. The step now recognises that specific error,
emits a warning plus a job summary with a compare URL, and exits 0. Any
other create failure still fails the job.

Two supporting fixes:

- --force-with-lease was failing with "stale info" because actions/checkout
  only fetches the default branch, so refs/remotes/origin/<branch> never
  existed on a same-day rerun. The ref is now seeded before the push.
- GH_TOKEN falls back to AUTOMATION_PAT when present, which sidesteps the
  policy entirely for repos that add one.

Also adds set -euo pipefail and drops an || true that was masking gh pr
list failures.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 68fa07a3-7ce8-442f-a9e8-f6b1775d8a1f
Half the recent Build catalog runs failed with:

  remote: error: GH006: Protected branch update failed for refs/heads/master
  - Changes must be made through a pull request.

The publish job pushes catalog.json, resource-stats.json,
resource-discussions.json and traffic-data/clarity-views.json directly to
master. Branch protection now forbids that, so the rebuild has been
silently not landing. The runs that appeared to pass only did so through
the "no changes" early exit.

The step now escalates: direct push, then pull --rebase and retry for a
plain race, then fall back to a catalog-refresh/<date>-<sha> branch with a
PR and auto-merge. If the repo also blocks Actions from opening PRs, it
warns with a compare URL and exits 0 rather than failing.

Errors are no longer swallowed. gh pr list dropped its || true, an
unconfirmed PR after a successful branch push is a hard failure, and a
failed auto-merge is a loud warning with the PR URL.

The PR number is parsed from gh pr create's output URL rather than by
re-listing, because listing immediately after creation races GitHub's PR
index and can report a phantom failure.

No retrigger loop: all four generated files sit outside this workflow's
paths filters, so merging the catalog PR will not start another run.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 68fa07a3-7ce8-442f-a9e8-f6b1775d8a1f
Three runs on 7/22 aborted with "Could not resolve a trusted PR number
from the workflow_run event". Fork PRs leave
workflow_run.pull_requests[0].number empty, so the job falls back to
looking the PR up by head SHA, and that lookup ran exactly once.

PR microsoft#495 was created at 16:49:09Z and the report job queried at 16:49:37Z.
Twenty-eight seconds. GitHub's commit-to-PR association index had not
propagated yet; the same SHA resolves fine now. An earlier run the same
afternoon hit the same empty value and resolved fine, which is what a
propagation race looks like.

The lookup now retries with backoff at 0, 3, 8 and 15 seconds and logs why
each attempt came back empty.

If it still cannot resolve, the job no longer exits 1. It warns, still
posts the commit status, and returns. This does not weaken the gate:
setStatus only needs the trusted head SHA, and the deterministic
guardrails remain the authoritative merge check. Only the sticky comment
is lost, and a red X on a healthy PR is worse than a missing comment.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 68fa07a3-7ce8-442f-a9e8-f6b1775d8a1f
@soyalejolopez
soyalejolopez merged commit b84b67a into microsoft:master Jul 27, 2026
3 checks passed
@soyalejolopez
soyalejolopez deleted the soyalejolopez-fictional-umbrella branch July 27, 2026 19:00
@github-actions

Copy link
Copy Markdown
Contributor

🛰️ PR Sweeper report

Risk: 🟠 MEDIUM · Security gate: ✅ passing · Files: 4

🔒 Automated guardrails (authoritative)

Category Location Finding
⚠️ scope Touches security-sensitive path(s): .github/scripts/pr-sweeper/report.mjs, .github/workflows/build-catalog.yml, .github/workflows/traffic-stats.yml, .github/workflows/update-message-center.yml. Maintainer + security review required.

🤖 Dual-model AI review (advisory)

AI review unavailable for this run (models not reachable or no diff). Guardrails above are unaffected.

  • GPT-class reviewer (openai/gpt-5): skipped — GitHub Models HTTP 413: {"error":{"code":"tokens_limit_reached","message":"Request body too large for gpt-5 model. Max size: 4000 tokens.","details":"Request body too large for gpt-5 model. Max size: 4000 tokens."}}
  • Deep-reasoning reviewer (openai/o3): skipped — GitHub Models HTTP 413: {"error":{"code":"tokens_limit_reached","message":"Request body too large for o3 model. Max size: 4000 tokens.","details":"Request body too large for o3 model. Max size: 4000 tokens."}}

The automated guardrails are authoritative and gate the security status. The AI review is advisory and never auto-merges. Thanks for contributing to FastTrack! 🛩️

@github-actions github-actions Bot added sweeper:risk-medium PR Sweeper: medium risk sweeper:scope-review PR Sweeper: scope/sensitive-path review labels Jul 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

sweeper:risk-medium PR Sweeper: medium risk sweeper:scope-review PR Sweeper: scope/sensitive-path review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants