Improve CI scanner duplicate KBE detection#131312
Open
svick wants to merge 1 commit into
Open
Conversation
Use the Build Analysis GitHub check as a positive-only dedup signal and search complete test identifiers before shortened stems. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d555b634-18ce-458d-a831-096e0b0c87b3
|
Azure Pipelines: Successfully started running 1 pipeline(s). 15 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
|
Tagging subscribers to this area: @dotnet/runtime-infrastructure |
svick
marked this pull request as ready for review
July 24, 2026 12:06
|
Azure Pipelines: Successfully started running 1 pipeline(s). 15 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
There was a problem hiding this comment.
Pull request overview
Updates the CI Outer-Loop Failure Scanner guidance to reduce duplicate Known Build Error (KBE) filings by improving how it searches for existing KBEs and by leveraging Build Analysis results exposed via a GitHub check instead of an AzDO attachment.
Changes:
- Add guidance to search full test method identifiers verbatim (especially underscore-delimited names) before deriving shorter stems.
- Update the scanner’s “Build Analysis” lookup instructions to use the GitHub check-runs API (instead of a referenced-but-nonexistent AzDO attachment).
- Refresh the generated workflow lock metadata hash to reflect the updated source workflow text.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
.github/workflows/shared/create-kbe.instructions.md |
Adds guidance to avoid truncating underscore-delimited test identifiers during existing-KBE searches. |
.github/workflows/ci-failure-scan.md |
Replaces Build Analysis “attachment” guidance with a best-effort lookup via the Build Analysis GitHub check. |
.github/workflows/ci-failure-scan.lock.yml |
Updates gh-aw metadata (body_hash) to match the edited workflow source. |
Comment on lines
+225
to
+229
| `Build Analysis` check's `output.text`. If the report links the source build's | ||
| failure to an existing issue, record `existing-kbe #<n>`. Reports omit some | ||
| known errors when they exceed GitHub's output limits, so absence is not proof | ||
| that Build Analysis did not match; always continue with the exact KBE searches | ||
| in Step 4.2 after a miss. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The CI failure scanner filed #130940 even though #129691 already covered the same failure. The run searched a shortened underscore-delimited method name, which GitHub did not match as a prefix of the complete test identifier.
The scanner instructions also claimed that Build Analysis results were available through an AzDO attachment named
Build_Analysis_KnownIssues_v1. Investigation found no producer or observed instance of that attachment. Build Analysis did recognize the failure, but that information was exposed through its GitHub check instead.Changes
Build AnalysisGitHub check.Validation
A targeted dry-mode eval replayed build 1505119. The rendered Build Analysis check omitted #129691, then the verbatim full-method search found it and produced
existing-kbe #129691with no issue creation. All 8 targeted graders passed.cc: @kotlarmilos
Note
This pull request description was generated with GitHub Copilot.