CI staging build succeeds even if some tests fails there

[tarekgh]

Here is example of that https://dev.azure.com/dnceng/public/_build/results?buildId=1905509&view=logs&j=b1ad8889-01f1-5331-c3a1-dcfbcdd4499f&t=affe8412-3853-58c1-7034-679fc1395f80.

This is causing a problem as after merging the changes, we get a new issues failed because of the ignored failed tests in the staging build.

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Here is example of that https://dev.azure.com/dnceng/public/_build/results?buildId=1905509&view=logs&j=b1ad8889-01f1-5331-c3a1-dcfbcdd4499f&t=affe8412-3853-58c1-7034-679fc1395f80.

This is causing a problem as after merging the changes, we get a new issues failed because of the ignored failed tests in the staging build.

Author:	tarekgh
Assignees:	-
Labels:	area-Infrastructure, untriaged
Milestone:	-

[tarekgh]

#72995 (comment)

[jkoritzinsky]

The runtime-staging pipeline is supposed to not be red when tests fail. The staging pipeline is where we put tests for legs that aren't yet reliable enough to be in the runtime or runtime-extra-platforms pipeline.

[steveisok]

@directhex Should we promote linix-bionic to the runtime pipeline or runtime-extra-platforms?

[tarekgh]

The runtime-staging pipeline is supposed to not be red when tests fail. The staging pipeline is where we put tests for legs that aren't yet reliable enough to be in the runtime or runtime-extra-platforms pipeline.

This was not the case in my instant. I considered the test to be stable and still failed in the staging build. It caused more work for me to get a new issue and handle it while would be easier if I did that in the original PR. How do we decide which test is stable and which is not? The concerned failed test for me was disabled for some time and I started to enable it after fixing it.

@directhex Jo Shields FTE Should we promote linix-bionic to the runtime pipeline or runtime-extra-platforms?

I think this will make sense.

[radical]

Keep in mind that runtime-extra-platforms doesn't run on PRs by default.
It would be very useful if we can set continueOnError for the runtime-staging jobs also.

[steveisok]

How do we decide which test is stable and which is not?

The way mobile in particular is set up, resolving failing tests may have to be done post merge. runtime-staging is meant to run per PR, but not fail the build, which means you have to look to see if any tests failed in azdo.

[tarekgh]

At least, I hope the CI gives any indication about failures in the staging build even when leg marked as succeeding. I don't think anyone is manually checking the staging build when seeing it green.

[ilonatommy]

@tarekgh, it's printing a text message next to runtime-staging job in the job list. Either: Build #XX.X had test failures or Build #XX.X succeeded. But yes, you need to scroll for it manually. I can see we are sorting jobs by failed - skipped - passed and then alphabetically in the jobs view. If the runtime-staging was higher, with the failed or skipped ones when the failure message is displayed, then probably people who are not aware of the need of checking it would see the message anyway.