Vscode e2e test broken again/still in release PR

[timotheeguerin]

https://dev.azure.com/azure-sdk/internal/_build/results?buildId=5748374&view=logs&j=6816d337-3d20-54b9-74bb-f1c7c74d29be&t=cf1798b5-868e-5cd2-0527-b94cd43a83f9&s=ff05ad62-bb9a-53b6-ce9f-72f329a63e7c

disabled them here #9353

[RodgeFu]

@v-xuto , could you help to have a look? thanks.

[zedy-wj]

@timotheeguerin - We enabled the e2e test, but we were unable to reproduce this error. (Tried five times) Could you double check this error now?

PR link: #9405. (Just enable e2e test)
Pipeline link: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=5771924&view=results.

If there is any issue with our reproduction steps, please let me know and I will investigate further. Thank you.

[timotheeguerin]

The problem only happened in the release build. This means the versions locally are not published yet. This is a pattern that cause issue with other e2e test which needed special handling

I retried the job to make sure it wasn't just a fluke.

[zedy-wj]

The problem only happened in the release build. This means the versions locally are not published yet. This is a pattern that cause issue with other e2e test which needed special handling

I retried the job to make sure it wasn't just a fluke.

Thank you for clarifying this again - the issue only occurs in the release build. If I want to simulate the release process locally or in a pipeline, how should I do that? Could you provide a detailed set of steps to reproduce it? I’m not entirely sure what the exact differences are between my successful (non-release) pipeline and the failing release pipeline you mentioned, or where the release-specific behavior is coming from. Thanks in advance!

[timotheeguerin]

hhm yeah I can't seem to reproduce anymore, the difference I would have thought was all the version have been bumped but not released yet which was the cuase of issues for e2e tests in the past. But right now wiht that it seems to pass #9427
The error in the CI wasn't very helpful so will update the test to report better so if it happens in the next release we can get more info

on another note I do get failure eitherway on OSX locally

[zedy-wj]

Thank you for the confirmation. Regarding the other issue you mentioned on macOS, based on the error message, my initial assumption is that this might be an expected behavior. Some of our UI automation test cases rely on UI-based element or button selectors, and since macOS has UI differences compared to Windows and is currently out of our test scope, certain methods have not been adapted. We will check whether the issue shown in your screenshot is related to this as well. Thanks again for your feedback.

[timotheeguerin]

well the repo is cross platform and having some of the build check not working everywhere is a problem. Everything should be designed to not fail. If that means skipping the test on certain platform because it doesn't work then its also ok

[zedy-wj]

The issue is no longer repro, closed it. If there are any further problems, please feel free to reopen it. Thanks all.