Add a CI check to keep the Python APIView parser version synchronized with APIView

[tjprescott]

Problem

The apiview-stub-generator version used to generate Python APIView token files can differ from the parser version configured for Python in APIView. This produces token files whose hashes do not match the approved APIView revisions and makes release failures difficult to diagnose.

We recently observed the following:

1. APIView was configured to use apiview-stub-generator 0.3.28 for Python.
2. azure-sdk-for-python/eng/apiview_reqs.txt pinned apiview-stub-generator 0.3.31.
3. The Python release pipeline ran the apistub check through the in-repository azpysdk tooling. That tooling installed eng/apiview_reqs.txt and generated a token file using 0.3.31.
4. The generated token file did not hash to the approved APIView revision, which had been produced with 0.3.28.
5. When APIView processed the token file, it detected that its parser version differed and initiated regeneration using its configured 0.3.28 version.
6. Several minutes later, APIView replaced the token file with one whose hash matched the approved revision.

The delayed regeneration obscured the parser mismatch and contributed to an initial misdiagnosis. During the delay, the release appeared to be using an unapproved API surface even though the underlying API had already been approved.

Current Python CI path

The SDK pipeline runs eng/scripts/dispatch_checks.py with --checks="apistub". The apistub implementation in eng/tools/azure-sdk-tools/azpysdk/apistub.py installs the dependencies pinned in eng/apiview_reqs.txt, including apiview-stub-generator, before generating the Python token file.

The generated token file is subsequently submitted to apiview.dev/autoreview by eng/common/scripts/Create-APIReview.ps1.

Proposed change

Add a required CI check that runs whenever eng/apiview_reqs.txt changes. The check should:

1. Read the pinned apiview-stub-generator version from eng/apiview_reqs.txt.
2. Read ApiStubVersion from eng/pipelines/apiview-review-gen-python.yml in Azure/azure-sdk-tools. This is the version installed by the Python APIView token regeneration pipeline.
3. Read PythonLanguageService.VersionString from src/dotnet/APIView/APIViewWeb/Languages/PythonLanguageService.cs in Azure/azure-sdk-tools. This is the parser version APIView expects for Python reviews.
4. Fail with a clear error unless all three versions match.
5. Report all three versions and explain that APIView must be updated and deployed before the Python dependency pin can be merged.

This check should prevent azure-sdk-for-python from moving to a parser version that APIView is not yet using.

Required update sequence

Parser upgrades should occur in this order:

1. Release the new apiview-stub-generator version from azure-sdk-tools.
2. Update the Python parser version in APIView and deploy APIView.
3. Update eng/apiview_reqs.txt in azure-sdk-for-python.

APIView must be updated first because it can automatically regenerate token files using its configured parser version. Updating the Python repository first creates a window where locally generated tokens and APIView-generated tokens differ.

Acceptance criteria

• A change to apiview_reqs.txt in the azure-sdk-for-python repository triggers the synchronization check.
• CI reads ApiStubVersion from Azure/azure-sdk-tools/eng/pipelines/apiview-review-gen-python.yml.
• CI reads PythonLanguageService.VersionString from Azure/azure-sdk-tools/src/dotnet/APIView/APIViewWeb/Languages/PythonLanguageService.cs.
• CI passes only when both azure-sdk-tools versions exactly match the version pinned in eng/apiview_reqs.txt.
• A mismatch fails before package release steps and displays all three versions and their source locations.
• The failure message documents the required update order.
• The check itself does not initiate or wait for an APIView parser upgrade.

Version sources

The synchronization check compares these values:

Repository	File	Value
Azure/azure-sdk-for-python	eng/apiview_reqs.txt	apiview-stub-generator==<version>
Azure/azure-sdk-tools	eng/pipelines/apiview-review-gen-python.yml	ApiStubVersion: '<version>'
Azure/azure-sdk-tools	src/dotnet/APIView/APIViewWeb/Languages/PythonLanguageService.cs	VersionString { get; } = "<version>"

The check should read the azure-sdk-tools files from its default branch. Passing the source check confirms that the coordinated APIView change has merged; the documented update sequence must still require APIView deployment before the azure-sdk-for-python update is merged.

[danieljurek]

Should PythonLangaugeService.cs always be kept in sync or would you accumulate changes for it leading up to the next version release?

If coordinating across azure-sdk-tools and azure-sdk-for-python the eng/common sync is a mechanical (albeit clunky) way to ensure things stay synchronized better.

If you're already checking the version used, why not just read the version used from the tools repo (or conversely, have the tools repo read the version that's expected to be used in the Python repo)?