Add ADO CI, SDL, and release pipelines with e2e test enablement

.Pipelines/CI-AND-RELEASE-PIPELINES.md

@@ -0,0 +1,126 @@
+# CI/CD Pipelines
+
+This document describes the pipeline structure for the `msal` Python package,
+including what each pipeline does, when it runs, and how to trigger a release.
+
+---
+
+## Pipeline Files
+
+| File | Purpose |
+|------|---------|
+| [`azure-pipelines.yml`](../azure-pipelines.yml) | PR gate and post-merge CI — calls the shared template with `runPublish: false` |
+| [`pipeline-publish.yml`](pipeline-publish.yml) | Release pipeline — manually queued, builds and publishes to PyPI |
+| [`template-pipeline-stages.yml`](template-pipeline-stages.yml) | Shared stages template — PreBuildCheck, Validate, and CI stages reused by both pipelines |
+| [`credscan-exclusion.json`](credscan-exclusion.json) | CredScan suppression file for known test fixtures |
+
+---
+
+## PR / CI Pipeline (`azure-pipelines.yml`)
+
+### Triggers
+
+| Event | Branches |
+|-------|----------|
+| Pull request opened / updated | all branches |
+| Push / merge | `dev`, `azure-pipelines` |
+| Scheduled | Daily at 11:45 PM Pacific, `dev` branch (only when there are new changes) |
+
+### Stages
+
+```
+PreBuildCheck ─► CI
+```
+
+| Stage | What it does |
+|-------|-------------|
+| **PreBuildCheck** | Runs SDL security scans: PoliCheck (policy/offensive content), CredScan (leaked credentials), and PostAnalysis (breaks the build on findings) |
+| **CI** | Runs the full test suite on Python 3.9, 3.10, 3.11, 3.12, 3.13, and 3.14 |
+
+The Validate stage is **skipped** on PR/CI runs (it only applies to release builds).
+
+> **SDL coverage:** The PreBuildCheck stage satisfies the OneBranch SDL requirement.
+> It runs on every PR, every merge to `dev`, and on the daily schedule — ensuring
+> continuous security scanning without a separate dedicated SDL pipeline.
+
+---
+
+## Release Pipeline (`pipeline-publish.yml`)
+
+### Triggers
+
+**Manual only** — no automatic branch or tag triggers. Must be queued explicitly
+with both parameters filled in.
+
+### Parameters
+
+| Parameter | Description | Example values |
+|-----------|-------------|----------------|
+| **Package version to publish** | Must exactly match `msal/sku.py __version__`. [PEP 440](https://peps.python.org/pep-0440/) format. | `1.36.0`, `1.36.0rc1`, `1.36.0b1` |
+| **Publish target** | Destination for this release. | `test.pypi.org (Preview / RC)` or `pypi.org (ESRP Production)` |
+
+### Stage Flow
+
+```
+PreBuildCheck ─► Validate ─► CI ─► Build ─┬─► PublishMSALPython  (publishTarget == 'test.pypi.org (Preview / RC)')
+                                           └─► PublishPyPI        (publishTarget == 'pypi.org (ESRP Production)')
+```
+
+| Stage | What it does | Condition |
+|-------|-------------|-----------|
+| **PreBuildCheck** | PoliCheck + CredScan scans | Always |
+| **Validate** | Asserts the `packageVersion` parameter matches `msal/sku.py __version__` | Always (release runs only) |
+| **CI** | Full test matrix (Python 3.9–3.14) | After Validate passes |
+| **Build** | Builds `sdist` and `wheel` via `python -m build`; publishes `python-dist` artifact | After CI passes |
+| **PublishMSALPython** | Uploads to test.pypi.org | `publishTarget == test.pypi.org (Preview / RC)` |
+| **PublishPyPI** | Uploads to PyPI via ESRP; requires manual approval | `publishTarget == pypi.org (ESRP Production)` |
+
+---
+
+## How to Publish a Release
+
+### Step 1 — Update the version
+
+Edit `msal/sku.py` and set `__version__` to the target version:
+
+```python
+__version__ = "1.36.0rc1"   # RC / preview
+__version__ = "1.36.0"      # production release
+```
+
+Push the change to the branch you intend to release from.
+
+### Step 2 — Queue the pipeline
+
+1. Go to the **MSAL.Python-Publish** pipeline in ADO.
+2. Click **Run pipeline**.
+3. Select the branch to release from.
+4. Enter the **Package version to publish** (must match `msal/sku.py` exactly).
+5. Select the **Publish target**:
+   - `test.pypi.org (Preview / RC)` — for release candidates and previews
+   - `pypi.org (ESRP Production)` — for final releases (requires approval gate)
+6. Click **Run**.
+
+### Step 3 — Approve (production releases only)
+
+The `pypi.org (ESRP Production)` path includes a required manual approval before
+the package is uploaded. An approver must review and approve in the ADO
+**Environments** panel before the `PublishPyPI` stage proceeds.
+
+### Step 4 — Verify
+
+- **test.pypi.org:** https://test.pypi.org/project/msal/
+- **PyPI:** https://pypi.org/project/msal/
+
+---
+
+## Version Format
+
+PyPI enforces [PEP 440](https://peps.python.org/pep-0440/). Versions with `-` (e.g. `1.36.0-Preview`) are rejected at upload time. Use standard suffixes:
+
+| Release type | Format |
+|-------------|--------|
+| Production | `1.36.0` |
+| Release candidate | `1.36.0rc1` |
+| Beta | `1.36.0b1` |
+| Alpha | `1.36.0a1` |

.Pipelines/credscan-exclusion.json

@@ -0,0 +1,13 @@
+{
+  "tool": "Credential Scanner",
+  "suppressions": [
+    {
+      "file": "tests/certificate-with-password.pfx",
+      "_justification": "Self-signed certificate used only in unit tests. Not a production credential."
+    },
+    {
+      "file": "tests/test_mi.py",
+      "_justification": "WWW-Authenticate challenge header value used as a mock HTTP response fixture in unit tests. Not a real credential."
+    }
+  ]
+}

.Pipelines/pipeline-publish.yml

@@ -0,0 +1,179 @@
+# pipeline-publish.yml
+#
+# Release pipeline for the msal Python package — manually triggered only.
+# Source: https://github.com/AzureAD/microsoft-authentication-library-for-python
+#
+# Publish targets:
+#   test.pypi.org (Preview / RC) — preview releases via MSAL-Test-Python-Upload SC
+#                                   (SC creation pending test.pypi.org API token)
+#   pypi.org (ESRP Production)   — production releases via ESRP (EsrpRelease@9) using MSAL-ESRP-AME SC
+#
+# For pipeline documentation, see .Pipelines/CI-AND-RELEASE-PIPELINES.md.
+
+parameters:
+- name: packageVersion
+  displayName: 'Package version to publish (must match msal/sku.py, e.g. 1.36.0 or 1.36.0rc1)'
+  type: string
+
+- name: publishTarget
+  displayName: 'Publish target'
+  type: string
+  values:
+  - 'test.pypi.org (Preview / RC)'
+  - 'pypi.org (ESRP Production)'
+
+trigger: none    # manual runs only — no automatic branch or tag triggers
+pr: none
+
+# Stage flow:
+#
+#   PreBuildCheck ─► Validate ─► CI ─► Build ─► PublishMSALPython  (publishTarget == Preview)
+#                                             └─► PublishPyPI       (publishTarget == ESRP Production)
+
+stages:
+
+# PreBuildCheck, Validate, and CI stages are defined in the shared template.
+- template: template-pipeline-stages.yml
+  parameters:
+    packageVersion: ${{ parameters.packageVersion }}
+    runPublish: true
+
+# ══════════════════════════════════════════════════════════════════════════════
+# Stage 3 · Build — build sdist + wheel
+# ══════════════════════════════════════════════════════════════════════════════
+- stage: Build
+  displayName: 'Build package'
+  dependsOn: CI
+  condition: eq(dependencies.CI.result, 'Succeeded')
+  jobs:
+  - job: BuildDist
+    displayName: 'Build sdist + wheel (Python 3.12)'
+    pool:
+      vmImage: ubuntu-latest
+    steps:
+    - task: UsePythonVersion@0
+      inputs:
+        versionSpec: '3.12'
+      displayName: 'Use Python 3.12'
+
+    - script: |
+        python -m pip install --upgrade pip build twine
+      displayName: 'Install build toolchain'
+
+    - script: |
+        python -m build
+      displayName: 'Build sdist and wheel'
+
+    - script: |
+        python -m twine check dist/*
+      displayName: 'Verify distribution (twine check)'
+
+    - task: PublishPipelineArtifact@1
+      displayName: 'Publish dist/ as pipeline artifact'
+      inputs:
+        targetPath: dist/
+        artifact: python-dist
+
+# ══════════════════════════════════════════════════════════════════════════════
+# Stage 4a · Publish to test.pypi.org (Preview / RC)
+#   Note: requires MSAL-Test-Python-Upload SC in ADO (pending test.pypi.org API token)
+# ══════════════════════════════════════════════════════════════════════════════
+- stage: PublishMSALPython
+  displayName: 'Publish to test.pypi.org (Preview)'
+  dependsOn: Build
+  condition: >
+    and(
+      eq(dependencies.Build.result, 'Succeeded'),
+      eq('${{ parameters.publishTarget }}', 'test.pypi.org (Preview / RC)')
+    )
+  jobs:
+  - deployment: DeployMSALPython
+    displayName: 'Upload to test.pypi.org'
+    pool:
+      vmImage: ubuntu-latest
+    environment: MSAL-Python
+    strategy:
+      runOnce:
+        deploy:
+          steps:
+          - task: DownloadPipelineArtifact@2
+            displayName: 'Download python-dist artifact'
+            inputs:
+              artifactName: python-dist
+              targetPath: $(Pipeline.Workspace)/python-dist
+
+          - task: UsePythonVersion@0
+            inputs:
+              versionSpec: '3.12'
+            displayName: 'Use Python 3.12'
+
+          - script: |
+              python -m pip install --upgrade pip twine
+            displayName: 'Install twine'
+
+          # TODO: create MSAL-Test-Python-Upload SC with test.pypi.org API token, then uncomment:
+          # - task: TwineAuthenticate@1
+          #   displayName: 'Authenticate with MSAL-Test-Python-Upload'
+          #   inputs:
+          #     pythonUploadServiceConnection: MSAL-Test-Python-Upload
+
+          # - script: |
+          #     python -m twine upload \
+          #       -r "MSAL-Test-Python-Upload" \
+          #       --config-file $(PYPIRC_PATH) \
+          #       --skip-existing \
+          #       $(Pipeline.Workspace)/python-dist/*
+          #   displayName: 'Upload to test.pypi.org'
+
+          - script: echo "Publish to test.pypi.org skipped — MSAL-Test-Python-Upload SC not yet created."
+            displayName: 'Skip upload (SC pending)'
+
+# ══════════════════════════════════════════════════════════════════════════════
+# Stage 4b · Publish to PyPI (ESRP Production)
+#   Uses EsrpRelease@9 via the MSAL-ESRP-AME service connection.
+#   IMPORTANT: configure a required manual approval on this environment in
+#   ADO → Pipelines → Environments → MSAL-Python-Release → Approvals and checks.
+#   IMPORTANT: EsrpRelease@9 requires a Windows agent.
+# ══════════════════════════════════════════════════════════════════════════════
+- stage: PublishPyPI
+  displayName: 'Publish to PyPI (ESRP Production)'
+  dependsOn: Build
+  condition: >
+    and(
+      eq(dependencies.Build.result, 'Succeeded'),
+      eq('${{ parameters.publishTarget }}', 'pypi.org (ESRP Production)')
+    )
+  jobs:
+  - deployment: DeployPyPI
+    displayName: 'Upload to PyPI via ESRP'
+    pool:
+      vmImage: windows-latest
+    environment: MSAL-Python-Release
+    strategy:
+      runOnce:
+        deploy:
+          steps:
+          - task: DownloadPipelineArtifact@2
+            displayName: 'Download python-dist artifact'
+            inputs:
+              artifactName: python-dist
+              targetPath: $(Pipeline.Workspace)/python-dist
+
+          - task: EsrpRelease@9
+            displayName: 'Publish to PyPI via ESRP'
+            inputs:
+              connectedservicename: 'MSAL-ESRP-AME'
+              usemanagedidentity: true
+              keyvaultname: 'MSALVault'
+              signcertname: 'MSAL-ESRP-Release-Signing'
+              clientid: '8650ce2b-38d4-466a-9144-bc5c19c88112'
+              intent: 'PackageDistribution'
+              contenttype: 'PyPi'
+              contentsource: 'Folder'
+              folderlocation: '$(Pipeline.Workspace)/python-dist'
+              waitforreleasecompletion: true
+              owners: 'ryauld@microsoft.com,avdunn@microsoft.com'
+              approvers: 'avdunn@microsoft.com,bogavril@microsoft.com'
+              serviceendpointurl: 'https://api.esrp.microsoft.com'
+              mainpublisher: 'ESRPRELPACMAN'
+              domaintenantid: '33e01921-4d64-4f8c-a055-5bdaffd5e33d'