feat(search): migrate Azure AI Search to the 2026-04-01 API and profile-based vector schema

[Rana Singh (ranadeepsingh)]

Related work

• Follow-up to feat: add backward-compatible AAD auth to Azure Search #2591, which added AAD authentication for Azure AI Search.

Problem

SynapseML defaulted to the deprecated 2023-07-01-Preview data-plane API. Moving to the stable 2026-04-01 API also requires adopting the profile-based vector schema without breaking the published Scala model or dropping valid service fields.

Changes

• Default Azure AI Search requests to 2026-04-01.
• Treat 2023-10-01-Preview and later as profile-based vector APIs:
  • vectorSearch.algorithms and vectorSearch.profiles
  • field-level vectorSearchProfile
  • searchable: true on vector fields
• Keep the published VectorSearch and IndexField case-class constructor, companion, and extractor shapes unchanged.
• Modernize REST payloads with lossless JSON-AST transformations that preserve algorithm parameters, vectorizers, compressions, and unknown future fields.
• Upgrade caller-supplied legacy JSON when sending it to a modern API.
• Reject lossy modern-to-legacy conversion with an actionable error; callers can still supply an explicit legacy definition with apiVersion=2023-07-01-Preview.
• Detect an existing legacy index before document upload. createIfNoneExists validates the actual remote schema with one LIST plus one GET and fails early with migration guidance instead of implying that a no-op create migrated the index.
• Use the selected API version consistently for index and document endpoints.
• Update Search docs and samples, including the 2026 vector query shape:
  vectorQueries: [{"kind":"vector","vector":[...],"fields":"embeddings","k":5}].

Release compatibility

The Spark 4.1 replay needs the AAD authentication precursor from #2591, which is already on master but not yet on the release branch. The compatibility job now reads an ordered manifest of full target-branch commit SHAs, applies only their release-relevant paths, skips content already present, and then replays this PR. This keeps the check generic and removes the prerequisite automatically once the release branch contains the backport.

Existing-index behavior

This PR does not destructively rewrite existing indexes. If a legacy vector index is opened with a modern API version, SynapseML explains the two safe options:

1. Continue temporarily with apiVersion=2023-07-01-Preview.
2. Explicitly migrate the index with Azure AI Search Create or Update Index after reviewing the schema changes.

Modern API responses that normalize a legacy vector field to dimensions plus a null profile are still detected as legacy. The shared document-QA sample pins writes to 2023-07-01-Preview until its persistent index is explicitly migrated; its vector query remains on 2026-04-01.

Validation

• cognitive/Test/compile
• VectorSchemaMigrationSuite: 15/15 secret-free tests
• tools/ci/tests/test_pipeline_yaml.py: 16/16 tests, including an isolated prerequisite-before-PR Git replay
• Actual feat: add backward-compatible AAD auth to Azure Search #2591 plus feat(search): migrate Azure AI Search to the 2026-04-01 API and profile-based vector schema #2604 patch replay onto spark4.1
• cognitive/scalastyle and cognitive/Test/scalastyle
• Coverage includes the API-version boundary, public case-class compatibility, lossless modern/legacy transformations, nested fields, unknown-field preservation, actual remote-schema validation, one-LIST/one-GET behavior, response cleanup, and the existing-index failure path.
• No Azure resources were created, updated, or deleted locally.

Dependencies

No dependency changes.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[github-actions]

Hey Rana Singh (@ranadeepsingh) 👋!
Thank you so much for contributing to our repository 🙌.
Someone from SynapseML Team will be reviewing this pull request soon.

We use semantic commit messages to streamline the release process.
Before your pull request can be merged, you should make sure your first commit and PR title start with a semantic prefix.
This helps us to create release messages and credit you for your hard work!

Examples of commit messages with semantic prefixes:

• fix: Fix LightGBM crashes with empty partitions
• feat: Make HTTP on Spark back-offs configurable
• docs: Update Spark Serving usage
• build: Add codecov support
• perf: improve LightGBM memory usage
• refactor: make python code generation rely on classes
• style: Remove nulls from CNTKModel
• test: Add test coverage for CNTKModel

To test your commit locally, please follow our guild on building from source.
Check out the developer guide for additional guidance on testing your change.

[Copilot]

Pull request overview

Updates SynapseML’s Azure AI Search integration to default to the 2026-04-01 data-plane API while preserving compatibility across the legacy (pre-2023-11-01) and modern vector index schema shapes by translating index definitions to the schema required by the selected api-version.

Changes:

• Switch default Azure AI Search API version to 2026-04-01 and add an api-version gate for profile-based vector schemas.
• Introduce vector schema alignment (legacy ⇄ modern), including dual parsing and a unified vectorReference accessor.
• Update writer behavior so document upload uses the same resolved api-version as index operations; refresh docs/notebooks to “Azure AI Search” and current sample API versions.

Show a summary per file

File	Description
docs/Quick Examples/transformers/cognitive/_AzureSearch.md	Renames service heading to Azure AI Search.
docs/Explore Algorithms/OpenAI/Quickstart - Understand and Search Forms.ipynb	Updates narrative + sample REST api-version to 2026-04-01.
docs/Explore Algorithms/AI Services/Quickstart - Document Question and Answering with PDFs.ipynb	Updates naming + vector-store REST api-version to 2026-04-01.
docs/Explore Algorithms/AI Services/Quickstart - Create a Visual Search Engine.ipynb	Updates naming + sample REST api-version to 2026-04-01.
docs/Explore Algorithms/AI Services/Overview.ipynb	Updates Azure AI Search naming in overview and comments.
cognitive/src/test/scala/com/microsoft/azure/synapse/ml/services/search/VectorSchemaMigrationSuite.scala	Adds secret-free tests validating schema migration + version gate behavior.
cognitive/src/test/scala/com/microsoft/azure/synapse/ml/services/search/split2/SearchWriterSuitePart2.scala	Uses version-agnostic vectorReference assertion for vectors.
cognitive/src/test/scala/com/microsoft/azure/synapse/ml/services/search/split1/SearchWriterSuitePart1.scala	Uses version-agnostic vectorReference assertions for vectors.
cognitive/src/main/scala/com/microsoft/azure/synapse/ml/services/search/AzureSearchSchemas.scala	Adds modern vector schema types (profiles/algorithms), vectorReference, and VectorSchema.align.
cognitive/src/main/scala/com/microsoft/azure/synapse/ml/services/search/AzureSearchAuth.scala	Updates user-facing auth error strings to “Azure AI Search”.
cognitive/src/main/scala/com/microsoft/azure/synapse/ml/services/search/AzureSearchAPI.scala	Defaults api-version to 2026-04-01; aligns schema pre-validation; updates vector validation to use vectorReference.
cognitive/src/main/scala/com/microsoft/azure/synapse/ml/services/search/AzureSearch.scala	Emits modern vector schema by default; pins AddDocuments endpoint to the selected api-version; switches vector detection to isVectorField.

Review details

Suppressed comments (1)

cognitive/src/main/scala/com/microsoft/azure/synapse/ml/services/search/AzureSearchAPI.scala:160

• validateIndexInfo aligns vector schemas but still allows vector fields to be sent without any vectorSearch.algorithms / legacy vectorSearch.algorithmConfigurations being defined (e.g., if a user supplies dimensions + vectorSearchProfile/config on fields but omits the vectorSearch section). That will pass current validation and fail later at request time with a less actionable service error. Consider rejecting this early with an explicit validation check after alignment.

  private def validateIndexInfo(indexJson: String, apiVersion: String): Try[IndexInfo] = {
    // Align first so the definition that is validated is exactly the one that gets sent.
    val schema = VectorSchema.align(parseIndexJson(indexJson), apiVersion)
    for {
      _ <- validName(schema.name.get)
      _ <- validIndexFields(schema.fields)
    } yield schema
  }

• Files reviewed: 12/12 changed files
• Comments generated: 0
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Codecov Comments Bot (codecov-commenter)]

Codecov Report

❌ Patch coverage is 77.00000% with 46 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.21%. Comparing base (0dfddd3) to head (bd75e92).

Files with missing lines	Patch %	Lines
...ynapse/ml/services/search/AzureSearchSchemas.scala	72.50%	33 Missing ⚠️
...re/synapse/ml/services/search/AzureSearchAPI.scala	78.00%	11 Missing ⚠️
...e/synapse/ml/services/search/AzureSearchAuth.scala	33.33%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2604      +/-   ##
==========================================
- Coverage   86.33%   86.21%   -0.12%     
==========================================
  Files         335      335              
  Lines       18007    18178     +171     
  Branches     1658     1706      +48     
==========================================
+ Hits        15546    15673     +127     
- Misses       2461     2505      +44     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Review details

• Files reviewed: 12/12 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Review details

Suppressed comments (2)

cognitive/src/main/scala/com/microsoft/azure/synapse/ml/services/search/AzureSearch.scala:368

• When the index does not exist, prepareDF calls getExisting(...) and then calls SearchIndex.createIfNoneExists(...), which performs its own client.getExisting(...) list call. This results in two index LIST requests on the create path (extra latency + rate-limit pressure). Consider refactoring so the create path reuses the already-fetched index list (e.g., call the private[search] createIfNoneExists(..., client) overload with a client that returns the cached list).

    val existingIndexJsonOpt = if (getExisting(auth, serviceName, apiVersion).contains(indexName)) {
      if (indexJsonOpt.isDefined) {
        println(f"indexJsonOpt is specified, however an index for $indexName already exists," +
          f"we will use the index definition obtained from the existing index instead")
      }
      val existingIndexJson = IndexJsonReader.get(auth, serviceName, indexName, apiVersion)
      VectorSchema.requireCompatibleExistingIndex(existingIndexJson.parseJson, apiVersion)
      Some(existingIndexJson)
    } else {
      None
    }

    val (indexJson, preppedDF) = resolveIndexDefinition(
      existingIndexJsonOpt, indexJsonOpt, vectorColsInfo, df, indexName, keyCol, actionCol)

    // TODO: Support vector search in nested fields
    // Throws an exception if any nested field is a vector in the schema
    parseIndexJson(indexJson).fields.foreach(_.fields.foreach(assertNoNestedVectors))

    if (existingIndexJsonOpt.isEmpty) {
      SearchIndex.createIfNoneExists(auth, serviceName, indexJson, apiVersion)
    }

cognitive/src/main/scala/com/microsoft/azure/synapse/ml/services/search/AzureSearch.scala:347

• apiVersion is used in getExisting(...) before it is validated. If a caller supplies an invalid apiVersion, this will attempt a network request and only fail later, even though supportsVectorProfiles is intended to reject invalid versions before sending requests. Validate apiVersion right before the first request.

    val existingIndexJsonOpt = if (getExisting(auth, serviceName, apiVersion).contains(indexName)) {

• Files reviewed: 12/12 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Review details

Suppressed comments (2)

tools/ci/tests/test_pipeline_yaml.py:360

• This test is named as a format/validation check, but it hard-codes the exact prerequisite SHA list. That will cause unrelated breakages whenever the prerequisites file is updated (e.g., when the backport lands or additional prerequisites are added). Prefer validating that prerequisites are present, unique, and full 40-character SHAs.

    assert lines == ["04897bae9baa08f0d67855566f7bad235791d508"]
    assert all(re.fullmatch(r"[0-9a-fA-F]{40}", line) for line in lines)

cognitive/src/main/scala/com/microsoft/azure/synapse/ml/services/search/AzureSearchSchemas.scala:284

• In rewriteField, the thrown error says it cannot create a vector profile for an algorithm name, but this code path also handles modern vectorSearchProfile values (which are profile names). The message is misleading and makes debugging invalid modern schemas harder.

      val reference = selectAlias(fields, ModernFieldReferenceKey, LegacyFieldReferenceKey).map {
        case JsString(name) =>
          JsString(references.getOrElse(name, throw new IllegalArgumentException(
            s"Cannot find or create a vector profile for algorithm '$name'")))

• Files reviewed: 15/15 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines will not run the associated pipelines, because the pull request was updated after the run command was issued. Review the pull request again and issue a new run command.

[Copilot]

Review details

Suppressed comments (2)

cognitive/src/main/scala/com/microsoft/azure/synapse/ml/services/search/AzureSearchSchemas.scala:241

• modernizeVectorSearch only auto-generates vectorSearch.profiles when upgradingLegacy (i.e., when only algorithmConfigurations is present). If a caller supplies a mixed schema (e.g., already-renamed vectorSearch.algorithms but still no profiles), rewriteField will fail with "Cannot find or create a vector profile..." even though default profiles can be generated losslessly from the algorithms list. Consider generating default profiles whenever profiles are absent, not only when upgrading legacy keys.

    val generatedProfiles = if (upgradingLegacy) {
      algorithms.toSeq.flatMap(asArray(_, ModernAlgorithmsKey)).flatMap { algorithm =>
        val algorithmName = objectString(algorithm, "name", ModernAlgorithmsKey)
        if (profileByAlgorithm.contains(algorithmName)) {
          None

cognitive/src/main/scala/com/microsoft/azure/synapse/ml/services/search/AzureSearch.scala:351

• prepareDF calls getExisting(auth, serviceName, apiVersion) to determine whether the index exists, but when the index does not exist it then calls SearchIndex.createIfNoneExists(...), which performs another index LIST internally. This adds an extra network round-trip on the create path (latency + potential throttling). Consider refactoring to reuse the already-fetched index list (e.g., by plumbing a cached result into SearchIndex.createIfNoneExists / its SearchIndexClient).

    val existingIndexJsonOpt = if (getExisting(auth, serviceName, apiVersion).contains(indexName)) {
      if (indexJsonOpt.isDefined) {
        println(f"indexJsonOpt is specified, however an index for $indexName already exists," +
          f"we will use the index definition obtained from the existing index instead")
      }

• Files reviewed: 15/15 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).