fix: skip VW barrier execution for single-partition training

[Rana Singh (ranadeepsingh)]

Summary

• use Vowpal Wabbit barrier execution only when it is enabled and training has more than one partition
• preserve the existing multi-partition barrier path and disabled-barrier path
• add execution-path coverage for enabled single-partition, enabled multi-partition, and disabled multi-partition training

Validation

• TDD red: the new one-partition test observed a BarrierTaskContext before the source guard
• vw/testOnly com.microsoft.azure.synapse.ml.vw.VerifyVowpalWabbitClassifier -- -z skips
• vw/testOnly com.microsoft.azure.synapse.ml.vw.VerifyVowpalWabbitClassifier -- -z libsvm
• vw/compile and vw/Test/compile
• scalastyle test:scalastyle

Compatibility

The production change is a source- and binary-compatible execution guard; no public API changes are introduced. This change is scoped only to Vowpal Wabbit and does not alter LightGBM.

Related

Recreates the still-relevant proposal from #1912 on current master without closing or modifying that original PR.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[github-actions]

Hey Rana Singh (@ranadeepsingh) 👋!
Thank you so much for contributing to our repository 🙌.
Someone from SynapseML Team will be reviewing this pull request soon.

We use semantic commit messages to streamline the release process.
Before your pull request can be merged, you should make sure your first commit and PR title start with a semantic prefix.
This helps us to create release messages and credit you for your hard work!

Examples of commit messages with semantic prefixes:

• fix: Fix LightGBM crashes with empty partitions
• feat: Make HTTP on Spark back-offs configurable
• docs: Update Spark Serving usage
• build: Add codecov support
• perf: improve LightGBM memory usage
• refactor: make python code generation rely on classes
• style: Remove nulls from CNTKModel
• test: Add test coverage for CNTKModel

To test your commit locally, please follow our guild on building from source.
Check out the developer guide for additional guidance on testing your change.

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Pull request overview

This PR adjusts Vowpal Wabbit training to avoid Spark barrier execution when training runs with a single partition, while preserving the existing behavior for multi-partition training (both barrier-enabled and barrier-disabled). This improves compatibility with single-partition workloads by preventing unnecessary barrier scheduling.

Changes:

• Guard barrier execution in VowpalWabbitBaseLearner so it only runs when barrier mode is enabled and the training DataFrame has more than one partition.
• Add execution-path tests that verify barrier usage (enabled multi-partition), non-barrier usage (disabled multi-partition), and skipping barrier usage (enabled single-partition) via a tracking subclass and accumulator.

Show a summary per file

File	Description
vw/src/main/scala/com/microsoft/azure/synapse/ml/vw/VowpalWabbitBaseLearner.scala	Adds a partition-count guard to skip barrier execution for single-partition training.
vw/src/test/scala/com/microsoft/azure/synapse/ml/vw/VerifyVowpalWabbitClassifier.scala	Adds tests that assert which execution path was used (barrier vs non-barrier) across partition and configuration combinations.

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 0
• Review effort level: Lite

[Codecov Comments Bot (codecov-commenter)]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.24%. Comparing base (d3ef6e3) to head (87a96ee).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2592      +/-   ##
==========================================
+ Coverage   86.07%   86.24%   +0.16%     
==========================================
  Files         331      331              
  Lines       17654    17654              
  Branches     1646     1631      -15     
==========================================
+ Hits        15195    15225      +30     
+ Misses       2459     2429      -30     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Review details

Suppressed comments (1)

vw/src/main/scala/com/microsoft/azure/synapse/ml/vw/VowpalWabbitBaseLearner.scala:187

• df.rdd is evaluated twice here (once for getNumPartitions and again for barrier()), which can rebuild the underlying RDD/query plan and is unnecessary. Assign it to a local val and reuse it for both the partition check and the barrier path.

    if (getUseBarrierExecutionMode && df.rdd.getNumPartitions > 1)
      df.rdd.barrier().mapPartitions(inputRows => trainIteration(inputRows, localInitialModel)).collect().toSeq
    else
      df.mapPartitions(inputRows => trainIteration(inputRows, localInitialModel))(encoder).collect().toSeq

• Files reviewed: 2/2 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).