feat: add persisted top-K categorical lumping

[Rana Singh (ranadeepsingh)]

Summary

Recreates the valuable categorical-lumping proposal from #1941 for the current SynapseML codebase and addresses #1891.

• implements LumpFeatures as a Spark ML Estimator plus persisted LumpFeaturesModel
• preserves the multi-column lumpRules API, including Scala map, Java map, legacy JSON, and generated Python dict bindings
• learns top-K values once during fit; transform never refits
• ranks deterministically by count descending and value ascending
• retains learned values, maps rare and unseen non-null values to an explicit otherValue, and makes null handling configurable
• rejects real-data collisions with the other bucket
• validates StringType-only v1 schemas and handles literal dotted, backtick, and count column names safely
• persists learned state and supports estimator/model/pipeline serialization and copy behavior

This is a new implementation; it does not alter the ancient PR branch.

Validation

• core/compile
• core/testOnly com.microsoft.azure.synapse.ml.stages.LumpFeaturesSuite — 18 tests passed
• core/scalastyle; core/Test/scalastyle
• codegen
• generated LumpFeatures.py and LumpFeaturesModel.py syntax-compiled with py_compile
• final SynapseML code review: no concrete issues

Design notes

The initial API intentionally supports string columns only so the explicit other bucket cannot silently change numeric or boolean column types. Nulls are excluded from frequency counts and preserved by default; unseen non-null values always use the other bucket. Fitting fails if otherValue already occurs in any configured column, avoiding ambiguous category merges.

Related: #1941, #1891

[github-actions]

Hey Rana Singh (@ranadeepsingh) 👋!
Thank you so much for contributing to our repository 🙌.
Someone from SynapseML Team will be reviewing this pull request soon.

We use semantic commit messages to streamline the release process.
Before your pull request can be merged, you should make sure your first commit and PR title start with a semantic prefix.
This helps us to create release messages and credit you for your hard work!

Examples of commit messages with semantic prefixes:

• fix: Fix LightGBM crashes with empty partitions
• feat: Make HTTP on Spark back-offs configurable
• docs: Update Spark Serving usage
• build: Add codecov support
• perf: improve LightGBM memory usage
• refactor: make python code generation rely on classes
• style: Remove nulls from CNTKModel
• test: Add test coverage for CNTKModel

To test your commit locally, please follow our guild on building from source.
Check out the developer guide for additional guidance on testing your change.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Pull request overview

Adds a new core Spark ML stage to address high-cardinality categorical features by learning per-column top‑K string values at fit time and applying deterministic lumping at transform time, with full persistence/serialization support consistent with SynapseML’s Estimator/Model patterns.

Changes:

• Introduces LumpFeatures (Estimator) and LumpFeaturesModel (Model) with multi-column lumpRules, deterministic top‑K selection, configurable null handling, and otherValue collision validation.
• Implements persisted learned state via keptValuesJson and supports load/save, copy, and schema transformation behavior.
• Adds a comprehensive Scala test suite covering ranking, null handling, collisions, schema/metadata behavior, special column names, and persistence round-trips.

Show a summary per file

File	Description
core/src/main/scala/com/microsoft/azure/synapse/ml/stages/LumpFeatures.scala	New Estimator/Model implementation for persisted top‑K categorical lumping with deterministic ranking, schema validation, and serialization.
core/src/test/scala/com/microsoft/azure/synapse/ml/stages/LumpFeaturesSuite.scala	New unit tests validating behavior, edge cases, schema expectations, and persistence round-trip.

Review details

Suppressed comments (1)

core/src/main/scala/com/microsoft/azure/synapse/ml/stages/LumpFeatures.scala:227

• handleNull='keep' currently only preserves nulls when the input schema marks the column as nullable; otherwise the expression coalesces nulls to otherValue. This makes runtime behavior depend on schema nullability and can violate the documented semantics that 'keep' preserves null. Refactor lumpExpr to preserve nulls whenever keepNulls is true, independent of schema nullability, and drop the inputNullable parameter.

      transformSchema(dataset.schema)
      val other = getOtherValue
      val keepNulls = getHandleNull == "keep"
      getKeptValues.foldLeft(dataset.toDF()) { case (acc, (name, values)) =>
        acc.withColumn(name, lumpExpr(name, values, other, keepNulls, acc.schema(name).nullable))

• Files reviewed: 2/2 changed files
• Comments generated: 1
• Review effort level: Lite

[Codecov Comments Bot (codecov-commenter)]

Codecov Report

❌ Patch coverage is 95.89552% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.47%. Comparing base (0dfddd3) to head (8560216).

Files with missing lines	Patch %	Lines
...crosoft/azure/synapse/ml/stages/LumpFeatures.scala	95.89%	11 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2596      +/-   ##
==========================================
+ Coverage   86.33%   86.47%   +0.14%     
==========================================
  Files         335      336       +1     
  Lines       18007    18275     +268     
  Branches     1658     1673      +15     
==========================================
+ Hits        15546    15803     +257     
- Misses       2461     2472      +11     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Rana Singh (ranadeepsingh)]

Addressed the independent API review finding in 5443214:

• fitted top-K is now persisted with retained values
• Scala, Java, and JSON setLumpRules overloads reject incompatible post-fit changes immediately
• generated-binding/generic Param mutations are rejected by model-state validation before scoring
• incompatible copy(extra) overrides are rejected
• persistence, loaded-model copy, no-op assignment, and mutation paths have regression coverage

Validation: 24 estimator tests, 4 model tests, 10 repository fuzzing checks, core compile/scalastyle, codegen, and generated-wrapper py_compile all passed.

copilot-pull-request-reviewer please re-review the updated head.

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Rana Singh (ranadeepsingh)]

Follow-up API fix is now fully green on head 54432143d3:

• all 76 GitHub/Azure checks passed
• Azure build 229247329 succeeded
• targeted estimator/model/fuzzing tests, scalastyle, codegen, and generated-wrapper syntax checks passed locally

Re-review requested for the fitted lumpRules immutability change.

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Review details

Suppressed comments (1)

core/src/main/scala/com/microsoft/azure/synapse/ml/stages/LumpFeatures.scala:297

• parseKeptJson will throw a NullPointerException if keptValuesJson contains a null value (e.g., set via Java/Param APIs), which bypasses the intended IllegalArgumentException validation path. It should explicitly reject null JSON strings with a clear message before calling parseJson.

  private def parseKeptJson(name: String, json: String): JsValue =
    try json.parseJson
    catch {
      case NonFatal(e) =>
        throw new IllegalArgumentException(s"keptValuesJson for column '$name' is not valid JSON: '$json'.", e)

• Files reviewed: 4/4 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Review details

• Files reviewed: 4/4 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines will not run the associated pipelines, because the pull request was updated after the run command was issued. Review the pull request again and issue a new run command.

[Copilot]

Review details

• Files reviewed: 4/4 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).