fix: preserve hand-written Python package initializers

[Rana Singh (ranadeepsingh)]

Related PRs

• Recreates the still-relevant init-file preservation intent from Fix init file codegen #2286 on current master.
• Preserves the OpenAI compatibility hook introduced by feat: Add v1 OpenAI endpoint support and remove legacy completions API #2560.

Problem

PyCodegen copies hand-written Python overrides and then rewrites nested __init__.py files, discarding required manual content. It also deletes every namespace-root initializer and the prepend approach proposed in #2286 would duplicate generated headers when init generation is repeated.

Before the fix, targeted regression tests reproduced three failures: nested manual UTF-8 content was missing, a non-empty manual namespace root was deleted, and OpenAI manual content was lost around the #2560 hook.

Changes

• Read manual initializer content from the source override tree, never from generated output.
• Emit generated imports in deterministic filename order using explicit UTF-8.
• Compose generated package metadata/imports, initFileExtra, then manual content.
• Keep empty/absent namespace roots as namespace packages while retaining non-empty manual roots.
• Keep the cognitive compatibility initializer entirely hand-written.
• Add focused tests for roots, nested packages, UTF-8, manual presence/absence, OpenAI extras, and repeat-run idempotency.

No files under target/ or other generated output are committed.

Validation

• core/testOnly com.microsoft.azure.synapse.ml.codegen.PyCodegenSuite: 11 passed.
• core/Test/compile and cognitive/Test/compile: passed.
• Canonical core/codegen, cognitive/codegen, and deepLearning/codegen: passed.
• core/scalastyle and core/Test/scalastyle: passed with zero findings.
• black==22.3.0 --check --extend-exclude docs/ .: 179 files unchanged.
• Strict UTF-8 generated-output assertions passed for namespace roots, manual suffixes, OpenAI hook/import ordering, and the cognitive compatibility package.
• Repeated fixed-version core/codegen was byte-idempotent across all 30 generated __init__.py files.

Review

The repository code-review checklist was run before commit; no concrete findings remained.

CI iteration

The first Azure Pipelines run exposed a stale source override at core/src/main/python/synapse/ml/automl/__init__.py: it imported _FindBestModel, but that generated module does not exist. The file had previously been harmless only because PyCodegen overwrote it. The override is now removed so deterministic codegen supplies the valid FindBestModel import. Local targeted tests and canonical core codegen passed after the fix.

Packaging review follow-up

Independent review found that find_namespace_packages(include=["synapse.ml.*"]) omitted a preserved synapse/ml/__init__.py from wheels. Component and aggregate setup templates now include both the exact synapse.ml package and synapse.ml.*. Regression tests build and inspect real fixture wheels for both templates, including UTF-8 root initializer content.

Codegen shard follow-up

Enabling com.microsoft.azure.synapse.ml.codegen.** exposed a dormant WrappableTests assertion that still expected TestRegressorModel to be nested under WrappableTests, even though #2195 moved the fixture to a package-level source file. The assertion now verifies the actual reflected top-level name. The correction is intentionally limited to that one line so the equivalent Spark 4.1 change replays cleanly. The combined WrappableTests and PyCodegenSuite run passes 12/12 tests.

The actual PR patch now applies cleanly onto spark4.1, including the codegen production and test sources.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[github-actions]

Hey Rana Singh (@ranadeepsingh) 👋!
Thank you so much for contributing to our repository 🙌.
Someone from SynapseML Team will be reviewing this pull request soon.

We use semantic commit messages to streamline the release process.
Before your pull request can be merged, you should make sure your first commit and PR title start with a semantic prefix.
This helps us to create release messages and credit you for your hard work!

Examples of commit messages with semantic prefixes:

• fix: Fix LightGBM crashes with empty partitions
• feat: Make HTTP on Spark back-offs configurable
• docs: Update Spark Serving usage
• build: Add codecov support
• perf: improve LightGBM memory usage
• refactor: make python code generation rely on classes
• style: Remove nulls from CNTKModel
• test: Add test coverage for CNTKModel

To test your commit locally, please follow our guild on building from source.
Check out the developer guide for additional guidance on testing your change.

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Pull request overview

This PR updates SynapseML’s Python code generation (PyCodegen) to preserve hand-written __init__.py content (including UTF-8 text) while still emitting deterministic generated imports and required OpenAI compatibility glue. It also adds targeted regression tests to validate manual-content retention, namespace-root behavior, ordering, and idempotency across repeated runs.

Changes:

• Refactors PyCodegen.makeInitFiles to read manual initializer content from the source override tree (src/main/python) and compose it after generated metadata/imports (and any initFileExtra).
• Makes generated import emission deterministic (sorted) and explicitly UTF-8 encoded for init-file writes/reads.
• Adds PyCodegenSuite covering UTF-8 preservation, namespace-root deletion/retention rules, OpenAI hook ordering, cognitive manual-only behavior, and repeat-run byte idempotency.

Show a summary per file

File	Description
core/src/main/scala/com/microsoft/azure/synapse/ml/codegen/PyCodegen.scala	Refactors init-file generation to preserve manual content from the override tree, enforce deterministic ordering, and use explicit UTF-8.
core/src/test/scala/com/microsoft/azure/synapse/ml/codegen/PyCodegenSuite.scala	Adds focused regression tests for manual __init__.py preservation, namespace-root handling, UTF-8, OpenAI hook behavior, and idempotency.

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 0
• Review effort level: Lite

[Codecov Comments Bot (codecov-commenter)]

Codecov Report

❌ Patch coverage is 87.50000% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.65%. Comparing base (d3ef6e3) to head (0d1f4f2).

Files with missing lines	Patch %	Lines
...ft/azure/synapse/ml/codegen/PythonInitMerger.scala	87.09%	16 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2590      +/-   ##
==========================================
- Coverage   86.07%   82.65%   -3.42%     
==========================================
  Files         331      332       +1     
  Lines       17654    17780     +126     
  Branches     1646     1607      -39     
==========================================
- Hits        15195    14696     -499     
- Misses       2459     3084     +625     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Rana Singh (ranadeepsingh)]

/azp run

[Rana Singh (ranadeepsingh)]

Ilya Matiach (@imatiach-msft) Re-review requested after addressing the namespace-root wheel packaging finding.

Commit 484c24f5ea now includes both synapse.ml and synapse.ml.* in component and aggregate discovery. The new regression coverage builds and inspects both wheel variants and verifies the preserved UTF-8 synapse/ml/__init__.py content.

Local validation: 7/7 targeted PyCodegen tests passed, component/test scalastyle passed with zero findings, Black passed, and canonical core codegen emitted the corrected discovery expression.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Review details

• Files reviewed: 6/7 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Review details

Suppressed comments (2)

core/src/test/scala/com/microsoft/azure/synapse/ml/codegen/PyCodegenSuite.scala:85

• buildWheel hard-depends on an external Python environment having both setuptools and wheel available. If either is missing (or python3 isn’t present), this Scala unit test will fail with a packaging error instead of being skipped with a clear reason, which can make CI and local runs flaky.

Consider pre-checking for required Python packaging deps and using assume(...) to skip the wheel-building assertions when unavailable.

  private def buildWheel(sourceDir: File, wheelDir: File): File = {
    wheelDir.mkdirs()
    val process = new ProcessBuilder(
      pythonExecutable, "setup.py", "bdist_wheel", "--universal", "-d", wheelDir.getAbsolutePath)
      .directory(sourceDir)
      .redirectErrorStream(true)
      .start()

core/src/main/scala/com/microsoft/azure/synapse/ml/codegen/PyCodegen.scala:159

• generateInitFiles is intended to make init generation deterministic and UTF-8 safe, but makeInitFiles still writes via FileUtilities.writeFile, which uses stuff.toString.getBytes() (platform default charset). On systems where the default charset isn’t UTF-8 (notably some Windows setups), this can produce non-UTF-8 __init__.py bytes for packages that don’t have a manual override (and therefore never get rewritten by PythonInitMerger).

To make encoding guarantees consistent, consider writing generated init files with StandardCharsets.UTF_8 (either by updating writeFile to accept a charset or by using Files.write(..., UTF_8) in makeInitFiles).

  private[codegen] def generateInitFiles(conf: CodegenConfig): Unit = {
    makeInitFiles(conf)
    PythonInitMerger.preserve(conf)
  }

• Files reviewed: 7/8 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Review details

Suppressed comments (1)

core/src/test/scala/com/microsoft/azure/synapse/ml/codegen/PyCodegenSuite.scala:85

• buildWheel assumes the bdist_wheel setuptools command is available, but that command only exists when the Python wheel package is installed. In environments where wheel is missing, this fails with a non-obvious "invalid command 'bdist_wheel'" error, which can make CI/local runs flaky and harder to diagnose.

Consider adding an explicit preflight check that wheel is importable (and fail with a clear message) before invoking setup.py bdist_wheel.

  private def buildWheel(sourceDir: File, wheelDir: File): File = {
    wheelDir.mkdirs()
    val process = new ProcessBuilder(
      pythonExecutable, "setup.py", "bdist_wheel", "--universal", "-d", wheelDir.getAbsolutePath)
      .directory(sourceDir)
      .redirectErrorStream(true)

• Files reviewed: 7/8 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).