Skip to content

fix: align BroadcastPartition routing in LinkConfig with broadcast semantics#5032

Open
Ma77Ball wants to merge 2 commits into
apache:mainfrom
Ma77Ball:fix/zip
Open

fix: align BroadcastPartition routing in LinkConfig with broadcast semantics#5032
Ma77Ball wants to merge 2 commits into
apache:mainfrom
Ma77Ball:fix/zip

Conversation

@Ma77Ball
Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

LinkConfig.toPartitioning's BroadcastPartition() arm was building channels with fromWorkerIds.zip(toWorkerIds), producing a 1:1 pairing rather than the full sender-by-receiver cross product that broadcast semantics imply.
The sibling ChannelConfig.generateChannelConfigs in the same package already emits a cross product for the broadcast arm, so the two helpers disagreed on routing for the same inputs. As with #4799 for OneToOnePartition,
List.zip also silently truncated the surplus side on unequal-length inputs. This PR replaces the zip with the same flatMap / map cross product already used by the HashPartition, RangePartition, and
UnknownPartition arms in this file, which fixes the semantics and removes the silent truncation hazard.

Any related issues, documentation, or discussions?

Closes: #4802

How was this PR tested?

The two LinkConfigSpec cases that pinned the previous behavior (with comments explicitly inviting this realignment) were flipped to assert the cross product: 3 senders by 3 receivers now produces 9 channels, and the
Asymmetric 3 by 2 case now produces 6 channels rather than silently truncating to 2. sbt scalafmtAll was run; CI should pass the existing LinkConfigSpec, ChannelConfigSpec, and downstream scheduling specs.

Was this PR authored or co-authored using generative AI tooling?

Co-authored with Claude Opus 4.7 in compliance with ASF

@Ma77Ball
Copy link
Copy Markdown
Contributor Author

/request-review @aglinxinyuan

@github-actions github-actions Bot requested a review from aglinxinyuan May 12, 2026 19:50
@codecov-commenter
Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 42.72%. Comparing base (7879c2a) to head (f290dd6).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #5032      +/-   ##
============================================
- Coverage     42.73%   42.72%   -0.02%     
+ Complexity     2191     2188       -3     
============================================
  Files          1045     1045              
  Lines         39979    39979              
  Branches       4218     4218              
============================================
- Hits          17087    17083       -4     
- Misses        21835    21838       +3     
- Partials       1057     1058       +1     
Flag Coverage Δ *Carryforward flag
access-control-service 39.53% <ø> (ø) Carriedforward from 7879c2a
agent-service 33.72% <ø> (ø) Carriedforward from 7879c2a
amber 43.36% <100.00%> (-0.03%) ⬇️
computing-unit-managing-service 0.00% <ø> (ø) Carriedforward from 7879c2a
config-service 0.00% <ø> (ø) Carriedforward from 7879c2a
file-service 32.18% <ø> (ø) Carriedforward from 7879c2a
frontend 34.04% <ø> (ø) Carriedforward from 7879c2a
python 88.90% <ø> (ø) Carriedforward from 7879c2a
workflow-compiling-service 47.72% <ø> (ø) Carriedforward from 7879c2a

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes LinkConfig.toPartitioning so that BroadcastPartition() produces the full sender×receiver channel cross product (broadcast semantics) instead of incorrectly using zip (1:1 pairing with silent truncation on length mismatch). This aligns LinkConfig routing behavior with the existing ChannelConfig.generateChannelConfigs behavior in the same package and resolves issue #4802.

Changes:

  • Update LinkConfig.toPartitioning’s BroadcastPartition() arm to generate channels via cross product (flatMap/map) rather than zip.
  • Update LinkConfigSpec broadcast test cases to assert the cross-product routing for symmetric and asymmetric worker list sizes.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
amber/src/main/scala/org/apache/texera/amber/engine/architecture/scheduling/config/LinkConfig.scala Fix broadcast routing to emit sender×receiver channel cross product instead of zip pairing.
amber/src/test/scala/org/apache/texera/amber/engine/architecture/scheduling/config/LinkConfigSpec.scala Update broadcast partition specs to match corrected broadcast semantics (including unequal list sizes).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

LinkConfig.toPartitioning emits broadcast channels via zip instead of cross product

3 participants