Skip to content

feat(udf): enable Python UDF UI parameters end to end#5912

Open
carloea2 wants to merge 12 commits into
apache:mainfrom
carloea2:ui-parameter-end2end
Open

feat(udf): enable Python UDF UI parameters end to end#5912
carloea2 wants to merge 12 commits into
apache:mainfrom
carloea2:ui-parameter-end2end

Conversation

@carloea2

@carloea2 carloea2 commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

This PR wires the existing Python UDF UI parameter support into the end-to-end execution path.

It changes:

Area Change
Python UDF descriptors Adds the uiParameters property to Python UDF, dual-input Python UDF, and Python UDF source operators.
Execution wiring Applies PythonUdfUiParameterInjector.inject(...) before creating Python physical operators.
Frontend wiring Connects the existing UI-parameter sync service to the code editor and property editor, and renders uiParameters with the existing UI-parameter editor.
Operator templates Adds commented UiParameter examples to Python UDF templates.

Any related issues, documentation, discussions?

Part of the Python UDF UI parameter feature split from feat/ui-parameter.

Related tracking issue / stack: #5044

Stack order:

  1. Frontend UI parameter building blocks: feat(frontend): add Python UDF UI parameter form support #5043
  2. Scala backend injection model: feat(workflow-operator): add Python UDF UI parameter injection model #5141
  3. Python runtime support: feat(pyamber): support Python UDF UI parameters #5603
  4. End-to-end execution wiring: this PR

How was this PR tested?

Manually +

Commands run:

sbt --no-server scalafmtAll scalafixAll
sbt --no-server scalafmtCheckAll "scalafixAll --check"

cd amber
ruff check src/main/python src/test/python
ruff format --check src/main/python src/test/python
cd ..

sbt --no-server "WorkflowExecutionService / Test / testOnly org.apache.texera.amber.engine.architecture.pythonworker.PythonWorkflowWorkerStartupConfigSpec"

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Me

@github-actions github-actions Bot added engine pyamber frontend Changes related to the frontend GUI common labels Jun 23, 2026
@github-actions

github-actions Bot commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @SarahAsad23, @kunwp1, @aglinxinyuan
    You can notify them by mentioning @SarahAsad23, @kunwp1, @aglinxinyuan in a comment.

@codecov-commenter

codecov-commenter commented Jun 23, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.45455% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 54.49%. Comparing base (6433e71) to head (de3717d).

Files with missing lines Patch % Lines
amber/src/main/python/pytexera/udf/udf_operator.py 96.80% 3 Missing ⚠️
...r/udf/python/DualInputPortsPythonUDFOpDescV2.scala 33.33% 2 Missing ⚠️
.../amber/operator/udf/python/PythonUDFOpDescV2.scala 66.66% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #5912      +/-   ##
============================================
+ Coverage     54.37%   54.49%   +0.12%     
- Complexity     2857     2861       +4     
============================================
  Files          1107     1107              
  Lines         42767    42885     +118     
  Branches       4599     4599              
============================================
+ Hits          23253    23369     +116     
- Misses        18160    18161       +1     
- Partials       1354     1355       +1     
Flag Coverage Δ *Carryforward flag
access-control-service 70.44% <ø> (ø)
agent-service 34.36% <ø> (ø) Carriedforward from 6433e71
amber 56.19% <75.00%> (+0.02%) ⬆️
computing-unit-managing-service 1.65% <ø> (ø)
config-service 57.35% <ø> (ø)
file-service 58.59% <ø> (ø)
frontend 48.27% <ø> (-0.01%) ⬇️ Carriedforward from 6433e71
pyamber 90.39% <97.50%> (+0.18%) ⬆️
python 90.75% <ø> (-0.02%) ⬇️ Carriedforward from 6433e71
workflow-compiling-service 58.69% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@carloea2 carloea2 force-pushed the ui-parameter-end2end branch from 808dcd6 to 46ccb80 Compare June 23, 2026 21:01
@github-actions

github-actions Bot commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

✅ No material benchmark regressions detected

🟢 5 better · 🔴 0 worse · ⚪ 10 noise (<±5%) · 0 without baseline

Compared against main 6433e71 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🟢 bs=10 sw=10 sl=64 428 0.261 22,335/34,925/34,925 us 🟢 -37.3% / 🟢 -6.1%
bs=100 sw=10 sl=64 933 0.569 106,873/122,979/122,979 us ⚪ within ±5% / 🟢 -12.0%
bs=1000 sw=10 sl=64 1,081 0.66 921,359/982,624/982,624 us ⚪ within ±5% / 🟢 -5.3%
Baseline details

Latest main 6433e71 from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 428 tuples/sec 362 tuples/sec 410.82 tuples/sec +18.2% +4.2%
bs=10 sw=10 sl=64 MB/s 0.261 MB/s 0.221 MB/s 0.251 MB/s +18.1% +4.1%
bs=10 sw=10 sl=64 p50 22,335 us 25,868 us 23,785 us -13.7% -6.1%
bs=10 sw=10 sl=64 p95 34,925 us 55,702 us 34,980 us -37.3% -0.2%
bs=10 sw=10 sl=64 p99 34,925 us 55,702 us 34,980 us -37.3% -0.2%
bs=100 sw=10 sl=64 throughput 933 tuples/sec 938 tuples/sec 891.94 tuples/sec -0.5% +4.6%
bs=100 sw=10 sl=64 MB/s 0.569 MB/s 0.572 MB/s 0.544 MB/s -0.5% +4.5%
bs=100 sw=10 sl=64 p50 106,873 us 105,827 us 112,277 us +1.0% -4.8%
bs=100 sw=10 sl=64 p95 122,979 us 129,387 us 139,802 us -5.0% -12.0%
bs=100 sw=10 sl=64 p99 122,979 us 129,387 us 139,802 us -5.0% -12.0%
bs=1000 sw=10 sl=64 throughput 1,081 tuples/sec 1,078 tuples/sec 1,041 tuples/sec +0.3% +3.8%
bs=1000 sw=10 sl=64 MB/s 0.66 MB/s 0.658 MB/s 0.635 MB/s +0.3% +3.9%
bs=1000 sw=10 sl=64 p50 921,359 us 924,790 us 972,714 us -0.4% -5.3%
bs=1000 sw=10 sl=64 p95 982,624 us 984,365 us 1,023,057 us -0.2% -4.0%
bs=1000 sw=10 sl=64 p99 982,624 us 984,365 us 1,023,057 us -0.2% -4.0%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,467.35,200,128000,428,0.261,22335.23,34925.27,34925.27
1,100,10,64,20,2143.96,2000,1280000,933,0.569,106872.89,122979.31,122979.31
2,1000,10,64,20,18508.33,20000,12800000,1081,0.660,921358.97,982624.32,982624.32

@github-actions github-actions Bot removed the engine label Jun 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common frontend Changes related to the frontend GUI pyamber

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants