From ad161f1e86fe69652a30230d3d5e3e35d63eaec7 Mon Sep 17 00:00:00 2001 From: Xinyuan Lin Date: Mon, 17 Aug 2026 03:42:18 +0000 Subject: [PATCH 1/2] fix(deps, ci): bump sbt/setup-sbt to v1.5.7 to restore CI (#7710) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ### What changes were proposed in this PR? CI has been red repo-wide since 2026-08-16 ~02:00 UTC — `main`, the merge queue, and every PR that runs a Scala stack. The jobs die in 3–7s during *Prepare all required actions*, before checkout, so they produce no test output at all. The cause is the ASF actions allowlist, not anything in this repository. `sbt/setup-sbt` reaches a nested `carabiner-dev` action that is no longer permitted: ``` .github/workflows/*.yml └─ sbt/setup-sbt@6444f4c v1.5.2 allowed └─ carabiner-dev/actions/ampel/verify@94f2939 v1.2.1 allowed └─ carabiner-dev/actions/install/ampel@2a11d59 v1.1.7 BLOCKED ``` > The action carabiner-dev/actions/install/ampel@2a11d59a135c5e291f305f249a92ad7903e3ee0f is not allowed in apache/texera because all actions must be from a repository owned by your enterprise, created by GitHub, or match one of the patterns: … Nothing in the tree changed — every link in that chain is SHA-pinned, and the `sbt/setup-sbt` pin last moved in #6710 on 2026-07-22. The same `2a11d59` downloaded successfully at `2026-08-15T02:01Z` (#7674's amber job) and is rejected at `2026-08-17T02:13Z`. Last clean run before the break: merge-queue `pr-7699` at `2026-08-16T01:52:00Z`; first failure: `main` at `2026-08-16T02:04:42Z`. This bumps all 9 call sites from v1.5.2 to v1.5.7, which resolves down a different path: | | v1.5.2 (current) | v1.5.7 (this PR) | | --- | --- | --- | | `ampel/verify` | `94f2939` (v1.2.1) | `36a39ef` (v1.2.6) | | `install/ampel` | **`2a11d59` — blocked** | `2fec8bd8` | | `install/bnd` | `2a11d59` | no longer referenced | Call sites: `benchmarks.yml` (1), `build-and-push-images.yml` (3), `build.yml` (4), `dependency-graph.yml` (1). > Reviewer note — one thing I could not verify up front. The org allowlist is not readable without repo-admin (`403`), and the error message truncates the pattern list before the `carabiner-dev` entries, so I could not confirm in advance that `2fec8bd8` is permitted. **This PR's own CI run is the experiment.** If it fails on the same error, the bump is a dead end and the fallback is an INFRA request to re-allow `carabiner-dev/actions@2a11d59a135c5e291f305f249a92ad7903e3ee0f`. ### Any related issues, documentation, discussions? No issue filed — the regression originates outside the repository and this PR is the unblock. The previous bump of this pin was #6710. ### How was this PR tested? By this PR's CI, which is the only place the failure reproduces — it depends on the `apache/texera` actions policy and cannot be triggered locally or on a fork. The change touches `.github/**`, so the labeler applies `ci`, and `ci` maps to every stack in `LABEL_STACKS` (`.github/workflows/precheck.yml`) — `frontend`, `amber`, `amber-integration`, `platform`, `platform-integration`, `pyamber`, `agent-service`, `infra`, `pyright-language-service` — plus `Bench`. Every job that consumes `sbt/setup-sbt` is therefore exercised here. ### Was this PR authored or co-authored using generative AI tooling? (backported from commit 7a2c8d03b5a81a895ba12e7c616c961bcc420048) Generated-by: Claude Code (Claude Opus 5) --- .github/workflows/benchmarks.yml | 418 ++++++++++++++++++++ .github/workflows/build-and-push-images.yml | 6 +- .github/workflows/build.yml | 165 +++++++- .github/workflows/dependency-graph.yml | 58 +++ 4 files changed, 641 insertions(+), 6 deletions(-) create mode 100644 .github/workflows/benchmarks.yml create mode 100644 .github/workflows/dependency-graph.yml diff --git a/.github/workflows/benchmarks.yml b/.github/workflows/benchmarks.yml new file mode 100644 index 00000000000..a427a18646b --- /dev/null +++ b/.github/workflows/benchmarks.yml @@ -0,0 +1,418 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Texera benchmarks — bench-agnostic umbrella workflow. +# +# This file is the single CI entry for ALL Texera performance benchmarks +# (currently Arrow Flight E2E; JMH and others land here as well). The +# workflow knows nothing about specific benches — bin/run-benchmarks.sh +# is the opaque entry point that owns which benches run and where their +# outputs land under bench-results/. Adding a new bench is: +# 1. Append the run command to bin/run-benchmarks.sh. +# 2. Add a `Publish ` step block below pointing at the +# bench's JSON output file with the right `tool:` setting. +# This workflow file otherwise stays unchanged. +# +# Triggering — mirrors amber-integration's label gate (NOT file paths): +# - PR: runs only when one of the labels mapped to the amber-integration +# stack in required-checks.yml's LABEL_STACKS is present on the PR. +# Labels are applied by the .github/labeler.yml workflow on opened / +# synchronize, so we wait for that workflow to complete before +# deciding (same pattern required-checks.yml uses). A PR run benches +# the PR head, then re-runs the IDENTICAL trimmed grid against the +# base-branch (main) commit it targets, in the SAME runner (see the +# "Benchmark main baseline in the same runner" step). The delta between +# those two cancels cross-runner hardware variance (the dominant source +# of CI bench noise), so the PR comment's main-vs-branch comparison is +# apples-to-apples rather than PR-here vs a stored baseline captured on +# some other runner. PRs never publish to gh-pages. +# - push to main: always runs (same trimmed grid as PR for quick post- +# merge signal) but does NOT publish to gh-pages; it only emits the +# job summary plus uploaded artifact. Publishing on every merge spammed +# the repo's Pulse / all-branches commit count with bot commits, so +# only the scheduled (daily) run persists the baseline now. +# - schedule (daily): runs the full 27-config sweep and is the sole +# writer that publishes to gh-pages (the authoritative long-term +# baseline). +# - workflow_dispatch: manual full-grid run (no publish; bring-your-own +# trigger for ad-hoc exploration). +# +# Two modes via BENCH_MODE env (read by the bench Scala main): +# pr — 3 configs × 20 batches, ~5 min (PR + push-to-main) +# full — 27 configs × 200 batches, ~40 min (schedule + dispatch) +# +# Non-blocking: this workflow is NOT included in required-checks.yml's +# `required-checks` aggregator, so its result doesn't gate merges even +# when it fails. Adding it to branch protection later is a deliberate +# .asf.yaml change. +# +# Permissions: +# contents: write — needed by benchmark-action's auto-push to gh-pages. +# PR runs (which GitHub auto-downgrades to read-only on forks) gate +# auto-push off via the event check, so the missing write is never +# exercised. + +name: Benchmarks + +on: + push: + branches: [main] + pull_request: + types: [opened, reopened, synchronize, labeled, unlabeled] + schedule: + # Daily full-grid baseline refresh, 12:00 UTC (05:00 PDT). PR and + # post-merge runs use a trimmed 3-config grid to stay around 5 min; the + # scheduled run covers the full 27-config sweep that the gh-pages + # dashboard tracks long-term. Daily (rather than weekly) keeps the + # baseline fresh and accumulates enough data points to average out CI + # noise; the extra bot commits on gh-pages are intentionally tolerated. + # 12:00 UTC lands in the early-morning PDT lull when GitHub runners are + # less contended (and thus less noisy) than during late-night dev hours. + # Bump to several times a day by adding more cron entries if denser + # sampling is wanted. + - cron: "0 12 * * *" + workflow_dispatch: + +permissions: + contents: write + +concurrency: + group: benchmarks-${{ github.ref }} + # On main: never cancel an in-flight baseline run; on PRs: supersede. + cancel-in-progress: ${{ github.ref != 'refs/heads/main' }} + +jobs: + precheck: + # Decide whether to run based on PR labels (push / dispatch always + # run). Lifted from required-checks.yml's precheck so the trigger + # surface matches amber-integration exactly. + name: Precheck + runs-on: ubuntu-latest + outputs: + run_bench: ${{ steps.decide.outputs.run_bench }} + steps: + - name: Wait for Pull Request Labeler + if: github.event_name == 'pull_request' + uses: actions/github-script@v9 + with: + script: | + const ref = context.payload.pull_request.head.sha; + const maxAttempts = 30; + for (let i = 0; i < maxAttempts; i++) { + const { data } = await github.rest.checks.listForRef({ + owner: context.repo.owner, + repo: context.repo.repo, + ref, + check_name: "labeler", + }); + const check = data.check_runs[0]; + if (check && check.status === "completed") { + core.info(`labeler ${check.conclusion}`); + return; + } + core.info(`labeler not ready (attempt ${i + 1}/${maxAttempts})`); + await new Promise((r) => setTimeout(r, 10000)); + } + core.warning("labeler did not complete within 5 minutes; proceeding with current labels."); + + - name: Decide whether to run bench + id: decide + uses: actions/github-script@v9 + with: + script: | + const eventName = context.eventName; + if (eventName !== "pull_request") { + // push to main / workflow_dispatch always run. + core.info(`event=${eventName} — running unconditionally`); + core.setOutput("run_bench", "true"); + return; + } + // Re-fetch labels: the labeler may have just added some. + const { data: pr } = await github.rest.pulls.get({ + owner: context.repo.owner, + repo: context.repo.repo, + pull_number: context.payload.pull_request.number, + }); + const labels = pr.labels.map((l) => l.name); + core.info(`PR labels: ${labels.join(", ") || "(none)"}`); + // Mirrors LABEL_STACKS in required-checks.yml: every label + // whose stack list contains "amber-integration" triggers this + // bench. Keep in sync if LABEL_STACKS there changes. + const TRIGGER_LABELS = new Set([ + "pyamber", + "engine", + "amber-integration", + "common", + "ddl-change", + "ci", + ]); + const matched = labels.filter((l) => TRIGGER_LABELS.has(l)); + const shouldRun = matched.length > 0; + core.info( + shouldRun + ? `Triggering on labels: ${matched.join(", ")}` + : "No trigger label present; skipping bench." + ); + core.setOutput("run_bench", shouldRun ? "true" : "false"); + + bench: + name: Bench + needs: precheck + if: ${{ needs.precheck.outputs.run_bench == 'true' }} + runs-on: ubuntu-latest + env: + JAVA_OPTS: -Xms2048M -Xmx2048M -Xss6M -XX:ReservedCodeCacheSize=256M -Dfile.encoding=UTF-8 + JVM_OPTS: -Xms2048M -Xmx2048M -Xss6M -XX:ReservedCodeCacheSize=256M -Dfile.encoding=UTF-8 + # `pr` mode = 3-config trimmed sweep (~5 min) for PR + post-merge. + # `full` mode = 27-config sweep (~40 min) for schedule + manual. + # Read by the bench Scala main (see GridSpec switch); workflow only + # decides which mode to pass. + BENCH_MODE: ${{ (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') && 'full' || 'pr' }} + services: + # The bench itself doesn't touch the DB, but sbt's transitive compile + # chain reaches `common/auth` which imports JOOQ-generated classes + # from `org.apache.texera.dao.jooq.generated.*`. JOOQ codegen at + # sbt compile time requires a live Postgres to introspect against; + # without it the auth module's `User` / `UserRoleEnum` symbols fail + # to resolve and the whole bench compile aborts. Mirrors the same + # service block from amber-integration in build.yml. + postgres: + image: postgres + env: + POSTGRES_PASSWORD: postgres + ports: + - 5432:5432 + options: >- + --health-cmd="pg_isready -U postgres" + --health-interval=10s + --health-timeout=5s + --health-retries=5 + steps: + - name: Checkout + uses: actions/checkout@v7 + with: + fetch-depth: 0 + - name: Setup JDK + uses: actions/setup-java@v5 + with: + distribution: "temurin" + java-version: 17 + - name: Setup Python + uses: actions/setup-python@v7 + with: + python-version: "3.12" + - name: Install Python dependencies + # Mirrors amber-integration's installer in build.yml so the bench + # subprocess imports resolve identically (pytorch CPU index + + # betterproto plugin via dev-requirements). + run: | + python -m pip install uv + if [ -f amber/requirements.txt ]; then uv pip install --system --index-strategy unsafe-best-match -r amber/requirements.txt; fi + if [ -f amber/operator-requirements.txt ]; then uv pip install --system --index-strategy unsafe-best-match -r amber/operator-requirements.txt; fi + if [ -f amber/dev-requirements.txt ]; then uv pip install --system --index-strategy unsafe-best-match -r amber/dev-requirements.txt; fi + - name: Install protoc + run: | + PROTOC_VERSION=$(cat bin/protoc-version.txt) + curl -fsSL -o /tmp/protoc.zip "https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOC_VERSION}/protoc-${PROTOC_VERSION}-linux-x86_64.zip" + sudo unzip -o /tmp/protoc.zip -d /usr/local + sudo chmod +x /usr/local/bin/protoc + sudo chmod -R a+rX /usr/local/include/google + - name: Create Database for JOOQ codegen + # Minimal subset of amber-integration's "Create Databases" step — + # JOOQ only introspects against texera_db, not iceberg/lakefs/ + # lakekeeper schemas which the bench never touches. + run: psql -h localhost -U postgres -f sql/texera_ddl.sql + env: + PGPASSWORD: postgres + - name: Generate Python proto bindings + run: bash bin/python-proto-gen.sh + - name: Setup sbt launcher + uses: sbt/setup-sbt@8feba82adc7f01ddcf8165b86f778bdb5b82cebc # v1.5.7 + - uses: coursier/cache-action@95e5b1029b6b86e7bac033ee44a0697d8a527d2d # v8.1.1 + with: + extraSbtFiles: '["*.sbt", "project/**.{scala,sbt}", "project/build.properties" ]' + + - name: Run benchmarks + # Single opaque entry point — this workflow doesn't know which + # benches exist. Adding a JMH suite later = appending one line + # inside bin/run-benchmarks.sh and adding a publish step below. + run: bash bin/run-benchmarks.sh + + - name: Benchmark main baseline in the same runner + # PR only: re-run the IDENTICAL trimmed grid against the base-branch + # (main) commit this PR targets, in THIS runner, right after the PR + # run above. Comparing two runs from the same machine cancels the + # cross-runner hardware variance that otherwise dominates CI bench + # deltas, so benchmarks-pr-comment.yml can show a trustworthy + # main-vs-branch comparison instead of PR-here vs a stored baseline + # captured on some other runner. + # + # The output convention is preserved: the PR's own outputs stay in + # bench-results/ untouched; we only ADD main's CSV as + # arrow-flight-e2e-main.csv (plus the base SHA in a sidecar file). + # The PR-mode grid is deterministic (see GridSpec in + # ArrowFlightActorBench.scala), so main's rows key 1:1 against the + # PR's rows for the comparison. + # + # Fail-soft by construction: no `set -e`, and a trap restores the + # PR's results plus the original checkout no matter where the main + # re-run dies (broken main, compile error, etc). On failure we emit + # no main CSV, and the comment workflow falls back to the stored + # gh-pages baseline. We also skip entirely if the PR run produced no + # CSV (e.g. the bench itself failed upstream). + if: ${{ github.event_name == 'pull_request' && !cancelled() }} + env: + BASE_SHA: ${{ github.event.pull_request.base.sha }} + run: | + set -uo pipefail + if [ ! -f bench-results/arrow-flight-e2e.csv ]; then + echo "::warning::no PR bench CSV; skipping same-runner main baseline." + exit 0 + fi + ORIG_REF=$(git rev-parse HEAD) + # Park the PR's outputs; main's re-run writes a fresh bench-results/. + mv bench-results bench-results-pr + restore() { + rm -rf bench-results + mv bench-results-pr bench-results 2>/dev/null || true + git checkout --force "$ORIG_REF" 2>/dev/null || true + } + trap restore EXIT + if ! git checkout --force "$BASE_SHA"; then + echo "::warning::could not check out base SHA $BASE_SHA; skipping main baseline." + exit 0 + fi + # Re-sync Python deps to main's requirements: the deps installed + # earlier are the PR's, and the bench subprocess imports must match + # the main code we're about to compile and run. Without this the + # "main" baseline would run main's Scala/Python sources against the + # PR's pinned Python packages, which is not a clean main measurement. + # (sbt recompiles main's Scala automatically when run-benchmarks.sh + # invokes it below; only the pip deps need an explicit re-sync.) + if [ -f amber/requirements.txt ]; then uv pip install --system --index-strategy unsafe-best-match -r amber/requirements.txt || { echo "::warning::main requirements install failed; skipping main baseline."; exit 0; }; fi + if [ -f amber/operator-requirements.txt ]; then uv pip install --system --index-strategy unsafe-best-match -r amber/operator-requirements.txt || { echo "::warning::main operator-requirements install failed; skipping main baseline."; exit 0; }; fi + if [ -f amber/dev-requirements.txt ]; then uv pip install --system --index-strategy unsafe-best-match -r amber/dev-requirements.txt || { echo "::warning::main dev-requirements install failed; skipping main baseline."; exit 0; }; fi + # Regenerate proto bindings against main's protos, then re-bench. + bash bin/python-proto-gen.sh || { echo "::warning::main proto-gen failed; skipping main baseline."; exit 0; } + if bash bin/run-benchmarks.sh && [ -f bench-results/arrow-flight-e2e.csv ]; then + cp bench-results/arrow-flight-e2e.csv bench-results-pr/arrow-flight-e2e-main.csv + printf '%s' "$BASE_SHA" > bench-results-pr/arrow-flight-e2e-main.commit.txt + echo "captured same-runner main baseline at $BASE_SHA" + else + echo "::warning::main baseline re-run failed; PR comment falls back to the gh-pages baseline." + fi + # trap restores the PR outputs (now incl. main CSV) plus original ref. + + - name: Stash PR number for downstream comment workflow + # PR fork workflows can't comment (GitHub forces read-only token); + # benchmarks-pr-comment.yml runs separately via workflow_run with + # proper write access, and needs the PR number to find the target. + # github.event.workflow_run.pull_requests is empty for fork PRs, + # so we ferry the number via artifact. + if: ${{ github.event_name == 'pull_request' && !cancelled() }} + env: + PR_NUMBER: ${{ github.event.pull_request.number }} + run: echo "$PR_NUMBER" > bench-results/pr-number.txt + + - name: Render bench summary + # Render the bench CSV into a markdown table on the workflow run + # page. Visible without further clicks — and doesn't need any + # extra permissions (writes to $GITHUB_STEP_SUMMARY only). + if: ${{ !cancelled() }} + run: | + { + echo "## Bench results (\`$BENCH_MODE\` mode)" + echo + if [ -f bench-results/arrow-flight-e2e.csv ]; then + echo '```csv' + cat bench-results/arrow-flight-e2e.csv + echo '```' + else + echo "_(no bench-results/arrow-flight-e2e.csv produced)_" + fi + } >> "$GITHUB_STEP_SUMMARY" + + - name: Upload bench artifacts + if: ${{ !cancelled() }} + uses: actions/upload-artifact@v7 + with: + name: bench-results-${{ github.run_id }} + path: bench-results/ + retention-days: 14 + + # Publish to the gh-pages dashboard. auto-push + save-data-file are + # gated on `schedule` ONLY: the daily full-grid run is the single + # authoritative baseline writer. PR *and* push-to-main runs only emit + # the job summary and the uploaded artifact, never touching the + # tracked baseline. This is deliberate: each gh-pages write is a bot + # commit (one per chart, so two per run), and persisting on every + # merge to main flooded the repo's Pulse / all-branches commit count + # with `github-action-benchmark` commits. The post-merge run still + # gives quick signal via the rendered summary + artifact; only the + # daily sweep persists. Adding a new benchmark = adding one publish + # block below matching the JSON filename convention in + # bin/run-benchmarks.sh. + # + # `skip-fetch-gh-pages: true` intentionally keeps baseline comparison + # OFF for now. When flipped to false, the action will fetch the stored + # gh-pages baseline, compare each run against main, post an alert + # comment when a result regresses past `alert-threshold`, and (with + # comment-on-alert / fail-on-alert) can block merge. We're deferring + # that until the baseline has accumulated enough daily data points to + # be trustworthy; turning it on is a deliberate follow-up to evaluate + # later. auto-push on the daily schedule still appends to the branch. + # + # `continue-on-error: true` keeps any other gh-pages-side surprise + # (permission glitches, transient git failures) from failing the + # bench job overall — the bench data itself is already in the + # uploaded artifact above. + - name: Publish throughput + if: ${{ !cancelled() }} + continue-on-error: true + uses: benchmark-action/github-action-benchmark@52576c92bccf6ac60c8223ec7eb2565637cae9ba # v1.22.1 + with: + name: Arrow Flight E2E Throughput + tool: customBiggerIsBetter + output-file-path: bench-results/arrow-flight-e2e-throughput.json + github-token: ${{ secrets.GITHUB_TOKEN }} + auto-push: ${{ github.event_name == 'schedule' }} + save-data-file: ${{ github.event_name == 'schedule' }} + skip-fetch-gh-pages: true + gh-pages-branch: gh-pages + benchmark-data-dir-path: dev/bench + alert-threshold: "150%" + # comment-on-alert needs pull-requests:write; skip and let + # results show up via summary-always instead. + comment-on-alert: false + summary-always: true + - name: Publish latency + if: ${{ !cancelled() }} + continue-on-error: true + uses: benchmark-action/github-action-benchmark@52576c92bccf6ac60c8223ec7eb2565637cae9ba # v1.22.1 + with: + name: Arrow Flight E2E Latency + tool: customSmallerIsBetter + output-file-path: bench-results/arrow-flight-e2e-latency.json + github-token: ${{ secrets.GITHUB_TOKEN }} + auto-push: ${{ github.event_name == 'schedule' }} + save-data-file: ${{ github.event_name == 'schedule' }} + skip-fetch-gh-pages: true + gh-pages-branch: gh-pages + benchmark-data-dir-path: dev/bench + alert-threshold: "150%" + comment-on-alert: false + summary-always: true diff --git a/.github/workflows/build-and-push-images.yml b/.github/workflows/build-and-push-images.yml index aa8335cfc1e..d81a3b20dca 100644 --- a/.github/workflows/build-and-push-images.yml +++ b/.github/workflows/build-and-push-images.yml @@ -126,7 +126,7 @@ jobs: java-version: 17 - name: Setup sbt launcher - uses: sbt/setup-sbt@6444f4c8111de4b9059c3975def104b03cfaa5f0 # v1.5.2 + uses: sbt/setup-sbt@8feba82adc7f01ddcf8165b86f778bdb5b82cebc # v1.5.7 - uses: coursier/cache-action@95e5b1029b6b86e7bac033ee44a0697d8a527d2d # v8.1.1 with: @@ -328,7 +328,7 @@ jobs: java-version: 17 - name: Setup sbt launcher - uses: sbt/setup-sbt@6444f4c8111de4b9059c3975def104b03cfaa5f0 # v1.5.2 + uses: sbt/setup-sbt@8feba82adc7f01ddcf8165b86f778bdb5b82cebc # v1.5.7 - uses: coursier/cache-action@95e5b1029b6b86e7bac033ee44a0697d8a527d2d # v8.1.1 with: @@ -408,7 +408,7 @@ jobs: java-version: 17 - name: Setup sbt launcher - uses: sbt/setup-sbt@6444f4c8111de4b9059c3975def104b03cfaa5f0 # v1.5.2 + uses: sbt/setup-sbt@8feba82adc7f01ddcf8165b86f778bdb5b82cebc # v1.5.7 - uses: coursier/cache-action@95e5b1029b6b86e7bac033ee44a0697d8a527d2d # v8.1.1 with: diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 788a8841069..19e4953145c 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -216,7 +216,7 @@ jobs: env: PGPASSWORD: postgres - name: Setup sbt launcher - uses: sbt/setup-sbt@6444f4c8111de4b9059c3975def104b03cfaa5f0 # v1.5.2 + uses: sbt/setup-sbt@8feba82adc7f01ddcf8165b86f778bdb5b82cebc # v1.5.7 - uses: coursier/cache-action@95e5b1029b6b86e7bac033ee44a0697d8a527d2d # v8.1.1 with: extraSbtFiles: '["*.sbt", "project/**.{scala,sbt}", "project/build.properties" ]' @@ -391,7 +391,7 @@ jobs: env: PGPASSWORD: postgres - name: Setup sbt launcher - uses: sbt/setup-sbt@6444f4c8111de4b9059c3975def104b03cfaa5f0 # v1.5.2 + uses: sbt/setup-sbt@8feba82adc7f01ddcf8165b86f778bdb5b82cebc # v1.5.7 - uses: coursier/cache-action@95e5b1029b6b86e7bac033ee44a0697d8a527d2d # v8.1.1 with: extraSbtFiles: '["*.sbt", "project/**.{scala,sbt}", "project/build.properties" ]' @@ -585,7 +585,7 @@ jobs: distribution: "temurin" java-version: 17 - name: Setup sbt launcher - uses: sbt/setup-sbt@6444f4c8111de4b9059c3975def104b03cfaa5f0 # v1.5.2 + uses: sbt/setup-sbt@8feba82adc7f01ddcf8165b86f778bdb5b82cebc # v1.5.7 - uses: coursier/cache-action@95e5b1029b6b86e7bac033ee44a0697d8a527d2d # v8.1.1 with: extraSbtFiles: '["*.sbt", "project/**.{scala,sbt}", "project/build.properties" ]' @@ -649,6 +649,165 @@ jobs: disable_search: true fail_ci_if_error: false +<<<<<<< HEAD +======= + platform-integration: + # Boot smoke test for the platform services (mirrors amber-integration: an + # independent, infra-provisioned integration job that builds on its own + # classpath — no artifact hand-off from the `platform` job, so the two run + # in parallel). Per-service matrix like `platform`: each service is launched + # from its freshly built dist and must reach a listening state without a + # runtime classpath/linkage crash (see #6220 and .github/scripts/smoke-boot.sh). + # postgres backs every service's JOOQ codegen + boot; file-service also + # fail-fasts on S3 / LakeFS during run(), so those are provisioned for it + # alone. Unit tests + coverage stay in `platform`. See #6273. + if: ${{ inputs.run_platform_integration }} + name: ${{ format('platform-integration{0} ({1})', inputs.job_name_suffix, matrix.service) }} + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + include: + - service: config-service + sbt_project: ConfigService + port: 9094 + - service: access-control-service + sbt_project: AccessControlService + port: 9096 + - service: file-service + sbt_project: FileService + port: 9092 + object_store: true + - service: computing-unit-managing-service + sbt_project: ComputingUnitManagingService + port: 8888 + - service: workflow-compiling-service + sbt_project: WorkflowCompilingService + port: 9090 + - service: notebook-migration-service + sbt_project: NotebookMigrationService + port: 9098 + env: + JAVA_OPTS: -Xms2048M -Xmx2048M -Xss6M -XX:ReservedCodeCacheSize=256M -Dfile.encoding=UTF-8 + JVM_OPTS: -Xms2048M -Xmx2048M -Xss6M -XX:ReservedCodeCacheSize=256M -Dfile.encoding=UTF-8 + services: + # Every platform service transitively depends on DAO, whose JOOQ code + # generation needs the live texera schema at compile time; the service + # also connects to it on boot. + postgres: + image: postgres + env: + POSTGRES_PASSWORD: postgres + ports: + - 5432:5432 + options: >- + --health-cmd="pg_isready -U postgres" + --health-interval=10s + --health-timeout=5s + --health-retries=5 + steps: + - name: Checkout + uses: actions/checkout@v7 + with: + ref: ${{ inputs.checkout_ref || github.sha }} + fetch-depth: 0 + - name: Prepare backport workspace + if: ${{ inputs.backport_target_branch != '' }} + working-directory: ${{ github.workspace }} + run: bash ./.github/scripts/prepare-backport-checkout.sh "${{ inputs.backport_target_branch }}" "${{ inputs.backport_commit_range }}" + - name: Setup JDK + uses: actions/setup-java@v5 + with: + distribution: "temurin" + java-version: 17 + - name: Setup sbt launcher + uses: sbt/setup-sbt@8feba82adc7f01ddcf8165b86f778bdb5b82cebc # v1.5.7 + - uses: coursier/cache-action@95e5b1029b6b86e7bac033ee44a0697d8a527d2d # v8.1.1 + with: + extraSbtFiles: '["*.sbt", "project/**.{scala,sbt}", "project/build.properties" ]' + - name: Create Databases + run: | + psql -h localhost -U postgres -f sql/texera_ddl.sql + psql -h localhost -U postgres -f sql/iceberg_postgres_catalog.sql + psql -h localhost -U postgres -f sql/texera_lakefs.sql + env: + PGPASSWORD: postgres + - name: Check ${{ matrix.sbt_project }} exists in this tree + # Same guard as the `platform` job. + id: module_check + env: + BACKPORT_TARGET_BRANCH: ${{ inputs.backport_target_branch }} + SBT_PROJECT: ${{ matrix.sbt_project }} + run: | + if [ -n "${BACKPORT_TARGET_BRANCH}" ] && ! grep -qE "^lazy val ${SBT_PROJECT}([[:space:]]|=)" build.sbt; then + echo "present=false" >> "$GITHUB_OUTPUT" + echo "::notice::${SBT_PROJECT} is not in this branch's build.sbt (branch cut before it existed); skipping" + else + echo "present=true" >> "$GITHUB_OUTPUT" + fi + - name: Build ${{ matrix.service }} dist + # Self-build on this job's own classpath (mirrors amber-integration + # compiling its own tests) rather than downloading an artifact from the + # `platform` job, so the two stay independent and run in parallel. No + # jacoco here — unit tests + coverage live in `platform`. + if: ${{ steps.module_check.outputs.present == 'true' }} + run: sbt "${{ matrix.sbt_project }}/dist" + - name: Unzip ${{ matrix.service }} dist + if: ${{ steps.module_check.outputs.present == 'true' }} + run: | + mkdir -p /tmp/dists + unzip -q ${{ matrix.service }}/target/universal/${{ matrix.service }}-*.zip -d /tmp/dists/ + - name: Start MinIO + # file-service's boot creates its S3 bucket via S3StorageClient; only + # that service needs an object store, so the rest of the matrix skips this. + if: ${{ matrix.object_store && steps.module_check.outputs.present == 'true' }} + run: | + docker run -d --name minio --network host \ + -e MINIO_ROOT_USER=texera_minio \ + -e MINIO_ROOT_PASSWORD=password \ + minio/minio:RELEASE.2025-02-28T09-55-16Z server /data + for i in $(seq 1 15); do + curl -sf http://localhost:9000/minio/health/live && break + echo "Waiting for MinIO... (attempt $i)"; sleep 1 + done + curl -sf http://localhost:9000/minio/health/live + - name: Start LakeFS + # lakeFS keeps its metadata in the texera_lakefs postgres DB (created + # above) and uses MinIO as its S3 blockstore. file-service's boot calls + # LakeFSStorageClient.healthCheck(), so this must be up for it to reach a + # listening state. Config mirrors bin/single-node (compose + .env), + # adapted to CI creds (postgres/postgres @ localhost). + if: ${{ matrix.object_store && steps.module_check.outputs.present == 'true' }} + run: | + docker run -d --name lakefs --network host \ + -e LAKEFS_DATABASE_TYPE=postgres \ + -e "LAKEFS_DATABASE_POSTGRES_CONNECTION_STRING=postgres://postgres:postgres@localhost:5432/texera_lakefs?sslmode=disable" \ + -e LAKEFS_BLOCKSTORE_TYPE=s3 \ + -e LAKEFS_BLOCKSTORE_S3_FORCE_PATH_STYLE=true \ + -e LAKEFS_BLOCKSTORE_S3_ENDPOINT=http://localhost:9000 \ + -e LAKEFS_BLOCKSTORE_S3_CREDENTIALS_ACCESS_KEY_ID=texera_minio \ + -e LAKEFS_BLOCKSTORE_S3_CREDENTIALS_SECRET_ACCESS_KEY=password \ + -e LAKEFS_AUTH_ENCRYPT_SECRET_KEY=random_string_for_lakefs \ + --entrypoint /bin/sh \ + treeverse/lakefs:1.51 \ + -c "lakefs setup --user-name texera-admin --access-key-id AKIAIOSFOLKFSSAMPLES --secret-access-key wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY || true; lakefs run" + for i in $(seq 1 20); do + curl -sf http://localhost:8000/api/v1/healthcheck && break + echo "Waiting for LakeFS... (attempt $i)"; sleep 1 + done + curl -sf http://localhost:8000/api/v1/healthcheck + - name: Smoke-test ${{ matrix.service }} boots + # Launch the packaged service and assert it reaches LISTEN on its port + # without a runtime classpath/linkage crash (#6220). + if: ${{ steps.module_check.outputs.present == 'true' }} + env: + TEXERA_HOME: ${{ github.workspace }} + # Quiet boot logs, same wiring as the amber jobs. Safe here: + # smoke-boot's verdict is LISTEN-based, never log-scraping (#6332). + TEXERA_SERVICE_LOG_LEVEL: ${{ runner.debug == '1' && 'DEBUG' || 'WARN' }} + run: .github/scripts/smoke-boot.sh "/tmp/dists/${{ matrix.service }}-*/bin/${{ matrix.service }}" "${{ matrix.port }}" + +>>>>>>> 7a2c8d03b (fix(deps, ci): bump sbt/setup-sbt to v1.5.7 to restore CI (#7710)) pyamber: if: ${{ inputs.run_pyamber }} strategy: diff --git a/.github/workflows/dependency-graph.yml b/.github/workflows/dependency-graph.yml new file mode 100644 index 00000000000..4034b15715f --- /dev/null +++ b/.github/workflows/dependency-graph.yml @@ -0,0 +1,58 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Submits the resolved sbt dependency graph (all modules, transitive +# dependencies included) to GitHub via the Dependency Submission API so +# that Dependabot alerts cover Scala dependencies, which GitHub cannot +# parse from build.sbt on its own. +name: Update Dependency Graph +on: + push: + branches: + - main + +permissions: {} + +jobs: + dependency-graph: + name: Update Dependency Graph + runs-on: ubuntu-latest + if: github.repository == 'apache/texera' + permissions: + # The Dependency Submission API requires write permission + # on the repository + contents: write + steps: + - uses: actions/checkout@v7 + with: + persist-credentials: false + + - name: Setup JDK + uses: actions/setup-java@v5 + with: + distribution: 'temurin' + java-version: 17 + + - name: Setup sbt launcher + uses: sbt/setup-sbt@8feba82adc7f01ddcf8165b86f778bdb5b82cebc # v1.5.7 + + - uses: coursier/cache-action@95e5b1029b6b86e7bac033ee44a0697d8a527d2d # v8.1.1 + + - name: Submit sbt dependency graph + uses: scalacenter/sbt-dependency-submission@d84eef4c09e633bcf5f113bcad7fd5e9af1baee9 # v3.2.3 + with: + configs-ignore: provided optional test compile-internal runtime-internal scala-tool scala-doc-tool From 26f10aced77e493583933d469e842bb6ccbf6481 Mon Sep 17 00:00:00 2001 From: Xinyuan Lin Date: Sun, 16 Aug 2026 21:20:23 -0700 Subject: [PATCH 2/2] fix conflicts: keep the backport to the six setup-sbt pins release/v1.2 has neither benchmarks.yml nor dependency-graph.yml, and its build.yml has no platform-integration job, so the cherry-pick hit a modify/delete on all three and pulled the main-only content in wholesale. Dropped both foreign workflows and took the HEAD side of the platform-integration hunk. What remains is the six pin bumps that exist on this branch: build.yml x3, build-and-push-images.yml x3. --- .github/workflows/benchmarks.yml | 418 ------------------------- .github/workflows/build.yml | 159 ---------- .github/workflows/dependency-graph.yml | 58 ---- 3 files changed, 635 deletions(-) delete mode 100644 .github/workflows/benchmarks.yml delete mode 100644 .github/workflows/dependency-graph.yml diff --git a/.github/workflows/benchmarks.yml b/.github/workflows/benchmarks.yml deleted file mode 100644 index a427a18646b..00000000000 --- a/.github/workflows/benchmarks.yml +++ /dev/null @@ -1,418 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -# Texera benchmarks — bench-agnostic umbrella workflow. -# -# This file is the single CI entry for ALL Texera performance benchmarks -# (currently Arrow Flight E2E; JMH and others land here as well). The -# workflow knows nothing about specific benches — bin/run-benchmarks.sh -# is the opaque entry point that owns which benches run and where their -# outputs land under bench-results/. Adding a new bench is: -# 1. Append the run command to bin/run-benchmarks.sh. -# 2. Add a `Publish ` step block below pointing at the -# bench's JSON output file with the right `tool:` setting. -# This workflow file otherwise stays unchanged. -# -# Triggering — mirrors amber-integration's label gate (NOT file paths): -# - PR: runs only when one of the labels mapped to the amber-integration -# stack in required-checks.yml's LABEL_STACKS is present on the PR. -# Labels are applied by the .github/labeler.yml workflow on opened / -# synchronize, so we wait for that workflow to complete before -# deciding (same pattern required-checks.yml uses). A PR run benches -# the PR head, then re-runs the IDENTICAL trimmed grid against the -# base-branch (main) commit it targets, in the SAME runner (see the -# "Benchmark main baseline in the same runner" step). The delta between -# those two cancels cross-runner hardware variance (the dominant source -# of CI bench noise), so the PR comment's main-vs-branch comparison is -# apples-to-apples rather than PR-here vs a stored baseline captured on -# some other runner. PRs never publish to gh-pages. -# - push to main: always runs (same trimmed grid as PR for quick post- -# merge signal) but does NOT publish to gh-pages; it only emits the -# job summary plus uploaded artifact. Publishing on every merge spammed -# the repo's Pulse / all-branches commit count with bot commits, so -# only the scheduled (daily) run persists the baseline now. -# - schedule (daily): runs the full 27-config sweep and is the sole -# writer that publishes to gh-pages (the authoritative long-term -# baseline). -# - workflow_dispatch: manual full-grid run (no publish; bring-your-own -# trigger for ad-hoc exploration). -# -# Two modes via BENCH_MODE env (read by the bench Scala main): -# pr — 3 configs × 20 batches, ~5 min (PR + push-to-main) -# full — 27 configs × 200 batches, ~40 min (schedule + dispatch) -# -# Non-blocking: this workflow is NOT included in required-checks.yml's -# `required-checks` aggregator, so its result doesn't gate merges even -# when it fails. Adding it to branch protection later is a deliberate -# .asf.yaml change. -# -# Permissions: -# contents: write — needed by benchmark-action's auto-push to gh-pages. -# PR runs (which GitHub auto-downgrades to read-only on forks) gate -# auto-push off via the event check, so the missing write is never -# exercised. - -name: Benchmarks - -on: - push: - branches: [main] - pull_request: - types: [opened, reopened, synchronize, labeled, unlabeled] - schedule: - # Daily full-grid baseline refresh, 12:00 UTC (05:00 PDT). PR and - # post-merge runs use a trimmed 3-config grid to stay around 5 min; the - # scheduled run covers the full 27-config sweep that the gh-pages - # dashboard tracks long-term. Daily (rather than weekly) keeps the - # baseline fresh and accumulates enough data points to average out CI - # noise; the extra bot commits on gh-pages are intentionally tolerated. - # 12:00 UTC lands in the early-morning PDT lull when GitHub runners are - # less contended (and thus less noisy) than during late-night dev hours. - # Bump to several times a day by adding more cron entries if denser - # sampling is wanted. - - cron: "0 12 * * *" - workflow_dispatch: - -permissions: - contents: write - -concurrency: - group: benchmarks-${{ github.ref }} - # On main: never cancel an in-flight baseline run; on PRs: supersede. - cancel-in-progress: ${{ github.ref != 'refs/heads/main' }} - -jobs: - precheck: - # Decide whether to run based on PR labels (push / dispatch always - # run). Lifted from required-checks.yml's precheck so the trigger - # surface matches amber-integration exactly. - name: Precheck - runs-on: ubuntu-latest - outputs: - run_bench: ${{ steps.decide.outputs.run_bench }} - steps: - - name: Wait for Pull Request Labeler - if: github.event_name == 'pull_request' - uses: actions/github-script@v9 - with: - script: | - const ref = context.payload.pull_request.head.sha; - const maxAttempts = 30; - for (let i = 0; i < maxAttempts; i++) { - const { data } = await github.rest.checks.listForRef({ - owner: context.repo.owner, - repo: context.repo.repo, - ref, - check_name: "labeler", - }); - const check = data.check_runs[0]; - if (check && check.status === "completed") { - core.info(`labeler ${check.conclusion}`); - return; - } - core.info(`labeler not ready (attempt ${i + 1}/${maxAttempts})`); - await new Promise((r) => setTimeout(r, 10000)); - } - core.warning("labeler did not complete within 5 minutes; proceeding with current labels."); - - - name: Decide whether to run bench - id: decide - uses: actions/github-script@v9 - with: - script: | - const eventName = context.eventName; - if (eventName !== "pull_request") { - // push to main / workflow_dispatch always run. - core.info(`event=${eventName} — running unconditionally`); - core.setOutput("run_bench", "true"); - return; - } - // Re-fetch labels: the labeler may have just added some. - const { data: pr } = await github.rest.pulls.get({ - owner: context.repo.owner, - repo: context.repo.repo, - pull_number: context.payload.pull_request.number, - }); - const labels = pr.labels.map((l) => l.name); - core.info(`PR labels: ${labels.join(", ") || "(none)"}`); - // Mirrors LABEL_STACKS in required-checks.yml: every label - // whose stack list contains "amber-integration" triggers this - // bench. Keep in sync if LABEL_STACKS there changes. - const TRIGGER_LABELS = new Set([ - "pyamber", - "engine", - "amber-integration", - "common", - "ddl-change", - "ci", - ]); - const matched = labels.filter((l) => TRIGGER_LABELS.has(l)); - const shouldRun = matched.length > 0; - core.info( - shouldRun - ? `Triggering on labels: ${matched.join(", ")}` - : "No trigger label present; skipping bench." - ); - core.setOutput("run_bench", shouldRun ? "true" : "false"); - - bench: - name: Bench - needs: precheck - if: ${{ needs.precheck.outputs.run_bench == 'true' }} - runs-on: ubuntu-latest - env: - JAVA_OPTS: -Xms2048M -Xmx2048M -Xss6M -XX:ReservedCodeCacheSize=256M -Dfile.encoding=UTF-8 - JVM_OPTS: -Xms2048M -Xmx2048M -Xss6M -XX:ReservedCodeCacheSize=256M -Dfile.encoding=UTF-8 - # `pr` mode = 3-config trimmed sweep (~5 min) for PR + post-merge. - # `full` mode = 27-config sweep (~40 min) for schedule + manual. - # Read by the bench Scala main (see GridSpec switch); workflow only - # decides which mode to pass. - BENCH_MODE: ${{ (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') && 'full' || 'pr' }} - services: - # The bench itself doesn't touch the DB, but sbt's transitive compile - # chain reaches `common/auth` which imports JOOQ-generated classes - # from `org.apache.texera.dao.jooq.generated.*`. JOOQ codegen at - # sbt compile time requires a live Postgres to introspect against; - # without it the auth module's `User` / `UserRoleEnum` symbols fail - # to resolve and the whole bench compile aborts. Mirrors the same - # service block from amber-integration in build.yml. - postgres: - image: postgres - env: - POSTGRES_PASSWORD: postgres - ports: - - 5432:5432 - options: >- - --health-cmd="pg_isready -U postgres" - --health-interval=10s - --health-timeout=5s - --health-retries=5 - steps: - - name: Checkout - uses: actions/checkout@v7 - with: - fetch-depth: 0 - - name: Setup JDK - uses: actions/setup-java@v5 - with: - distribution: "temurin" - java-version: 17 - - name: Setup Python - uses: actions/setup-python@v7 - with: - python-version: "3.12" - - name: Install Python dependencies - # Mirrors amber-integration's installer in build.yml so the bench - # subprocess imports resolve identically (pytorch CPU index + - # betterproto plugin via dev-requirements). - run: | - python -m pip install uv - if [ -f amber/requirements.txt ]; then uv pip install --system --index-strategy unsafe-best-match -r amber/requirements.txt; fi - if [ -f amber/operator-requirements.txt ]; then uv pip install --system --index-strategy unsafe-best-match -r amber/operator-requirements.txt; fi - if [ -f amber/dev-requirements.txt ]; then uv pip install --system --index-strategy unsafe-best-match -r amber/dev-requirements.txt; fi - - name: Install protoc - run: | - PROTOC_VERSION=$(cat bin/protoc-version.txt) - curl -fsSL -o /tmp/protoc.zip "https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOC_VERSION}/protoc-${PROTOC_VERSION}-linux-x86_64.zip" - sudo unzip -o /tmp/protoc.zip -d /usr/local - sudo chmod +x /usr/local/bin/protoc - sudo chmod -R a+rX /usr/local/include/google - - name: Create Database for JOOQ codegen - # Minimal subset of amber-integration's "Create Databases" step — - # JOOQ only introspects against texera_db, not iceberg/lakefs/ - # lakekeeper schemas which the bench never touches. - run: psql -h localhost -U postgres -f sql/texera_ddl.sql - env: - PGPASSWORD: postgres - - name: Generate Python proto bindings - run: bash bin/python-proto-gen.sh - - name: Setup sbt launcher - uses: sbt/setup-sbt@8feba82adc7f01ddcf8165b86f778bdb5b82cebc # v1.5.7 - - uses: coursier/cache-action@95e5b1029b6b86e7bac033ee44a0697d8a527d2d # v8.1.1 - with: - extraSbtFiles: '["*.sbt", "project/**.{scala,sbt}", "project/build.properties" ]' - - - name: Run benchmarks - # Single opaque entry point — this workflow doesn't know which - # benches exist. Adding a JMH suite later = appending one line - # inside bin/run-benchmarks.sh and adding a publish step below. - run: bash bin/run-benchmarks.sh - - - name: Benchmark main baseline in the same runner - # PR only: re-run the IDENTICAL trimmed grid against the base-branch - # (main) commit this PR targets, in THIS runner, right after the PR - # run above. Comparing two runs from the same machine cancels the - # cross-runner hardware variance that otherwise dominates CI bench - # deltas, so benchmarks-pr-comment.yml can show a trustworthy - # main-vs-branch comparison instead of PR-here vs a stored baseline - # captured on some other runner. - # - # The output convention is preserved: the PR's own outputs stay in - # bench-results/ untouched; we only ADD main's CSV as - # arrow-flight-e2e-main.csv (plus the base SHA in a sidecar file). - # The PR-mode grid is deterministic (see GridSpec in - # ArrowFlightActorBench.scala), so main's rows key 1:1 against the - # PR's rows for the comparison. - # - # Fail-soft by construction: no `set -e`, and a trap restores the - # PR's results plus the original checkout no matter where the main - # re-run dies (broken main, compile error, etc). On failure we emit - # no main CSV, and the comment workflow falls back to the stored - # gh-pages baseline. We also skip entirely if the PR run produced no - # CSV (e.g. the bench itself failed upstream). - if: ${{ github.event_name == 'pull_request' && !cancelled() }} - env: - BASE_SHA: ${{ github.event.pull_request.base.sha }} - run: | - set -uo pipefail - if [ ! -f bench-results/arrow-flight-e2e.csv ]; then - echo "::warning::no PR bench CSV; skipping same-runner main baseline." - exit 0 - fi - ORIG_REF=$(git rev-parse HEAD) - # Park the PR's outputs; main's re-run writes a fresh bench-results/. - mv bench-results bench-results-pr - restore() { - rm -rf bench-results - mv bench-results-pr bench-results 2>/dev/null || true - git checkout --force "$ORIG_REF" 2>/dev/null || true - } - trap restore EXIT - if ! git checkout --force "$BASE_SHA"; then - echo "::warning::could not check out base SHA $BASE_SHA; skipping main baseline." - exit 0 - fi - # Re-sync Python deps to main's requirements: the deps installed - # earlier are the PR's, and the bench subprocess imports must match - # the main code we're about to compile and run. Without this the - # "main" baseline would run main's Scala/Python sources against the - # PR's pinned Python packages, which is not a clean main measurement. - # (sbt recompiles main's Scala automatically when run-benchmarks.sh - # invokes it below; only the pip deps need an explicit re-sync.) - if [ -f amber/requirements.txt ]; then uv pip install --system --index-strategy unsafe-best-match -r amber/requirements.txt || { echo "::warning::main requirements install failed; skipping main baseline."; exit 0; }; fi - if [ -f amber/operator-requirements.txt ]; then uv pip install --system --index-strategy unsafe-best-match -r amber/operator-requirements.txt || { echo "::warning::main operator-requirements install failed; skipping main baseline."; exit 0; }; fi - if [ -f amber/dev-requirements.txt ]; then uv pip install --system --index-strategy unsafe-best-match -r amber/dev-requirements.txt || { echo "::warning::main dev-requirements install failed; skipping main baseline."; exit 0; }; fi - # Regenerate proto bindings against main's protos, then re-bench. - bash bin/python-proto-gen.sh || { echo "::warning::main proto-gen failed; skipping main baseline."; exit 0; } - if bash bin/run-benchmarks.sh && [ -f bench-results/arrow-flight-e2e.csv ]; then - cp bench-results/arrow-flight-e2e.csv bench-results-pr/arrow-flight-e2e-main.csv - printf '%s' "$BASE_SHA" > bench-results-pr/arrow-flight-e2e-main.commit.txt - echo "captured same-runner main baseline at $BASE_SHA" - else - echo "::warning::main baseline re-run failed; PR comment falls back to the gh-pages baseline." - fi - # trap restores the PR outputs (now incl. main CSV) plus original ref. - - - name: Stash PR number for downstream comment workflow - # PR fork workflows can't comment (GitHub forces read-only token); - # benchmarks-pr-comment.yml runs separately via workflow_run with - # proper write access, and needs the PR number to find the target. - # github.event.workflow_run.pull_requests is empty for fork PRs, - # so we ferry the number via artifact. - if: ${{ github.event_name == 'pull_request' && !cancelled() }} - env: - PR_NUMBER: ${{ github.event.pull_request.number }} - run: echo "$PR_NUMBER" > bench-results/pr-number.txt - - - name: Render bench summary - # Render the bench CSV into a markdown table on the workflow run - # page. Visible without further clicks — and doesn't need any - # extra permissions (writes to $GITHUB_STEP_SUMMARY only). - if: ${{ !cancelled() }} - run: | - { - echo "## Bench results (\`$BENCH_MODE\` mode)" - echo - if [ -f bench-results/arrow-flight-e2e.csv ]; then - echo '```csv' - cat bench-results/arrow-flight-e2e.csv - echo '```' - else - echo "_(no bench-results/arrow-flight-e2e.csv produced)_" - fi - } >> "$GITHUB_STEP_SUMMARY" - - - name: Upload bench artifacts - if: ${{ !cancelled() }} - uses: actions/upload-artifact@v7 - with: - name: bench-results-${{ github.run_id }} - path: bench-results/ - retention-days: 14 - - # Publish to the gh-pages dashboard. auto-push + save-data-file are - # gated on `schedule` ONLY: the daily full-grid run is the single - # authoritative baseline writer. PR *and* push-to-main runs only emit - # the job summary and the uploaded artifact, never touching the - # tracked baseline. This is deliberate: each gh-pages write is a bot - # commit (one per chart, so two per run), and persisting on every - # merge to main flooded the repo's Pulse / all-branches commit count - # with `github-action-benchmark` commits. The post-merge run still - # gives quick signal via the rendered summary + artifact; only the - # daily sweep persists. Adding a new benchmark = adding one publish - # block below matching the JSON filename convention in - # bin/run-benchmarks.sh. - # - # `skip-fetch-gh-pages: true` intentionally keeps baseline comparison - # OFF for now. When flipped to false, the action will fetch the stored - # gh-pages baseline, compare each run against main, post an alert - # comment when a result regresses past `alert-threshold`, and (with - # comment-on-alert / fail-on-alert) can block merge. We're deferring - # that until the baseline has accumulated enough daily data points to - # be trustworthy; turning it on is a deliberate follow-up to evaluate - # later. auto-push on the daily schedule still appends to the branch. - # - # `continue-on-error: true` keeps any other gh-pages-side surprise - # (permission glitches, transient git failures) from failing the - # bench job overall — the bench data itself is already in the - # uploaded artifact above. - - name: Publish throughput - if: ${{ !cancelled() }} - continue-on-error: true - uses: benchmark-action/github-action-benchmark@52576c92bccf6ac60c8223ec7eb2565637cae9ba # v1.22.1 - with: - name: Arrow Flight E2E Throughput - tool: customBiggerIsBetter - output-file-path: bench-results/arrow-flight-e2e-throughput.json - github-token: ${{ secrets.GITHUB_TOKEN }} - auto-push: ${{ github.event_name == 'schedule' }} - save-data-file: ${{ github.event_name == 'schedule' }} - skip-fetch-gh-pages: true - gh-pages-branch: gh-pages - benchmark-data-dir-path: dev/bench - alert-threshold: "150%" - # comment-on-alert needs pull-requests:write; skip and let - # results show up via summary-always instead. - comment-on-alert: false - summary-always: true - - name: Publish latency - if: ${{ !cancelled() }} - continue-on-error: true - uses: benchmark-action/github-action-benchmark@52576c92bccf6ac60c8223ec7eb2565637cae9ba # v1.22.1 - with: - name: Arrow Flight E2E Latency - tool: customSmallerIsBetter - output-file-path: bench-results/arrow-flight-e2e-latency.json - github-token: ${{ secrets.GITHUB_TOKEN }} - auto-push: ${{ github.event_name == 'schedule' }} - save-data-file: ${{ github.event_name == 'schedule' }} - skip-fetch-gh-pages: true - gh-pages-branch: gh-pages - benchmark-data-dir-path: dev/bench - alert-threshold: "150%" - comment-on-alert: false - summary-always: true diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 19e4953145c..5e3ada2c319 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -649,165 +649,6 @@ jobs: disable_search: true fail_ci_if_error: false -<<<<<<< HEAD -======= - platform-integration: - # Boot smoke test for the platform services (mirrors amber-integration: an - # independent, infra-provisioned integration job that builds on its own - # classpath — no artifact hand-off from the `platform` job, so the two run - # in parallel). Per-service matrix like `platform`: each service is launched - # from its freshly built dist and must reach a listening state without a - # runtime classpath/linkage crash (see #6220 and .github/scripts/smoke-boot.sh). - # postgres backs every service's JOOQ codegen + boot; file-service also - # fail-fasts on S3 / LakeFS during run(), so those are provisioned for it - # alone. Unit tests + coverage stay in `platform`. See #6273. - if: ${{ inputs.run_platform_integration }} - name: ${{ format('platform-integration{0} ({1})', inputs.job_name_suffix, matrix.service) }} - runs-on: ubuntu-latest - strategy: - fail-fast: false - matrix: - include: - - service: config-service - sbt_project: ConfigService - port: 9094 - - service: access-control-service - sbt_project: AccessControlService - port: 9096 - - service: file-service - sbt_project: FileService - port: 9092 - object_store: true - - service: computing-unit-managing-service - sbt_project: ComputingUnitManagingService - port: 8888 - - service: workflow-compiling-service - sbt_project: WorkflowCompilingService - port: 9090 - - service: notebook-migration-service - sbt_project: NotebookMigrationService - port: 9098 - env: - JAVA_OPTS: -Xms2048M -Xmx2048M -Xss6M -XX:ReservedCodeCacheSize=256M -Dfile.encoding=UTF-8 - JVM_OPTS: -Xms2048M -Xmx2048M -Xss6M -XX:ReservedCodeCacheSize=256M -Dfile.encoding=UTF-8 - services: - # Every platform service transitively depends on DAO, whose JOOQ code - # generation needs the live texera schema at compile time; the service - # also connects to it on boot. - postgres: - image: postgres - env: - POSTGRES_PASSWORD: postgres - ports: - - 5432:5432 - options: >- - --health-cmd="pg_isready -U postgres" - --health-interval=10s - --health-timeout=5s - --health-retries=5 - steps: - - name: Checkout - uses: actions/checkout@v7 - with: - ref: ${{ inputs.checkout_ref || github.sha }} - fetch-depth: 0 - - name: Prepare backport workspace - if: ${{ inputs.backport_target_branch != '' }} - working-directory: ${{ github.workspace }} - run: bash ./.github/scripts/prepare-backport-checkout.sh "${{ inputs.backport_target_branch }}" "${{ inputs.backport_commit_range }}" - - name: Setup JDK - uses: actions/setup-java@v5 - with: - distribution: "temurin" - java-version: 17 - - name: Setup sbt launcher - uses: sbt/setup-sbt@8feba82adc7f01ddcf8165b86f778bdb5b82cebc # v1.5.7 - - uses: coursier/cache-action@95e5b1029b6b86e7bac033ee44a0697d8a527d2d # v8.1.1 - with: - extraSbtFiles: '["*.sbt", "project/**.{scala,sbt}", "project/build.properties" ]' - - name: Create Databases - run: | - psql -h localhost -U postgres -f sql/texera_ddl.sql - psql -h localhost -U postgres -f sql/iceberg_postgres_catalog.sql - psql -h localhost -U postgres -f sql/texera_lakefs.sql - env: - PGPASSWORD: postgres - - name: Check ${{ matrix.sbt_project }} exists in this tree - # Same guard as the `platform` job. - id: module_check - env: - BACKPORT_TARGET_BRANCH: ${{ inputs.backport_target_branch }} - SBT_PROJECT: ${{ matrix.sbt_project }} - run: | - if [ -n "${BACKPORT_TARGET_BRANCH}" ] && ! grep -qE "^lazy val ${SBT_PROJECT}([[:space:]]|=)" build.sbt; then - echo "present=false" >> "$GITHUB_OUTPUT" - echo "::notice::${SBT_PROJECT} is not in this branch's build.sbt (branch cut before it existed); skipping" - else - echo "present=true" >> "$GITHUB_OUTPUT" - fi - - name: Build ${{ matrix.service }} dist - # Self-build on this job's own classpath (mirrors amber-integration - # compiling its own tests) rather than downloading an artifact from the - # `platform` job, so the two stay independent and run in parallel. No - # jacoco here — unit tests + coverage live in `platform`. - if: ${{ steps.module_check.outputs.present == 'true' }} - run: sbt "${{ matrix.sbt_project }}/dist" - - name: Unzip ${{ matrix.service }} dist - if: ${{ steps.module_check.outputs.present == 'true' }} - run: | - mkdir -p /tmp/dists - unzip -q ${{ matrix.service }}/target/universal/${{ matrix.service }}-*.zip -d /tmp/dists/ - - name: Start MinIO - # file-service's boot creates its S3 bucket via S3StorageClient; only - # that service needs an object store, so the rest of the matrix skips this. - if: ${{ matrix.object_store && steps.module_check.outputs.present == 'true' }} - run: | - docker run -d --name minio --network host \ - -e MINIO_ROOT_USER=texera_minio \ - -e MINIO_ROOT_PASSWORD=password \ - minio/minio:RELEASE.2025-02-28T09-55-16Z server /data - for i in $(seq 1 15); do - curl -sf http://localhost:9000/minio/health/live && break - echo "Waiting for MinIO... (attempt $i)"; sleep 1 - done - curl -sf http://localhost:9000/minio/health/live - - name: Start LakeFS - # lakeFS keeps its metadata in the texera_lakefs postgres DB (created - # above) and uses MinIO as its S3 blockstore. file-service's boot calls - # LakeFSStorageClient.healthCheck(), so this must be up for it to reach a - # listening state. Config mirrors bin/single-node (compose + .env), - # adapted to CI creds (postgres/postgres @ localhost). - if: ${{ matrix.object_store && steps.module_check.outputs.present == 'true' }} - run: | - docker run -d --name lakefs --network host \ - -e LAKEFS_DATABASE_TYPE=postgres \ - -e "LAKEFS_DATABASE_POSTGRES_CONNECTION_STRING=postgres://postgres:postgres@localhost:5432/texera_lakefs?sslmode=disable" \ - -e LAKEFS_BLOCKSTORE_TYPE=s3 \ - -e LAKEFS_BLOCKSTORE_S3_FORCE_PATH_STYLE=true \ - -e LAKEFS_BLOCKSTORE_S3_ENDPOINT=http://localhost:9000 \ - -e LAKEFS_BLOCKSTORE_S3_CREDENTIALS_ACCESS_KEY_ID=texera_minio \ - -e LAKEFS_BLOCKSTORE_S3_CREDENTIALS_SECRET_ACCESS_KEY=password \ - -e LAKEFS_AUTH_ENCRYPT_SECRET_KEY=random_string_for_lakefs \ - --entrypoint /bin/sh \ - treeverse/lakefs:1.51 \ - -c "lakefs setup --user-name texera-admin --access-key-id AKIAIOSFOLKFSSAMPLES --secret-access-key wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY || true; lakefs run" - for i in $(seq 1 20); do - curl -sf http://localhost:8000/api/v1/healthcheck && break - echo "Waiting for LakeFS... (attempt $i)"; sleep 1 - done - curl -sf http://localhost:8000/api/v1/healthcheck - - name: Smoke-test ${{ matrix.service }} boots - # Launch the packaged service and assert it reaches LISTEN on its port - # without a runtime classpath/linkage crash (#6220). - if: ${{ steps.module_check.outputs.present == 'true' }} - env: - TEXERA_HOME: ${{ github.workspace }} - # Quiet boot logs, same wiring as the amber jobs. Safe here: - # smoke-boot's verdict is LISTEN-based, never log-scraping (#6332). - TEXERA_SERVICE_LOG_LEVEL: ${{ runner.debug == '1' && 'DEBUG' || 'WARN' }} - run: .github/scripts/smoke-boot.sh "/tmp/dists/${{ matrix.service }}-*/bin/${{ matrix.service }}" "${{ matrix.port }}" - ->>>>>>> 7a2c8d03b (fix(deps, ci): bump sbt/setup-sbt to v1.5.7 to restore CI (#7710)) pyamber: if: ${{ inputs.run_pyamber }} strategy: diff --git a/.github/workflows/dependency-graph.yml b/.github/workflows/dependency-graph.yml deleted file mode 100644 index 4034b15715f..00000000000 --- a/.github/workflows/dependency-graph.yml +++ /dev/null @@ -1,58 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -# Submits the resolved sbt dependency graph (all modules, transitive -# dependencies included) to GitHub via the Dependency Submission API so -# that Dependabot alerts cover Scala dependencies, which GitHub cannot -# parse from build.sbt on its own. -name: Update Dependency Graph -on: - push: - branches: - - main - -permissions: {} - -jobs: - dependency-graph: - name: Update Dependency Graph - runs-on: ubuntu-latest - if: github.repository == 'apache/texera' - permissions: - # The Dependency Submission API requires write permission - # on the repository - contents: write - steps: - - uses: actions/checkout@v7 - with: - persist-credentials: false - - - name: Setup JDK - uses: actions/setup-java@v5 - with: - distribution: 'temurin' - java-version: 17 - - - name: Setup sbt launcher - uses: sbt/setup-sbt@8feba82adc7f01ddcf8165b86f778bdb5b82cebc # v1.5.7 - - - uses: coursier/cache-action@95e5b1029b6b86e7bac033ee44a0697d8a527d2d # v8.1.1 - - - name: Submit sbt dependency graph - uses: scalacenter/sbt-dependency-submission@d84eef4c09e633bcf5f113bcad7fd5e9af1baee9 # v3.2.3 - with: - configs-ignore: provided optional test compile-internal runtime-internal scala-tool scala-doc-tool