Skip to content

[VPEX] localenv: target a specific job task (--job-task <job-id>.<task-key>)#6048

Open
rugpanov wants to merge 1 commit into
mainfrom
dbconnect/job-task
Open

[VPEX] localenv: target a specific job task (--job-task <job-id>.<task-key>)#6048
rugpanov wants to merge 1 commit into
mainfrom
dbconnect/job-task

Conversation

@rugpanov

Copy link
Copy Markdown
Contributor

What

Replace the whole-job --job-id <id> flag with a task-scoped --job-task <job-id>.<task-key> (the task key is required).

Why

A single job can bind multiple tasks to different serverless environment versions (see the Workflows notebook-environments PRD). So a bare job ID is ambiguous — it can't resolve to one Python/serverless environment. Targeting a specific task removes the ambiguity: the CLI reads that task's environment directly.

This implements spec task D5 ([P0] CLI Changes.md, cli-spec-internal.md §2.2/§2.3/§4.3).

Behavior

  • --job-task <job-id>.<task-key> resolves the named task. The value splits on the first dot, so task keys may themselves contain dots.
  • Serverless task → version read directly from the environment it binds (environment_keyspec version). The job path no longer applies the serverless-vN default fallback — that now belongs solely to the bundle path. A task whose environment records no version is E_RESOLVE, not a guess.
  • Classic task → resolved from its own new_cluster (or existing_cluster_id, via the Clusters API).
  • Bare --job-task <job-id> (no task key) → E_USAGE, listing the job's task keys. Because listing requires a Jobs API call, this E_USAGE is emitted at the resolve phase; the error-code annotation in result.go is updated to note E_USAGE can also come from resolve for this case.
  • Unknown task keyE_RESOLVE, also listing the available keys.

The ComputeClient seam gains JobTaskEnvironment (replacing GetJobSparkVersion) and a typed ErrTaskKeyRequired, so ResolveTarget can distinguish the missing-key usage error from a genuine resolve failure.

Tests

  • job-classic-check / job-serverless-check now bind a task and resolve it.
  • The old whole-job ambiguity cases (both-compute, multi-cluster mismatch, serverless-version mismatch) are obsolete under task scoping and are replaced by job-task-missing-key (E_USAGE), job-task-unknown (E_RESOLVE), and job-task-unpinned (E_RESOLVE, no fallback).
  • serverless-default-check removed: the job path no longer defaults; the bundle default remains covered by TestResolveBundleServerless.
  • Unit tests updated for the new interface + split/enumerate semantics. Build, unit, acceptance (localenv/help), lint, deadcode all pass.

Still Hidden; part of the pre-unhide hardening of environments setup-local.

This pull request and its description were written by Isaac.

Replace the whole-job --job-id flag with a task-scoped
--job-task <job-id>.<task-key>. A job can bind multiple tasks to different
serverless environment versions (Workflows notebook-environments PRD), so a bare
job ID cannot resolve to a single environment; the target must be a specific
task.

Behavior (per [P0] CLI Changes spec, task D5):
- --job-task <job-id>.<task-key> resolves the named task's compute. The value is
  split on the FIRST dot, so task keys may contain dots.
- A serverless task's environment version is read directly from the environment
  it binds (environment_key -> spec version), so the job path no longer applies
  the serverless-vN default fallback — that now belongs solely to the bundle
  path. A task whose environment records no version is an E_RESOLVE, not a guess.
- A classic task resolves from its own new_cluster (or existing_cluster_id, via
  the Clusters API).
- A bare --job-task <job-id> (no task key) is E_USAGE, listing the job's task
  keys so the user can pick one. Because listing requires a Jobs API call, this
  E_USAGE is emitted at the resolve phase; the error-code annotation in result.go
  is updated to note E_USAGE can also come from resolve for this case.
- An unknown task key is E_RESOLVE, also listing the available keys.

The ComputeClient seam gains JobTaskEnvironment (replacing GetJobSparkVersion)
and a typed ErrTaskKeyRequired so ResolveTarget can classify the missing-key case
as E_USAGE vs a genuine E_RESOLVE.

Acceptance tests reworked: job-classic-check and job-serverless-check now bind a
task; the old whole-job ambiguity cases (both-compute, multi-cluster mismatch,
serverless-version mismatch) are obsolete under task scoping and are replaced by
job-task-missing-key (E_USAGE), job-task-unknown (E_RESOLVE), and
job-task-unpinned (E_RESOLVE, no fallback). serverless-default-check is removed:
the job path no longer defaults, and the bundle default remains unit-tested.

Co-authored-by: Isaac
@github-actions

Copy link
Copy Markdown
Contributor

Approval status: pending

/acceptance/localenv/ - needs approval

31 files changed
Eligible: @rclarey, @anton-107, @misha-db

/cmd/environments/ - needs approval

Files: cmd/environments/compute.go, cmd/environments/sync.go
Eligible: @rclarey, @anton-107, @misha-db

/libs/localenv/ - needs approval

Files: libs/localenv/result.go, libs/localenv/target.go, libs/localenv/target_test.go
Eligible: @rclarey, @anton-107, @misha-db

Any maintainer (@andrewnester, @anton-107, @denik, @pietern, @shreyas-goenka, @simonfaltum, @renaudhartert-db, @janniklasrose, @lennartkats-db) can approve all areas.
See OWNERS for ownership rules.

@rugpanov
rugpanov requested a review from misha-db July 23, 2026 16:35
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: d0a7690

Run: 30025629102

Env 💚​RECOVERED 🙈​SKIP ✅​pass 🙈​skip Time
💚​ aws linux 4 4 322 1058 5:44
💚​ aws windows 4 4 324 1056 4:49
💚​ azure linux 4 4 322 1057 5:57
💚​ azure windows 4 4 324 1055 4:34
💚​ gcp linux 1 5 321 1059 5:18
💚​ gcp windows 1 5 323 1057 5:06
8 interesting tests: 4 RECOVERED, 4 SKIP
Test Name aws linux aws windows azure linux azure windows gcp linux gcp windows
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🙈​ TestAccept/bundle/invariant/no_drift 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/ssh/connection 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
💚​ TestFetchRepositoryInfoAPI_FromRepo 💚​R 💚​R 💚​R 💚​R 🙈​S 🙈​S
💚​ TestFetchRepositoryInfoAPI_FromRepo/root 💚​R 💚​R 💚​R 💚​R
💚​ TestFetchRepositoryInfoAPI_FromRepo/subdir 💚​R 💚​R 💚​R 💚​R
Top 3 slowest tests (at least 2 minutes):
duration env testname
3:58 gcp windows TestAccept
3:18 aws windows TestAccept
3:14 azure windows TestAccept

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants