[VPEX] localenv: target a specific job task (--job-task <job-id>.<task-key>)#6048
Open
rugpanov wants to merge 1 commit into
Open
[VPEX] localenv: target a specific job task (--job-task <job-id>.<task-key>)#6048rugpanov wants to merge 1 commit into
rugpanov wants to merge 1 commit into
Conversation
Replace the whole-job --job-id flag with a task-scoped --job-task <job-id>.<task-key>. A job can bind multiple tasks to different serverless environment versions (Workflows notebook-environments PRD), so a bare job ID cannot resolve to a single environment; the target must be a specific task. Behavior (per [P0] CLI Changes spec, task D5): - --job-task <job-id>.<task-key> resolves the named task's compute. The value is split on the FIRST dot, so task keys may contain dots. - A serverless task's environment version is read directly from the environment it binds (environment_key -> spec version), so the job path no longer applies the serverless-vN default fallback — that now belongs solely to the bundle path. A task whose environment records no version is an E_RESOLVE, not a guess. - A classic task resolves from its own new_cluster (or existing_cluster_id, via the Clusters API). - A bare --job-task <job-id> (no task key) is E_USAGE, listing the job's task keys so the user can pick one. Because listing requires a Jobs API call, this E_USAGE is emitted at the resolve phase; the error-code annotation in result.go is updated to note E_USAGE can also come from resolve for this case. - An unknown task key is E_RESOLVE, also listing the available keys. The ComputeClient seam gains JobTaskEnvironment (replacing GetJobSparkVersion) and a typed ErrTaskKeyRequired so ResolveTarget can classify the missing-key case as E_USAGE vs a genuine E_RESOLVE. Acceptance tests reworked: job-classic-check and job-serverless-check now bind a task; the old whole-job ambiguity cases (both-compute, multi-cluster mismatch, serverless-version mismatch) are obsolete under task scoping and are replaced by job-task-missing-key (E_USAGE), job-task-unknown (E_RESOLVE), and job-task-unpinned (E_RESOLVE, no fallback). serverless-default-check is removed: the job path no longer defaults, and the bundle default remains unit-tested. Co-authored-by: Isaac
Contributor
Approval status: pending
|
Collaborator
Integration test reportCommit: d0a7690
8 interesting tests: 4 RECOVERED, 4 SKIP
Top 3 slowest tests (at least 2 minutes):
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Replace the whole-job
--job-id <id>flag with a task-scoped--job-task <job-id>.<task-key>(the task key is required).Why
A single job can bind multiple tasks to different serverless environment versions (see the Workflows notebook-environments PRD). So a bare job ID is ambiguous — it can't resolve to one Python/serverless environment. Targeting a specific task removes the ambiguity: the CLI reads that task's environment directly.
This implements spec task D5 (
[P0] CLI Changes.md,cli-spec-internal.md§2.2/§2.3/§4.3).Behavior
--job-task <job-id>.<task-key>resolves the named task. The value splits on the first dot, so task keys may themselves contain dots.environment_key→specversion). The job path no longer applies theserverless-vNdefault fallback — that now belongs solely to the bundle path. A task whose environment records no version isE_RESOLVE, not a guess.new_cluster(orexisting_cluster_id, via the Clusters API).--job-task <job-id>(no task key) →E_USAGE, listing the job's task keys. Because listing requires a Jobs API call, thisE_USAGEis emitted at the resolve phase; the error-code annotation inresult.gois updated to noteE_USAGEcan also come from resolve for this case.E_RESOLVE, also listing the available keys.The
ComputeClientseam gainsJobTaskEnvironment(replacingGetJobSparkVersion) and a typedErrTaskKeyRequired, soResolveTargetcan distinguish the missing-key usage error from a genuine resolve failure.Tests
job-classic-check/job-serverless-checknow bind a task and resolve it.job-task-missing-key(E_USAGE),job-task-unknown(E_RESOLVE), andjob-task-unpinned(E_RESOLVE, no fallback).serverless-default-checkremoved: the job path no longer defaults; the bundle default remains covered byTestResolveBundleServerless.localenv/help), lint, deadcode all pass.Still
Hidden; part of the pre-unhide hardening ofenvironments setup-local.This pull request and its description were written by Isaac.