Skip to content

ateapi: pair ate.workerpool.namespace with ate.workerpool.name - #953

Open
Jeff Luo (JeffLuoo) wants to merge 2 commits into
agent-substrate:mainfrom
JeffLuoo:fix/workerpool-namespace-label
Open

ateapi: pair ate.workerpool.namespace with ate.workerpool.name#953
Jeff Luo (JeffLuoo) wants to merge 2 commits into
agent-substrate:mainfrom
JeffLuoo:fix/workerpool-namespace-label

Conversation

@JeffLuoo

@JeffLuoo Jeff Luo (JeffLuoo) commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator

ate.actor.crashes, ate.actor.lifecycle.operation.duration and ate.scheduler.assignment.duration named a WorkerPool by name alone. A WorkerPool is namespaced, so same-named pools in different namespaces merged into one series, and the three could not join the instruments that already carry both keys.

ateattr.WorkerPoolAttributes now builds the pair, and omits both keys when no pool is assigned, so a crash before the actor reached a worker no longer reports an empty-string pool. This changes the series identity of the three instruments.

Fixes #951

It's a good idea to open an issue first for discussion.

  • Tests pass
  • Appropriate changes to documentation are included in the PR

Fixes agent-substrate#951.

ate.actor.crashes, ate.actor.lifecycle.operation.duration and
ate.scheduler.assignment.duration named a WorkerPool by name alone. A
WorkerPool is namespaced, so same-named pools in different namespaces
merged into one series, and the three could not join the instruments
that already carry both keys.

ateattr.WorkerPoolAttributes now builds the pair, and omits both keys
when no pool is assigned, so a crash before the actor reached a worker
no longer reports an empty-string pool. This changes the series identity
of the three instruments.
@JeffLuoo
Jeff Luo (JeffLuoo) force-pushed the fix/workerpool-namespace-label branch from 44a2259 to e3c36b6 Compare August 14, 2026 14:45
@JeffLuoo

Copy link
Copy Markdown
Collaborator Author

cc - Krisztian F (@krisztianfekete)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added some minor comments, otherwise LGTM!

Comment thread internal/ateattr/ateattr.go Outdated
Comment thread docs/observability.md Outdated
Comment on lines +189 to +190
ass := actor.GetWorkerAssignment()
attrs = append(attrs, ateattr.WorkerPoolAttributes(ass.GetWorkerNamespace(), ass.GetWorkerPool())...)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't have to be in this PR, can be a follow-up, but suspend and pause reassign actor to the finalized record whose WorkerAssignment was just nil'd, so the pool pair lands only on failed suspends, leaving resume the only one of five operations where it's reliable.

Should be fixable like this:

// Snapshot crash attributes before pod and pool pointers are cleared below;
// the counter itself is emitted only after the transition commits.
crashAttrs := ateattr.ActorMetricAttributes(actor, sandboxClass, opName, reason)
?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can create a new issue to track it later.

WorkerPoolAttributes guarded only the empty name, so a name without a
namespace still emitted the pair with an empty namespace -- the same
merged series the pair exists to prevent, and one that joins to nothing.
Both keys are now set together or not at all.

Also separate the two states the docs ran together: an absent pair means
the operation has no pool, while the empty-valued pair on
ate.scheduler.eligible_workers means no pool matched the constraints.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Metrics: ate.workerpool.name is emitted without ate.workerpool.namespace on three metrics

2 participants