Skip to content

Report the app name with command stats - #5136

Open
clouvet wants to merge 1 commit into
masterfrom
command-stats-app-name
Open

Report the app name with command stats#5136
clouvet wants to merge 1 commit into
masterfrom
command-stats-app-name

Conversation

@clouvet

@clouvet clouvet commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator

Follows #5134, which populated app_id/org_id on command/stats from token scope.

The gap

config.ScopedIDs only names an org when every permission macaroon we hold agrees on one. Tokens loaded from the config file fan out to one macaroon per org the user belongs to (fetchOrgTokens), so an interactive user in more than one org reports nothing rather than guessing. Since everyone has a personal org alongside any org they work in, that's nearly every human at a business — exactly the population we most want to attribute.

Worth being precise: this is a property of the credential, not the person. fetchOrgTokens early-returns when t.FromFile() == "", so a token supplied through the environment never fans out. Tightly scoped tokens used for CI and agentic work already report both IDs today and are unaffected by this change.

The change

Send the app name rather than trying harder to resolve an ID on the client.

RequireAppName already resolves it from --app / FLY_APP / fly.toml with no I/O at all, and app names are unique, so a name identifies an app and its org regardless of how many macaroons we're holding. The warehouse already resolves names this way for deploys and launches (against raw_web_pg.apps and base__flyio__organizations), so this needs no new lookup anywhere.

Coverage is the 40 app-scoped command packages — including status, logs, ssh, scale, machine, volumes, ips, certificates and config, where command/stats is the only event covering the invocation.

Why this complements the IDs rather than replacing them

  • The token-scoped IDs are first-hand and accurate as of the moment the command ran. A name resolves to whichever org owns the app today, which drifts if an app moves orgs.
  • Roughly half our command packages aren't app-scoped at all. For those there is no name to send, and an org-scoped token remains the only attribution available.

Resolution downstream should prefer the event's own id and fall back to the name — the two-step pattern the existing staging models already use, rather than coalescing output columns.

Alternatives considered

  • A GetApp call in a shared preparer. Adds a network round trip to the highest-volume event we have (~2.09M command invocations/day vs ~50k deploys/day) and a network failure mode to RequireAppName, which today does no I/O.
  • Decorating FlapsClient.GetApp. Free, but only covers commands that already fetch an app — which overlap with deploy and launch, and notably exclude status/logs/ssh. Doesn't fix the multi-org case at all.
  • Having the server return the authorizing org on responses. Would give a definitive answer, but needs changes in three repos, puts telemetry on the flaps auth hot path, and risks becoming an oracle if it's ever emitted on a failed or unauthorized response.

Follow-ups (not in this PR)

  • flyctl-metrics must restore the sendToRudderstack("flyctl_command_stats", …) block — command/stats is currently accepted but not forwarded, so nothing on this payload reaches Snowflake yet.
  • A stg_rudderstack__flyctl_commands dbt model doing the same deduped, id-first resolution as the deploys/launches models. Name joins against raw_web_pg.apps must be deduped (a deleted app frees its name) or they silently fan out.
  • Consider an org_id_source provenance column. Attribution mechanism correlates with interactive-vs-automation use, so an unlabelled mix will quietly bias any comparison across those segments.

Testing

go build ./internal/..., go vet, go test ./internal/metrics/... ./internal/command/ ./internal/config/... all pass; golangci-lint v2.11.3 (the pinned CI version) reports 0 issues.

🤖 Generated with Claude Code

The app_id and org_id added to command stats in #5134 come from token
scope, which only names an org when every permission macaroon we hold
agrees on one. Tokens loaded from the config file fan out to one macaroon
per org the user belongs to (fetchOrgTokens), so an interactive user in
more than one org reports nothing rather than guessing. Since everyone
has a personal org alongside any org they work in, that is nearly every
human at a business -- exactly the population we most want to attribute.

Send the app name instead of trying harder to resolve an ID on the
client. RequireAppName already resolves it from the --app flag, FLY_APP
or fly.toml with no I/O at all, and app names are unique, so a name
identifies an app and its org regardless of how many macaroons we are
holding. The warehouse already resolves names this way for deploys and
launches, against raw_web_pg.apps and base__flyio__organizations, so
this needs no new lookup anywhere.

This complements the token-scoped IDs rather than replacing them. The
IDs stay first-hand and are accurate as of the moment the command ran,
where a name resolves to whichever org owns the app today; they also
cover commands that have no app name at all. Roughly half our command
packages are not app-scoped, and for those an org-scoped token remains
the only attribution available. Resolution should prefer the event's own
id and fall back to the name, which is the pattern the existing staging
models already use.

Note that this is a property of the credential, not the person: a token
supplied through the environment skips the org fan-out entirely, so
tightly scoped tokens used for CI and agentic work already report both
IDs today and are unaffected by this.

The field is omitempty, so a command with no app sends what it sends
today.

Co-authored-by: Sprite <noreply@sprites.dev>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant