Skip to content

feat(api): API tokens v2 (personal + service accounts, scopes) and the /v1 versioned API (E1.3) - #332

Open
hoangsnowy wants to merge 5 commits into
mainfrom
claude/e1-3-api-tokens
Open

hoangsnowy wants to merge 5 commits into
mainfrom
claude/e1-3-api-tokens

Conversation

@hoangsnowy

Copy link
Copy Markdown
Owner

Change description

E1.3 from docs/plans/enterprise-readiness.md (scorecard rows 4–5).

API tokens and service accounts. Tokens built on the existing aos_ token (#212):

  • Personal tokens for every member. They act as their owner (sub = the owner's user id) and are revoked when the member is removed, before the account is deleted (GAP-04 order). Service accounts are admin-only workspace identities (sub = apitoken:<id>, name service:<name>).
  • Scopes read / run / admin, closed upwards (admin includes run, run includes read). ApiTokenScopeMiddleware enforces them per HTTP method: read covers GET/HEAD/OPTIONS, and anything else needs run. The role claim enforces the admin scope (admin scope ⇒ admin role, never platform-admin). A token without the needed scope gets a 403 problem+json response.
  • Storage: HMAC-SHA256 with a per-token 16-byte random salt, compared in constant time; an unknown prefix costs the same hash work. Older unsalted rows still validate. Migration AddApiTokenScopesAndSalt backfills scopes from the old role.
  • Expiry is capped by a new platform setting Tenants:ApiTokens:MaxLifetimeDays (a SettingDescriptor, default 366, range 1–1825, per ADR-0011; no appsettings key).
  • Audit: apitoken.created / revoked / used / refused. Use and refusal rows are written at most once per token per 5 minutes, with IP and user agent, so a stale CI secret can't flood the trail. Every use is also logged (token id and prefix only).
  • Self-contained auth registration (so it won't collide with GAP-13's fallback policy): services.AddApiTokenAuthentication<JwtBearerOptions>(scheme) adds the scheme and chains the JWT ForwardDefaultSelector, and app.UseApiTokenScopes() adds the scope check. Neither touches the default scheme or any authorization policy.
  • Desktop: Settings › API tokens now opens to every member (personal tokens). Admins can also create service accounts and admin-scoped tokens and see / revoke every token. The expiry choices follow the max-lifetime setting. /tenants/me now returns authMethod and scopes.

/v1 versioned API.

  • ModuleLoader.MapVersionedModuleEndpoints serves every module REST route under /v1 and again at its unversioned path. The unversioned twin is deprecated: it answers with Deprecation: @<unix> (RFC 9745), Sunset (RFC 8594, 2027-05-03), Link: rel="successor-version" and rel="deprecation" headers. Its endpoint names are stripped (via a Finally convention) so names stay unique, and it is excluded from OpenAPI.
  • SignalR hubs (new IProtocolEndpointModule marker), /mcp and the probes stay unversioned.
  • OpenAPI document v1 (title, bearer scheme, policy) is committed as docs/api/openapi-v1.json. A test fails when the running Api drifts from it (regenerate with AGENTOS_UPDATE_OPENAPI=1). Live /openapi/v1.json and Scalar stay non-Production, per the hardening checklist.
  • docs/API.md covers versions, authentication, scopes, a GitHub Actions example and the deprecation policy: additive-only within a version, six months' minimum notice, CHANGELOG Deprecated / Removed entries.

VS Code extension: new commands Set / Check / Clear API token (stored in VS Code SecretStorage, checked against /v1/tenants/me) and Run the pipeline on a user story (POST /v1/pipeline), plus a new agentos.apiUrl setting.

Docs: API.md (new), CONFIG.md, SECURITY.md, hardening checklist, scorecard rows 4–5 plus the shipped table, CHANGELOG (Added / Deprecated / Migration notes), README API section, extension README, CLAUDE.md wiring sentence.

Type of change

  • New feature (new agent / endpoint / capability)
  • Bugfix
  • Refactor (no behavior change)
  • Documentation / README
  • Infrastructure / CI configuration

Related issues

Enterprise readiness E1.3 (scorecard #4 Machine access, #5 API contract).

Checklist

  • dotnet build passes locally in Release mode (dotnet build AgentOs.slnx -c Release: 0 errors, on the official 10.0.402 SDK)
  • dotnet test passes (dotnet test --solution AgentOs.slnx -c Release: 3195 total, 0 failed, 97 skipped = the gated E2E/lab suites)
  • No secrets committed (API keys, connection strings)
  • README / docs updated if public behavior changed
  • Measurable results provided when related to agent output quality (n/a)

Test plan

Automated (in this PR)

  • ApiTokenTests: format; salted HMAC vs legacy SHA-256; constant-time compare; scope normalisation and the role mapping; method → scope; create / validate / revoke / expire; legacy-row validation; the max-lifetime setting (default cap + custom cap); members can't create service accounts or the admin scope; who can list and revoke what; RevokeForOwnerAsync; the audit trail (created / used / refused / revoked, with use and refusal throttled); claims for personal vs service tokens; the forward selector.
  • ApiV1IntegrationTests boots the real Api host with WebApplicationFactory (Development, no DB; token store = the real ApiTokenService on EF InMemory). It covers: token auth next to the JWT scheme; the tenant and scopes in /v1/tenants/me; a personal token acting as its owner; a read token gets 403 on POST; a run token reaches the endpoint; the admin route needs the admin scope; revoked, expired, unknown and malformed tokens get 401 with invalid_token; no credentials gets 401; legacy routes return 200 with Deprecation / Sunset / Link (even on a 401) and /v1 returns none; probes aren't deprecated; /v1 endpoint names are unique, legacy twins are unnamed, and the hub is unversioned; OpenAPI documents only /v1 and matches the committed copy.
  • MemberRemovalTests: personal tokens are revoked before the Keycloak delete; a failed token revoke removes nothing.

Manual, run in this session (Web + Api standalone against a local Postgres 16)

  • Migration applied on real Postgres (new columns and index verified with \d).
  • Desktop as the dev admin: created a service account (run scope) and a personal token (read scope, 30 days); shown once; copied.
  • Real Api results:
    • service token on GET /v1/tenants/me → 200 (service:…, scopes read run); legacy /tenants/me → 200 with the deprecation headers.
    • service token on POST /v1/requirement {} → 400 (got past auth); on the admin route → 403.
    • personal token on POST → 403 Insufficient token scope.
    • after Revoke in the UI → 401.
  • tenants.audit_events held created ×2, used ×2 (with IP), revoked, refused; rows are salted (32-hex salt) and the personal row carries OwnerUserId.
  • Screenshots in light and dark (Settings › API tokens: create, list, revoked state).

Still to verify on the lab / full Aspire stack (persistence, auth and tenancy, so a standalone run isn't proof)

  • E1.3 acceptance: a CI pipeline (GitHub Actions on the lab) triggers a run with a service-account token via POST /v1/pipeline, and the run shows under the tenant.
  • Real Keycloak: a non-admin member sees only "Personal" and Read/Run, sees and revokes only their own tokens; an admin sees all.
  • ApiTokensRealAuthTests (updated for /v1 and service:<name>) green on the full stack (RUN_AGENTOS_E2E_REAL=true).
  • Two tenants: a token of tenant A never reads tenant B (RLS on); member removal in the Users app revokes that member's personal tokens (audit row apiTokensRevoked=N).
  • Settings › Platform › Safety shows API token max lifetime (days); lowering it shrinks the expiry choices.
  • Keycloak JWT callers still work unchanged on /v1/* and on the legacy routes (forward-selector chaining).
  • VS Code extension against the lab Api: Set API token → Check → Run the pipeline.

Notes / follow-ups

  • Claude Design: Claude Design auth wasn't available in this non-interactive session (the design connectors failed to connect), so the changed settings-api-tokens screen (Type and Scope selects, the workspace-wide list) still needs pushing up with /design-login → DesignSync write_files.
  • A personal token's role comes from its scope, not the owner's current roles: if an admin is demoted, their admin-scoped personal tokens keep working until revoked (removal does revoke them). Re-checking against RBAC v2 / SCIM is a candidate for E1.2 / E2.1.
  • Pre-existing, not changed here: at 390 px the Settings hub keeps its sidebar (the Users page behaves the same), so it isn't a full-screen sheet yet.

Generated by Claude Code

…e /v1 versioned API (E1.3)

API tokens
- Personal tokens for every member (act as their owner) and service accounts for
  admins; scopes read / run / admin, closed upwards, enforced per HTTP method by
  ApiTokenScopeMiddleware and by the role claim (admin scope => admin role, never
  platform-admin).
- Stored as HMAC-SHA256 with a per-token random salt, compared in constant time;
  legacy unsalted rows still validate. Migration AddApiTokenScopesAndSalt backfills
  scopes from the old role.
- Expiry capped by the platform setting Tenants:ApiTokens:MaxLifetimeDays (default
  366). Create / revoke / use / refusal audited (use + refusal at most once per token
  per 5 minutes, with IP + user agent). Member removal revokes the member's personal
  tokens before deleting the account.
- Self-contained registration: AddApiTokenAuthentication<TPrimaryOptions>() chains
  the JWT forward selector; UseApiTokenScopes(). No default-scheme or policy changes.
- Settings > API tokens opens to every member (personal tokens); admins also create
  service accounts and see / revoke every token. /tenants/me reports the auth method
  and scopes.

/v1
- Every module REST route is served under /v1 and at its unversioned path, which is
  deprecated (Deprecation / Sunset / Link successor-version headers, sunset
  2027-05-03) and left out of OpenAPI. SignalR hubs (IProtocolEndpointModule), /mcp
  and the probes stay unversioned.
- OpenAPI document v1 with title, bearer scheme and policy; committed as
  docs/api/openapi-v1.json and kept in sync by a test.
- docs/API.md: versions, auth, scopes, CI example, deprecation policy.

VS Code extension: Set / Check / Clear API token (SecretStorage) and Run the pipeline
via POST /v1/pipeline.

Tests: token lifecycle, scopes, salting, max lifetime, audit, claims; integration on
the real Api host (WebApplicationFactory): token auth next to JWT, scope per method,
admin route, revoked / expired -> 401, /v1 + deprecated twins with headers, unique
endpoint names, hub unversioned, OpenAPI = committed copy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Twi9jFk8LiG81JBRi6Ddh
…-select)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Twi9jFk8LiG81JBRi6Ddh
@hoangsnowy
hoangsnowy marked this pull request as ready for review November 3, 2026 15:48
…/e1-3-api-tokens

Docs conflicts resolved keeping both sides: CHANGELOG + SECURITY bullets, migration
notes, scorecard rows (RBAC row from #328, machine access + API contract from E1.3).
#328 adds no Api routes (approvals are decided in the desktop), so nothing new needs a
/v1 twin; its Approver policy works unchanged for JWT callers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Twi9jFk8LiG81JBRi6Ddh
… into claude/e1-3-api-tokens

Conflicts resolved keeping both sides:
- ApiTokensApp: GAP-17 fail-closed tenant (no tenant claim => alert, never "default")
  kept on top of personal tokens for every member + scopes.
- AgentOs.Tests.csproj: both Mvc.Testing (E1.3) and TestHost (#329).
- CHANGELOG migration notes: #329's deploy notes + the API-token note.
docs/api/openapi-v1.json regenerated: #334 adds optional tracker fields + the
TrackerBinding schema to /v1/workspaces (additive only). #329's new routes (sign-out,
exports, pairing) live on the Web host, not the versioned Api.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Twi9jFk8LiG81JBRi6Ddh
…into claude/e1-3-api-tokens

Kept both sides:
- The API-token scheme registers alongside #331's FallbackPolicy (RequireAuthenticatedUser):
  a token principal is authenticated, so it passes; the explicit AllowAnonymous routes stay
  public on both twins (/tenants/register, /tenants/invitations/preview and their /v1 twins
  added to the reviewed anonymous inventory — nothing else becomes public).
- #331's tenant + user limiter partitions a token principal exactly like a JWT (tenant; a
  personal token shares its owner's user bucket, a service account is its own user).
- Tests: one Api reference with #331's ApiHost alias, one Mvc.Testing package version.
- README API intro combines #331's public-route list and the /v1 + API-token wording.
New tests: limiter keys for token principals == JWT; fallback policy with tokens; public
routes stay public on both twins.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Twi9jFk8LiG81JBRi6Ddh
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants