Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,12 @@ Everything on `main` before the first tagged release. AgentOS.NET has not shippe
without a tenant claim is refused instead of falling back to the `default` tenant.
- Sign-out is a POST with an antiforgery token, the VS Code approve form is antiforgery-protected, and plugins load
only when their SHA-256 is listed in `Plugins:AllowedSha256` (GAP-19).
- Invitations are single-use, bound to the invited address and revocable from Settings › Users (GAP-11); the
audit trail is append-only for the application's database role and hash-chained per tenant, with **Verify
chain** in Settings › Activity log (GAP-11b).
- Row-level security stage 2 (GAP-14): a connection with no tenant sees nothing; settings, the audit trail and
invitations are under RLS; cross-tenant platform work runs in explicit, counted platform scopes; a failed RLS
bootstrap stops the host outside Development and `/health` reports `row-level-security`.

### Migration notes

Expand All @@ -128,5 +134,19 @@ Everything on `main` before the first tagged release. AgentOS.NET has not shippe
[docs/CONFIG.md](docs/CONFIG.md#set-at-deploy-bootstrap).
- The `approver` realm role (E1.5) is created by the azd post-deploy hook and the Helm realm job; a local realm
imported before it needs the role added through the Keycloak admin API, or its volume dropped.
- Invitation links minted before invitations became single-use rows stop working — send new ones.
- Audit events recorded before the hash chain stay readable but are reported as "not chained"; the chain starts
with the next event.
- **Row-level security stage 2 is not a rolling upgrade.** The first new replica rewrites the database policies when
it boots; a replica of the previous version still running then reads platform settings, model keys and scheduled
work as nothing, and the audit rows it writes have no chain link (**Verify chain** reports a break there). Stop
every old Api / Web replica before the first new one starts — a stop-then-start cut-over: compose recreates
single containers that way already; on Helm scale the `api` / `web` Deployments to 0 before `helm upgrade`; on
Azure Container Apps deactivate the active revisions before `azd deploy`. Steps in
[docs/ONPREM.md](docs/ONPREM.md#upgrading-to-row-level-security-stage-2--stop-the-old-version-first) and
[docs/DEPLOYMENT.md](docs/DEPLOYMENT.md#a4--subsequent-deploys).
- Outside Development the hosts now refuse to start when the row-level-security bootstrap fails (the database user
needs `CREATEROLE`). Tests or scripts that delete audit rows will make **Verify chain** report a break — that is
the point; don't clean audit rows up.

[Unreleased]: https://github.com/hoangsnowy/AgentOS.NET/commits/main
5 changes: 4 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -455,7 +455,10 @@ Roles: `admin` (tenant admin), `member`, and `platform-admin` (platform-wide set

Row-level isolation: pipeline runs, run metrics and orchestrations, sessions/runners and workspaces
carry a `TenantId` with an EF global query filter on `ITenantContext.TenantId`; writes stamp the
tenant. Settings and keys are stored per `(tenant, key)` and always read for an explicit tenant.
tenant. Settings and keys are stored per `(tenant, key)` and always read for an explicit tenant. Below the
application, Postgres row-level security binds every connection to its tenant on every tenant table — settings,
audit trail and invitations included — and a connection with no tenant sees nothing
([ADR-0012](docs/adr/0012-postgres-row-level-security.md)).

## API

Expand Down
11 changes: 8 additions & 3 deletions SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,11 +46,16 @@ When reporting, include: affected component, reproduction steps, and impact.
- **Audit trail** — every settings change (old → new, secrets masked), every direct config write (LLM keys,
tokens, prompts, tool policy, budget — recorded as *set / cleared*, never the value), member lifecycle,
invitations and sign-in land in an append-only audit log, viewable in **Settings → Audit log** and
exportable as formula-safe CSV. Platform-scope changes are visible to platform admins only.
exportable as formula-safe CSV. Platform-scope changes are visible to platform admins only. The application's
database role may only add and read audit rows (no update / delete), and each tenant's events form a SHA-256
hash chain that **Verify chain** checks — it names the first event that was changed or removed.
- **Invitations** — single-use, accepted only for the address they were sent to, revocable by a tenant admin until
used, and expiring after 1–168 hours (default 24); refused attempts are audited.
- **Tenant isolation** — row-level EF query filters **and Postgres row-level security** on the tenant tables
([ADR-0012](docs/adr/0012-postgres-row-level-security.md)): every connection runs as a restricted role bound to
the caller's tenant, so a forgotten filter returns only the caller's rows and a write for another tenant is
refused. Per-tenant keys are never mixed with another tenant's or the platform's. Cross-tenant endpoints (list / provision tenants) require `platform-admin`,
the caller's tenant, so a forgotten filter returns only the caller's rows, a write for another tenant is
refused, and a connection with no tenant sees nothing; the few cross-tenant platform operations run in an
explicit, counted platform scope. Outside Development the hosts refuse to start without it. Per-tenant keys are never mixed with another tenant's or the platform's. Cross-tenant endpoints (list / provision tenants) require `platform-admin`,
never the per-tenant `admin` role; a real-auth E2E proves a second tenant sees none of the first's data.
- **QA gate before every PR** — a pipeline result becomes a pull request only when QA passed on real model
output (`PrGate`, enforced inside the PR service itself): failed runs, the iteration cap with QA still
Expand Down
2 changes: 1 addition & 1 deletion docs/CONFIG.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ to the next request.
|---|---|---|
| `ConnectionStrings:DefaultConnection` | — | Postgres. Absent → modules fall back to no-op repositories (stateless boot) unless `RequireDatabase` is true. Aspire injects it on the full stack. |
| `Persistence:RequireDatabase` | `true` (code); Web base `false` for the standalone run, `true` in Web Production | Fail-loud guard: refuse to boot stateless when a DB is expected. |
| `Persistence:RowLevelSecurity` | `true` | Postgres RLS on the tenant tables ([ADR-0012](adr/0012-postgres-row-level-security.md)); the DB user needs `CREATEROLE`. `false` = escape hatch (application filters only). |
| `Persistence:RowLevelSecurity` | `true` | Postgres RLS on the tenant tables ([ADR-0012](adr/0012-postgres-row-level-security.md)); a connection with no tenant sees nothing. The DB user needs `CREATEROLE`. A failed bootstrap **stops the host outside Development** (Development logs an error and runs on application filters). `false` = escape hatch (application filters only; logged as a warning every boot, `/health` reports `row-level-security` Degraded). |
| `Auth:Keycloak:Authority` | empty | **Required outside Development.** Dev fallback `http://localhost:8080/realms/agentic` applies in Development only. |
| `Auth:Keycloak:Audience` | `agentic-api` | JWT audience (Api). |
| `Auth:Keycloak:ClientId` / `ClientSecret` | `agentic-web` / empty | Web OIDC client. Secret required outside Development; the dev default secret is rejected by `DevSecretGuard`. |
Expand Down
17 changes: 17 additions & 0 deletions docs/DEPLOYMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,23 @@ azd deploy # re-build + re-push images only, no infra changes
azd up # provision + deploy (safe to re-run; Postgres and ACA env are idempotent)
```

**Upgrading to row-level security stage 2** (the release that brings [ADR-0012](adr/0012-postgres-row-level-security.md)
stage 2 + the audit hash chain) is the one deploy that must not overlap: the new revision rewrites the database
policies when it boots, and a revision of the previous version still serving traffic then reads platform settings,
model keys and scheduled work as *nothing* and writes audit rows the chain verifier reports as a break. Container Apps
keeps the old revision live until the new one is healthy, so deactivate the running revisions first, then deploy:

```bash
for app in api web; do
for rev in $(az containerapp revision list -n $app -g <rg> --query "[?properties.active].name" -o tsv); do
az containerapp revision deactivate -n $app -g <rg> --revision "$rev"
done
done
azd deploy # the new revisions start against the stage-2 policies; nothing old is left to read through them
```

Expect a few minutes of downtime. Later deploys roll as usual. (On-prem compose / Helm: [ONPREM.md §4](ONPREM.md#4-upgrade).)

Tear down when done:

```bash
Expand Down
34 changes: 30 additions & 4 deletions docs/ONPREM.md
Original file line number Diff line number Diff line change
Expand Up @@ -221,8 +221,9 @@ external Keycloak without its URL, several Web replicas without sticky sessions)
- **External Postgres** — `postgresql.enabled=false` and `externalDatabase.{host,port,database,username,sslMode}`
(password: `postgres-password` in the secret). Create the `agentos` and `keycloak` databases first. The user
must **own** them and hold **`CREATEROLE`**: the hosts create the `agentos_rls` role, its grants and the
row-level-security policies at every boot. Without it RLS stays off and the hosts log
`Row-level security bootstrap FAILED` (isolation falls back to application filters only).
row-level-security policies at every boot. Without it the hosts **refuse to start** with "Row-level security
could not be set up, so AgentOS.NET will not start" (`Persistence:RowLevelSecurity=false` runs without it, on
application filters only — on purpose, and reported Degraded by `/health`).
- **External Keycloak** — `keycloak.enabled=false`, `keycloak.publicUrl=https://sso.corp.example`. Import
[`infra/keycloak/agentic-realm.json`](../infra/keycloak/agentic-realm.json) and copy
[`infra/keycloak/themes/agentos`](../infra/keycloak/themes/agentos) into its themes directory first; the realm
Expand Down Expand Up @@ -285,9 +286,34 @@ Nothing in the stack calls the internet at runtime except the LLM providers you
- compose: set `AGENTOS_VERSION` in `.env` (bundle installs) → `docker compose up -d --wait`
- Helm: `helm upgrade agentos <chart> -n agentos -f my-values.yaml --wait --timeout 15m`
4. Database migrations run at boot, per module, serialised by an advisory lock — rolling several replicas is
safe. The realm step re-runs (idempotent; seed passwords are never re-applied). Verify with `/health` and a
safe **between versions on the same row-level-security stage**; the one upgrade that is not rolling-safe is
below. The realm step re-runs (idempotent; seed passwords are never re-applied). Verify with `/health` and a
sign-in.

### Upgrading to row-level security stage 2 — stop the old version first

The release that brings RLS stage 2 ([ADR-0012](adr/0012-postgres-row-level-security.md): a connection with no
tenant sees **nothing**; settings, the audit trail and invitations under RLS; the audit hash chain) rewrites the
database policies when its **first** replica boots. From that moment an Api or Web replica still running the
previous version reads through the new policies with its old assumptions: its tenant-less reads (platform settings
and model keys resolved off a request, the scheduler, the Autopilot scan, the runner handshake) come back empty, and
the audit rows it writes carry no chain link, so **Verify chain** reports the trail broken from there. Do this
upgrade as a stop-then-start cut-over, never side by side:

- **compose:** `docker compose up -d --wait` recreates each container (the old one stops before the new one starts),
which is enough with one `api` and one `web`. If you scaled them, run `docker compose stop api web` first.
- **Helm:** a Deployment rolls new pods up before the old ones go. Scale the old version to zero first, then
upgrade (the chart puts `replicaCount` back):

```bash
kubectl -n agentos scale deployment -l app.kubernetes.io/component=api --replicas=0
kubectl -n agentos scale deployment -l app.kubernetes.io/component=web --replicas=0
kubectl -n agentos wait --for=delete pod -l 'app.kubernetes.io/component in (api,web)' --timeout=5m
helm upgrade agentos <chart> -n agentos -f my-values.yaml --wait --timeout 15m
```

Expect a few minutes without the desktop and the Api. Later upgrades on stage 2 roll as usual.

Notes:

- **Keycloak never re-imports an existing realm.** Realm changes shipped in a release (a new role, client
Expand Down Expand Up @@ -393,7 +419,7 @@ card without a version under the service account.
| Sign-in loops, *IDX10205 issuer invalid* | `KEYCLOAK_PUBLIC_URL` differs from how browsers reach Keycloak — the issuer is fixed to it. |
| Api/Web exit: *RequireKeyWrapping is on but the key ring is unwrapped* | The certificate is not mounted or its path/password is wrong. Check `dataprotection/` and `DATAPROTECTION_CERTIFICATE_*`, or the Helm Secret. |
| HTTP 431 on Keycloak | Oversized cookie jar on a shared host name — compose/Helm raise Keycloak's limit to 128K; use dedicated host names per install (and `COOKIE_PREFIX` / `web.cookiePrefix`). |
| Log: `Row-level security bootstrap FAILED` | The database user lacks `CREATEROLE`/ownership (external Postgres). Grant it, restart. |
| Host exits: `Row-level security could not be set up` (log: `Row-level security bootstrap FAILED`) | The database user lacks `CREATEROLE`/ownership (external Postgres). Grant it (or have a DBA create `agentos_rls` and grant it), restart. |
| Log: `libgssapi_krb5.so.2: cannot open shared object file` | Harmless — Npgsql probing Kerberos; the app uses password auth. |
| Api/Web exit: *Hosting:PublicBaseUrl is required outside Development* / *must use https://* | Set `WEB_PUBLIC_URL` / `web.publicUrl` to the desktop's `https://` URL (`http://` only for `localhost`). |
| HTTP 400 with an empty body on every page | The browser's host name is not the public URL's (nor `localhost`) — fix `WEB_PUBLIC_URL` / `web.publicUrl`, or add the extra name to `AllowedHosts`. |
Expand Down
7 changes: 5 additions & 2 deletions docs/OPERATIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,10 @@ dashboard or rule references a series no instrument emits.
| | `agentos.evidence.remote_refused` (`agentos_evidence_remote_refused_total`) | counter | `reason` = `unknown_run` · `not_dispatched` · `wrong_connection` · `grace_passed` · `run_cap` · `batch_cap` | Tool calls a paired runner reported that the server refused to record (each refusal is also logged). Cap hits on very long runs are expected; the rest mean a runner is out of step. |
| | `agentos.remote.result_refused` (`agentos_remote_result_refused_total`) | counter | `kind` = `request` · `tool_call` · `progress` · `unknown` | Runner results (`CompleteRequest`, `CompleteToolCall`) and progress steps (`ReportProgress`) the server refused (GAP-06). `request` / `tool_call` / `progress`: it came from a connection the work was not dispatched to — neither that runner nor its own reconnect (same runner id, same tenant); each is also logged as a warning. `unknown`: no one waits for that id any more — a runner finishing after the timeout or a Stop, a duplicate, or an id this replica never dispatched (logged at Information). |
| | `agentos.tools.output.saved_chars` (`agentos_tools_output_saved_chars_total`) | counter `{char}` | `kind` = `builtin` · `mcp` · `remote` | Tool-output characters the gateway kept away from the model (log shaping + size cap). Dashboards show **≈ tokens = chars ÷ 4** — an estimate of tokens not sent, never billed usage. |
| `AgentOs.Health` | `agentos.health.status` (`agentos_health_status`) | gauge | `agentos.health.check` = `self` · `postgres` · `keycloak` · `overall` | 1 healthy · 0.5 degraded · 0 unhealthy, published every 60 s (each publish runs every check, so not more often). |
| `AgentOs.Health` | `agentos.health.status` (`agentos_health_status`) | gauge | `agentos.health.check` = `self` · `postgres` · `row-level-security` · `keycloak` · `overall` | 1 healthy · 0.5 degraded · 0 unhealthy, published every 60 s (each publish runs every check, so not more often). |
| `AgentOs.Persistence` | `agentos.rls.connections` (`agentos_rls_connections_total`) | counter | `scope` = `tenant` · `platform` · `none` | Connections bound by row-level security (ADR-0012). `none` sees **nothing** (stage 2): a rising `none` is a code path that lost its tenant. |
| | `agentos.rls.platform_scope` (`agentos_rls_platform_scope_total`) | counter | `reason` = `autopilot-scan` · `scheduler` · `runner-handshake` · `runner-revalidation` · `platform-settings` · `audit-platform` | Explicit cross-tenant scopes entered. A new reason is a new cross-tenant code path — review it. |
| `AgentOs.Audit` | `agentos.audit.lost` (`agentos_audit_lost_total`) | counter `{event}` | `reason` = `no_tenant` · `database` | Audit rows that were not recorded. Writes are best-effort (the surrounding operation still completes), so every loss is counted and logged instead. **Should stay 0**: `no_tenant` is a code path that lost its tenant; `database` an outage or a refused insert. |
| `AgentOs.Http` | `agentos.http.rate_limited` (`agentos_http_rate_limited_total`) | counter `{request}` | `authenticated` = `true` · `false` | Requests the global request limiter refused with 429 (Settings → Platform → Request limits). A steady anonymous rate is scraping or a sign-up flood; an authenticated one means a limit too low for real use. |

Traces: the `AgentOs.Llm` ActivitySource emits one `chat {model}` client span per provider attempt (attributes
Expand All @@ -81,7 +84,7 @@ Traces: the `AgentOs.Llm` ActivitySource emits one `chat {model}` client span pe
| Endpoint | Checks | Use |
|---|---|---|
| `/alive` | `self` (tag `live`) | Liveness probe — restart the container when it fails. |
| `/health` | `self` + `postgres` (when `ConnectionStrings:DefaultConnection` is set) + `keycloak` (OIDC discovery, outside Development) | Readiness probe + the outside-in availability probe. JSON body names each check. |
| `/health` | `self` + `postgres` and `row-level-security` (when `ConnectionStrings:DefaultConnection` is set; Degraded = turned off by `Persistence:RowLevelSecurity=false`) + `keycloak` (OIDC discovery, outside Development) | Readiness probe + the outside-in availability probe. JSON body names each check. |

Every check is also the `agentos_health_status` gauge, so an unhealthy dependency alerts even when nobody probes.

Expand Down
Loading
Loading