Skip to content

Feature/docker stats #1436

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
alexcos20 wants to merge 11 commits into
base: next-4
Choose a base branch
from
Open

Feature/docker stats #1436

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
d110a44
docker stats
alexcos20 Jul 29, 2026
c3df901
fix deps
alexcos20 Jul 29, 2026
f3b1958
expose metrics
alexcos20 Jul 29, 2026
0c9cba0
fix bugs
alexcos20 Jul 29, 2026
d63bb50
fix bug
alexcos20 Jul 29, 2026
728eb79
fix review comments
alexcos20 Jul 29, 2026
4bd00a2
use sqlite transaction
alexcos20 Aug 5, 2026
675ae8d
includeMetric default on & address checksum
alexcos20 Aug 13, 2026
2dbcd1f
add more logging
alexcos20 Aug 13, 2026
34c5707
fix persistent bug
alexcos20 Aug 13, 2026
3127ed2
fix docs for nvidia
alexcos20 Aug 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
199 changes: 187 additions & 12 deletions docs/API.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,16 @@
# Ocean Node Api

## Address casing

Every EVM address you send (`consumerAddress`, `owner`, `address`, `decrypterAddress`,
`dataNftAddress`, `publisherAddress`, `consumerAddrs`, `additionalViewers`) is accepted in **any
casing** — checksummed (EIP-55), all-lowercase, all-uppercase — and canonicalized to its checksummed
form by the node before it is used as a lookup key or compared against an owner. A lowercased
address therefore matches the same jobs, services and buckets as the checksummed one.

Signatures are unaffected: the node verifies the signed message against the casing you actually
signed, so clients that build the message from a lowercase address keep working.

---

## State DDO
Expand Down Expand Up @@ -1577,11 +1588,30 @@ returns job status

Required at least one of the following parameters:

| name | type | required | description |
| --------------- | ------ | -------- | ------------------------------------ |
| consumerAddress | string | | consumer address to use as filter |
| jobId | string | | jobId address to use as filter |
| agreementId | string | | agreementId address to use as filter |
| name | type | required | description |
| --------------- | ------- | -------- | ------------------------------------------------------------------------------------------------------------ |
| consumerAddress | string | | consumer address to use as filter |
| jobId | string | | jobId address to use as filter |
| agreementId | string | | agreementId address to use as filter |
| includeMetrics | boolean | | override the runtime-metrics default (`true` = require them, `false` = never). See note below. |
| signature | string | | signature over `consumerAddress` + `nonce` + `command` (or an auth token) — authenticates the owner |
| nonce | string | | request nonce, paired with `signature` |

**Runtime metrics are owner-only and returned BY DEFAULT.** If the request carries owner credentials
(`consumerAddress` plus `signature`/`nonce`, or an `Authorization` header token), each job owned by
(or shared with) that address comes back with a `runtimeMetrics` object — no flag needed.

`includeMetrics` only overrides that default:

| `includeMetrics` | behavior |
| --- | --- |
| omitted (default) | Metrics attached when owner credentials are present and valid. A request without credentials — the plain, unauthenticated status call — returns `200` with no metrics, exactly as before. Invalid credentials likewise just mean no metrics. |
| `true` | Metrics are **required**: missing `consumerAddress` answers `400`, failed authentication `401`. Use it when you want to know *why* metrics are absent instead of getting a silently trimmed response. |
| `false` | Metrics are never attached (and the node skips the auth round-trip). |

Metrics never reach a non-owner, and are never part of the on-chain escrow claim proof. They are
best-effort and up to one sampling interval stale (see [compute.md](compute.md) and
`C2D_METRICS_INTERVAL_SECONDS` in [env.md](env.md)).

#### Response

Expand Down Expand Up @@ -1621,6 +1651,146 @@ Required at least one of the following parameters:
]
```

When called with owner credentials, each owned job additionally carries a `runtimeMetrics` object
(see [The `runtimeMetrics` object](#the-runtimemetrics-object) below).

---

### The `runtimeMetrics` object

`runtimeMetrics` is an optional snapshot of live container stats, returned on
`COMPUTE_GET_STATUS` / `SERVICE_GET_STATUS` to the **authenticated owner** of the job or service —
by default, without asking for it. It never reaches anyone else, and `includeMetrics=false` opts out.
Clients MUST treat every part as optional and render a field only when present.

**Semantics clients should surface to users:**

- **Best-effort & slightly stale.** Sampled on a fixed cadence (`C2D_METRICS_INTERVAL_SECONDS`,
default 10s), so values can be up to one interval old. `collectedAt` is the sample time — show it
(e.g. "as of 8s ago").
- **May be missing entirely.** No snapshot yet (job just started), collection disabled on the node
(`C2D_METRICS_INTERVAL_SECONDS=0`), or a transient sampling failure ⇒ no `runtimeMetrics` field.
This is normal, not an error.
- **`null` vs absent for GPU numbers.** Inside a `gpu[]` entry, a `null` metric means "the backend
could not read it" — display as "n/a", never as `0`.
- **Bytes are raw bytes**; percentages are rounded to two decimals — memory/disk/GPU are `0–100`,
but CPU `usagePercent` can exceed `100` across multiple cores (see the CPU table); durations are
seconds; the final snapshot after a job/service ends carries the peak/exit values.

#### Top-level fields

| field | type | unit / notes |
| --------------- | ------- | ------------------------------------------------------------------------------------------------ |
| collectedAt | string | ISO-8601 timestamp of the sample |
| containerState | object | see below — status + structured exit info |
| cpu | object | see below |
| memory | object | see below |
| disk | object | see below |
| network | object? | `{ rxBytes, txBytes }`; **absent** when the container runs with no network (`NetworkMode: none`) |
| blockIO | object | `{ readBytes, writeBytes }` — cumulative disk I/O in bytes |
| pids | object | `{ current, limit }` — process/thread count vs the container PID limit (512) |
| gpu | array? | one entry per GPU the job/service holds; **absent** for CPU-only jobs or when GPU metrics are off |

`containerState`:

| field | type | notes |
| ------------ | -------- | ------------------------------------------------------------------------ |
| status | string | e.g. `running`, `exited` |
| startedAt | string? | ISO-8601 |
| finishedAt | string? | ISO-8601; present once the container has stopped |
| exitCode | number? | process exit code (present after exit) |
| oomKilled | boolean | `true` if the kernel OOM-killed the container |
| error | string? | Docker-reported error string, if any |
| restartCount | number | container restarts |
| health | string? | Docker HEALTHCHECK status when the image defines one (e.g. `healthy`) |

`cpu`:

| field | type | unit / notes |
| ---------------------- | ------ | ------------------------------------------------------------------------------- |
| usagePercent | number | % of one host CPU-second per wall-second (docker-stats formula), `0–N×100` |
| allocated | number | CPU cores requested by the job (`0` when unconstrained) |
| usagePercentOfAllocated| number | `usagePercent / allocated` — "how saturated is what you paid for" (`0` if alloc 0)|
| cumulativeSeconds | number | total CPU-seconds consumed since start (monotonic; billing-grade) |
| throttledPeriods | number | CFS quota throttling events — high ⇒ the CPU request is too small |
| throttledSeconds | number | total time throttled, seconds |

`memory`:

| field | type | unit / notes |
| -------------- | ------ | -------------------------------------------------------- |
| usageBytes | number | working-set bytes (`usage − inactive_file`, cgroup v2) |
| limitBytes | number | memory limit (= allocated RAM) in bytes |
| usagePercent | number | `usageBytes / limitBytes × 100` |
| peakUsageBytes | number | max `usageBytes` observed across samples |

`disk`:

| field | type | unit / notes |
| ------------ | ------- | ---------------------------------------------------------------------------------- |
| usedBytes | number | compute jobs: bytes written under `/` (excludes base image); services: writable layer |
| quotaBytes | number? | present only for jobs with a `disk` resource |
| usagePercent | number? | present only when `quotaBytes` is known |

`gpu[]` entry:

| field | type | unit / notes |
| ------------------ | --------------- | --------------------------------------------------------------------- |
| resourceId | string | the requested resource id (`gpu0`, `gpu1`, …) — maps the entry to a device |
| vendor | string | `nvidia` (only NVIDIA is emitted today; `amd`/`intel` reserved) |
| utilizationPercent | number \| null | GPU busy % (`null` = unreadable) |
| memoryUsedBytes | number \| null | VRAM used |
| memoryTotalBytes | number \| null | total VRAM |
| temperatureC | number? | °C, when available |
| powerWatts | number? | current draw, W, when available |
| shared | boolean? | `true` ⇒ device is shareable and the number may include other jobs |

> Note: the node also keeps an internal delta accumulator on the stored snapshot; it is stripped
> from the response and clients will never see it.

#### Example `runtimeMetrics`

```json
{
"collectedAt": "2026-07-29T12:00:10.000Z",
"containerState": {
"status": "running",
"startedAt": "2026-07-29T11:59:30.000Z",
"oomKilled": false,
"restartCount": 0
},
"cpu": {
"usagePercent": 182.4,
"allocated": 2,
"usagePercentOfAllocated": 91.2,
"cumulativeSeconds": 73.1,
"throttledPeriods": 12,
"throttledSeconds": 0.4
},
"memory": {
"usageBytes": 734003200,
"limitBytes": 1073741824,
"usagePercent": 68.36,
"peakUsageBytes": 812345678
},
"disk": { "usedBytes": 524288000, "quotaBytes": 10737418240, "usagePercent": 4.88 },
"network": { "rxBytes": 10485760, "txBytes": 2097152 },
"blockIO": { "readBytes": 41943040, "writeBytes": 8388608 },
"pids": { "current": 24, "limit": 512 },
"gpu": [
{
"resourceId": "gpu0",
"vendor": "nvidia",
"utilizationPercent": 77,
"memoryUsedBytes": 1073741824,
"memoryTotalBytes": 3221225472,
"temperatureC": 55,
"powerWatts": 90
}
]
}
```

### `HTTP` GET /api/services/computeResult

### `P2P` command: getComputeResult
Expand Down Expand Up @@ -2002,16 +2172,21 @@ by the authenticated `consumerAddress` are returned.

#### Query Parameters

| name | type | required | description |
| --------------- | ------ | -------- | ----------- |
| consumerAddress | string | v | owner address |
| nonce | string | v | request nonce |
| signature | string | v | signed message (or use an `Authorization` auth-token header) |
| serviceId | string | | filter to a single service; omit to list all owned services |
| name | type | required | description |
| --------------- | ------- | -------- | ----------- |
| consumerAddress | string | v | owner address |
| nonce | string | v | request nonce |
| signature | string | v | signed message (or use an `Authorization` auth-token header) |
| serviceId | string | | filter to a single service; omit to list all owned services |
| includeMetrics | boolean | | runtime metrics (`runtimeMetrics`) are included by default; pass `false` to omit them |

#### Response (200)

Array of `ServiceJob` (with `userData` stripped).
Array of `ServiceJob` (with `userData` stripped). Each entry also carries a sanitized
`runtimeMetrics` object — see [The `runtimeMetrics` object](#the-runtimemetrics-object) for its full
structure. Included by default here because this command is already authenticated and owner-scoped
(pass `includeMetrics=false` to omit); the node-wide `serviceList` never returns metrics. Metrics are
best-effort (see [compute.md](compute.md) and `C2D_METRICS_INTERVAL_SECONDS` in [env.md](env.md)).

---

Expand Down
57 changes: 57 additions & 0 deletions docs/Logs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,3 +100,60 @@ npm run logs
```
npm run logs http://localhost:8000 "2023-11-01T00:00:00Z" "2023-11-30T23:59:59Z" 50 "http" "info"
```

## Compute/Service runtime metrics in the logs

While compute jobs and services run, the node samples live container stats (CPU, RAM, disk,
network, block I/O, PIDs, exit info, GPU) every `C2D_METRICS_INTERVAL_SECONDS` — see
[env.md](env.md). Every one of those lines is logged at **debug** level by the `CORE` module
with a `[metrics]` tag, so one filter shows the whole picture:

```bash
# everything metrics-related (needs LOG_LEVEL=debug)
grep '\[metrics\]' logs/*.log

# just the engine-wide roll-up: one line per sampling interval
grep '\[metrics\] summary' logs/*.log

# just the workloads close to a limit (mem/disk/pids/cpu-throttling)
grep '\[metrics\] pressure' logs/*.log

# one specific job or service, sample by sample
grep '\[metrics\] job 88ee41c8' logs/*.log
```

What each tag means:

| line | when | what it tells you |
| --- | --- | --- |
| `[metrics] C2D Engine <hash>: sampling every Ns` / `collection DISABLED` | engine start | whether metrics are being collected at all — the first thing to check when a job shows no `runtimeMetrics` |
| `[metrics] summary engine <hash>: …` | once per interval | totals across every sampled job/service on that engine: cpu % of host, cores allocated, memory used vs allocated, disk, network, GPU count, how many containers are cpu-throttled, and the age of the oldest sample |
| `[metrics] pressure job\|service <id>: …` | once per interval, only when relevant | that workload is ≥90% of its memory limit (OOM-kill risk), ≥90% of its disk quota (stop risk), ≥80% of its PID limit, or is being cpu-throttled (undersized `cpu` request) |
| `[metrics] job\|service <id>: cpu … mem … disk … pids … net … blkio … state …` | per sample | the full `docker stats` view of that container, plus throttling, peak memory, disk vs quota, exit info and GPU. `[final]` marks the last snapshot taken before teardown |
| `[metrics] job <id>: first snapshot …` | first sample of a job | reminder that `cpu` reads 0% until the second sample (deltas need two samples) |
| `[metrics] … dropping sample …` / `collection failed …` | on failure | why a snapshot is missing: the container vanished, or a lifecycle operation was in flight |
| `[metrics] gpu: …` and `GPU metrics (nvidia): …` | on failure | no GPU numbers, and why. Each message is explained — with its fix — in [compute.md → Troubleshooting GPU metrics](compute.md#troubleshooting-gpu-metrics) |

These are pure diagnostics — collection is best-effort and never affects job or service
execution. The same numbers are available over the API to the owner of a job/service
(`runtimeMetrics` on `COMPUTE_GET_STATUS` / `SERVICE_GET_STATUS`, see [API.md](API.md)).

### When a status response has no `runtimeMetrics`

Snapshots are persisted on the job record, so you can check the stored state directly instead
of guessing whether they were never written or dropped on the way out:

```bash
npm run job-metrics # the 5 most recent jobs
npm run job-metrics -- 018c0121 # one job (full id or any trailing part)
```

For each job it prints whether the row holds a snapshot, its values, and whether the delta
accumulator that CPU % needs is there. Read-only; run it from the node's working directory.

- **`runtimeMetrics: ABSENT`** — nothing was ever stored for that job, so the API cannot return
it and `cpu usagePercent` stays 0 (each sample would be a "first sample"). The node also warns
about this itself: `[metrics] job <id>: no previous snapshot after Ns of runtime`.
- **present, but missing from the response** — the caller was not recognised as the owner.
Metrics only go to the job's owner (or an `additionalViewers` address), proven by
`consumerAddress` + signature/nonce or an `Authorization` token issued to that same address.
Loading
Loading