Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ Resource usage depends on accepted browser traffic and the configured measuremen

#### Install the DEM plugin

DEM is experimental and opt-in. Build Netdata with `-DENABLE_PLUGIN_DEM=ON`, or use `--enable-plugin-dem` with the source installer. Install the resulting DEM plugin package.
Linux builds include DEM by default. Native installations include the `netdata-plugin-dem` package. RUM does not require the optional browser runtime.

#### Expose the receiver

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ Resource usage depends on accepted browser traffic and the configured measuremen

#### Install the DEM plugin

DEM is experimental and opt-in. Build Netdata with `-DENABLE_PLUGIN_DEM=ON`, or use `--enable-plugin-dem` with the source installer. Install the resulting DEM plugin package.
Linux builds include DEM by default. Native installations include the `netdata-plugin-dem` package. RUM does not require the optional browser runtime.

#### Configure the receiver and site

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ Module: ipmi

Monitor hardware sensor readings, sensor health and System Event Log entries through IPMI.

Queries a local Linux OpenIPMI device or a remote BMC over IPMI 1.5 LAN or IPMI 2.0 LAN+ using the configured account. The collector reads sensors and SEL metadata; it does not change BMC configuration or clear the event log.
Queries a local Linux OpenIPMI device or a remote BMC over IPMI 1.5 LAN or IPMI 2.0 LAN+ using the configured account. The collector reads sensors and the System Event Log (SEL); it does not change BMC configuration or clear the event log.

This collector is only supported on the following platforms:

Expand Down Expand Up @@ -59,7 +59,7 @@ This plugin requires Linux. Keep FreeIPMI for legacy direct drivers, bridged sen

#### Performance Impact

IPMI command latency grows with sensor count. Increase update_every when a complete sensor scan takes longer than the configured interval.
IPMI command latency grows with sensor count. Increase update_every when a complete sensor scan takes longer than the configured interval. Sensor thresholds are read with the SDR inventory, about every five minutes, at most one command per threshold sensor. With collect_sel enabled, the first collections after a start also read the existing System Event Log, for up to two seconds each (a quarter of update_every when shorter), until all records are read. Later collections read only new records, or the whole log again after the BMC clears or rewrites it.

## Setup

Expand Down Expand Up @@ -98,7 +98,7 @@ Options apply to each IPMI job. Configure one local device or remote BMC per job
| | privilege_level | Session privilege requested from the BMC. User privilege is sufficient for standard sensor and SEL commands. | user | no |
| **Collection** | update_every | Data collection frequency in seconds. | 10 | no |
| | timeout | Timeout for session setup and for each IPMI command. | 5s | no |
| | collect_sel | Collect the System Event Log entry count. | yes | no |
| | collect_sel | Read the System Event Log: the entry count and new records charts, and the ipmi-sel Live Data view. | yes | no |
| **Virtual Node** | vnode | Associates this job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |


Expand Down Expand Up @@ -167,6 +167,7 @@ The following alerts are available:
|:------------|:----------|:------------|
| [ ipmi_sensor_state ](https://github.com/netdata/netdata/blob/master/src/health/health.d/ipmi.conf) | ipmi.sensor_state | IPMI sensor $\{label:sensor} ($\{label:component}) state |
| [ ipmi_events ](https://github.com/netdata/netdata/blob/master/src/health/health.d/ipmi.conf) | ipmi.sel | number of events in the IPMI System Event Log (SEL) |
| [ ipmi_sel_critical_events ](https://github.com/netdata/netdata/blob/master/src/health/health.d/ipmi.conf) | ipmi.sel_new_records | Number of critical events, other than threshold crossings, observed by the collector in the last 10 minutes. Failed SEL reads can delay reporting. The alert clears when no new one is observed, not when the problem is resolved; the ipmi-sel Live Data view lists the records |



Expand All @@ -176,7 +177,17 @@ Metrics grouped by *scope*.

The scope defines the instance that the metric belongs to. An instance is uniquely identified by a set of labels.

Each sensor has its own reading and state chart. SEL is the raw repository entry count, including records FreeIPMI may filter during interpretation. The stock ipmi_events alert template matches the ipmi.sel context and applies to every Go job and to the C collector.
Each sensor has its own reading and state chart. SEL is the raw repository entry count, including
records FreeIPMI may filter during interpretation. The stock ipmi_events alert template matches the ipmi.sel
context and applies to every Go job and to the C collector.

The SEL new records chart counts records newly observed since the previous value, by default FreeIPMI state.
Records already in the log when collection starts are not counted: the chart appears once the first read of
the log completes. After the BMC log is cleared or rewritten, rereading the same record does not count it
again. After the initial baseline, observed events are reported during rescans even if rollover prevents the
whole log from being read. A failed SEL read leaves a gap, and records observed during it count in the next
successful value.



### Per sensor
Expand Down Expand Up @@ -218,14 +229,31 @@ Metrics:
| ipmi.sel | IPMI System Event Log Entries | events | events |


### Per SEL event type

New System Event Log records of one event type on the selected BMC.

Labels:

| Label | Description |
|:-----------|:----------------|
| event_type | `threshold` for threshold events, which also change the state of their sensor; `discrete` for every other record, OEM records included. |

Metrics:

| Metric | Description | Dimensions | Unit |
|:------|:------------|:----------|:----|
| ipmi.sel_new_records | IPMI System Event Log New Records | nominal, warning, critical, unknown | records |



## Live Data

Inspect the latest successful collection without issuing more IPMI commands.

### IPMI Sensors

Show current readings and default FreeIPMI health states, with sensor counts by component and state.
Show current readings, the thresholds the BMC uses for each threshold sensor, and default FreeIPMI health states, with sensor counts by component and state. Compare a reading with its thresholds to see how close it is to a limit; by default only crossing a critical or non-recoverable threshold makes the state critical.

| Aspect | Description |
|:-------|:------------|
Expand All @@ -247,7 +275,7 @@ No additional configuration is required.

#### Returns

One row per discovered full or compact sensor record, expanded for sharing. Unavailable numeric readings are null.
One row per discovered full or compact sensor record, expanded for sharing. Unavailable numeric readings are null. Thresholds are in the reading's units and are null when the sensor has none or they could not be read.

| Column | Type | Unit | Visibility | Description |
|:-------|:-----|:-----|:-----------|:------------|
Expand All @@ -257,9 +285,64 @@ One row per discovered full or compact sensor record, expanded for sharing. Unav
| Reading | float | | | Current reading; null when unavailable. |
| Units | string | | | Reading units; null for discrete sensors. |
| State | string | | | Default FreeIPMI health interpretation. |
| LNR | float | | | Lower non-recoverable threshold. |
| LCR | float | | | Lower critical threshold. |
| LNC | float | | | Lower non-critical threshold. |
| UNC | float | | | Upper non-critical threshold. |
| UCR | float | | | Upper critical threshold. |
| UNR | float | | | Upper non-recoverable threshold. |
| Key | string | | hidden | Stable sensor identity within the collection job. |
| rowOptions | none | | hidden | Row health severity. |

### IPMI System Event Log

Browse the hardware events the BMC recorded, such as threshold crossings, fan, power and memory errors, processor throttling and boots, with default FreeIPMI states. Use it to see when a problem started and how often it recurs; the sensor charts show only the current state.

| Aspect | Description |
|:-------|:------------|
| Name | `Ipmi-sel` |
| Require Cloud | no |
| Performance | Reads the records the collector already read from the BMC; sends no BMC requests. |
| Security | Read-only hardware event records from the configured target. |
| Availability | Available for jobs with collect_sel enabled while they run; when no job collects the SEL, requests return HTTP 404. Returns HTTP 503 before the first successful collection, after a collection failure, or while the BMC has not answered SEL requests. On the first collections of a large log, it shows the oldest records read so far. |

#### Prerequisites

No additional configuration is required.

#### Parameters

| Parameter | Type | Description | Required | Default | Options |
|:---------|:-----|:------------|:--------:|:--------|:--------|
| Instance | select | Select the IPMI collection job. | no | First available job | |
| State | select | Show only records with this default FreeIPMI state. | no | all | All states (default), Critical, Warning, Nominal, Unknown |
| Search | string | Case-sensitive Netdata simple patterns against sensor, type, event, direction, state, generator and record ID. Use `*DIMM*` for a substring. Space-separated patterns are checked in order; the first match decides, `!` excludes, and no match excludes the row. | no | No text filter | |

#### Returns

One row per SEL record the BMC retains, newest first in log order. The dashboard time range does not
filter the table, because SEL records are often weeks apart. Times are converted to UTC with the BMC's
SEL UTC offset. When the BMC reports none, times are the BMC clock, and the view shows how far that
clock is from the Agent clock. Records logged before a daylight saving change can be off by an hour,
because the BMC reports only its current offset. Records without a date (OEM records without a
timestamp, or records logged before the BMC clock was set) show Time as Unavailable.


| Column | Type | Unit | Visibility | Description |
|:-------|:-----|:-----|:-----------|:------------|
| Order | integer | | hidden | Position in the log, newest first. |
| Time | timestamp | | | When the BMC logged the record, in UTC; Unavailable when the record has no date. |
| Sensor | string | | | SDR sensor name, or the event generator and sensor number (for example BIOS #34) when the BMC has no record of that sensor. |
| Sensor Type | string | | | IPMI sensor type. |
| Event | string | | | Event description; threshold events include the trigger reading and threshold when the BMC records them and the sensor's SDR record describes their conversion. |
| Direction | string | | | Assertion (the condition started) or Deassertion (it ended). |
| State | string | | | Default FreeIPMI interpretation of the record. |
| Record ID | string | | hidden | SEL record identifier. |
| Generator | string | | hidden | Event generator, such as the BMC, BIOS or SMI handler, and its ID. |
| BMC Time | string | | hidden | Timestamp as the BMC recorded it, before UTC conversion. |
| Raw | string | | hidden | Record bytes as returned by the BMC. |
| rowOptions | none | | hidden | Row health severity. |



## Troubleshooting
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -63,20 +63,34 @@ Each attempt starts a fresh browser and consumes CPU, memory and target website

### Prerequisites

#### Prepare the browser sandbox policy

Managed bundles and preloads on Ubuntu hosts with restricted user namespaces, including Ubuntu 24.04,
require administrator-prepared AppArmor policy. DEM does not automatically allow service-writable
browser paths. The native runtime package supplies its own profile for its root-owned browser.
Containers also require compatible host and container namespace, AppArmor and seccomp policies.
Follow the [runtime setup guide](https://github.com/netdata/netdata/blob/master/src/go/plugin/dem/README.md#runtime-setup-and-retirement)
and keep Chromium's sandbox enabled when qualifying the actual service deployment.


#### Set up the browser runtime

Browser monitoring requires an explicitly installed runtime on Linux amd64 or arm64, running as the
unprivileged Netdata service account with Chromium sandboxing enabled. Delivery is experimental;
production downloads and native packages are not provided yet.
unprivileged Netdata service account with Chromium sandboxing enabled. Native installations use the optional
`netdata-dem-runtime` package on live systemd hosts; package hooks may restart the Agent. Other installation
methods use managed bundles or protected preloads. Production publication still requires release and platform qualification.

Obtain a trusted release descriptor and matching archive, then run as the service account:
For offline managed installation, set `runtime.offline_keyring` in `dem.conf` to the path of an
administrator-provisioned public OpenPGP keyring. Obtain a signed release descriptor and matching archive,
then run as the service account:

```sh
dem.plugin runtime install --manifest /absolute/path/manifest.json --archive /absolute/path/runtime.tar.gz
dem.plugin runtime install --release /absolute/path/release.json --archive /absolute/path/runtime.tar.zst
dem.plugin runtime status
```

Restart DEM to activate the installation. Enabling a monitor never downloads a runtime.
Restart DEM to activate the installation. Explicit install enables automatic update preparation; add
--manual-updates for manual control. Enabling a monitor never downloads a runtime.
Use `dem.plugin runtime qualify` under the service's execution policy to check both journeys and Lighthouse.
See the [runtime setup guide](https://github.com/netdata/netdata/blob/master/src/go/plugin/dem/README.md#runtime-setup-and-retirement)
for updates, explicit downloads, installation paths and protected preloads.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -62,20 +62,34 @@ Each attempt starts a fresh browser and consumes CPU, memory and target website

### Prerequisites

#### Prepare the browser sandbox policy

Managed bundles and preloads on Ubuntu hosts with restricted user namespaces, including Ubuntu 24.04,
require administrator-prepared AppArmor policy. DEM does not automatically allow service-writable
browser paths. The native runtime package supplies its own profile for its root-owned browser.
Containers also require compatible host and container namespace, AppArmor and seccomp policies.
Follow the [runtime setup guide](https://github.com/netdata/netdata/blob/master/src/go/plugin/dem/README.md#runtime-setup-and-retirement)
and keep Chromium's sandbox enabled when qualifying the actual service deployment.


#### Set up the browser runtime

Browser monitoring requires an explicitly installed runtime on Linux amd64 or arm64, running as the
unprivileged Netdata service account with Chromium sandboxing enabled. Delivery is experimental;
production downloads and native packages are not provided yet.
unprivileged Netdata service account with Chromium sandboxing enabled. Native installations use the optional
`netdata-dem-runtime` package on live systemd hosts; package hooks may restart the Agent. Other installation
methods use managed bundles or protected preloads. Production publication still requires release and platform qualification.

Obtain a trusted release descriptor and matching archive, then run as the service account:
For offline managed installation, set `runtime.offline_keyring` in `dem.conf` to the path of an
administrator-provisioned public OpenPGP keyring. Obtain a signed release descriptor and matching archive,
then run as the service account:

```sh
dem.plugin runtime install --manifest /absolute/path/manifest.json --archive /absolute/path/runtime.tar.gz
dem.plugin runtime install --release /absolute/path/release.json --archive /absolute/path/runtime.tar.zst
dem.plugin runtime status
```

Restart DEM to activate the installation. Enabling a monitor never downloads a runtime.
Restart DEM to activate the installation. Explicit install enables automatic update preparation; add
--manual-updates for manual control. Enabling a monitor never downloads a runtime.
Use `dem.plugin runtime qualify` under the service's execution policy to check both journeys and Lighthouse.
See the [runtime setup guide](https://github.com/netdata/netdata/blob/master/src/go/plugin/dem/README.md#runtime-setup-and-retirement)
for updates, explicit downloads, installation paths and protected preloads.
Expand Down
2 changes: 1 addition & 1 deletion docs/Netdata Agent/Logging.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -231,7 +231,7 @@ Example logrotate configuration:
| `ND_SRC_IP` | `src_ip` | `SourceIP` | 24 | when the event happened during an inbound request, this is the IP the request came from |
| `ND_SRC_PORT` | `src_port` | `SourcePort` | 25 | when the event happened during an inbound request, this is the port the request came from |
| `ND_SRC_FORWARDED_HOST` | `src_forwarded_host` | `SourceForwardedHost` | 26 | the contents of the HTTP header `X-Forwarded-Host` |
| `ND_SRC_FORWARDED_FOR` | `src_forwarded_for` | `SourceForwardedFor` | 27 | the contents of the HTTP header `X-Forwarded-For` |
| `ND_SRC_FORWARDED_FOR` | `src_forwarded_for` | `SourceForwardedFor` | 27 | the first address in the HTTP header `X-Forwarded-For`, if it is a valid IPv4/IPv6 address (port dropped) |
| `ND_SRC_CAPABILITIES` | `src_capabilities` | `SourceCapabilities` | 28 | when the request came from a child, this is the communication capabilities of the child |
| `ND_DST_TRANSPORT` | `dst_transport` | `DestinationTransport` | 29 | when the event happened during an outbound request, this is the outbound request transport |
| `ND_DST_IP` | `dst_ip` | `DestinationIP` | 30 | when the event happened during an outbound request, this is the IP the request destination |
Expand Down
Loading
Loading