diff --git a/content/reference/components/cf-reactor.markdown b/content/reference/components/cf-reactor.markdown index e25c5211e..6f79e4c5b 100644 --- a/content/reference/components/cf-reactor.markdown +++ b/content/reference/components/cf-reactor.markdown @@ -88,17 +88,82 @@ The daemon is then signaled to handle the event and run the bundles of the `then **Notes:** -- Event detection is based on polling, but `cf-reactor` does not use busy-waiting. +- Event detection is based on polling, sped up by [wakeup sources](#wakeup-sources), but `cf-reactor` does not use busy-waiting. Instead, the polling thread sleeps between polling cycles and remains dormant when no events are registered. - `cf-reactor` does not guarantee that an event will be handled immediately after it occurs. The delay depends on factors such as the polling interval, policy reloads, and slow code execution. However, `cf-reactor` guarantees that changes will be detected as fast as possible and the corresponding code executed, provided that execution does not fail. +- After a wakeup source wakes it up, `cf-reactor` waits 200 milliseconds before checking, so that a burst of changes is checked once. +- An event that fires more than 10 times in 60 seconds is delayed, starting at 10 seconds and doubling up to 5 minutes, until it calms down. + The changes it holds back are handled in one run when the delay ends. + Meanwhile, and while its bundles run, the event is not checked again: it is checked once when it can run. + +### Wakeup sources + +A wakeup source wakes an event's watcher up as soon as the operating system reports a change to the watched file or its directory, so that `cf-reactor` checks the event right away instead of at its next [`interval`][events#interval]. +Each event uses the first available wakeup source of its [`wakeup_sources`][events#wakeup_sources] attribute. +Wakeup sources only decide _when_ an event is checked, never _whether_ it happened, and polling continues underneath all of them. +Changing the wakeup source never changes which events fire, only how soon. + +- `ebpf` (Linux, Enterprise) loads a small program into the kernel that matches changes against the watched files' inodes before telling `cf-reactor`. + It needs kernel 5.8 or later (or the RHEL 8 kernel) with BTF, and `CAP_SYS_ADMIN` (or `CAP_BPF` and `CAP_PERFMON`). + It has no per-directory watch limits, and watches up to 65,536 files; further events use their next wakeup source, and `cf-reactor` logs a warning. +- `fanotify` and `inotify` (Linux, Enterprise) watch each file for writes and changes of its permissions or owner, and its parent directory for files being created, deleted and renamed. + Files created and deleted in that directory are reported to `cf-reactor`, which ignores the ones for other files. + `inotify` is limited by `fs.inotify.max_user_watches` and `fs.inotify.max_user_instances`, and `fanotify` by `fs.fanotify.max_user_marks`. + When out of watches or marks, they watch files through their directories only, and `cf-reactor` logs a warning; a write through a file descriptor opened before the file was renamed into place is then only seen at the next interval. + Raise the limit to watch more than about half as many files as it allows. +- `usn` (Windows, Enterprise) reads the change journal of NTFS and ReFS volumes, waiting for new records as they are written. + It sees changes made while `cf-reactor` was stopped, and doesn't lose changes during bursts, so it comes before `rdcw`. + It needs administrator rights and a change journal on the volume; `cf-reactor` never creates one. + If the journal is deleted, recreated or overwritten past the last record read, every event on that volume is checked. +- `rdcw` (Windows, Enterprise) uses `ReadDirectoryChangesW` on the watched files' directories, and works on every local file system, including FAT32 and exFAT. + When its buffer overflows during a burst, the events of that directory are checked. + It isn't told when a watched directory, or one above it, is renamed or moved; `cf-reactor` notices within 5 seconds and watches the new location. + Changes made while `cf-reactor` was stopped are found by the first check after it starts. +- `poll` only checks at the interval. It is the only wakeup source in CFEngine Community. + +`package_installed` and `package_uninstalled` events always use `poll`: the installed packages are listed every 10 seconds whatever the wakeup source, so no wakeup source can make them faster. + +**Cost:** away from watched files, no wakeup source adds measurable CPU time to the rest of the system. +In a directory that holds a watched file, each file operation costs about 1 µs more with `ebpf`, `fanotify` or `inotify` (about 10%), counting the process making the change and `cf-reactor`. +At 1,000 file operations per second in one watched directory this is about 0.1% of a CPU core, so the choice of wakeup source makes no practical difference to CPU time. +Where the wakeup sources differ: + +| Source | Wakes the watcher after a change | Kernel resources | +| ---------- | -------------------------------- | ------------------------------------------------------------------------ | +| `ebpf` | under 2 ms | up to 65,536 files, about 0.7 MB of locked memory for 1,000 | +| `fanotify` | about 20 ms | one mark per file and one per directory (`fs.fanotify.max_user_marks`) | +| `inotify` | about 20 ms | one watch per file and one per directory (`fs.inotify.max_user_watches`) | +| `usn` | under 2 ms | one reader per volume, none per file | +| `rdcw` | under 3 ms | one directory handle per watched directory | +| `poll` | at the next interval | none | + +`cf-reactor` then waits 200 milliseconds before checking, so in practice the native wakeup sources react equally fast. +`fanotify` and `inotify` gather the changes of 20 milliseconds into one wakeup. +Measured on Ubuntu 24.04 (kernel 6.8) and Windows Server 2019, in virtual machines. + +`file_content_modified` hashes the whole file when its size or modification time changed. +For a large file that is written to all the time, such as a log, this costs about as much with every wakeup source, since the event is limited to 10 runs a minute (see above) and checked once per run: hashing a 200 MB file appended to 20 times a second took about 20% of a CPU core. +Use `file_modified` for such files, which only compares the modification time. + +**Changes only seen at the next interval:** + +- writes through a memory mapping (`mmap`) that stays open, with every wakeup source +- with `ebpf`, writes on file systems that don't report them to the kernel's generic write path (for example some network file systems) until the file is closed +- with every wakeup source, changes on network file systems made by other hosts + +**Notes:** + +- On CFEngine Enterprise, the wakeup source each event uses is reported to the hub, in the `reactor_watchers` table of the reporting database. +- SELinux or other mandatory access control may deny `cf-reactor` a wakeup source; it then falls back to the next one in the list. ## Performance `cf-reactor` uses these threads: - A watcher thread checks for events, and queues the events it detects. + Events about files on network file systems are checked by a second watcher thread, so that a server that stops responding doesn't hold up the other events. - The main thread takes the events from the queue and runs their [`then`][events#then] bundles, one event at a time. - For `package_installed` and `package_uninstalled` events, a listing thread lists the installed packages every 10 seconds. There is one listing thread per [package module][Package modules] (e.g. `apt_get`), shared by all the `events` promises using it, whatever the number of packages they watch. @@ -109,9 +174,10 @@ How many events per second it can handle depends on two things: - The cost of `cf-reactor` itself: checking for the event, which depends on the event type, and setting up and running the `then` bundles. - The work done by the `then` bundles, and anything outside of `cf-reactor`, such as how fast the package manager lists the installed packages. -Each `events` promise fires at most once per check, so at most once per [`interval`][events#interval]. -With `N` `events` promises and an interval of `I` seconds, `cf-reactor` has at most `N / I` events per second to handle. +Each `events` promise fires at most once per check. +Without a [wakeup source](#wakeup-sources), it is checked once per [`interval`][events#interval]; a wakeup source can check it sooner. Changes happening between two checks are seen as one event. +Either way, an `events` promise that fires more than 10 times in 60 seconds is delayed (see [Under the hood](#under-the-hood)), so with `N` `events` promises that keep firing, `cf-reactor` has at most about `N / 6` events per second to handle. The figures below were measured on an Amazon EC2 `t2.micro` instance (1 vCPU, Intel Xeon E5-2686 v4 at 2.30 GHz, 1 GB of memory) running Ubuntu 24.04. Each `events` promise watched its own file, user or package, with the default `interval` of 1 second, and ran this `then` bundle: @@ -151,6 +217,8 @@ The limit is the most events per second `cf-reactor` can handle for each event t - Above the limit, `cf-reactor` doesn't lose events: they pile up in the queue and are handled late. The queue has no size limit: if events keep coming faster than `cf-reactor` handles them, the delay keeps growing, and so does the memory used by the queue, until `cf-reactor` runs out of memory. - Slower `then` bundles lower these limits. +- These figures were measured with polling only, before an `events` promise was limited to 10 runs in 60 seconds. + They are totals across many `events` promises, each watching its own file, user or package: no single `events` promise reaches these rates, since one that fires more than 10 times a minute is delayed. ## Enterprise hub tasks diff --git a/content/reference/promise-types/events.markdown b/content/reference/promise-types/events.markdown index 193dbb3dd..c76d7965d 100644 --- a/content/reference/promise-types/events.markdown +++ b/content/reference/promise-types/events.markdown @@ -119,6 +119,7 @@ bundle reactor main Each event has its own interval. A lower value means the event is noticed sooner but checked more often. +A [wakeup source](#wakeup_sources) can wake the check up earlier, so with one the interval mostly bounds how late a missed change is noticed. For how polling works, see [Under the hood][cf-reactor#Under the hood]. ```cf3 @@ -135,6 +136,50 @@ body when deleted_slow(file) **Default value:** `1` +#### wakeup_sources + +**Description:** The wakeup sources that may wake this event's watcher up before its next interval, in order of preference. + +A wakeup source is a way for the operating system to tell `cf-reactor` that the watched file, or its directory, changed; `poll` means none, only the interval. +It makes `cf-reactor` check the event right away instead of at the next interval. +It never decides whether the event happened: the check does, so the same events fire whatever the wakeup source. +Polling at the [`interval`](#interval) continues underneath every wakeup source, and catches what a wakeup source misses. + +`cf-reactor` uses the first wakeup source in the list that is available on the host and can watch the event's target. +Without `wakeup_sources`, it tries `ebpf`, `fanotify`, `inotify`, `usn`, `rdcw` and `poll`, in that order. +CFEngine Community only has `poll`. +If none of the listed wakeup sources is available, the event is not watched and `cf-reactor` logs an error; list `poll` last to fall back to polling instead. + +| Source | Platform | Edition | +| ---------- | -------- | ---------- | +| `ebpf` | Linux | Enterprise | +| `fanotify` | Linux | Enterprise | +| `inotify` | Linux | Enterprise | +| `usn` | Windows | Enterprise | +| `rdcw` | Windows | Enterprise | +| `poll` | All | All | + +For how the wakeup sources differ, see [Wakeup sources][cf-reactor#Wakeup sources]. + +```cf3 +body when passwd_modified +{ + file_content_modified => "/etc/passwd"; + interval => 300; + wakeup_sources => { "inotify", "poll" }; +} +``` + +**Type:** `slist` + +**Allowed input range:** `poll`, `inotify`, `fanotify`, `ebpf`, `usn`, `rdcw` + +**Notes:** + +- changing `wakeup_sources` does not reset the event: on policy reload, a watcher keeps its state when only its wakeup sources change. +- with `wakeup_sources => { "poll" }` the event is only checked every `interval` seconds. +- no wakeup source can watch `package_installed` and `package_uninstalled` events: they use `poll`, so a list without `poll` leaves them unwatched. + #### file_deleted **Description:** Creates an event when the file at the given path is deleted.