diff --git a/docs/Collecting Metrics/AWS CloudWatch Profile Format.mdx b/docs/Collecting Metrics/AWS CloudWatch Profile Format.mdx index bee87a2b2..55f26356c 100644 --- a/docs/Collecting Metrics/AWS CloudWatch Profile Format.mdx +++ b/docs/Collecting Metrics/AWS CloudWatch Profile Format.mdx @@ -203,7 +203,7 @@ An explicitly empty list is invalid; omit the field for unrestricted profiles. | Field | Required | Description | |:--------------------|:--------------------:|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `period` | yes at profile level | CloudWatch aggregation period. It must be from `1m` through `24h` and an exact multiple of `1m`. | -| `lookback` | no | Rolling retrieval horizon. It must be positive, at least one period, an exact multiple of the effective period, and no more than 1,440 buckets. Omission follows the resolved period. | +| `lookback` | no | Rolling retrieval horizon. It must be positive, at least one period, an exact multiple of the effective period, and no more than the collector's 1,440-bucket safety cap. Omission follows the resolved period. | | `publication_delay` | no | Delay before querying a closed bucket -- collector scheduling policy, not an AWS publication SLA. Default `10m`; `0` is valid. | Durations accept duration strings or numeric seconds and must resolve to whole seconds. Stock profiles use canonical @@ -212,14 +212,22 @@ strings such as `1m`, `10m`, and `24h`; custom profiles should do the same for r Every metric may define an optional `query` object with the same fields. Job configuration resolves each field independently in this order, from highest to lowest precedence: -1. `rules[].query` (job configuration) -2. `rule_defaults.query` (job configuration) -3. `metrics[].query` (profile) -4. profile `query` +1. `rules[].metrics[].include[].query` (job metric/statistic item) +2. `rules[].metrics[].query` (job profile-metric group, for explicit `include[]` expansions only) +3. `rules[].query` (job rule) +4. `rule_defaults.query` (job defaults) +5. `metrics[].query` (profile metric) +6. profile `query` After inheritance, the collector validates the complete policy. The combined `publication_delay + lookback + period` horizon must not exceed 14 days. +The query window is `end = align_down(now - publication_delay, period)` and `start = end - lookback`; CloudWatch +`EndTime` is exclusive. One-period lookback is valid. Longer lookback improves tolerance for sparse or late data but +increases response work; it is not needed to repair alignment. Policies are evaluated only on the job's shared +`update_every` ticks. A `period` shorter than `update_every` still controls CloudWatch aggregation and rate +normalization, but intermediate eligible windows may be skipped. + ## Instance dimensions `instance.dimensions` is the exact CloudWatch dimension-name set accepted during discovery. Metrics with missing or @@ -298,6 +306,14 @@ metrics: Set `defaults: false` when the profile must use only the listed MetricNames. If both the group and metric omit `statistics`, the collector uses every statistic declared for that metric in the profile. +A metric group and each included metric may also define `query` with `period`, `lookback`, and/or +`publication_delay`. The group query applies only to series expanded from `include[]`, even when `defaults: true`; +unlisted default-selected series keep rule/profile timing. The item query overrides the group field by field. + +Repeat one MetricName only to assign policies to disjoint statistic sets. Statistic expansion is item statistics, +then group statistics, then all profile-declared statistics. For example, separate `Average` and `Sum` items are +valid; two items that both expand to `Average` are rejected with their configuration paths. + ## Chart template rules The `template` uses Netdata's dynamic chart-template format. See the complete @@ -321,4 +337,3 @@ are ignored (see "How profiles work"). The collector divides rate totals by the effective query period before emission and marks all emitted CloudWatch metric families as floating-point values. - diff --git a/docs/Collecting Metrics/Collectors/Cloud and DevOps/Amazon CloudWatch.mdx b/docs/Collecting Metrics/Collectors/Cloud and DevOps/Amazon CloudWatch.mdx index 2019573cb..939f877de 100644 --- a/docs/Collecting Metrics/Collectors/Cloud and DevOps/Amazon CloudWatch.mdx +++ b/docs/Collecting Metrics/Collectors/Cloud and DevOps/Amazon CloudWatch.mdx @@ -111,6 +111,7 @@ The defaults are designed so a minimal configuration collects something useful: - A rule that omits `metrics` collects the default-enabled metrics from those profiles. - A metric group changes only its named profile; other selected profiles keep their defaults. The group keeps the profile's defaults unless `defaults: false`; either way, it adds the exact AWS MetricNames it lists. - Statistics resolve from the metric entry, then the group, then the profile declaration. +- Query timing resolves field-by-field from the metric item, metric group, rule, rule defaults, profile metric, and profile defaults. A group query applies only to its explicit `include[]` expansions; unlisted defaults keep lower-precedence timing. - Charts appear only for profiles with live metrics. - Discovery and the query blueprint are cached; discovery refreshes every `discovery.refresh_every` seconds (default 300). @@ -119,8 +120,10 @@ The defaults are designed so a minimal configuration collects something useful: **Timing** -- The minimum collection interval is 60 seconds (CloudWatch's minimum metric period). -- Query timing resolves field-by-field: `rules[].query`, then `rule_defaults.query`, then metric and profile defaults, then the built-in 10-minute publication delay. The combined `publication_delay + lookback + period` horizon cannot exceed 14 days. +- The minimum collection interval is 60 seconds. Every per-series policy is evaluated on the job's shared `update_every` ticks; there are no independent per-selection timers. +- Query timing resolves field-by-field, from highest to lowest precedence: `rules[].metrics[].include[].query`, `rules[].metrics[].query`, `rules[].query`, `rule_defaults.query`, profile metric query, profile query, then built-in fallbacks. The combined `publication_delay + lookback + period` horizon cannot exceed 14 days. +- The rolling window is `end = align_down(now - publication_delay, period)` and `start = end - lookback`; `end` is exclusive. One-period lookback is valid. Longer lookback tolerates sparse or late data but increases response work. +- If `period < update_every`, period still controls CloudWatch aggregation and rate normalization, but the first available collection tick queries only the newest eligible window; intermediate windows may be skipped. - The stock S3 storage profile uses a conservative one-day delay policy: AWS documents that S3 storage metrics are reported once per day, without guaranteeing publication within one day. - A successful sparse query can replay its newest eligible CloudWatch value for up to `lookback`. During transient AWS failures, the retained value can be replayed longer, until a successful query replaces or expires it. @@ -150,19 +153,19 @@ AWS bills CloudWatch API usage. `GetMetricData` is the cost driver -- roughly $0 To estimate a job's normal daily cost: ```text -billable metric requests/day ≈ instances × billable metrics per instance × (86,400 / period seconds) +billable metric requests/day ≈ Σ effective-policy groups (instances × calculated metric-request units per instance × 86,400 / max(period seconds, update_every seconds)) ``` -For example, one Billing series at the stock 10-minute period is 86,400 / 600 = 144 requests per day. For a running job, skip the arithmetic and read the **CloudWatch Metric Requests** chart described below -- it reports the billable metric requests actually submitted. +This is a healthy steady-state estimate; retries add work, and alignment near the observation boundary can shift a short sample. For example, one Billing series at the stock 10-minute period with `update_every \<= 600` is 86,400 / 600 = 144 requests per day. For a running job, skip the arithmetic and read the **CloudWatch Metric Requests** chart described below -- it reports the billable metric requests actually submitted. **How the collector keeps cost down** -- Each series is queried once per newly eligible period window, not once per Netdata collection cycle. +- Each series is queried on the first shared collection tick that observes a newer eligible period window. If several windows become eligible between ticks, only the newest is queried. - Curated profiles, exact metric/statistic/resource-tag selection, and single-statistic defaults keep the selected set small. - Compatible rules and profiles share discovery scans; discovery and query plans are cached; `recently_active_only` narrows scans to active resources. -- A transient failure retries after one `update_every`, then doubles the delay within the same eligible window, capped at the effective period. Retries are billable, so `update_every` affects failure-time cost even though it does not set the normal query cadence. +- A transient failure retries after one `update_every`, then doubles the delay within the same eligible window, capped at the effective period. Retries are billable. `update_every` also limits successful request frequency when it is longer than the effective period. -Cost scales with selected targets, instances, metrics, statistics beyond AWS's grouping, periods, and lookback length. Longer lookbacks increase requested datapoints and can disable CloudWatch's three-hour recently-active discovery filter. To reduce cost, narrow `rules[].targets`, `rules[].profiles`, `rules[].metrics`, `rules[].regions`, or configure resource tag filters. +Cost scales with selected targets, instances, metrics, statistics beyond AWS's grouping, effective request frequency, and lookback length. Statistics of one structural metric can share a calculated request unit only when their complete effective policies match; per-selection differences split the group and can increase cost. Longer lookbacks increase requested datapoints and can disable CloudWatch's three-hour recently-active discovery filter. To reduce cost, narrow `rules[].targets`, `rules[].profiles`, `rules[].metrics`, `rules[].regions`, or configure resource tag filters. **Watching collector-issued work** @@ -280,7 +283,7 @@ A user profile file with the same basename as a stock profile overrides it. | Group | Option | Description | Default | Required | |:------|:-----|:------------|:--------|:---------:| -| **Collection** | update_every | Data collection interval (seconds). Must be at least 60 (CloudWatch's minimum period). | 60 | no | +| **Collection** | update_every | Shared Netdata collection interval (seconds), minimum 60. Every per-series policy is evaluated on these ticks; a shorter period does not create an independent timer and intermediate eligible windows may be skipped. | 60 | no | | | autodetection_retry | Recheck interval (seconds) when the job fails to start. Default `0` means no retry; set a positive value to keep retrying. | 0 | no | | | timeout | AWS operation timeout (seconds). Identity, resource-tag, and query operations use it for their operation scope; discovery shares one timeout across its whole refresh stage. | 30 | no | | **Credentials** | credentials | Up to 64 named credential sources. Every source has a `type` of `default` (AWS SDK default chain) or `static` (explicit access/session credentials in `type_static`). Multiple targets can share one credential source. | | yes | @@ -305,17 +308,25 @@ A user profile file with the same basename as a stock profile overrides it. | | rules[].metrics[].profile | Profile basename. It must already be selected by `rules[].profiles` and may appear in only one metrics group per rule. Other selected profiles keep their default-enabled metrics. | | yes | | | rules[].metrics[].defaults | Include this profile's default-enabled metrics before adding the exact MetricNames below. Set `false` for an exact-only selection. | yes | no | | | rules[].metrics[].statistics | Optional non-empty AWS statistics inherited by included metrics that omit their own list. When both lists are omitted, the profile-declared statistics are used. Named statistics are case-insensitive. | | no | -| | rules[].metrics[].include | Non-empty list of exact, case-sensitive AWS CloudWatch MetricNames added by this group. Duplicate names are rejected. | | yes | +| **Rules / Query Policy** | rules[].metrics[].query | Optional field-by-field timing override applied only to series expanded from this group's explicit `include[]` items. Unlisted default-selected series keep rule/profile timing. | | no | +| | rules[].metrics[].query.period | CloudWatch aggregation period for explicitly included series, from `1m` through `24h` as an exact multiple of `1m`. Eligibility is still evaluated only on shared `update_every` ticks. | | no | +| | rules[].metrics[].query.lookback | Rolling window for explicitly included series. It must be at least the effective period, an exact period multiple, and no more than the collector's 1,440-bucket safety cap. | | no | +| | rules[].metrics[].query.publication_delay | Collector wait after a bucket closes for explicitly included series. This is a scheduling policy, not an AWS publication guarantee; explicit `0s` is allowed. | | no | +| **Rules** | rules[].metrics[].include | Non-empty list of exact, case-sensitive AWS CloudWatch MetricNames added by this group. A MetricName may repeat only when the items expand to disjoint statistic sets. | | yes | | | rules[].metrics[].include[].name | Exact, case-sensitive AWS CloudWatch MetricName exported by the profile. | | yes | | | rules[].metrics[].include[].statistics | Optional non-empty replacement for the group statistics. When both are omitted, inherit every statistic declared for the metric by the profile. Use `Average`, `Minimum`, `Maximum`, `Sum`, `SampleCount`, or `p`; named statistics are case-insensitive. | | no | -| | rules[].regions | Canonical lowercase AWS region codes selected by this rule. The compiler intersects them with intrinsic profile restrictions; CloudFront and the Billing profiles support only `us-east-1`. | | yes | +| **Rules / Query Policy** | rules[].metrics[].include[].query | Optional field-by-field timing override for this item's expanded metric/statistic series. It overrides the enclosing metric-group query; omitted fields inherit independently. | | no | +| | rules[].metrics[].include[].query.period | CloudWatch aggregation period for this item's expanded series, from `1m` through `24h` as an exact multiple of `1m`. It remains the aggregation and rate divisor when shorter than `update_every`. | | no | +| | rules[].metrics[].include[].query.lookback | Rolling window for this item's expanded series. It must be at least the effective period, an exact period multiple, and no more than the collector's 1,440-bucket safety cap. | | no | +| | rules[].metrics[].include[].query.publication_delay | Collector wait after a bucket closes for this item's expanded series. This is a scheduling policy, not an AWS publication guarantee; explicit `0s` is allowed. | | no | +| **Rules** | rules[].regions | Canonical lowercase AWS region codes selected by this rule. The compiler intersects them with intrinsic profile restrictions; CloudFront and the Billing profiles support only `us-east-1`. | | yes | | **Rules / Query Policy** | rule_defaults.query | Shared query timing inherited field-by-field by collection rules. Omitted fields fall through to profile or built-in fallbacks. The resolved `publication_delay + lookback + period` horizon cannot exceed 14 days. | | no | -| | rule_defaults.query.period | Default CloudWatch aggregation period from `1m` through `24h`, as an exact multiple of `1m`. An omitted rule period inherits this value before nested metric and profile query defaults. | | no | -| | rule_defaults.query.lookback | Default rolling window searched for the newest eligible datapoint. It must be at least the effective period, an exact period multiple, and no more than 1,440 buckets (a bucket is one period). | | no | +| | rule_defaults.query.period | Default CloudWatch aggregation period from `1m` through `24h`, as an exact multiple of `1m`. It overrides profile timing and is inherited unless a rule, metric group, or metric item replaces it. | | no | +| | rule_defaults.query.lookback | Default rolling window searched for the newest eligible datapoint. It must be at least the effective period, an exact period multiple, and no more than the collector's 1,440-bucket safety cap. | | no | | | rule_defaults.query.publication_delay | Default collector wait after a bucket closes before it becomes eligible. This is a scheduling policy, not an AWS publication guarantee. Omission falls through to the profile value and then the built-in `10m` fallback. Setting this option overrides profile-specific delays for every inheriting rule, including the stock S3 storage profile's conservative `1d`; AWS documents only that S3 storage metrics are reported once per day, so use a shorter default only after verifying each workload's publication timing. | | no | -| | rules[].query | Optional query timing overrides for this rule. Each omitted field independently inherits `rule_defaults.query`, then the relevant profile or built-in fallback. | | no | -| | rules[].query.period | CloudWatch aggregation period for every series selected by this rule, from `1m` through `24h` as an exact multiple of `1m`. Rate metrics are normalized using this effective period. | | no | -| | rules[].query.lookback | Rolling window searched for the newest complete finite datapoint. It must be at least the effective period, an exact period multiple, and no more than 1,440 buckets (a bucket is one period). Successful queries may present the retained datapoint as current for up to this duration; longer lookbacks increase response work. | | no | +| | rules[].query | Optional query timing overrides for this rule. Each omitted field independently inherits `rule_defaults.query`, then the relevant profile or built-in fallback; explicit metric-group and item values take precedence. | | no | +| | rules[].query.period | CloudWatch aggregation period for series selected by this rule, from `1m` through `24h` as an exact multiple of `1m`; metric-group and item values can replace it. Rate metrics are normalized using the final effective period. | | no | +| | rules[].query.lookback | Rolling window searched for the newest complete finite datapoint. It must be at least the effective period, an exact period multiple, and no more than the collector's 1,440-bucket safety cap. Successful queries may present the retained datapoint as current for up to this duration; longer lookbacks increase response work. | | no | | | rules[].query.publication_delay | Collector wait after a bucket closes before querying it. This is a scheduling policy, not an AWS publication guarantee. Explicit `0s` is allowed for metrics known to publish immediately. | | no | | **Rules / Resource Filters** | rule_defaults.filters.resource_tags | Job-wide list of exact, case-sensitive AWS resource tag predicates inherited by rules that omit `rules[].filters.resource_tags`. All keys must match; any listed value for one key may match. The Resource Groups Tagging API performs the focused lookup and requires `tag:GetResources`. | | no | | | rule_defaults.filters.resource_tags[].key | Exact AWS resource tag key. A filter list supports at most 50 distinct keys. | | yes | @@ -465,7 +476,7 @@ jobs: ###### Different timing for metrics in one profile -Use disjoint exact metric selections when one service needs different query timing. Earlier rules own only the metric/statistic series they select, so these two Lambda rules do not shadow each other. +Use one profile metric group with disjoint statistic items when one AWS MetricName needs different timing. The group query is the default for its explicit items; the p90 item overrides it. Repeated MetricNames whose expanded statistic sets overlap are rejected.
Config @@ -480,7 +491,7 @@ jobs: - name: base credentials: sdk_default rules: - - name: lambda-activity + - name: lambda-duration targets: [base] profiles: defaults: false @@ -488,37 +499,27 @@ jobs: metrics: - profile: lambda defaults: false - statistics: [Sum] - include: - - name: Invocations - regions: [us-east-1] - query: - period: 5m - lookback: 30m - publication_delay: 10m - - name: lambda-latency - targets: [base] - profiles: - defaults: false - include: [lambda] - metrics: - - profile: lambda - defaults: false - statistics: [Average, p90] + query: + period: 5m + lookback: 30m + publication_delay: 10m include: - name: Duration + statistics: [Average] + - name: Duration + statistics: [p90] + query: + period: 1m + lookback: 5m + publication_delay: 5m regions: [us-east-1] - query: - period: 1m - lookback: 5m - publication_delay: 5m ```
###### Lower resolution to reduce cost -Collect the default profiles at five-minute resolution and refresh discovery less often. `rule_defaults.query` applies field-by-field to every rule that does not override it, replacing profile timing defaults -- the daily S3 storage profile is excluded here so the job-wide five-minute policy does not query it before AWS publishes. `update_every` controls how often charts update and the failure-retry cadence, not the normal query cost. +Collect the default profiles at five-minute resolution and refresh discovery less often. `rule_defaults.query` applies field-by-field to every rule that does not override it, replacing profile timing defaults -- the daily S3 storage profile is excluded here so the job-wide five-minute policy does not query it before AWS publishes. Matching `update_every` to the period lets every newly eligible window be observed without extra cached-only collection ticks.
Config @@ -551,7 +552,7 @@ jobs: ###### AWS Billing estimated charges -Collect each available exact Billing grain independently. Billing metrics must be enabled first, are published only in `us-east-1`, and do not support resource-tag filters or resource-tag-derived labels. The stock profiles use a 10-minute period and 24-hour retrieval window; a `rules[].query` block would override only the fields it sets. +Collect each available exact Billing grain independently. Billing metrics must be enabled first, are published only in `us-east-1`, and do not support resource-tag filters or resource-tag-derived labels. The stock profiles use a 10-minute period and 24-hour retrieval window; job query blocks override only the fields they set according to selection precedence.
Config @@ -606,7 +607,7 @@ jobs: - key: environment values: [production] rules: - - name: endpoint-averages + - name: endpoint-split-timing targets: [base] profiles: defaults: false @@ -614,32 +615,24 @@ jobs: metrics: - profile: privatelink_endpoint defaults: false - statistics: [Average] + query: + period: 1m + lookback: 5m + publication_delay: 5m include: - name: ActiveConnections + statistics: [Average] - name: BytesProcessed + statistics: [Average] - name: NewConnections - regions: [us-east-1] - query: - period: 1m - lookback: 5m - publication_delay: 5m - - name: endpoint-six-hour-bytes - targets: [base] - profiles: - defaults: false - include: [privatelink_endpoint] - metrics: - - profile: privatelink_endpoint - defaults: false - include: + statistics: [Average] - name: BytesProcessed statistics: [Sum] + query: + period: 6h + lookback: 6h + publication_delay: 5m regions: [us-east-1] - query: - period: 6h - lookback: 6h - publication_delay: 5m - name: endpoint-subnets targets: [base] profiles: @@ -675,7 +668,7 @@ jobs: - key: environment values: [production] rules: - - name: service-traffic-averages + - name: service-split-timing targets: [base] profiles: defaults: false @@ -683,49 +676,32 @@ jobs: metrics: - profile: privatelink_service defaults: false - statistics: [Average] + query: + period: 1m + lookback: 5m + publication_delay: 5m include: - name: ActiveConnections + statistics: [Average] - name: BytesProcessed + statistics: [Average] - name: NewConnections + statistics: [Average] - name: RstPacketsSent - regions: [us-east-1] - query: - period: 1m - lookback: 5m - publication_delay: 5m - - name: service-endpoint-count - targets: [base] - profiles: - defaults: false - include: [privatelink_service] - metrics: - - profile: privatelink_service - defaults: false - include: + statistics: [Average] - name: EndpointsCount statistics: [Average] - regions: [us-east-1] - query: - period: 5m - lookback: 5m - publication_delay: 5m - - name: service-six-hour-bytes - targets: [base] - profiles: - defaults: false - include: [privatelink_service] - metrics: - - profile: privatelink_service - defaults: false - include: + query: + period: 5m + lookback: 5m + publication_delay: 5m - name: BytesProcessed statistics: [Sum] + query: + period: 6h + lookback: 6h + publication_delay: 5m regions: [us-east-1] - query: - period: 6h - lookback: 6h - publication_delay: 5m labels: resource_tags: - key: Name @@ -983,7 +959,7 @@ These disabled opt-in profiles are collected when a rule names them in `profiles **PrivateLink service grains.** The default `privatelink_service` profile identifies the provider service by `service_id` and is the only grain that exports `EndpointsCount`. The opt-in profiles split by `availability_zone`, `load_balancer_arn`, both, or consumer `vpc_endpoint_id`. All five share one `AWS/PrivateLinkServices` discovery scan and join tags through the parent endpoint service (`ec2:vpc-endpoint-service`). The collector deliberately does not attach endpoint or load-balancer tags to detailed children -- a `vpc_endpoint_id` can identify a consumer endpoint outside the service-owning account. `EndpointsCount` is read on its documented five-minute cadence and records zero when no datapoint is published; traffic gauges show gaps when absent. -**PrivateLink cost.** At stock timing (five-minute period, lookback, and publication delay), an endpoint, subnet, or default service instance has five structural CloudWatch metrics: (86,400 / 300) × 5 = 1,440 billable metric requests per day before retries. Each opt-in detailed service instance has four, or 1,152 per day. A one-minute override runs its selected metrics five times as often; narrow profiles, metrics, regions, grains, and resource tags when that freshness is not required. +**PrivateLink cost.** At stock timing (five-minute period, lookback, and publication delay) and `update_every \<= 300`, an endpoint, subnet, or default service instance has five structural CloudWatch metrics: (86,400 / 300) × 5 = 1,440 billable metric requests per day before retries. Each opt-in detailed service instance has four, or 1,152 per day. With `update_every \<= 60`, a one-minute override runs its selected metrics five times as often; a slower collection interval limits execution and skips intermediate windows. Narrow profiles, metrics, regions, grains, and resource tags when that freshness is not required. **Cardinality warning.** These opt-in profiles include potentially high-cardinality data. **S3 Request Metrics** additionally require per-bucket request-metrics configuration in AWS and are billed at CloudWatch custom-metric rates; they collect nothing until enabled on the bucket. PrivateLink cardinality grows with endpoint subnets, service Availability Zones, load balancers, and consumer endpoints; the combined Availability Zone/load-balancer grain multiplies those dimensions. The Billing service/account grains grow with the payer's services and linked accounts. @@ -1145,10 +1121,10 @@ Check the following: CloudWatch publishes metrics with a delay. -- Keep `rules[].query.period` at or above the metric's real publication cadence. A shorter override increases billed query frequency but cannot make AWS publish more often, so it can create empty windows. -- Set `rules[].query.publication_delay` when a workload publishes completed buckets later than its profile or the built-in `10m` fallback. +- Keep the effective query `period` at or above the metric's real publication cadence. A shorter override can increase billed query frequency but cannot make AWS publish more often, so it can create empty windows. When it is also shorter than `update_every`, request execution remains limited to shared collection ticks and intermediate windows may be skipped. +- Set a rule, metric-group, or metric-item `publication_delay` when a workload publishes completed buckets later than its profile or the built-in `10m` fallback. - Check `rule_defaults.query.publication_delay` before overriding an individual rule. A job-wide value replaces profile-specific delays for inheriting rules, including the stock S3 storage profile's conservative `1d`, and a shorter value can query daily data before it is published. -- Set `rules[].query.lookback` to search a wider rolling window for sparse datapoints. It must be an exact multiple of the effective period. +- Set `lookback` at the narrowest applicable query level to search a wider rolling window for sparse datapoints. It must be at least one effective period and an exact period multiple; one period is valid. - **Stale-looking values** -- a successful query presents its newest eligible datapoint on every Netdata collection cycle, so an old CloudWatch value can appear current for up to `lookback`. During transient AWS failures, replay can continue longer until a successful query replaces or expires it. - Longer lookbacks increase response work and may disable `recently_active_only` for the shared discovery scan. - Transient query failures preserve the retained value and retry after one `update_every`; later delays double within the same eligible window up to the effective period. A newly eligible window resets the backoff. diff --git a/docs/Collecting Metrics/Secrets Management/Secret Stores/Vault.mdx b/docs/Collecting Metrics/Secrets Management/Secret Stores/Vault.mdx index 7c2396647..478a070eb 100644 --- a/docs/Collecting Metrics/Secrets Management/Secret Stores/Vault.mdx +++ b/docs/Collecting Metrics/Secrets Management/Secret Stores/Vault.mdx @@ -34,6 +34,12 @@ This page covers Vault specific setup. For the full resolver overview and syntax Netdata reads existing secrets from Vault. It does not create or renew Vault tokens. If the configured token expires or becomes invalid, secret resolution fails until Netdata can read a valid token again. If you use `token_file` mode, Netdata re-reads the file on every secret resolution, so an external process (e.g. Vault Agent, a cron job) can renew the token by writing to the file. For KV v2 secrets, Netdata does not add `/data/` to the path automatically. +The Dynamic Configuration **Test** action initiates one real authenticated `GET /v1/auth/token/lookup-self` request using the configured token, namespace header, TLS settings, proxy path, and timeout. HTTP 200 is operational success by status; its response body is not inspected. An exact permission-only HTTP 403 body is reported as validation-only because it cannot reliably distinguish a valid token without self-lookup permission from an invalid token on older Vault versions or another authentication restriction. Invalid-token, ambiguous, malformed, or oversized HTTP 403 bodies fail, as does every other HTTP status. + +The request does not read a secret or prove access to any secret path. It also cannot prove that an intermediary preserved the namespace header or that Vault used it. A gateway or allowlist can block the self-lookup route even when configured secret reads would work. When the request reaches Vault, it creates normal audit and activity evidence. For a limited-use service token, Test can consume the final remaining use and cause later secret resolution to fail; an external proxy or service mesh can also retry independently. + +A configured token file must resolve to a regular file no larger than 1 MiB. Symlinks to regular files are supported. + ## Setup @@ -90,7 +96,7 @@ The following options can be defined for this secretstore backend. | | [tls_skip_verify](#option-tls-skip-verify) | Disable TLS certificate verification for Vault requests. | no | no | | | timeout | Timeout in seconds for HTTP requests made by this secretstore backend. | 3 | no | | **Token** | mode_token.token | Vault token value. Required when `mode` is `token`. | | yes | -| **Token File** | mode_token_file.path | Path to a file containing the Vault token. Required when `mode` is `token_file`. | | yes | +| **Token File** | [mode_token_file.path](#option-token-file-mode-token-file-path) | Path to a file containing the Vault token. Required when `mode` is `token_file`. | | yes | ##### mode @@ -109,6 +115,12 @@ Prefer `token_file` for production so the token is stored separately from the se This is insecure. Use it only as a temporary workaround or in a non-production environment. + +##### mode_token_file.path + +The path may be a regular file or a symlink to a regular file. The file must be no larger than 1 MiB. + +
@@ -256,6 +268,13 @@ Check the Netdata Agent logs when the collector starts or restarts. Vault resolv Check the Vault token policy and, if you use Vault Enterprise namespaces, confirm that `namespace` is correct. If you use a short-lived token, make sure the token is renewed or replaced before it expires. +### The Dynamic Configuration Test fails + +The Test action makes a real authenticated request to Vault's token self-lookup endpoint. HTTP 200 is operational success by status; its response body is not inspected. An exact permission-only HTTP 403 body is reported as validation-only because it cannot reliably prove whether the token is valid. Invalid-token, ambiguous, malformed, or oversized HTTP 403 bodies fail. An unreachable endpoint and every other HTTP status also fail. + +Operational success proves only that the responding endpoint returned HTTP 200 for self-lookup at that time. It does not prove that Vault accepted the token or that the token can read a particular secret path. Check secret-path policies separately. A gateway or deployment allowlist can reject the self-lookup route even when configured secret reads would work. + + ### Secret or key is not found Check the operand carefully: diff --git a/docs/Collecting Metrics/Service Discovery/Discoverer/HTTP endpoint.mdx b/docs/Collecting Metrics/Service Discovery/Discoverer/HTTP endpoint.mdx index a56256cf4..3249fedc9 100644 --- a/docs/Collecting Metrics/Service Discovery/Discoverer/HTTP endpoint.mdx +++ b/docs/Collecting Metrics/Service Discovery/Discoverer/HTTP endpoint.mdx @@ -46,8 +46,10 @@ Each discovery cycle, the discoverer: - Only **one URL per pipeline**. To pull from multiple sources, configure multiple HTTP discovery pipelines (each as its own UI entry, or split the file into one job per source). - **Response size** is capped at 10 MiB. +- A DynCfg **test** performs one complete fetch only when `method` is empty/default or exactly `GET`. It uses the configured authentication, headers, TLS, proxy, redirects, and timeout; requires HTTP 200; and parses every returned item. Redirects can make up to 10 HTTP requests. The test discards the targets without evaluating `services:` rules or creating jobs. Other configured methods are intentionally validation-only. - **One-shot mode** (`interval: 0`) fetches a single time when the pipeline starts. It does **not** refetch on SD reload — recreate the pipeline to refresh. -- **`bearer_token_file`** under `/var/run/secrets/` is treated as optional when Netdata is **not** running in Kubernetes (so the same config can be used in a Helm deployment without erroring out on dev hosts). +- A missing **`bearer_token_file`** under `/var/run/secrets/` is treated as optional when Netdata is **not** running in Kubernetes (so the same config can be used in a Helm deployment without erroring out on dev hosts). +- Configured bearer-token and TLS files have [file-safety requirements](#option-headers-username-password-bearer-token-file-proxy-url-tls-skip-verify-etc). - The discoverer does not introspect the items it received — anything beyond what the upstream endpoint provides must be inferred via service rules. @@ -112,7 +114,7 @@ With `auto`, the decoder uses `Content-Type` when it is unambiguous (`applicatio ##### headers / username / password / bearer_token_file / proxy_url / tls_skip_verify / etc. -See any go.d HTTP-based collector (`httpcheck`, `prometheus`, `nginx`, …) for the full set. Notable: when `bearer_token_file` points under `/var/run/secrets/` and Netdata is **not** running inside Kubernetes, missing token files are silently ignored. +See any go.d HTTP-based collector (`httpcheck`, `prometheus`, `nginx`, …) for the full set. Bearer-token and TLS CA paths must resolve to regular files no larger than 1 MiB. When both `tls_cert` and `tls_key` are configured, each path has the same requirement. Symlinks to regular files are supported. When `bearer_token_file` points under `/var/run/secrets/` and Netdata is **not** running inside Kubernetes, a missing token file is silently ignored. @@ -311,6 +313,13 @@ Your endpoint mixes shapes — some items target `httpcheck`, some target `prome After enabling the discoverer, confirm the endpoint is reachable and items are being parsed. +### Test a UI-managed configuration + +DynCfg test performs one complete production fetch for an empty/default or exact `GET` method. Success proves that the request could be built, the configured auth/TLS/proxy/redirect path worked at that instant, the endpoint returned HTTP 200 with at most 10 MiB, and every item parsed into a target. + +The fetched targets are discarded: the test does not evaluate `services:` rules, render or validate collector jobs, publish targets, or install jobs. Non-GET methods are intentionally reported as validation-only because executing them could mutate the endpoint. + + ### Confirm the endpoint is being fetched Watch the agent log for `discoverer=http` messages. With systemd: diff --git a/docs/Collecting Metrics/Service Discovery/Discoverer/Local listening processes.mdx b/docs/Collecting Metrics/Service Discovery/Discoverer/Local listening processes.mdx index 3041a835d..3a5ace211 100644 --- a/docs/Collecting Metrics/Service Discovery/Discoverer/Local listening processes.mdx +++ b/docs/Collecting Metrics/Service Discovery/Discoverer/Local listening processes.mdx @@ -34,7 +34,7 @@ This page covers `net_listeners`-specific setup. For the broader Service Discove Each discovery cycle, the discoverer: -1. **Reads the kernel's TCP/UDP listening-socket table** via the bundled `local-listeners` helper (which reads `/proc` on Linux, `netstat`-equivalents elsewhere). +1. **Reads the kernel's TCP/UDP listening-socket table** via the bundled `local-listeners` helper, which reads Linux procfs socket and process data. 2. **Builds one target per `(protocol, IP, port, process)` tuple**, exposing `.Protocol`, `.IPAddress`, `.Port`, `.Comm` (process basename), `.Cmdline` (full command line), and `.Address` (the convenience `IPAddress:Port`). 3. **Caches** each target for 10 minutes so a brief disappearance (process restart) does not churn collector jobs. 4. **Runs the `services:` rules** against each target. The stock conf carries ~100 curated rules covering the bulk of go.d modules (databases, web servers, caches, message queues, exporters). @@ -45,6 +45,7 @@ Each discovery cycle, the discoverer: - Only **local** listeners are visible. Discovering services on other hosts requires another discoverer (`http`, `snmp`, `k8s`, or a custom one). - The discoverer needs to **read kernel socket information**. On Linux this works for processes owned by other users only when Netdata can read the appropriate `/proc//net` files; the Netdata installer configures this via the `local-listeners` setuid helper. +- The bundled `local-listeners` helper is currently built for Linux. A missing or non-executable helper makes both normal discovery and the DynCfg operational test fail. - **Containerised services in `host` networking** appear as listeners and are picked up here, not by the Docker discoverer. Services in private container networks must be discovered by the Docker discoverer instead. - The discoverer does not introspect process runtime — anything beyond port/`comm`/`cmdline` (e.g. config-file path, version, runtime URL prefix) must be inferred via service rules or known by convention. @@ -244,6 +245,13 @@ The last rule in the stock conf catches generic Prometheus exporters by port. `p After enabling the discoverer, confirm it is finding listeners and producing jobs. +### Test a candidate configuration without applying it + +A DynCfg `test` runs the same installed `nd-run` and `local-listeners` path once, with the production flags and configured `timeout`, then parses the complete snapshot and discards it. Success proves that this one helper invocation and parse worked on the local host at that instant. + +The test does not start discovery, cache or publish targets, evaluate `services:` rules, create collector jobs, or prove that a generated job can connect to its service. It requires no permissions beyond normal local-listener discovery. Work and buffered output scale with the host's socket/process inventory; caller cancellation and `timeout` bound elapsed time, not the number of inspected entries. + + ### Confirm listeners are being scanned Watch the agent log for `discoverer=net_listeners` messages. With systemd: @@ -292,4 +300,4 @@ The stock `exporter` catch-all rule (last in the file) is greedy by design — a ### Generated jobs fail to start -The discoverer creates jobs but does not run them. Common causes: the rendered template assumes credentials the local service rejects (e.g. RabbitMQ default `guest:guest`); `0.0.0.0` listeners produce `0.0.0.0:port` addresses that the collector cannot connect to (use `127.0.0.1` in the template if appropriate); the service has TLS but the template uses HTTP. +The discoverer creates jobs but does not run them. Common causes: the rendered template assumes credentials the local service rejects (e.g. RabbitMQ default `guest:guest`); a rule renders the wrong port or protocol; or the service has TLS but the template uses HTTP. diff --git a/docs/Collecting Metrics/Service Discovery/Service Discovery.mdx b/docs/Collecting Metrics/Service Discovery/Service Discovery.mdx index 6f1c8ebf5..8c9d6fd52 100644 --- a/docs/Collecting Metrics/Service Discovery/Service Discovery.mdx +++ b/docs/Collecting Metrics/Service Discovery/Service Discovery.mdx @@ -18,7 +18,7 @@ Each SD pipeline is a `discoverer:` (where to look) plus a list of `services:` r ### Jump To -[How it works](#how-it-works) • [Configuration file structure](#configuration-file-structure) • [Rule evaluation semantics](#rule-evaluation-semantics) • [Template helper reference](#template-helper-reference) • [config_template rendering](#config_template-rendering) • [Supported discoverers](#supported-discoverers) • [Mixing discoverers](#mixing-discoverers) • [Troubleshooting](#troubleshooting) +[How it works](#how-it-works) • [Configuration file structure](#configuration-file-structure) • [Testing configurations](#testing-configurations) • [Rule evaluation semantics](#rule-evaluation-semantics) • [Template helper reference](#template-helper-reference) • [config_template rendering](#config_template-rendering) • [Supported discoverers](#supported-discoverers) • [Mixing discoverers](#mixing-discoverers) • [Troubleshooting](#troubleshooting) ## How it works @@ -57,6 +57,16 @@ services: - Where each discoverer's stock conf ships (with the Netdata package, with the Helm chart, or not at all) is documented on its per-discoverer page. +## Testing configurations + +Testing a UI-managed pipeline builds a complete temporary pipeline but does not publish targets, install jobs, or make the configuration persistent. The response says explicitly when only configuration validation was possible. + +Operational guarantees depend on the discoverer. Docker runs one bounded container-list query. Local-listener discovery runs and parses one helper snapshot. HTTP discovery runs one complete production fetch only when `method` is empty/default or exactly `GET`; it uses the configured authentication, headers, TLS, proxy, redirects, and timeout, requires HTTP 200, enforces the 10 MiB response limit, and parses every returned item. Redirect handling can issue at most 10 requests. + +The HTTP test discards the parsed targets. It does not evaluate `services:` rules, render or validate collector jobs, publish targets, or install jobs. A non-empty HTTP method other than exact `GET`, plus Kubernetes and SNMP, is intentionally validation-only. + + + ## Rule evaluation semantics For each discovered target, the engine walks the `services:` array top-to-bottom. The two rule shapes behave differently: diff --git a/docs/Network Performance Monitoring/Network Flows/Configuration.mdx b/docs/Network Performance Monitoring/Network Flows/Configuration.mdx index c911de2b0..08b4fd911 100644 --- a/docs/Network Performance Monitoring/Network Flows/Configuration.mdx +++ b/docs/Network Performance Monitoring/Network Flows/Configuration.mdx @@ -190,13 +190,15 @@ Per-tier values: | Key | Default per tier | Notes | |---|---|---| -| `size_of_journal_files` | `10GB` | Disk budget for this tier. Minimum `100MB`. Set to `null` to disable size-based retention on this tier. | +| `size_of_journal_files` | `10GB` | Retained-artifact budget for journal data and finalized per-journal facet sidecars. Minimum `100MB`. Set to `null` to disable size-based retention on this tier. | | `duration_of_journal_files` | `null` | Optional time budget for this tier. The default disables time-based eviction; set a duration such as `24h` or `14d` to add an age cap. | Either configured limit can evict old files. The tier expires whichever configured limit is hit first. At least one of the two must be set per tier (validation enforces this). If you omit a tier entry entirely, that tier uses the built-in defaults (`10GB` / no time limit). If you provide a tier entry but omit one of the two knobs, the omitted knob falls back to its built-in default. Setting either to `null` explicitly disables that limit on that tier. +The size budget is not a strict filesystem quota. The active journal counts toward the budget but is protected from deletion, so a tier can temporarily exceed its limit until rotation. Shared `facet-state.bin`, temporary sidecar files, and unrelated files are not charged to an individual tier. + Standalone CLI runs still accept the legacy uniform retention flags: `--netflow-retention-size-of-journal-files` and `--netflow-retention-duration-of-journal-files`. They apply the same value to diff --git a/docs/Network Performance Monitoring/Network Flows/Field Reference.mdx b/docs/Network Performance Monitoring/Network Flows/Field Reference.mdx index f3f2e7f9b..0f97cdd1a 100644 --- a/docs/Network Performance Monitoring/Network Flows/Field Reference.mdx +++ b/docs/Network Performance Monitoring/Network Flows/Field Reference.mdx @@ -282,20 +282,20 @@ Column legend: | Field | Type | v5 | v7 | v9 | IPFIX | sFlow | Source | Tiers | Selectivity | Notes | |---|---|---|---|---|---|---|---|---|---|---| -| `BYTES` | uint64 | ✓ | ✓ | ✓ | ✓ | ✓ | decoder | all | metric, filter | Canonical byte counter, scaled by `SAMPLING_RATE` at ingest. Ordinary v9/IPFIX prefer IEs 1/2 and fall back to IEs 23/24 as a matched family. Cisco ASA NSEL event 5 uses initiator/responder IEs 231/232. sFlow derives it from decoded L3 length | +| `BYTES` | uint64 | ✓ | ✓ | ✓ | ✓ | ✓ | decoder | all | metric | Canonical byte counter, scaled by `SAMPLING_RATE` at ingest. Ordinary v9/IPFIX prefer IEs 1/2 and fall back to IEs 23/24 as a matched family. Cisco ASA NSEL event 5 uses initiator/responder IEs 231/232. sFlow derives it from decoded L3 length | | `DIRECTION` | string | — | — | ◐ | ◐ | — | decoder | all | facet, group-by, filter | v9 IE 61, IPFIX IE 61/239. sFlow has no native direction | | `DST_ADDR` | IP | ✓ | ✓ | ◐ | ◐ | ◐ | decoder | raw | facet, group-by, filter | v9/IPFIX IE 12/28; sFlow `SampledHeader`/`SampledIPv4`/`SampledIPv6`. Raw-only | | `DST_ADDR_NAT` | IP | — | — | ◐ | ◐ | — | decoder | raw | facet, group-by, filter | v9 IE 226/282; IPFIX `postNATdestinationIPv4/IPv6Address` | | `DST_AS` | uint32 | ✓ | ✓ | ◐ | ◐ | ◐ | both | all | facet, group-by, filter | decoder IE 17 / sFlow `ExtendedGateway` last AS in path. Enrichment chain: `asn_providers` (default `[flow, routing, geoip]`); per-CIDR `enrichment.networks..asn` overrides | | `DST_AS_NAME` | string | — | — | — | — | — | enrichment | all | facet, group-by, filter | `format_as_name(DST_AS, attrs.asn_name)` → `AS{n} {name}`; falls back to `AS0 Unknown ASN` or `AS0 Private IP Address Space` | -| `DST_AS_PATH` | string | — | — | — | — | ◐ | both | raw | filter | sFlow `ExtendedGateway` BGP path. Routing enrichment overlay (BMP / BioRIS) for non-sFlow exporters | -| `DST_COMMUNITIES` | string | — | — | — | — | ◐ | both | raw | filter | sFlow `ExtendedGateway` communities. Routing enrichment overlay (BMP / BioRIS) | +| `DST_AS_PATH` | string | — | — | — | — | ◐ | both | raw | facet, group-by, filter | sFlow `ExtendedGateway` BGP path. Routing enrichment overlay (BMP / BioRIS) for non-sFlow exporters | +| `DST_COMMUNITIES` | string | — | — | — | — | ◐ | both | raw | facet, group-by, filter | sFlow `ExtendedGateway` communities. Routing enrichment overlay (BMP / BioRIS) | | `DST_COUNTRY` | string | — | — | — | — | — | enrichment | all | facet, group-by, filter | GeoIP MMDB on `DST_ADDR` → optional override from `enrichment.networks..country` | | `DST_GEO_CITY` | string | — | — | — | — | — | enrichment | raw | facet, group-by, filter | GeoIP city MMDB. Raw-only (dropped at rollup) | | `DST_GEO_LATITUDE` | string | — | — | — | — | — | enrichment | raw | filter, hidden | GeoIP coordinates. Raw-only; hidden in default table view | | `DST_GEO_LONGITUDE` | string | — | — | — | — | — | enrichment | raw | filter, hidden | GeoIP coordinates. Raw-only; hidden in default table view | | `DST_GEO_STATE` | string | — | — | — | — | — | enrichment | all | facet, group-by, filter | GeoIP subdivision. Preserved in rollups | -| `DST_LARGE_COMMUNITIES` | string | — | — | — | — | — | enrichment | raw | filter | RFC 8092 large communities from routing enrichment (BMP / BioRIS) | +| `DST_LARGE_COMMUNITIES` | string | — | — | — | — | — | enrichment | raw | facet, group-by, filter | RFC 8092 large communities from routing enrichment (BMP / BioRIS) | | `DST_MAC` | MAC | — | — | ◐ | ◐ | ◐ | decoder | raw | facet, group-by, filter | v9 IE 80/57; IPFIX same. sFlow from `SampledHeader` datalink or `SampledEthernet` | | `DST_MASK` | uint8 | ✓ | ✓ | ◐ | ◐ | ◐ | both | raw | facet, group-by, filter | v9 IE 13/29; sFlow `ExtendedRouter`. Enrichment overlay via `net_providers` (default `[flow, routing]`) plus per-CIDR overrides | | `DST_NET_NAME` | string | — | — | — | — | — | enrichment | all | facet, group-by, filter | `enrichment.networks..name` (static) merged with network sources by ascending prefix length | @@ -305,7 +305,7 @@ Column legend: | `DST_NET_TENANT` | string | — | — | — | — | — | enrichment | all | facet, group-by, filter | `enrichment.networks..tenant` from static + network sources | | `DST_PORT` | uint16 | ✓ | ✓ | ◐ | ◐ | ◐ | decoder | raw | facet, group-by, filter | v9/IPFIX IE 11. sFlow from `SampledIPv4`/`SampledIPv6` or `SampledHeader` transport parse. Raw-only | | `DST_PORT_NAT` | uint16 | — | — | ◐ | ◐ | — | decoder | raw | facet, group-by, filter | v9 IE 228; IPFIX `postNAPTdestinationTransportPort` | -| `DST_PREFIX` | IP | ✓ | ✓ | ◐ | — | — | decoder | raw | filter | v5/v7 derived from `DST_ADDR` & `DST_MASK`. v9 IE 45 (`Ipv4DstPrefix`). IPFIX has no canonical mapping; sFlow none | +| `DST_PREFIX` | IP | ✓ | ✓ | ◐ | — | — | decoder | raw | facet, group-by, filter | v5/v7 derived from `DST_ADDR` & `DST_MASK`. v9 IE 45 (`Ipv4DstPrefix`). IPFIX has no canonical mapping; sFlow none | | `DST_VLAN` | uint16 | — | — | ◐ | ◐ | ◐ | decoder | all | facet, group-by, filter | v9 IE 59; IPFIX IE 254 (`PostVlanId`/`PostDot1qVlanId`). sFlow only via `ExtendedSwitch` (NOT from 802.1Q tag in `SampledHeader`) | | `ETYPE` | uint16 | ✓ (IPv4) | ✓ (IPv4) | ◐ | ◐ | ◐ | decoder | all | facet, group-by, filter | v5/v7 hardcoded to 2048. v9/IPFIX IE 60 `IpProtocolVersion` (4→2048, 6→34525). sFlow from sampled L2 etype | | `EXPORTER_GROUP` | string | — | — | — | — | — | enrichment | all | facet, group-by, filter | `metadata_static.exporters..group`. Classifiers fill it when static metadata didn't | @@ -316,7 +316,7 @@ Column legend: | `EXPORTER_ROLE` | string | — | — | — | — | — | enrichment | all | facet, group-by, filter | `metadata_static.exporters..role`. Classifiers may fill | | `EXPORTER_SITE` | string | — | — | — | — | — | enrichment | all | facet, group-by, filter | `metadata_static.exporters..site`. Classifiers may fill | | `EXPORTER_TENANT` | string | — | — | — | — | — | enrichment | all | facet, group-by, filter | `metadata_static.exporters..tenant`. Classifiers may fill | -| `FLOWS` | uint64 | ✓ | ✓ | ✓ | ✓ | ✓ | decoder | all | metric, filter | Always 1 for raw records; sums during rollup aggregation | +| `FLOWS` | uint64 | ✓ | ✓ | ✓ | ✓ | ✓ | decoder | raw | metric | Always 1 for raw records; not stored in rollups | | `FLOW_END_USEC` | uint64 | ✓ | ✓ | ◐ | ◐ | — | decoder | raw | hidden | v5/v7 from header `sysUpTime` + `LastSwitched`. v9 `LastSwitched` is relative to system init; `flowEndMilliseconds` is an absolute Unix timestamp. IPFIX uses the flow-end time family. Not populated for sFlow | | `FLOW_START_USEC` | uint64 | ✓ | ✓ | ◐ | ◐ | — | decoder | raw | hidden | v5/v7 from header `sysUpTime` + `FirstSwitched`. v9 `FirstSwitched` is relative to system init; `flowStartMilliseconds` is an absolute Unix timestamp. IPFIX uses the flow-start time family. Not populated for sFlow | | `FLOW_VERSION` | string | ✓ | ✓ | ✓ | ✓ | ✓ | decoder | all | facet, group-by, filter | One of `v5`, `v7`, `v9`, `ipfix`, `sflow` | @@ -337,7 +337,7 @@ Column legend: | `IPV6_FLOW_LABEL` | uint32 | — | — | ◐ | ◐ | ◐ | decoder | raw | facet, group-by, filter | v9/IPFIX IE 31 `FlowLabelIpv6`. sFlow from parsed IPv6 header | | `IP_FRAGMENT_ID` | uint32 | — | — | ◐ | ◐ | ◐ | decoder | raw | facet, group-by, filter | v9 IE 54 `Ipv4Ident`. IPFIX IE 54 `FragmentIdentification`. sFlow from parsed IPv4 header | | `IP_FRAGMENT_OFFSET` | uint16 | — | — | ◐ | ◐ | ◐ | decoder | raw | facet, group-by, filter | v9/IPFIX IE 88 `FragmentOffset`. sFlow from parsed IPv4 header | -| `MPLS_LABELS` | string | — | — | ◐ | ◐ | ◐ | decoder | raw | filter | v9 IE 70-79 `MplsLabel1..10`. IPFIX IE 70 `MplsTopLabelStackSection` + 71-79 `MplsLabelStackSection2..10`. sFlow from MPLS in `SampledHeader`. Comma-separated decimal labels | +| `MPLS_LABELS` | string | — | — | ◐ | ◐ | ◐ | decoder | raw | facet, group-by, filter | v9 IE 70-79 `MplsLabel1..10`. IPFIX IE 70 `MplsTopLabelStackSection` + 71-79 `MplsLabelStackSection2..10`. sFlow from MPLS in `SampledHeader`. Comma-separated decimal labels | | `NEXT_HOP` | IP | ✓ | ✓ | ◐ | ◐ | ◐ | both | all | facet, group-by, filter | v9 IE 15/18/62/63; IPFIX same. sFlow `ExtendedRouter`/`ExtendedGateway`. Enrichment overlay via `net_providers` chain (default `[flow, routing]`) | | `OBSERVATION_TIME_MILLIS` | uint64 | — | — | ◐ | — | — | decoder | raw | hidden | v9 IE 323 `ObservationTimeMilliseconds`. For Cisco ASA NSEL this remains metadata; collector receive time determines journal/query placement. IPFIX observation-time fields are not exposed | | `OUT_IF` | uint32 | ✓ | ✓ | ◐ | ◐ | ◐ | decoder | all | facet, group-by, filter | v9 IE 14 `OutputSnmp`; IPFIX IE 14/253. sFlow flow-sample `output` (single index only; LOCAL→0) | @@ -347,7 +347,7 @@ Column legend: | `OUT_IF_NAME` | string | — | — | — | — | — | enrichment | all | facet, group-by, filter | `metadata_static.exporters..if_indexes..name` | | `OUT_IF_PROVIDER` | string | — | — | — | — | — | enrichment | all | facet, group-by, filter | Static metadata or interface classifier provider tag | | `OUT_IF_SPEED` | uint64 | — | — | — | — | — | enrichment | all | facet, group-by, filter | `metadata_static.exporters..if_indexes..speed` (bps) | -| `PACKETS` | uint64 | ✓ | ✓ | ✓ | ✓ | ✓ | decoder | all | metric, filter | Canonical packet counter, selected with `BYTES` from the same family and scaled by `SAMPLING_RATE`. Cisco ASA NSEL event 5 uses initiator/responder IEs 298/299 without sampling. sFlow always 1 per sample | +| `PACKETS` | uint64 | ✓ | ✓ | ✓ | ✓ | ✓ | decoder | all | metric | Canonical packet counter, selected with `BYTES` from the same family and scaled by `SAMPLING_RATE`. Cisco ASA NSEL event 5 uses initiator/responder IEs 298/299 without sampling. sFlow always 1 per sample | | `PROTOCOL` | uint8 | ✓ | ✓ | ✓ | ✓ | ◐ | decoder | all | facet, group-by, filter | IP protocol number: v5/v7 protocol_number; v9 IE 4; IPFIX IE 4 `ProtocolIdentifier`. sFlow from `SampledIPv4`/`SampledIPv6` or parsed L3. Zero (`HOPOPT`) is retained explicitly | | `RAW_BYTES` | uint64 | ✓ | ✓ | ✓ | ✓ | ✓ | decoder | raw | metric | Unscaled byte value from the selected canonical counter family; the non-selected family is discarded. Equal to `BYTES` for Cisco ASA NSEL | | `RAW_PACKETS` | uint64 | ✓ | ✓ | ✓ | ✓ | ✓ | decoder | raw | metric | Unscaled packet value from the selected canonical counter family; the non-selected family is discarded. Equal to `PACKETS` for Cisco ASA NSEL | @@ -370,7 +370,7 @@ Column legend: | `SRC_NET_TENANT` | string | — | — | — | — | — | enrichment | all | facet, group-by, filter | `enrichment.networks..tenant` from static + network sources | | `SRC_PORT` | uint16 | ✓ | ✓ | ◐ | ◐ | ◐ | decoder | raw | facet, group-by, filter | v9/IPFIX IE 7. sFlow from `SampledIPv4`/`SampledIPv6` or transport parse. Raw-only | | `SRC_PORT_NAT` | uint16 | — | — | ◐ | ◐ | — | decoder | raw | facet, group-by, filter | v9 IE 227; IPFIX `postNAPTsourceTransportPort` | -| `SRC_PREFIX` | IP | ✓ | ✓ | ◐ | — | — | decoder | raw | filter | v5/v7 derived from `SRC_ADDR` & `SRC_MASK`. v9 IE 44 (`Ipv4SrcPrefix`). IPFIX has no canonical mapping; sFlow none | +| `SRC_PREFIX` | IP | ✓ | ✓ | ◐ | — | — | decoder | raw | facet, group-by, filter | v5/v7 derived from `SRC_ADDR` & `SRC_MASK`. v9 IE 44 (`Ipv4SrcPrefix`). IPFIX has no canonical mapping; sFlow none | | `SRC_VLAN` | uint16 | — | — | ◐ | ◐ | ◐ | decoder | all | facet, group-by, filter | v9 IE 58; IPFIX IE 58/243 (`VlanId`/`Dot1qVlanId`). sFlow only via `ExtendedSwitch` (NOT from 802.1Q tag in `SampledHeader`) | | `TCP_FLAGS` | uint8 | ✓ | ✓ | ◐ | ◐ | ◐ | decoder | all | facet, group-by, filter | OR of all TCP control bits seen in the flow. v9/IPFIX IE 6. sFlow from parsed TCP header in `SampledHeader` | diff --git a/docs/Network Performance Monitoring/Network Flows/Flow Protocols/IPFIX.mdx b/docs/Network Performance Monitoring/Network Flows/Flow Protocols/IPFIX.mdx index 3d56a5250..71a9691c2 100644 --- a/docs/Network Performance Monitoring/Network Flows/Flow Protocols/IPFIX.mdx +++ b/docs/Network Performance Monitoring/Network Flows/Flow Protocols/IPFIX.mdx @@ -92,7 +92,7 @@ Enable IPFIX via the `protocols.ipfix` option. | protocols.sampling_cache_max_entries | Maximum learned NetFlow v9/IPFIX sampling-rate entries across all exporter streams. Must be positive. | 100000 | no | | protocols.sampling_cache_max_entries_per_stream | Maximum learned sampling-rate entries for one exporter stream. Must be positive; values above the global limit are clamped. | 65536 | no | | journal.journal_dir | Directory for journal files (relative to NETDATA_CACHE_DIR). | flows | no | -| journal.tiers.<tier>.size_of_journal_files | Per-tier hard size cap. Replace `` with `raw`, `minute_1`, `minute_5`, or `hour_1`. Set to `null` for time-only retention. | 10GB | no | +| journal.tiers.<tier>.size_of_journal_files | Per-tier retained-artifact budget for journal data and finalized per-journal facet sidecars. Replace `` with `raw`, `minute_1`, `minute_5`, or `hour_1`. The protected active journal can temporarily exceed the budget. Set to `null` for time-only retention. | 10GB | no | | journal.tiers.<tier>.duration_of_journal_files | Per-tier maximum age. Replace `` with `raw`, `minute_1`, `minute_5`, or `hour_1`. The default `null` disables time-based eviction; set a duration to add an age cap. | null | no | diff --git a/docs/Network Performance Monitoring/Network Flows/Flow Protocols/NetFlow.mdx b/docs/Network Performance Monitoring/Network Flows/Flow Protocols/NetFlow.mdx index bdc113ca4..4b5831d24 100644 --- a/docs/Network Performance Monitoring/Network Flows/Flow Protocols/NetFlow.mdx +++ b/docs/Network Performance Monitoring/Network Flows/Flow Protocols/NetFlow.mdx @@ -64,7 +64,7 @@ Operational limits are driven by sustained flow records/s, exporter batching, ca #### Performance Impact -Disabled until exporters send traffic. Once active, CPU and disk I/O scale with flow-record rate and cardinality; size retention and storage from observed flow records/s. +Disabled until exporters send traffic. Once active, CPU and disk I/O scale with flow-record rate, template volume, and cardinality; size retention and storage from observed flow records/s. ## Setup @@ -100,7 +100,7 @@ The plugin is configured via `netflow.yaml` in the Netdata configuration directo | protocols.sampling_cache_max_entries | Maximum learned NetFlow v9/IPFIX sampling-rate entries across all exporter streams. Must be positive. | 100000 | no | | protocols.sampling_cache_max_entries_per_stream | Maximum learned sampling-rate entries for one exporter stream. Must be positive; values above the global limit are clamped. | 65536 | no | | journal.journal_dir | Directory for journal files (relative to NETDATA_CACHE_DIR). | flows | no | -| journal.tiers.<tier>.size_of_journal_files | Per-tier hard size cap. Replace `` with `raw`, `minute_1`, `minute_5`, or `hour_1`. Set to `null` for time-only retention. | 10GB | no | +| journal.tiers.<tier>.size_of_journal_files | Per-tier retained-artifact budget for journal data and finalized per-journal facet sidecars. Replace `` with `raw`, `minute_1`, `minute_5`, or `hour_1`. The protected active journal can temporarily exceed the budget. Set to `null` for time-only retention. | 10GB | no | | journal.tiers.<tier>.duration_of_journal_files | Per-tier maximum age. Replace `` with `raw`, `minute_1`, `minute_5`, or `hour_1`. The default `null` disables time-based eviction; set a duration to add an age cap. | null | no | diff --git a/docs/Network Performance Monitoring/Network Flows/Flow Protocols/sFlow.mdx b/docs/Network Performance Monitoring/Network Flows/Flow Protocols/sFlow.mdx index 58850264e..d92f70af0 100644 --- a/docs/Network Performance Monitoring/Network Flows/Flow Protocols/sFlow.mdx +++ b/docs/Network Performance Monitoring/Network Flows/Flow Protocols/sFlow.mdx @@ -91,7 +91,7 @@ Enable sFlow via the `protocols.sflow` option. | listener.listen | UDP listener endpoints for NetFlow/IPFIX and sFlow datagrams. YAML accepts either a scalar endpoint or a list of endpoints; CLI accepts repeated `--netflow-listen` flags or comma-delimited values. | 0.0.0.0:2055, 0.0.0.0:6343 | no | | protocols.sflow | Enable sFlow decoding. | yes | no | | journal.journal_dir | Directory for journal files (relative to NETDATA_CACHE_DIR). | flows | no | -| journal.tiers.<tier>.size_of_journal_files | Per-tier hard size cap. Replace `` with `raw`, `minute_1`, `minute_5`, or `hour_1`. Set to `null` for time-only retention. | 10GB | no | +| journal.tiers.<tier>.size_of_journal_files | Per-tier retained-artifact budget for journal data and finalized per-journal facet sidecars. Replace `` with `raw`, `minute_1`, `minute_5`, or `hour_1`. The protected active journal can temporarily exceed the budget. Set to `null` for time-only retention. | 10GB | no | | journal.tiers.<tier>.duration_of_journal_files | Per-tier maximum age. Replace `` with `raw`, `minute_1`, `minute_5`, or `hour_1`. The default `null` disables time-based eviction; set a duration to add an age cap. | null | no | diff --git a/docs/Network Performance Monitoring/Network Flows/Network Flows.mdx b/docs/Network Performance Monitoring/Network Flows/Network Flows.mdx index 69dc6c50f..a670f37b9 100644 --- a/docs/Network Performance Monitoring/Network Flows/Network Flows.mdx +++ b/docs/Network Performance Monitoring/Network Flows/Network Flows.mdx @@ -89,7 +89,7 @@ Each flow record is enriched at ingestion with: - **Live BGP attributes** (AS path, communities, next-hop) — from BMP, BioRIS, or static prefix configuration - **Decapsulated inner-packet fields** for SRv6 / VXLAN traffic -Flow records land in a four-tier journal: raw + 1-minute + 5-minute + 1-hour rollups, with independent retention per tier. Rollup tiers drop a few high-cardinality fields (IPs, ports, city/coordinates) to stay compact, so any query that filters or groups by those fields is served from the raw tier; everything else can use a coarser tier. The dashboard auto-picks the best tier for each query. +Flow records land in a four-tier journal: raw + 1-minute + 5-minute + 1-hour rollups, with independent retention per tier. Rollup tiers drop high-cardinality and protocol-specific details such as addresses, prefixes, ports, cities/coordinates, MACs, BGP paths/communities, NAT fields, fragmentation details, and MPLS labels. Any query that filters or groups by a field absent from rollups is served from the raw tier; everything else can use a coarser tier. The dashboard auto-picks the best tier for each query. ## What sampling does to your numbers diff --git a/docs/Network Performance Monitoring/Network Flows/Retention and Tiers.mdx b/docs/Network Performance Monitoring/Network Flows/Retention and Tiers.mdx index 7cbb28a53..e191ae0e4 100644 --- a/docs/Network Performance Monitoring/Network Flows/Retention and Tiers.mdx +++ b/docs/Network Performance Monitoring/Network Flows/Retention and Tiers.mdx @@ -34,16 +34,14 @@ Rollup tiers (1m, 5m, 1h) deliberately drop the high-cardinality and protocol-sp **Forced to the raw tier** (any query that filters on, groups by, or runs full-text search against these fields is rerouted to the raw tier — see [Field Reference](/docs/network-performance-monitoring/network-flows/field-reference) for the per-field matrix): -- `SRC_ADDR`, `DST_ADDR`, `SRC_PORT`, `DST_PORT` +- Addresses and prefixes: `SRC_ADDR`, `DST_ADDR`, `SRC_PREFIX`, `DST_PREFIX`, `SRC_MASK`, `DST_MASK` +- NAT: `SRC_ADDR_NAT`, `DST_ADDR_NAT`, `SRC_PORT_NAT`, `DST_PORT_NAT` +- Transport and link details: `SRC_PORT`, `DST_PORT`, `SRC_MAC`, `DST_MAC` - `SRC_GEO_CITY`, `DST_GEO_CITY`, `SRC_GEO_LATITUDE`, `DST_GEO_LATITUDE`, `SRC_GEO_LONGITUDE`, `DST_GEO_LONGITUDE` +- Routing and encapsulation: `DST_AS_PATH`, `DST_COMMUNITIES`, `DST_LARGE_COMMUNITIES`, `MPLS_LABELS` +- IP header details: `IPTTL`, `IPV6_FLOW_LABEL`, `IP_FRAGMENT_ID`, `IP_FRAGMENT_OFFSET` - All `V9_*` and `IPFIX_*` raw-protocol fields. -**Dropped from rollup output but do not switch tier** (the field comes back as null on rollup tiers; the planner does not reroute the query to raw): - -- AS path, BGP community fields (`SRC_COMMUNITIES`, `DST_COMMUNITIES`, etc.), MPLS labels, MAC addresses, NAT / post-NAT addresses, and any other field not in the preserved set below. - -If you need any of these fields populated in the result, force the raw tier explicitly (open a city map, add a port filter, type something into the search ribbon, or pick a window that fits inside raw-tier retention). - **Preserved in rollup tiers** (these queries can use coarser tiers): - Core: `PROTOCOL`, `DIRECTION`, `ETYPE`, `FORWARDING_STATUS`, `FLOW_VERSION`, `IPTOS`, `TCP_FLAGS`, `ICMPV4_TYPE/CODE`, `ICMPV6_TYPE/CODE`, `SRC_AS` / `DST_AS` (ASN number), `SRC_AS_NAME` / `DST_AS_NAME`. @@ -52,9 +50,9 @@ If you need any of these fields populated in the result, force the raw tier expl - Network: `SRC_NET_*` / `DST_NET_*` (name / role / site / region / tenant), `SRC_COUNTRY` / `DST_COUNTRY`, `SRC_GEO_STATE` / `DST_GEO_STATE`, `NEXT_HOP`, `SRC_VLAN` / `DST_VLAN`. - Aggregates: bytes / packets / flow-count sums per bucket. -So rollups are fine for most country / state / ASN / interface / VLAN / protocol questions, but useless if you need to ask "which IP", "which port", "which AS path", "which MPLS label", or "where in the city". +So rollups are fine for most country / state / ASN / interface / VLAN / protocol questions, but cannot answer "which IP", "which port", "which AS path", "which MPLS label", or "where in the city". -This is why filtering or grouping by IP/port/city/lat/lon forces the query to the raw tier — there is no other tier that has those fields. +This is why filtering or grouping by any field absent from the preserved set forces the query to the raw tier — there is no other tier that can answer it correctly. For the per-field tier-preservation matrix, see [Field Reference](/docs/network-performance-monitoring/network-flows/field-reference). @@ -64,7 +62,7 @@ For every query the dashboard sends to the plugin, the planner makes a single de **Rules:** -1. **Any raw-only field used as a filter or group-by → raw tier.** No exception. See the "Forced to the raw tier" list above. Selecting the city map or filtering on any IP / port / city / lat / lon field (plus the `V9_*` / `IPFIX_*` raw protocol fields) falls in this category. +1. **Any raw-only field used as a filter or group-by → raw tier.** No exception. See the complete "Forced to the raw tier" list above. 2. **A non-empty full-text search → raw tier.** Full-text search runs as a regex against the raw journal payload, which only the raw tier carries. 3. **Otherwise, pick the coarsest tier that satisfies the time range alignment.** - **Time-Series view** additionally needs at least 100 buckets in the window. The planner walks the tiers from coarsest to finest and picks the first that delivers ≥100 buckets, falling back to 1-minute when no tier qualifies: @@ -82,7 +80,7 @@ The plugin reports the chosen tier in the response stats (`query_tier` = `0`, `1 If you ask for a 30-day window with an IP filter and raw-tier retention is 24 hours, you get an empty response. No error, no banner reading "data has expired" — just an empty result set. The dashboard renders this as "No data". -The planner does not fall back to a coarser tier for raw-only queries. When a span requires the raw tier (because the query filters or groups on an IP / port / city / lat / lon / V9_* / IPFIX_* field, or runs a full-text search) and that span's raw-tier files have been rotated out, the planner returns no flows for that span. Rollups never carry raw-only fields, so they cannot satisfy the query anyway. Conversely, when a span only needs preserved fields (country, ASN, exporter, interface, protocol…), the planner can fall back from a coarser tier to a finer one if the coarser files have rotated out — finer tiers are supersets of coarser tiers for the preserved fields. +The planner does not fall back to a coarser tier for raw-only queries. When a span requires the raw tier (because the query filters or groups on any field absent from rollups, or runs a full-text search) and that span's raw-tier files have been rotated out, the planner returns no flows for that span. Rollups never carry raw-only fields, so they cannot satisfy the query anyway. Conversely, when a span only needs preserved fields (country, ASN, exporter, interface, protocol…), the planner can fall back from a coarser tier to a finer one if the coarser files have rotated out — finer tiers are supersets of coarser tiers for the preserved fields. Other spans within the same query that don't need raw data may still return flows. So it's also possible to see partial coverage — half the time range filled, half empty. @@ -92,8 +90,7 @@ For Time-Series, "no data" appears as zero values in the affected buckets, not a Quick reference for "why is my query slow / showing less time?": -- Adding `SRC_ADDR`, `DST_ADDR`, `SRC_PORT`, or `DST_PORT` as a filter -- Adding any of those fields to the group-by +- Adding any field from the raw-only list above as a filter or group-by - Switching to the city map (it uses `SRC_GEO_CITY`/`DST_GEO_CITY` plus latitudes/longitudes) - Typing anything into the global search ribbon @@ -101,7 +98,7 @@ If you see the time depth in your dashboard suddenly shrink after you applied a ## Default retention and the most common misconfiguration -Each tier has its own `size_of_journal_files` and `duration_of_journal_files`. The built-in defaults are uniform: `10GB` on every tier with no time-based age limit. That is safe for first validation because disk is still capped, but production deployments should usually set tier-specific size and duration budgets; the whole point of having rollup tiers is to keep them around longer than raw. +Each tier has its own `size_of_journal_files` and `duration_of_journal_files`. The built-in defaults are uniform: `10GB` on every tier with no time-based age limit. The size budget includes journal data and finalized per-journal facet sidecars. The protected active journal can temporarily exceed it, and shared facet state is outside the per-tier budget. Production deployments should usually set tier-specific size and duration budgets; the whole point of having rollup tiers is to keep them around longer than raw. A more useful production profile: diff --git a/docs/Network Performance Monitoring/Network Flows/Sizing and Capacity Planning.mdx b/docs/Network Performance Monitoring/Network Flows/Sizing and Capacity Planning.mdx index d5acd3d28..bf9533d4f 100644 --- a/docs/Network Performance Monitoring/Network Flows/Sizing and Capacity Planning.mdx +++ b/docs/Network Performance Monitoring/Network Flows/Sizing and Capacity Planning.mdx @@ -90,7 +90,7 @@ journal: hour_1: { size_of_journal_files: 20GB, duration_of_journal_files: 365d } ``` -For lighter loads, scale `size_of_journal_files` on the raw tier down proportionally — at 10 000 flow records/s ~350 GB / 24 h is enough; at 1 000 flow records/s ~35 GB / 24 h is enough. Whichever limit (size or duration) is hit first triggers rotation; size your raw tier so the **duration limit fires first** under normal load and the size cap is a safety net for traffic surges. +For lighter loads, scale `size_of_journal_files` on the raw tier down proportionally — at 10 000 flow records/s ~350 GB / 24 h is enough; at 1 000 flow records/s ~35 GB / 24 h is enough. The size budget covers journal data and finalized per-journal facet sidecars. Whichever limit (size or duration) is hit first triggers rotation; size your raw tier so the **duration limit fires first** under normal load and the size budget is a safety net for traffic surges. ### Use fast NVMe for the raw tier diff --git a/docs/Network Performance Monitoring/Network Flows/Troubleshooting.mdx b/docs/Network Performance Monitoring/Network Flows/Troubleshooting.mdx index 3e1b75439..8641738bc 100644 --- a/docs/Network Performance Monitoring/Network Flows/Troubleshooting.mdx +++ b/docs/Network Performance Monitoring/Network Flows/Troubleshooting.mdx @@ -228,7 +228,7 @@ Note: the live (open) tier rows shown by queries refresh on a 1-second cadence, sudo du -sh /var/cache/netdata/flows/* ``` -Default retention is `10GB` per tier with no time-based age limit. The default is applied separately to raw, 1m, 5m, and 1h tiers, so total can reach roughly 40 GB plus some. If your config left this default and your collector is busy, expect to hit the size cap quickly; quiet collectors may keep data for longer than seven days. See [Configuration](/docs/network-performance-monitoring/network-flows/configuration) for per-tier overrides — most production deployments need them. +Default retention is `10GB` per tier with no time-based age limit. The default is applied separately to raw, 1m, 5m, and 1h tiers, and each budget includes journal files plus finalized per-journal facet sidecars. Retained artifacts therefore target roughly 40 GB total. Protected active journals can temporarily push usage above that target, and shared facet state is outside the per-tier budgets. If your collector is busy, expect to hit the size budget quickly; quiet collectors may keep data for longer than seven days. See [Configuration](/docs/network-performance-monitoring/network-flows/configuration) for per-tier overrides — most production deployments need them. ## Things that look like bugs but aren't diff --git a/docs/Network Performance Monitoring/Network Flows/Visualization/Filters and Facets.mdx b/docs/Network Performance Monitoring/Network Flows/Visualization/Filters and Facets.mdx index 0168c1798..b8299918e 100644 --- a/docs/Network Performance Monitoring/Network Flows/Visualization/Filters and Facets.mdx +++ b/docs/Network Performance Monitoring/Network Flows/Visualization/Filters and Facets.mdx @@ -24,15 +24,23 @@ Around 80 fields are available as facets. They're a subset of the full 91-field Everything else — IPs, ports, protocol, AS numbers and names, country, state, city, exporter labels, interfaces, MACs, VLANs, NAT addresses, TCP flags, ToS, etc. — is filterable. +The filter ribbon shows only fields that have at least one value in the collector's retained facet vocabulary. A field appears after its first value is retained and disappears after its last value ages out. An actively selected field remains visible even when its retained vocabulary is empty, so you can clear the filter. This availability is collector-retention-wide, not limited to the time window currently shown. + ## Filter logic Within a single field: **OR**. Selecting `PROTOCOL = TCP` and `PROTOCOL = UDP` shows TCP-or-UDP. Across different fields: **AND**. Adding `SRC_COUNTRY = US` to the above shows TCP-or-UDP from the US. +The six address fields — `EXPORTER_IP`, `SRC_ADDR`, `DST_ADDR`, `NEXT_HOP`, `SRC_ADDR_NAT`, and `DST_ADDR_NAT` — accept either exact IPv4/IPv6 addresses or CIDRs. Exact addresses and CIDRs selected in the same field are ORed together. For example, selecting `10.0.0.0/8` and `2001:db8::7` on `SRC_ADDR`, plus `PROTOCOL = TCP`, means: + +`(source is inside 10.0.0.0/8 OR source is 2001:db8::7) AND protocol is TCP`. + +CIDR selection does not change these Boolean rules. `SRC_PREFIX` and `DST_PREFIX` are stored labels and continue to use exact-value filtering; they do not mean "address overlaps this network." + ## No negative match -You cannot directly say "everything except X". The workaround is to select all values and remove the unwanted one — works for low-cardinality fields like `PROTOCOL` (a handful of values). For high-cardinality fields like `SRC_AS_NAME`, the autocomplete only surfaces the top 100 values, so there's no practical way to "select all and remove". +You cannot directly say "everything except X". The workaround is to select all values and remove the unwanted one — works for low-cardinality fields like `PROTOCOL` (a handful of values). For high-cardinality fields like `SRC_AS_NAME`, the autocomplete only surfaces the top 256 values, so there's no practical way to "select all and remove". Negative matching is not supported. @@ -40,14 +48,28 @@ Negative matching is not supported. Type into a facet field and the dashboard suggests existing values from your live data. The list: -- Shows up to **100 matching values**, sorted alphabetically. +- Shows up to **256 results**. Retained values are sorted alphabetically; generated CIDRs use the ranking described below. +- Retained values for every facet come from the same retention-wide vocabulary. Non-IP facets with up to **256 retained values** return the complete static list; facets with 257 or more values switch to autocomplete. A facet can switch back to a static list when older values leave retention. +- IP address facets keep autocomplete enabled so `/` can suggest CIDRs. They still return their complete inline value list while the retained vocabulary has at most 256 values; above that limit, the inline list is omitted. - Matching policy is per-field. Free-form text fields (`SRC_AS_NAME`, `EXPORTER_NAME`, `IN_IF_DESCRIPTION`, MAC addresses, AS paths, BGP communities, country/city/state names) match by **substring**, so typing `Akamai` finds `AS20940 Akamai International`. IPs and short numeric fields (ports, protocols, ASN numbers, interface speeds) match by **prefix**, so typing `10.0.` narrows to that range. - Runs against an **in-memory snapshot of the live journal** plus on-disk FST sidecars for promoted high-cardinality fields. Autocomplete never reads the raw flow tiers, and is fast even on busy collectors. - The autocomplete `term` is hard-capped at 256 bytes; longer requests are rejected. For high-cardinality fields, autocomplete is the only practical way to discover values. You can't scroll a list of millions of IP addresses, but you can find one by typing what you remember. -**Autocomplete and regular filtering are different paths.** When you select a value from the dropdown, the resulting filter is **exact equality**, not substring. The dropdown only helps you discover values; the filter that gets applied is `key = value` (or `key in [values]`) and uses indexes — never a substring scan over flow data. +### CIDR autocomplete + +Typing `/` in an IP address facet switches that request from retained-value lookup to CIDR generation. The backend generates a bounded candidate set, then returns only networks containing at least one address in that field's compact retention-wide vocabulary. It does not scan journal rows or flow tiers. + +- Loose IPv4 input fills missing octets: `10/8` can suggest `10.0.0.0/8`; `10.1/` can suggest `10.1.0.0/16`. One trailing dot before `/` is tolerated after one through three octets, so `10./`, `10.1./`, and `10.1.2./` behave like the same inputs without that dot. +- Loose IPv6 input fills omitted trailing groups: `2001:db8/32` can suggest `2001:db8::/32`. +- Digits after `/` are an autocomplete fragment. `10.1/1` generates the naturally implied `10.1.0.0/16` first, then the exact prefix-length match `0.0.0.0/1`, followed by the other canonical matches `/10` through `/19`. Candidates with no retained address in the selected facet are omitted, so the first returned value may differ. +- Suggestions are always canonical IPv4 or IPv6 CIDRs. Loose forms are accepted only while searching; a direct Function selection must contain a complete address and prefix length. The backend normalizes a valid non-network address such as `10.1.2.99/24` to its network before matching. +- Candidate validation is retention-wide, like ordinary facet discovery. A suggested CIDR can still have no data in the current time window or after other filters are applied. + +**Autocomplete and regular filtering are different paths.** The dropdown uses prefix or substring matching only to discover retained values. A selected non-IP value uses exact equality. A selected IP value uses exact-address or CIDR containment, never substring matching over flow data. + +Small IP networks containing at most 256 addresses can be represented as exact indexed matches. Broader networks are checked by the typed CIDR matcher after any other exact filters have narrowed the scan. Exact addresses and CIDRs return the same results in table, Sankey, and time-series views. ## Full-text search @@ -58,7 +80,7 @@ The search box at the top of the filter ribbon performs a regex match against th - Any non-empty search **forces raw tier**. The full-text search only works against the raw journal — it doesn't apply to the rollup tiers. Time depth is therefore bounded by raw-tier retention. - The plugin's "fast aggregation" path is also disabled when full-text search is active, because aggregation needs to scan every record. Expect somewhat slower responses than tier-based aggregation queries. -For "find anything containing this string in any field", the search is the right tool. For "filter by an exact value of a specific field", use the facet on that field — it's faster and doesn't trigger raw-tier mode. +For "find anything containing this string in any field", the search is the right tool. For an exact value or IP network, use the facet on that field. Facet filters follow that field's tier availability: source, destination, and NAT address fields are currently raw-tier-only; exporter IP and next hop are available in rollups. ## URL preservation @@ -71,6 +93,8 @@ A practical note: filters use a structured representation (per-field IN-list) th - **Search for `192.168.1.1` matches unrelated rows.** Regex semantics: each `.` is "any byte". Escape: `192\.168\.1\.1`. - **Time depth shrinks unexpectedly after typing in search.** Full-text search forces raw tier. Clear the search to use rollup tiers and longer time ranges. - **Negative match is unsupported.** Workaround: select-all-minus-one for low-cardinality fields. For high-cardinality fields, use a positive filter that narrows the result set instead. +- **A loose CIDR works in autocomplete but fails as a direct selection.** Send a complete address and prefix such as `10.0.0.0/8`, not `10/8`. +- **A suggested CIDR returns no data in the current view.** Suggestions prove that the selected field contains a matching address somewhere in retained data. They are not recalculated for the current time window or other active filters. - **Filter on an ICMP virtual facet seems slower than expected.** `ICMPV4` / `ICMPV6` virtual facets aren't optimised by the journal index — they're evaluated per-record. The query still returns; the cost shows up as longer wall time on busy collectors. - **`query_max_groups` exceeded.** Result rows after the limit fold into `__overflow__`. Narrow the filter or reduce group-by depth. - **GET-style args don't carry selections.** When integrating the function call yourself, send a JSON payload — the dashboard does this automatically. diff --git a/docs/Network Performance Monitoring/Network Flows/Visualization/Maps and Globe.mdx b/docs/Network Performance Monitoring/Network Flows/Visualization/Maps and Globe.mdx index 690e4c25c..9c0ffc21f 100644 --- a/docs/Network Performance Monitoring/Network Flows/Visualization/Maps and Globe.mdx +++ b/docs/Network Performance Monitoring/Network Flows/Visualization/Maps and Globe.mdx @@ -53,7 +53,7 @@ The country map and state map can use the rollup tiers. They're cheap over long The city map and the globe **need raw-tier data**. City, latitude, and longitude are dropped from the rollup tiers (1m / 5m / 1h) to keep cardinality manageable. So: - Country / state map over the last 30 days — fine, uses the 1-hour tier. -- City map over the last 30 days — likely empty on busy collectors. Raw-tier retention defaults to its own 10GB size cap with no time-based age limit, so high flow volume can exhaust raw history quickly. +- City map over the last 30 days — likely empty on busy collectors. Raw-tier retention defaults to its own 10GB retained-artifact budget with no time-based age limit, so high flow volume can exhaust raw history quickly. If your city map looks empty over a long window, try the country map first to confirm data is arriving, then narrow the time range until the city map fills in. diff --git a/docs/Network Performance Monitoring/Network Flows/Visualization/Sankey and Table.mdx b/docs/Network Performance Monitoring/Network Flows/Visualization/Sankey and Table.mdx index f0b180bc7..248af6e87 100644 --- a/docs/Network Performance Monitoring/Network Flows/Visualization/Sankey and Table.mdx +++ b/docs/Network Performance Monitoring/Network Flows/Visualization/Sankey and Table.mdx @@ -80,7 +80,7 @@ A filter strip sits between the Sankey and the table. Three things you can do he Filter logic is "AND across fields, OR within a field". Selecting `PROTOCOL = TCP` and `PROTOCOL = UDP` shows TCP-or-UDP. Selecting `PROTOCOL = TCP` plus `SRC_COUNTRY = US` shows only US-source TCP. -There is no negative match. To exclude a value, select all values and remove the unwanted one — works for low-cardinality fields, becomes impractical for high-cardinality ones (the autocomplete cap is 100). +There is no negative match. To exclude a value, select all values and remove the unwanted one — works for low-cardinality fields, becomes impractical for high-cardinality ones (the autocomplete cap is 256). See [Filters and Facets](/docs/network-performance-monitoring/network-flows/visualization/filters-and-facets) for the full mechanics. diff --git a/docs/Network Performance Monitoring/Network Flows/Visualization/Time-Series.mdx b/docs/Network Performance Monitoring/Network Flows/Visualization/Time-Series.mdx index e282a6ed4..21039dd87 100644 --- a/docs/Network Performance Monitoring/Network Flows/Visualization/Time-Series.mdx +++ b/docs/Network Performance Monitoring/Network Flows/Visualization/Time-Series.mdx @@ -54,10 +54,10 @@ The minimum bucket size is **60 seconds**. Zoom in past one minute and the chart Some queries can't use the rollup tiers. They drop to raw tier and inherit raw-tier retention: -- Filtering or grouping by `SRC_ADDR`, `DST_ADDR`, `SRC_PORT`, `DST_PORT`, or any geo city / latitude / longitude field +- Filtering or grouping by any [raw-only field](/docs/network-performance-monitoring/network-flows/retention-and-tiers#what-survives-the-rollup) - Any non-empty full-text search -In those cases the 100-bucket rule still applies, but the source tier is raw tier. Time depth is bounded by raw-tier retention (default: the raw tier has a 10GB size cap and no time-based age limit, so busy collectors can exhaust raw history quickly). +In those cases the 100-bucket rule still applies, but the source tier is raw tier. Time depth is bounded by raw-tier retention (default: the raw tier has a 10GB retained-artifact budget and no time-based age limit, so busy collectors can exhaust raw history quickly). If you've been working at a higher tier and add an IP filter, the time depth on your chart may suddenly shrink — that's the tier switch. diff --git a/docs/Network Performance Monitoring/Network Flows/Visualization/Visualization.mdx b/docs/Network Performance Monitoring/Network Flows/Visualization/Visualization.mdx index b5d1b56ce..0724cefe0 100644 --- a/docs/Network Performance Monitoring/Network Flows/Visualization/Visualization.mdx +++ b/docs/Network Performance Monitoring/Network Flows/Visualization/Visualization.mdx @@ -21,13 +21,13 @@ The Network Flows view exposes the same query engine through five panel types: S The dashboard sends one of two query modes to the plugin: - **`flows`** — the normal aggregation request. Returns top-N groups, sums of bytes and packets, optional facet counts. -- **`autocomplete`** — for the filter ribbon. Returns up to 100 facet values matching the user's term. Matching policy is per-field: text fields use substring matching, IP and numeric fields use prefix. Term is capped at 256 bytes. Runs against in-memory facet snapshots and on-disk FST sidecars; never scans tier files. Resulting filters apply as exact equality, not substring. +- **`autocomplete`** — for the filter ribbon. Returns up to 256 retained facet values or generated CIDR candidates. Matching policy for retained values is per-field: text fields use substring matching, IP and numeric fields use prefix. For IP address fields, `/` generates canonical IPv4/IPv6 CIDRs and returns only networks containing an address in that field's compact retention-wide vocabulary. Term is capped at 256 bytes. Retained-value lookup runs against in-memory facet snapshots and on-disk FST sidecars; CIDR validation uses only the in-memory typed IP vocabulary. Neither path scans flow tiers. Selected non-IP values use exact equality; selected IP values use exact-address or CIDR containment. A `flows` query carries: - A time range (`after` / `before`). If you omit both, the plugin uses the last 15 minutes. - A list of `group_by` fields (up to 10). -- A list of `selections` — per-field IN-lists for filtering. +- A list of `selections` — per-field IN-lists for filtering. The six IP address facets accept exact IPv4/IPv6 addresses and canonical CIDRs. - Optional `facets` to enrich the response with per-facet value counts. - A `top_n` (one of 25, 50, 100, 200, 500). - A `sort_by` (`bytes` or `packets`).