Design an InfluxDB downsampling cascade in the browser: get the Flux tasks that implement it, an auditable storage estimate, and a check for the misconfigurations that quietly corrupt a rollup.
Open the live tool — no install, no account, no backend. Everything is computed in the page.
Every time-series deployment eventually has the same conversation. Raw telemetry arrives every few seconds, the disk is filling, and somebody proposes "let's just downsample it". Then the questions start, and none of them have obvious answers:
- If we keep raw for 14 days and minute data for 90, how much does that actually save? Nobody can answer this without a spreadsheet, and the spreadsheet is always wrong, because the thing that dominates the number is not the sample interval — it is the series count, and the series count is multiplied by every aggregate function in every tier of the cascade.
- Do we roll up hour data from the minute tier, or from raw? Cascading is cheaper. Cascading is also how you end up storing the mean of a mean, which is not the mean.
- What does the Flux actually look like? The example in the docs is four lines with an
unbounded
range(start: -task.every). A production task needs its window boundaries truncated so it never writes a half-filled window, an offset sized to your ingest lag, and a lookback wide enough that late writes are not silently dropped.
Nothing about this is hard, exactly. It is just fiddly enough that people guess, and the mistakes are the kind you only discover months later — when the raw bucket has expired and you find out the rollup that was supposed to preserve the history has a gap in it, or that the "average" on your two-year dashboard has been quietly biased the whole time.
This tool makes the whole thing concrete. Describe your data, get the tasks, see the numbers with the arithmetic shown, and get told about the specific mistakes your design contains.
The screenshots below are captured from the running application by
scripts/screenshots.mjs, against a worked configuration of
4 000 devices sampling three fields every five seconds.
The same view in dark theme
Each tier produces a complete, copy-pasteable task with a copy button and a download.
The saving, the per-bucket breakdown, the cascade diagram — and an expandable panel that shows every step of the arithmetic, because an unexplained number is not usable for capacity planning.
Every warning explains what is wrong in terms of your own numbers and links to a write-up of that class of problem.
Four frames captured while changing the first tier's window from 1m to 1h. The Flux,
the estimate and the findings all update on every keystroke.
 |
 |
 |
 |
The tool is a static site. There is no server component and nothing to configure.
git clone https://github.com/taskautomation-org/downsample-designer.git
cd downsample-designer
npm install
npm run dev # http://localhost:5173To produce a deployable build:
npm run build # writes dist/
npm run preview # serves dist/ on http://localhost:4173dist/ is plain HTML, CSS and JS. Drop it behind any static host, or open it from a
file:// URL if your browser allows module scripts from disk. If you are hosting it under
a sub-path, set BASE_PATH at build time:
BASE_PATH=/tools/downsampler/ npm run buildRequires Node 20 or newer.
| Field | Meaning |
|---|---|
| Raw bucket | Bucket your writes land in. Becomes the from(bucket:) of the first task. |
| Organisation | Used in the generated to(org:). |
| Measurement | The measurement to roll up. |
| Fields | Comma-separated field keys. Each field is a separate series per tag combination. |
| Tag keys | Each tag's key, its number of distinct values, and whether the rollup keeps it. |
| Sample interval | How often a point arrives per series, e.g. 10s. |
| Raw retention | Retention of the raw bucket. 0s means keep forever. |
The keep checkbox on each tag is the most consequential control on the page. Series
count is the product of every retained tag, and it is what the index has to hold in
memory. A firmware_version tag is useful on raw data and almost never useful on a daily
rollup; unticking it divides the rollup's cardinality by its distinct-value count and adds
a drop(columns:) to the first generated task.
Tiers form a cascade: the first reads the raw bucket, and every later tier reads the one before it. For each tier you set the target bucket, the aggregation window, the aggregate functions and the retention.
Aggregates are where the storage estimate gets interesting. Each function you tick writes
its own series, and that multiplier compounds down the cascade — a three-aggregate minute
tier feeding a two-aggregate hour tier writes six series for every input series. When a
tier has more than one aggregate, the generated task tags each stream with an agg label
and unions them into a single write.
| Field | Meaning |
|---|---|
| every | Task interval. Left blank, each tier uses its own window, which is almost always what you want. |
| offset | How long after the window closes the task fires. Exists to wait out late writes. |
| cron | Optional. Replaces every in the task option for wall-clock scheduling. |
| Late-data tolerance | How late a write may arrive and still be picked up. Widens the query range; the extra windows are simply rewritten. |
Copy each task, or download them all as a single file, and create them with:
influx task create -f downsample_sensor_reading_to_telemetry_1m.fluxThe whole configuration is encoded into the URL hash as you type. Copy the address bar — or use the Copy permalink button — and whoever opens the link sees exactly the numbers you were looking at. Nothing is stored anywhere; the link is the storage.
For a first-hop tier with three aggregates, the tool produces:
// downsample_sensor_reading_to_plant_telemetry_1m
//
// Rolls up raw sensor_reading from "plant_telemetry_raw"
// (5 seconds resolution) into "plant_telemetry_1m"
// at 1 minute resolution, kept for 90 days.
// Aggregates: mean, min, max.
//
// Background on the window-alignment and idempotency pattern used below:
// https://taskautomation.org/automated-task-scheduling-orchestration/flux-scripting-for-task-automation/writing-robust-flux-scripts-for-automated-data-rollups/
import "date"
option task = {
name: "downsample_sensor_reading_to_plant_telemetry_1m",
every: 1m,
offset: 45s,
}
// Truncating `stop` to the 1m window means the task only ever writes whole
// windows — a window that is still filling is never written and then rewritten with
// a different value. The 3m lookback re-reads recently closed windows so late
// arrivals are folded in; rewriting a window with `to()` is idempotent.
stop = date.truncate(t: now(), unit: 1m)
start = date.sub(d: 3m, from: stop)
source =
from(bucket: "plant_telemetry_raw")
|> range(start: start, stop: stop)
|> filter(fn: (r) => r._measurement == "sensor_reading")
|> filter(fn: (r) => r._field == "temperature" or r._field == "vibration")
// Dropped at the first hop: these tags multiply rollup cardinality.
|> drop(columns: ["firmware"])
agg_mean =
source
|> aggregateWindow(every: 1m, fn: mean, createEmpty: false)
|> set(key: "agg", value: "mean")
agg_min =
source
|> aggregateWindow(every: 1m, fn: min, createEmpty: false)
|> set(key: "agg", value: "min")
agg_max =
source
|> aggregateWindow(every: 1m, fn: max, createEmpty: false)
|> set(key: "agg", value: "max")
union(tables: [agg_mean, agg_min, agg_max])
|> sort(columns: ["_time"])
|> to(bucket: "plant_telemetry_1m", org: "acme-industrial")
Three details are deliberate and worth calling out, because they are what separates this from the four-line example in the documentation:
The range is bounded on both ends and aligned to the window. date.truncate snaps
stop down to a window boundary, so the task never aggregates a window that is still
filling — the classic cause of a rollup value that changes after the fact. start is
stop minus a lookback rounded up to a whole number of windows, so boundaries always
line up.
The lookback is wider than the interval. It covers every plus your late-data
tolerance. Recently closed windows get re-derived and rewritten every run, which is
harmless: to() overwrites a point with the same timestamp and tag set, so the task is
idempotent and late arrivals are folded in on the next pass rather than lost.
createEmpty is always stated. Leaving it to the default is how a rollup ends up
either silently gapped or silently padded with nulls that consume a point per series per
window for the whole retention period.
The model is intentionally simple enough to check by hand:
points/day = (86 400 / resolution_seconds) × series
bytes/day = points/day × bytesPerPoint
resident = bytes/day × retention_days
Series counts follow the cascade. The raw bucket carries every combination of every tag multiplied by the field count. A rollup carries only the retained tags, times the product of the aggregate counts of every tier up to and including itself.
The baseline it compares against — "raw only" — is the cost of keeping raw data at full
resolution for the entire horizon, which is the thing the cascade exists to avoid. The
tiered figure is the raw bucket at its own retention plus every rollup bucket at theirs.
Infinite retention (0s) is priced at the horizon, since an unbounded number is not a
useful comparison.
bytesPerPoint is an input, not a constant, and that is the point. Real TSM
compression depends entirely on the value distribution: a slow-moving counter can land
under 2 bytes per point while a jittery float costs 16 or more. Measure your own buckets
and put the real number in. The default of 16 is deliberately pessimistic.
What the model does not capture: WAL, index and tombstone overhead; series that report intermittently (real gaps make the estimate conservative); replication factor; and shard-group boundary effects, since a bucket only expires whole shard groups and so overshoots its nominal retention slightly. It is a well-founded order of magnitude for capacity planning, not an invoice.
Every rule fires only on something that is genuinely wrong or genuinely surprising, and every finding links to a write-up of that class of problem. A panel full of noise gets ignored, and then the one real finding gets ignored with it.
| Rule | Severity | Fires when |
|---|---|---|
parse-error |
error | A duration field is not a valid Flux literal. Calendar units (mo, y) are rejected because their length varies, which breaks window alignment. |
no-tiers |
error | The cascade is empty, so the raw bucket carries the whole retention window at full resolution. |
offset-ge-every |
error | The task offset is not shorter than its interval, pushing each run into the following schedule slot. |
window-finer-than-source |
error | A tier aggregates into windows finer than the data feeding it, fabricating points instead of reducing them. |
window-not-multiple |
error | A tier window is not a whole multiple of its input resolution, so consecutive windows hold different sample counts and every aggregate is weighted by that count. |
retention-gap |
error | A tier expires before the bucket it summarises, so the long-range history it exists to provide disappears first. |
count-of-count |
error | A tier applies count to data that was already counted or summed, counting rollup rows instead of original events. |
sum-of-mean |
error | A tier sums averaged data, producing a value that scales with the number of parent windows rather than with the data. |
mean-of-mean |
warning | A tier applies mean or median to already-averaged data. Unbiased only if every parent window held the same number of samples; min, max, sum and last cascade cleanly. |
every-shorter-than-window |
warning | The task runs more often than a window closes, re-deriving unchanged windows. |
cardinality |
warning | A tier exceeds 100 000 series. Names the highest-cardinality retained tag. |
negative-saving |
warning | The tiers together cost more than simply keeping raw for the horizon. |
mean-only |
note | A first-hop tier keeps nothing but the mean, so excursions inside a window vanish permanently once raw expires. |
create-empty |
note | createEmpty is on, so a series that stops reporting keeps consuming a point per window for its whole retention. |
no-offset |
note | No offset anywhere, so every task fires the instant its window closes and misses writes still in flight. |
The hash is v<version>.<base64url>, where the payload is short-keyed JSON. The default
design encodes to roughly 300 characters. It is versioned so old links keep working, and
compact rather than binary so you can decode one by hand and read what it says.
Decoding is total: a truncated, corrupted or foreign hash produces an explanatory message and the worked example, never a throw and never a half-applied configuration. A link written by a newer version says so rather than guessing at it. Missing individual values fall back to sensible defaults, and out-of-range numbers are clamped rather than rejected — a link that lost a character in a chat client should still be mostly readable.
src/lib/ Pure logic. No React import anywhere in here.
duration.ts Flux duration parsing and formatting
types.ts The domain model
resolve.ts Config -> resolved plan: durations, cascade wiring, series counts
flux.ts Plan -> Flux task scripts
storage.ts Plan -> storage estimate and growth curve
warnings.ts Plan + estimate -> findings
permalink.ts Config <-> URL hash
palette.ts Tier colours and axis rounding
src/components/ Presentation only. Charts and the cascade diagram are hand-written SVG.
resolvePlan() is the hinge. It parses every duration once, wires each tier to its
predecessor, and computes series counts — so the generator, the storage model and the
warning rules all read the same resolved structure instead of re-deriving it three times
and disagreeing. Durations that fail to parse are collected with a dotted path and
substituted with a neutral fallback, which is why the app keeps rendering while you are
halfway through typing 1h30m instead of blanking out on every keystroke.
There is no state management library and no chart library. Charts are a few dozen lines of SVG each, which keeps the shipped bundle around 77 kB gzipped.
npm run lint # ESLint
npm run format:check # Prettier
npm test # Vitest unit tests
npm run coverage # with a 85% floor on src/lib
npm run e2e # Playwright, against the production build
npm run buildUnit tests cover the pure modules: Flux generation against golden files, the storage model against hand-computed values, every warning rule with both a positive and a negative case, and the permalink codec round-tripping plus a battery of malformed input. The E2E suite drives the built application through the main flow, the copy button, permalink sharing, theme switching and a set of accessibility invariants.
Background on the problems this tool encodes:
- Writing robust Flux scripts for automated data rollups — the window-alignment and idempotency pattern the generated tasks use.
- Cascading hourly-to-daily rollups without double-counting — why
meanofmeanis not the mean, and which aggregates survive a second pass. - Bounding series cardinality in Flux rollup tasks — why dropping one tag beats every other optimisation.
- Mapping InfluxQL GROUP BY time() to Flux aggregateWindow — window semantics, and what changes when you migrate off continuous queries.
- Per-measurement offset tuning for late IoT data — how to size the offset and the late-data tolerance.
- How to configure retention policies in InfluxDB 2.x — the bucket retention semantics the storage model assumes.
See CONTRIBUTING.md. New warning rules are especially welcome — if you have been bitten by a downsampling misconfiguration that this does not catch, that is a gap worth filling.
MIT. See LICENSE.