heron-polyfeed-reader

A small local feed reader and site monitor. heron polls RSS, Atom and RDF feeds, watches sites for downtime, content change and TLS certificate expiry (honouring robots.txt), keeps everything in an append-only NDJSON log with a SQLite (FTS5) index rebuilt from it, and serves a localhost-only web UI. Written in Zig; Linux first.

Table of Contents

• What this is
• What this is not
• Quick start
  • Commands
  • Container
• Layout
• Where to go next
• Licence
• AI-Assisted Installation (Recommended)
  • Just say it
  • Privacy and security
  • Troubleshooting

What this is

• A feed reader. Tolerant XML, encoding and date parsing (recorded real feeds and hand-made edge cases are fixtures under src/testdata/); guid/id then link+date de-duplication; conditional requests (ETag, Last-Modified); per-source intervals with jitter, exponential backoff and Retry-After.

• A site monitor. Up/down transitions (DOWN after two failures, first success is a silent baseline), content-change hashes (alerts rate-limited to one per hour), robots.txt compliance (RFC 9309: a BLOCKED alert when it disallows the monitored path) and leaf-certificate expiry. Alerts are stored as items of kind alert, so they appear in the same lists and searches as articles.

• A daemon with a CLI. One binary, one JSON config, one data directory. The web UI is off unless configured and binds 127.0.0.1:8484 by default.

What this is not

• Not a hosted or multi-user service: one user, one machine, loopback only unless you say otherwise (web.allow_non_loopback).

• Not a WebSub subscriber (deliberately), and not a browser: the web UI has no JavaScript and only follows http(s) links.

• Not yet proven beyond HTTP: the adapter seam is designed for NNTP, Gopher, Gemini, ActivityPub and AT Protocol, but no second adapter exists (roadmap M8).

Quick start

Needs Zig 0.17.0 exactly (build.zig.zon says so; the std.Io, std.http and TLS APIs changed from 0.15/0.16). mise.toml pins it; just setup installs it.

zig build test --summary all            # 85 unit tests, fixtures from src/testdata/
zig build -Doptimize=ReleaseSafe        # -> zig-out/bin/heron
cp examples/heron.json ~/.config/heron.json  # edit sources, monitors, web
heron --config ~/.config/heron.json check-config
heron --config ~/.config/heron.json daemon   # foreground: poll, serve the UI if enabled

Through just: just test, just lint (zig fmt --check), just build, just build-release, just test-smoke, just quality.

Note	just test and just build run every language the estate provisioning library detects. This tree also carries the template’s Idris2 ABI seam (src/interface/abi.ipkg), so those recipes need idris2 on PATH as well; zig build test alone runs the project’s own suite.

Commands

heron [--config FILE] [--data DIR] <command> [args]

  daemon                    run in the foreground: poll sources, serve the web UI if enabled
  fetch [NAME...]           poll the named sources (default: all) once, now
  list [--source NAME] [--unread] [--limit N]
  search TEXT               substring search over titles and text (needs 3+ characters)
  show ID                   print an item and mark it read
  read ID | unread ID       change an item's read state
  mark-all-read [NAME]
  sources                   per-source status: last check, failures, next poll
  monitors                  monitor status: up/down, HTTP status, TLS certificate expiry
  rebuild                   rebuild the index and search tables from the log
  check-config              validate the config file
  help

Environment: HERON_CONFIG and HERON_DATA stand in for --config and --data. The data directory holds the NDJSON log (source of truth), the SQLite index and the heron.lock lock file.

Container

podman build -f Containerfile --target test    -t heron-test .   # runs the unit tests
podman build -f Containerfile --target runtime -t heron .        # static binary on scratch (~23 MB)

The toolchain is pinned by version and verified by SHA-256 inside the Containerfile; CI (.github/workflows/zig-ci.yml) pins the same digest. Inside a container the default loopback bind is unreachable from the host: set "web": {"bind": "0.0.0.0", "allow_non_loopback": true} only together with -p 127.0.0.1:8484:8484 on the host side.

Layout

Path	What lives there
build.zig, build.zig.zon	The build: core module heron (everything but main.zig), the vendored SQLite compiled by zig cc, test and run steps.
src/item.zig, src/adapter.zig	Item model; Adapter vtable, Registry, Cond (etag, last_modified, opaque state), Outcome (items / not_modified / failed), FailureClass, parseRetryAfter.
src/transport.zig, src/http.zig	Fakeable Transport interface; the HTTP(S) implementation (select-based wall-clock timeout, manual redirects, size cap, certificate probe).
src/xml.zig, src/encoding.zig, src/dates.zig, src/feed.zig, src/feed_adapter.zig, src/dedupe.zig	Tolerant XML DOM (XXE-safe), encoding detection and conversion, tolerant dates, RSS2/Atom/RDF mapping, de-duplication.
src/scheduler.zig	Pure logic over an injected clock and PRNG; FakeClock for tests.
src/store.zig, src/sqlite.zig	NDJSON log plus SQLite state and FTS5 trigram index; crash replay via meta.indexed_offset; hand-written SQLite externs (@cImport no longer exists in Zig 0.17).
src/monitor.zig, src/robots.zig	Monitor adapter and robots.txt.
src/engine.zig, src/app.zig, src/daemon.zig, src/cli.zig, src/web.zig, src/localserver.zig, src/config.zig, src/util.zig	Poll orchestration, wiring, daemon loop and signals, CLI, web UI, tiny HTTP server, JSON config.
src/vendor/tls_client.zig	Patched copy of std’s TLS client used only to read the leaf certificate’s expiry; re-sync on every Zig upgrade (src/vendor/NOTICE).
vendor/sqlite/	SQLite 3.50.4 amalgamation (FTS5, THREADSAFE=2, no load-extension).
src/testdata/	Recorded real feeds (real/) and hand-made edge cases.
examples/heron.json	A complete example configuration.
src/interface/, docs/, .machine_readable/, build/, www/	Estate scaffolding from the RSR template: ABI/FFI seam, documentation, the repo deed’s machine-readable companions, just phase modules, site-operations bundle.

Where to go next

• ROADMAP — milestones, blockers, next actions, known gaps.

• Usage and Architecture — configuration reference and internals.

• CONTRIBUTING — toolchain, conventions, signing.

• The repo deed and CLAUDE.md — what this repo IS, for tools and agents.

• The repository map — generated; what every directory is for.

Licence

Code, configuration and scripts are Mozilla Public License 2.0 (MPL-2.0); prose documentation is CC-BY-SA-4.0. Both texts live in LICENSES/, and per-file SPDX-License-Identifier headers are authoritative. Long-term attribution uses Quantum-Safe Provenance — see the Quantum-Safe Provenance exhibit. SQLite is public domain (vendor/sqlite); the patched TLS client in src/vendor/ carries the Zig standard library’s MIT licence (src/vendor/NOTICE).

AI-Assisted Installation (Recommended)

Just say it

Say this to any AI assistant that can read a URL and run commands (or give you commands to paste):

Set up heron-polyfeed-reader from https://github.com/hyperpolymath/heron-polyfeed-reader — follow docs/AI_INSTALLATION_GUIDE.adoc in that repo.

The URL leads the assistant to docs/AI_INSTALLATION_GUIDE.adoc. The assistant checks your system, installs Zig 0.17.0 through mise (just setup), builds and tests heron, and proves the result with just doctor. ./launcher.sh --ai-setup (or just ai-setup) prints the same line.

Privacy and security

Important

Setting up heron-polyfeed-reader clones this repository and installs its developer tools per-user with mise (no administrator rights, nothing sent anywhere). heron itself talks only to the feeds and sites you configure, and its web UI listens on loopback unless you say otherwise.

Troubleshooting

Run just doctor; each FAIL carries a code explained in docs/SETUP.adoc §Troubleshooting, and just heal applies the automatic fixes. The manual route is docs/SETUP.adoc; docs/troubleshooting.adoc lists `heron’s own known failure modes.