Ship the search indexer as a CLI and Docker image (object grain, config-driven)

[ddeboer]

Motivation

#647 shipped searchIndexerPipeline: the composition layer that turns a schema + datasets + engine writer into a ready-to-run Pipeline. What remains for the write path is the same final layer the read path got in #646 (ADR 15): a bootable process and prebuilt Docker image, so a deployment configures the indexer instead of writing a Node program around it.

The symmetry is the point: mount the same schema-declaration module (.mjs default-exporting SearchType declarations, validated at boot by searchSchema()) into both the indexer image and @lde/search-api-server, point both at the same TYPESENSE_* coordinates, and the write and read sides cannot disagree about the schema. Linked Open Limburg is the object-grain consumer that would drop its last hand-rolled shell; deployments with bespoke root selectors (e.g. the Dataset Register’s deployment-fact entry points) stay on the library path, which remains first-class.

Proposal

A new @lde/search-indexer package (name open): a Commander CLI wrapping searchIndexerPipeline, published to npm and shipped as a Docker image that installs the published package (same few-line Dockerfile + docker.yml release-tag trigger as #646). Engine-bound to Typesense, like search-api-server – that binding cannot live in @lde/search-pipeline without breaking its engine-agnosticism.

Configuration (all boot-or-fail, every problem reported in one error, mirroring ADR 15):

• schema: mounted declaration module (when GraphQL search API #495 lands, mounted SHACL becomes a second source);
• datasets: a list of dataset IRIs, or registry criteria for a RegistrySelector;
• engine: TYPESENSE_* env vars + rebuild mode (in-place / blue-green), optional collection-name prefix;
• provenance: a mounted file path (FileProvenanceStore) + PIPELINE_VERSION;
• reporter: ConsoleReporter by default.

The QLever question (the import path)

The one genuinely open design problem: distributionResolver. Endpoint-only (SparqlDistributionResolver, the Pipeline default) works in any container unchanged. The import path (ImportResolver + @lde/sparql-qlever) does not, because importing is not “have a QLever running” but a per-dataset control loop: download the dump, run qlever index (offline index build), (re)start the server on the new index – one QLever instance per dataset.

That rules out the obvious-looking option:

• Statically-declared sidecar (Docker Compose service / k8s sidecar): does not fit. A compose-managed QLever only exposes the SPARQL port; the pipeline has no control plane to rebuild its index or restart it per dataset. Feeding a running server instead is not viable either: SPARQL UPDATE’s update-triples overlay performs badly and deleted triples can reappear after restart, and Graph Store HTTP POST is unusable for large datasets (Graph Store Protocol POST with large data ad-freiburg/qlever#2014). So “a separate QLever container” only works when the pipeline itself creates and controls it, not when the orchestrator declares it.

Which leaves three real options:

1. Docker socket mount – DockerTaskRunner as-is: the indexer container mounts /var/run/docker.sock and spawns/controls sibling QLever containers. Zero code changes, Compose-friendly (the QLever container simply isn’t in the compose file). Unavailable on containerd-based Kubernetes; socket access is a security concern.
2. Native mode in-image – ship the QLever engine + qlever CLI in the indexer image (e.g. base on adfreiburg/qlever, add Node): NativeTaskRunner keeps full lifecycle control, works anywhere a container runs, including k8s. Costs: image size, engine-version coupling, and QLever shares the container’s cgroup – a QLever OOM kills the indexer process, so --memory-max-size must be set below the container limit for graceful aborts.
3. A controllable sidecar – a QLever sidecar with a small control agent (shared volume for dumps, an endpoint to trigger index/restart). Cleanest isolation (own cgroup), but new moving parts and a new protocol; only worth designing if 1 and 2 both prove unacceptable.

Suggested scoping: v1 ships endpoint-only plus option 1 behind a flag (it is free); evaluate option 2 as the k8s answer in a follow-up. Option 3 only on demonstrated need.

Out of scope

• Bespoke root selectors and per-stage tuning – the library composition path (searchStages + searchIndexWriter + Pipeline).
• SHACL as schema source (GraphQL search API #495).
• Any engine other than Typesense.

Relationships

• Completes the write-path counterpart of Batteries-included served GraphQL search API + self-contained playground #600 / feat(search-api-server): serve the search API as a bootable process and Docker image #646 (ADR 15 layering); builds on Ship a searchIndexerPipeline convenience that wires the common indexer (schema + datasets -> Pipeline) #640 / feat(search-pipeline): add searchIndexerPipeline convenience #647.
• First publish needs the manual Trusted-Publishing bootstrap, like every new @lde/* package.