Skip to content

feat(search-pipeline): add searchIndexerPipeline convenience#647

Open
ddeboer wants to merge 1 commit into
mainfrom
feat/search-indexer-pipeline
Open

feat(search-pipeline): add searchIndexerPipeline convenience#647
ddeboer wants to merge 1 commit into
mainfrom
feat/search-indexer-pipeline

Conversation

@ddeboer

@ddeboer ddeboer commented Jul 23, 2026

Copy link
Copy Markdown
Member

Fix #640

What

searchIndexerPipeline in @lde/search-pipeline wires the common object-grain search indexer and returns the ready-to-run Pipeline<TypedSearchDocument>:

  • one stage per RootType in the schema, each selecting the type’s non-blank roots by class (selectByClass, blank-node-safe since Blank-node roots crash framing; selectByClass selects them #641 landed) and extracting with the schema-generated CONSTRUCT;
  • the single searchIndexWriter terminal, one engine writer per type via writerFor;
  • datasets as a plain Dataset[] (wrapped in ManualDatasetSelection) or any DatasetSelector;
  • pass-through of distributionResolver (e.g. ImportResolver + @lde/sparql-qlever), provenanceStore + pipelineVersion, and reporter.

Consumers now supply only the domain (schema, datasets) and the deployment shell (engine writer, SPARQL/import adapter), instead of repeating the wiring that surfaced #633/#638 downstream. Anything beyond the object grain – bespoke selectors, per-stage tuning, non-SPARQL readers, plugins – still composes searchStages/searchIndexWriter/Pipeline directly; the convenience owns no capability of its own.

Supporting docs (the issue’s original scope)

  • @lde/search-pipeline README: a Quick start for the convenience plus the role → package mapping table (selection, resolution/import via @lde/sparql-qlever, stages, engine writer via @lde/search-typesense, provenance, reporting).
  • The stub READMEs of @lde/sparql-server and @lde/sparql-importer now say they are interface-only and point at @lde/sparql-qlever as the concrete adapter.

Tests

  • Unit: wiring (one engine writer per root type, Dataset[] and DatasetSelector forms, pipelineVersion required with a provenance store).
  • Integration (new port 3008, registered in AGENTS.md): full searchIndexerPipeline(...).run() against a local SPARQL endpoint over the Drapo fixture – both types indexed into their own committed collections, and the fixture’s blank-node CreativeWork excluded by the baked-in blank-node-safe selection.

- Wire the common object-grain indexer into a ready-to-run Pipeline: one
  stage per root type in the schema, roots selected by class, extraction
  generated from the schema, one engine writer per type, with optional
  provenance and reporting passed through.
- Exclude blank-node subjects in selectByClass via FILTER(!isBlank(...)):
  a blank node has no stable document key, so it can never become a
  search document and would crash framing.
- Document the role-to-package mapping in the search-pipeline README and
  point the stub READMEs of sparql-server and sparql-importer at
  sparql-qlever as the concrete adapter.
@ddeboer
ddeboer force-pushed the feat/search-indexer-pipeline branch from 8d81448 to 6feedfe Compare July 23, 2026 18:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ship a searchIndexerPipeline convenience that wires the common indexer (schema + datasets -> Pipeline)

1 participant