Skip to content

feat: add retrieval-quality + version A/B evaluation harness (#98)#110

Merged
dcfocus merged 1 commit into
lance-format:mainfrom
dcfocus:feat/issue-98-eval-harness
Jun 27, 2026
Merged

feat: add retrieval-quality + version A/B evaluation harness (#98)#110
dcfocus merged 1 commit into
lance-format:mainfrom
dcfocus:feat/issue-98-eval-harness

Conversation

@dcfocus

@dcfocus dcfocus commented Jun 25, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds a retrieval-quality evaluation harness that measures search/retrieve quality against a labeled query set and compares quality across dataset versions — closing #98. Cross-version regression eval is the differentiator a stateless vector DB can't offer.

This PR is independent of #108 (#100) and #109 (#102) — it branches from main and can be reviewed/merged on its own.

Core (lance-context-core::eval)

  • Eval dataset formatEvalQuerySet / EvalQuery / RelevanceLabel, referencing records by stable external_id with optional graded relevance, loadable from / serializable to JSONL.
  • Metrics — recall@k, precision@k, MRR, nDCG@k (linear gain), hit-rate, as pure functions.
  • RunnerContextStore::evaluate(query_set, config) scores a set at the current version for vector (search) or hybrid (retrieve) mode, with configurable k, filters, and lifecycle options. Returns per-query + aggregate scores and a manifest (query-set id, version, k, mode, distance metric).
  • Version A/BContextStore::evaluate_versions(..., baseline, candidate) checks out each version, evaluates, restores the original version, and reports per-metric deltas. MetricScores::delta also enables config A/B (two evaluate runs).

Python

  • Context.evaluate(queries, *, k, mode, filters, include_expired, include_retired) → report dict.
  • Context.evaluate_versions(queries, baseline_version, candidate_version, ...){baseline, candidate, deltas}.

Query sets and reports are marshaled as JSON across the pyo3 boundary, reusing the serde-derived core types.

Tests

  • core (12): metric correctness vs hand-computed fixtures (perfect / rank-2 / none / graded nDCG / precision@k), JSONL round-trip, vector + hybrid runners, filter and lifecycle handling, and the version-A/B mechanism (zero-delta + version restore).
  • python (5): test_eval.py — vector mode, graded nDCG, config A/B delta, version A/B, invalid-mode error.

README gains an eval + version-A/B example.

Checks

  • cargo test -p lance-context-core --lib — 59 passed
  • cargo fmt --all -- --check and cargo clippy --workspace --all-targets -- -D warnings — clean
  • ruff format --check, ruff check, pyright — clean
  • test_eval.py — 5 passed

Acceptance criteria (#98)

  • Run a labeled query set → recall@k / MRR / nDCG for vector and hybrid retrieval
  • A/B two versions (or two configs) → per-metric deltas
  • Results pinned to a context version + config in a reproducible report
  • Tests cover metric correctness against fixtures, filter/lifecycle handling, and the version-A/B mechanism

Notes

Closes #98

@dcfocus dcfocus force-pushed the feat/issue-98-eval-harness branch from 3faf2cf to a3422b5 Compare June 27, 2026 06:34
…ormat#98)

Adds an evaluation harness that measures retrieval quality against a
labeled query set and compares quality across dataset versions — the
cross-version regression eval a stateless vector DB can't do.

Core (`lance-context-core::eval`):
- Eval dataset format: `EvalQuerySet` / `EvalQuery` / `RelevanceLabel`,
  referencing records by stable `external_id` with optional graded
  relevance, loadable from / serializable to JSONL.
- Metrics: recall@k, precision@k, MRR, nDCG@k (linear gain), and
  hit-rate, as pure functions tested against hand-computed fixtures.
- Runner: `ContextStore::evaluate(query_set, config)` scores a set at the
  current version for vector (`search`) or hybrid (`retrieve`) mode, with
  configurable `k`, filters, and lifecycle options; returns per-query and
  aggregate scores plus a manifest (query-set id, version, k, mode,
  distance metric) for reproducibility.
- Version A/B: `ContextStore::evaluate_versions(..., baseline, candidate)`
  checks out each version, evaluates, restores the original version, and
  reports per-metric deltas. `MetricScores::delta` supports config A/B
  (two `evaluate` runs) as well.

Python:
- `Context.evaluate(queries, ...)` and
  `Context.evaluate_versions(queries, baseline, candidate, ...)` returning
  report dicts (query sets and reports are marshaled as JSON across the
  pyo3 boundary, reusing the serde types).

Tests:
- core: metric correctness (perfect / rank-2 / none / graded / precision),
  JSONL round-trip, vector + hybrid runners, filter and lifecycle
  handling, and the version-A/B mechanism (zero-delta + version restore).
- python: `test_eval.py` covering vector mode, graded nDCG, config A/B,
  version A/B, and invalid-mode handling.

README gains an eval + version-A/B example.

Note: cross-version A/B with *differing data* isn't unit-tested because
MemWAL-backed writes don't reliably advance the base-table manifest
version (the same limitation `test_time_travel_checkout` xfails on); the
A/B mechanism and metric deltas are covered via the same-version and
config-A/B tests.

Closes lance-format#98
@dcfocus dcfocus force-pushed the feat/issue-98-eval-harness branch from a3422b5 to 9884c56 Compare June 27, 2026 06:43
@dcfocus dcfocus merged commit 3926b3b into lance-format:main Jun 27, 2026
9 checks passed
@dcfocus dcfocus deleted the feat/issue-98-eval-harness branch June 27, 2026 19:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Retrieval-quality + version A/B evaluation harness (recall@k / MRR / nDCG)

1 participant