Hoist per-row namespace resolution and batch the ClinVar fetch in CSV export #845

Two avoidable per-row costs in the CSV export.

Namespace and key resolution repeats per row

variant_to_csv_row (src/mavedb/lib/csv/columns.py) calls namespace_spec(namespace) for every namespace on every row, and _format_column_key runs CLINVAR_NS_PATTERN.match and CALIBRATION_NS_PATTERN.match — and, on the namespaced path, a second namespace_spec — for every cell. None of it varies across rows: the plan is fixed once plan_csv_columns returns.

Measured at 6.4 µs per row for 9 namespaces / 23 columns, which is 0.6 s per 100k variants and 6.4 s per million, in pure resolution overhead before any data is touched.

ClinVar is fetched one query per namespace

fetch_variant_csv_data (src/mavedb/lib/csv/fetch.py) loops for ns, db_version in clinvar_namespaces.items() and issues a separate query per release. Bounded by the number of ingested releases, so the constant is small today, but the queries are identical apart from db_version and collapse into one IN.

This is most visible in the public dump, which composes every ClinVar release a score set holds.

On the allele data model branch (RT), this loop is _clinvar_by_allele in the same file, keyed by allele id instead of mapped variant id. The same single-query fix applies.

Work

• Resolve the spec and the emitted header key once per (namespace, column) when the plan is built, and have variant_to_csv_row walk a flat pre-resolved list of (header, source, resolver) triples.
• Replace the per-namespace ClinVar loop with a single query filtering db_version.in_(...), keyed by (mapped_variant_id, db_version).

Acceptance criteria

• Existing tests/lib/csv and tests/scripts/test_export_public_data.py coverage passes unchanged; the emitted bytes do not change.
• Composing the widest namespace set over a large score set issues one ClinVar query regardless of how many releases are requested.