Stream /scores & /counts CSV responses (no row cap) #770

Context

The CSV endpoints build the entire response in memory (StreamingResponse(iter([csv_str]))) and return it as a fake stream. This is the memory bomb that let a ChatGPT-User crawler pull full, unpaginated CSVs and saturate prod-api. Full-CSV download is a legitimate feature (the UI download button pulls with no pagination), so we stream rather than cap rows.

Part of the effort to make /scores & /counts safe to reopen to AI agents.

The CSV code has since moved out of src/mavedb/lib/score_sets.py into the src/mavedb/lib/csv/ package (columns, fetch, specs, namespaces, annotations, score_set, variant), and drop_na_columns is now drop_unused_hgvs_columns — the old name survives only as a deprecated query parameter. Paths below reflect that layout.

src/mavedb/scripts/export_public_data.py builds full CSV strings through the same get_score_set_variants_as_csv, so it inherits whatever this lands. It is not an endpoint and is out of scope here, but it is the largest single consumer.

Streaming spreads a full export's CPU over the download but doesn't reduce it, so the CSV build cap (#898) stays until exports are cheap or cached.

Scope

• Extract column derivation from get_score_set_variants_as_csv so the buffered and streaming paths share it — done by the namespace refactor: plan_csv_columns and assemble_csv_headers in src/mavedb/lib/csv/columns.py are already pure functions over a namespace plan.
• Add stream_score_set_variants_as_csv(...) beside get_score_set_variants_as_csv (src/mavedb/lib/csv/score_set.py): write the header from assemble_csv_headers, iterate the Variant query with stream_results=True, flush ~1k-row batches via variant_to_csv_row. rows_to_csv (src/mavedb/lib/csv/columns.py) is the buffered serializer to split against — it writes every row into one io.StringIO and returns getvalue().
• Have fetch_variant_csv_data (src/mavedb/lib/csv/fetch.py) yield batches rather than materialize list[Variant], so ORM rows are released as the file is written.
• Use the request-scoped db session (no extra session).
• Make drop_unused_hgvs_columns streamable via a single up-front aggregate query — bool_or(...) over hgvs_nt / hgvs_splice / hgvs_pro, with a Postgres regex mirroring _is_output_null / _OUTPUT_NULL_STRINGS (not validation.utilities.is_null, which the export no longer uses: it additionally counts - as null) — instead of the all-rows Python scan; use DictWriter(extrasaction='ignore'). This is the one genuine all-rows dependency in the export.
• Re-evaluate whether the buffered fallback is still needed. The old rationale was namespaces requiring all-rows side-queries (clingen/vep/gnomad/clinvar/post-mapped); those are now served by fetch_variant_csv_data's single joined query plus ClinVar lookups keyed by mapped-variant id, both of which batch by variant. Confirm the same for annotations_for_rows (src/mavedb/lib/csv/annotations.py) before assuming the fallback can be dropped.
• Index the row order: ix_variants_scoreset_number on (scoreset_id, variant number, id), with Variant.variant_number as the shared expression (perf(db): index variants by score set and variant number #899).
• Point all four CSV download endpoints at the generator: GET /score-sets/{urn}/variants/data, GET /score-sets/{urn}/scores, GET /score-sets/{urn}/counts (src/mavedb/routers/score_sets.py), and GET /variants/{urn}/csv (src/mavedb/routers/variants.py).
• Give the stream a mid-flight error convention. CatchAllErrorMiddleware (src/mavedb/lib/middleware/errors.py) re-raises once bytes are on the wire, so a genuinely streaming export cannot convert a late failure into a 500. lib/annotation already reports a failed record in-band; follow that.

Acceptance criteria

• Byte-for-byte parity with current output across scores / counts / custom-namespace + drop_unused_hgvs_columns, including a test case with an all-"NA" hgvs column.
• A streaming test on a score set large enough to force multiple chunks streams fully and correctly, and fails loudly if the request DB session closes mid-stream (guards the FastAPI 0.121 yield-dependency timing).
• The first chunk reaches the client before the last row is serialized.
• EXPLAIN of the export query on a large score set shows an index scan in variant-number order, with no sort node.
• Peak memory for /scores & /counts no longer scales with full CSV size (spot-checked).
• Existing tests/lib/csv/ coverage passes unchanged.