You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Stream the public dump's NDJSON files into the archive #192
The public dump builds each score set's NDJSON files as one string in memory. va_ndjson and vrs_ndjson in scripts/export_public_data.py collect every line into a list, join it and return a str, which is then written into the archive. Peak memory grows with the largest score set in the dump. The api's own NDJSON routes already stream: see _stream_score_set_variant_details in routers/score_sets.py.
Scope
Make va_ndjson and vrs_ndjson generators that yield lines in batches.
Write each file into the archive through ZipFile.open(name, "w"), one batch at a time, instead of writestr with a full string.
Leave the JSON metadata files as they are. They're small.
Acceptance criteria
The archive's bytes are unchanged for the existing tests/scripts/test_export_public_data.py fixtures.
Peak memory while exporting the largest score set doesn't grow with its variant count (spot-checked).
changed the title [-]Consider using custom JSON serialization that supports streaming.[/-][+]Stream the public dump's NDJSON files into the archive[/+]on Sep 24, 2026
Activity
afrubin commentedon May 8, 2024
I think this is going to be pretty important once we start serving VRS representations of whole score sets.
[-]Consider using custom JSON serialization that supports streaming.[/-][+]Stream the public dump's NDJSON files into the archive[/+]