Skip to content

Stream the public dump's NDJSON files into the archive #192

Description

@jstone-dev

Problem

The public dump builds each score set's NDJSON files as one string in memory. va_ndjson and vrs_ndjson in scripts/export_public_data.py collect every line into a list, join it and return a str, which is then written into the archive. Peak memory grows with the largest score set in the dump. The api's own NDJSON routes already stream: see _stream_score_set_variant_details in routers/score_sets.py.

Scope

  • Make va_ndjson and vrs_ndjson generators that yield lines in batches.
  • Write each file into the archive through ZipFile.open(name, "w"), one batch at a time, instead of writestr with a full string.
  • Leave the JSON metadata files as they are. They're small.

Acceptance criteria

  • The archive's bytes are unchanged for the existing tests/scripts/test_export_public_data.py fixtures.
  • Peak memory while exporting the largest score set doesn't grow with its variant count (spot-checked).

Activity

  1. afrubin commented on May 8, 2024

    @afrubin
    Member

    I think this is going to be pretty important once we start serving VRS representations of whole score sets.

  2. added this to the milestone on Oct 1, 2024
  3. removed this from the milestone on Jul 27, 2026
  4. added theissue type on Aug 7, 2026
  5. added this to the R2026.4 milestone on Sep 24, 2026
  6. changed the title [-]Consider using custom JSON serialization that supports streaming.[/-] [+]Stream the public dump's NDJSON files into the archive[/+] on Sep 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    app: backendTask implementation touches the backend

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions