Skip to content

perf: overwrite file features in place - #2317

Open
John Lyu (PaleNeutron) wants to merge 2 commits into
microsoft:mainfrom
PaleNeutron:fix/file-feature-storage-overwrite
Open

perf: overwrite file features in place#2317
John Lyu (PaleNeutron) wants to merge 2 commits into
microsoft:mainfrom
PaleNeutron:fix/file-feature-storage-overwrite

Conversation

@PaleNeutron

Copy link
Copy Markdown

Description

  • Replace the pandas-based overlap rewrite in FileFeatureStorage.write with a fixed-width float32 seek/write for the common case.
  • Preserve the file header and use a NumPy-only full rewrite when extending before the current start index.
  • Treat a cleared zero-byte storage as a new file so FeatureStorage.rewrite remains valid.
  • Add focused regression tests for overlapping writes, explicit NaN values, left extension, and rewrite after clear.

Motivation and Context

The existing overlap path reads the complete feature file and builds multiple pandas objects before rewriting it. It also passes the float32 header value to range, which raises TypeError for any overlapping write. Casting that value to int alone is insufficient because the current rewrite omits the start-index header and does not truncate stale trailing bytes.

Since each entry is a fixed-width float32, writes that start at or after the current start index can seek directly to the target byte offset and write only the new values. Only left extension needs to shift existing data and rewrite the file.

How Has This Been Tested?

  • Pass the test by running: pytest qlib/tests/test_all_pipeline.py under upper directory of qlib.
  • Added and ran focused regression tests: python -m pytest -q tests/storage_tests/test_file_feature_storage.py (4 passed).
  • Ran the existing storage tests with their provider directory redirected to /tmp (3 passed).
  • Verified the three overlap regression cases fail against the previous implementation with TypeError.

Screenshots of Test Results (if appropriate):

N/A

Types of changes

  • Fix bugs
  • Add new feature
  • Update documentation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant