Skip to content

feat(python): write TsFiles to file-like and object-storage targets - #1107

Merged
ColinLeeo merged 16 commits into
developfrom
colin/python-write-sink
Nov 16, 2026
Merged

ColinLeeo merged 16 commits into
developfrom
colin/python-write-sink

Conversation

@ColinLeeo

@ColinLeeo ColinLeeo commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Problem

The C++ write stack already accepts any RandomAccessWriteFile sink (merged in #1094), and Python can read from file-like objects — but TsFileWriter(pathname) only took a path. Writing a TsFile to an object store (for example through fsspec/s3fs) therefore required a local temp file and a copy.

Change

  • C API: add write_file_new_memory() / write_file_get_buffer() on top of MemoryRandomAccessWriteFile, so a C caller can serialize into memory and upload the bytes.
  • TsFileWriter::init(std::unique_ptr<RandomAccessWriteFile>): hand sink ownership to the writer so the Python bridge cannot leak it.
  • free_write_file() and the writer factories now cast the opaque handle to RandomAccessWriteFile (its destructor is virtual); before, they assumed WriteFile and would have been wrong for an in-memory sink.
  • Python: python_random_access_write_file.{h,cc} implements RandomAccessWriteFile on top of write/tell (and optional seek/flush/truncate); TsFileWriter(pathname) dispatches on the target type — str/bytes/PathLike keep the old path, anything else must be a writable binary file object. Targets are reset to an empty file before writing, and a failing flush() is reported instead of ignored.
  • Tests: in-memory and file-like round trips (python/tests/test_write_sources.py): BytesIO round trip, path-vs-object byte equality, bytes paths, the pathname= keyword, an existing buffer being reset, and flush-failure propagation.

Usage

import s3fs

fs = s3fs.S3FileSystem(...)
with fs.open("bucket/data.tsfile", "wb") as target:
    with TsFileWriter(target) as writer:
        writer.register_table(table)
        writer.write_table(tablet)

Testing

  • C++ suite: 1537 tests passed; spotless:check clean.
  • Python: 616 passed, 16 skipped (pytest python/tests).

Expose the C++ write sink to Python so TsFileWriter(target) accepts a
writable binary file object (fsspec/S3-style) as well as a path.

- Add the C API in-memory handle (write_file_new_memory /
  write_file_get_buffer) on top of MemoryRandomAccessWriteFile, and take
  ownership of the sink through TsFileWriter::init(unique_ptr).
- free_write_file() and the writer factories now go through
  RandomAccessWriteFile so an in-memory handle is destroyed correctly.
- Add the Python bridge (python_random_access_write_file) implementing
  RandomAccessWriteFile over write/tell/seek/flush/truncate, wire it into
  the tsfile_writer extension, and let TsFileWriter(target) dispatch on
  the target type.
- Tests: in-memory and file-like round trips, plus a MinIO compose file
  and an env-gated integration test for a real object store.
The RAT check failed on the new README.md and docker-compose.yml; add
the standard headers.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Flush and tell failures are silently masked, allowing writes to report success despite uncommitted data or incorrect metadata offsets.

8 open issues
What changed in this PR

Extends the existing C++ write-sink abstraction to Python file-like targets, enabling object-storage writes through clients such as s3fs without an intermediate local file.

Changes:

  • Adds C APIs for creating memory-backed sinks and copying their serialized bytes.
  • Adds an owning C++ writer initializer and a Python file-object bridge.
  • Adds file-like round-trip tests and an optional MinIO integration setup.
File Description
python/​tsfile/​tsfile_writer.pyx Dispatches writer construction between paths and file objects.
python/​tsfile/​python_random_access_write_file.h Declares the Python-target writer factory.
python/​tsfile/​python_random_access_write_file.cc Implements the native adapter for Python write targets.
python/​tests/​test_write_sources.py Tests BytesIO round trips and output equivalence.
python/​tests/​test_minio_integration.py Adds an environment-gated object-storage round trip.
python/​tests/​integration/​minio/​README.md Documents local MinIO setup and testing.
python/​tests/​integration/​minio/​docker-compose.yml Provisions MinIO for integration testing.
python/​setup.py Compiles the write adapter into the Python extension.
cpp/​src/​writer/​tsfile_writer.h Declares ownership-taking sink initialization.
cpp/​src/​writer/​tsfile_writer.cc Implements sink ownership transfer.
cpp/​src/​cwrapper/​tsfile_cwrapper.h Declares memory-sink creation and buffer APIs.
cpp/​src/​cwrapper/​tsfile_cwrapper.cc Implements memory APIs and generalized sink handling.
.gitignore Allows tracking the write adapter header.

Review effort: Balanced

Comment thread cpp/src/cwrapper/tsfile_cwrapper.cc Resolved
Comment thread cpp/src/writer/tsfile_writer.cc Resolved
Comment thread python/tsfile/tsfile_writer.pyx Resolved
Comment thread python/tsfile/tsfile_writer.pyx Resolved
Comment thread python/tsfile/tsfile_writer.pyx Resolved
Review follow-ups on the write bridge:

- sync() only ignores a missing flush(); a flush that raises now returns
  E_FILE_SYNC_ERR instead of pretending the data was committed.
- get_position() reports failure instead of returning 0, the factory
  validates tell() and starts from an empty file (seek+truncate when the
  target supports them), and it applies memory_threshold like the
  path-based writer.
- TsFileWriter keeps the pathname keyword (it still accepts bytes and
  PathLike) and tsfile_writer_new_c now uses os.fsencode, so bytes paths
  work.
- write_file_get_buffer() rejects a sink larger than 4 GiB instead of
  truncating the reported length, and init(unique_ptr) keeps ownership
  until initialization succeeds so a throw cannot leak the sink.
It would add a Docker/MinIO dependency to the test tree for a path the
file-like unit tests already cover; keep only test_write_sources.py.
setup.py lists python_random_access_write_file.cc as an Extension source,
so setuptools copies it into an sdist automatically, but the header is
only referenced through depends= and was left out; CI builds the sdist
and failed with 'python_random_access_write_file.h: No such file'.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

…ailure

- A forward-only io.RawIOBase inherits callable seek()/truncate() that
  raise io.UnsupportedOperation; treat a failed rewind as "not seekable"
  and accept the target when it already starts at 0, and only require
  truncate when earlier content has to be cleared. Added a regression
  test with a write-only stream.
- _tsfile_writer_close() now deletes the writer even when close() fails,
  and TsFileWriterPy drops its handle before raising, so a persistent
  flush error no longer leaks the native writer or its Python target.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The truncation fallback can accept nonempty seekable targets after truncate() fails, allowing shorter writes to leave stale trailing bytes and produce an unreadable TsFile despite a successful close.

1 open issue
2 resolved since last review

Review effort: Balanced

python/pom.xml's clean plugin deletes tsfile/*.h to drop generated SDK
headers, with an exception only for the read bridge header. CI runs
clean verify, so the write header was removed before build_ext. Add it
to the excludes.
mvn clean used to delete every *.h there, which removed the hand-written
bridge headers (each needed its own exception) along with the generated
Cython public-API headers. Switch to an explicit list: compiled
extensions, generated .cpp, and the two Cython-generated headers; the
bridge headers now survive clean without per-file exceptions.
Review pass over the write bridge:

- get_position() no longer asks the target for tell() after creation.
  TsFileIOWriter records chunk offsets from it after every flush, and a
  stream whose tell() becomes unavailable mid-write would have silently
  produced -1 offsets; track bytes written internally instead (updated
  by seek_to_end too).
- write_file_new_memory() now runs the same lazy init_tsfile_config() as
  write_file_new(), so a writer built on a memory handle in a fresh
  process does not read zero-initialized global config.
- Document writable file objects in python/README (reset semantics,
  forward-only streams, FileSyncError), and add C-API and Python tests:
  memory-handle round trip through tsfile_writer_new, memory_threshold
  propagation, and a target whose tell() fails after the first call.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

3 open issues
1 resolved since last review

Review effort: Lite

Comment thread python/tsfile/tsfile_py_cpp.pyx Resolved
- The sink guard mixed PyGILState_Ensure/Release with
  PyEval_SaveThread/RestoreThread. Every access already happens while
  the GIL is held, so the GIL alone serializes target_/position_ access;
  use a plain GIL guard and drop the mutex.
- A failing write()/flush() no longer discards the underlying exception:
  its text is stored per thread and passed as the context of the raised
  FileSyncError, so e.g. the OSError message survives. Test asserts it.
Second review pass:

- The per-thread error text is cleared at the start of every sink
  operation, so a previous failure cannot be attached to an unrelated
  one; truncate/seek failures now record their message too.
- libtsfile_init() runs before the target is rewound/truncated, so a
  failed initialization no longer destroys the caller's file content.
- The C API memory round-trip test uses a per-process unique file name so
  parallel CTest processes cannot collide.
Third review pass:

- The bridge destructor skips close() once the interpreter is
  finalized, where PyGILState_Ensure is undefined.
- seek_to_end() restores the previous offset when tell() cannot report
  the end position, so a fallback append does not start at the wrong
  place.
- The error text is read only after the failing close/flush call, and
  only for writers created from a file object, so a path writer cannot
  inherit an unrelated sink message.
- write_file_get_buffer() documents E_OVERFLOW and has a test for the
  non-memory (RET_NOT_SUPPORT) path.
Fourth review pass:

- TsFileWriter::init(unique_ptr) deleted nothing when a writer that
  already owned a sink was initialized again, leaking the old sink (and
  its Python target); release it once the new init succeeds.
- Tests: tree-model write (register_timeseries + write_row_record) to a
  file object, and an explicit flush() mid-write followed by close and a
  successful read-back.
A text stream would fail on the first bytes with an opaque TypeError;
check the target's mode (when it exposes one) and raise ValueError
before touching the file. Added a test with a "w"-mode file.
The new C API tests use _getpid() under _WIN32 but the file only included
io.h, breaking the MSVC build (the py-msvc jobs build the C++ tests too).
@codecov-commenter

codecov-commenter commented Nov 16, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 47.22222% with 19 lines in your changes missing coverage. Please review.
✅ Project coverage is 67.34%. Comparing base (4ef3102) to head (49252b5).
⚠️ Report is 4 commits behind head on develop.

Files with missing lines Patch % Lines
cpp/src/writer/tsfile_writer.cc 0.00% 10 Missing ⚠️
cpp/src/cwrapper/tsfile_cwrapper.cc 65.38% 2 Missing and 7 partials ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #1107      +/-   ##
===========================================
+ Coverage    66.89%   67.34%   +0.44%     
===========================================
  Files          788      788              
  Lines        57630    57812     +182     
  Branches      9310     9360      +50     
===========================================
+ Hits         38551    38931     +380     
+ Misses       17399    17199     -200     
- Partials      1680     1682       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ColinLeeo
ColinLeeo requested a balanced review from Copilot November 16, 2026 05:43
It forwards its target to TsFileWriter, so the new sink support already
worked; update the annotation/docstring, the README and add a round-trip
test through TsFileTableWriter.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because repeated writer initialization can leak resources and retain prior state, while Python target truncation and flush behavior introduce additional data-integrity and error-reporting risks.

No open issues 🎉

3 resolved since last review

Review effort: Balanced

@ColinLeeo
ColinLeeo merged commit 6ece2cf into develop Nov 16, 2026
52 of 53 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants