Repository navigation
feat(python): write TsFiles to file-like and object-storage targets - #1107
Conversation
Expose the C++ write sink to Python so TsFileWriter(target) accepts a writable binary file object (fsspec/S3-style) as well as a path. - Add the C API in-memory handle (write_file_new_memory / write_file_get_buffer) on top of MemoryRandomAccessWriteFile, and take ownership of the sink through TsFileWriter::init(unique_ptr). - free_write_file() and the writer factories now go through RandomAccessWriteFile so an in-memory handle is destroyed correctly. - Add the Python bridge (python_random_access_write_file) implementing RandomAccessWriteFile over write/tell/seek/flush/truncate, wire it into the tsfile_writer extension, and let TsFileWriter(target) dispatch on the target type. - Tests: in-memory and file-like round trips, plus a MinIO compose file and an env-gated integration test for a real object store.
The RAT check failed on the new README.md and docker-compose.yml; add the standard headers.
There was a problem hiding this comment.
🟡 Changes recommended
Flush and tell failures are silently masked, allowing writes to report success despite uncommitted data or incorrect metadata offsets.
8 open issues
Propagate errors from an existing flush method · New Handle tell failures without corrupting metadata offsets · New Reject memory sink sizes above UINT32_MAX · New Keep sink ownership until writer initialization succeeds · New Prepare seekable targets as empty files · New Preserve the pathname keyword for compatibility · New Normalize bytes paths before opening the writer · New Apply memory threshold to file-object writers · New
What changed in this PR
Extends the existing C++ write-sink abstraction to Python file-like targets, enabling object-storage writes through clients such as s3fs without an intermediate local file.
Changes:
- Adds C APIs for creating memory-backed sinks and copying their serialized bytes.
- Adds an owning C++ writer initializer and a Python file-object bridge.
- Adds file-like round-trip tests and an optional MinIO integration setup.
| File | Description |
|---|---|
python/tsfile/tsfile_writer.pyx |
Dispatches writer construction between paths and file objects. |
python/tsfile/python_random_access_write_file.h |
Declares the Python-target writer factory. |
python/tsfile/python_random_access_write_file.cc |
Implements the native adapter for Python write targets. |
python/tests/test_write_sources.py |
Tests BytesIO round trips and output equivalence. |
python/tests/test_minio_integration.py |
Adds an environment-gated object-storage round trip. |
python/tests/integration/minio/README.md |
Documents local MinIO setup and testing. |
python/tests/integration/minio/docker-compose.yml |
Provisions MinIO for integration testing. |
python/setup.py |
Compiles the write adapter into the Python extension. |
cpp/src/writer/tsfile_writer.h |
Declares ownership-taking sink initialization. |
cpp/src/writer/tsfile_writer.cc |
Implements sink ownership transfer. |
cpp/src/cwrapper/tsfile_cwrapper.h |
Declares memory-sink creation and buffer APIs. |
cpp/src/cwrapper/tsfile_cwrapper.cc |
Implements memory APIs and generalized sink handling. |
.gitignore |
Allows tracking the write adapter header. |
Review effort: Balanced
Review follow-ups on the write bridge: - sync() only ignores a missing flush(); a flush that raises now returns E_FILE_SYNC_ERR instead of pretending the data was committed. - get_position() reports failure instead of returning 0, the factory validates tell() and starts from an empty file (seek+truncate when the target supports them), and it applies memory_threshold like the path-based writer. - TsFileWriter keeps the pathname keyword (it still accepts bytes and PathLike) and tsfile_writer_new_c now uses os.fsencode, so bytes paths work. - write_file_get_buffer() rejects a sink larger than 4 GiB instead of truncating the reported length, and init(unique_ptr) keeps ownership until initialization succeeds so a throw cannot leak the sink.
It would add a Docker/MinIO dependency to the test tree for a path the file-like unit tests already cover; keep only test_write_sources.py.
setup.py lists python_random_access_write_file.cc as an Extension source, so setuptools copies it into an sdist automatically, but the header is only referenced through depends= and was left out; CI builds the sdist and failed with 'python_random_access_write_file.h: No such file'.
There was a problem hiding this comment.
🟡 Changes recommended
Forward-only streams are rejected when optional seek/truncate methods are unsupported, and persistent flush failures can leak the native writer and retained Python target.
2 open issues
8 resolved since last review
Handle tell failures without corrupting metadata offsets Propagate errors from an existing flush method Apply memory threshold to file-object writers Normalize bytes paths before opening the writer Preserve the pathname keyword for compatibility Prepare seekable targets as empty files Keep sink ownership until writer initialization succeeds Reject memory sink sizes above UINT32_MAX
Review effort: Balanced
…ailure - A forward-only io.RawIOBase inherits callable seek()/truncate() that raise io.UnsupportedOperation; treat a failed rewind as "not seekable" and accept the target when it already starts at 0, and only require truncate when earlier content has to be cleared. Added a regression test with a write-only stream. - _tsfile_writer_close() now deletes the writer even when close() fails, and TsFileWriterPy drops its handle before raising, so a persistent flush error no longer leaks the native writer or its Python target.
There was a problem hiding this comment.
🟡 Changes recommended
The truncation fallback can accept nonempty seekable targets after truncate() fails, allowing shorter writes to leave stale trailing bytes and produce an unreadable TsFile despite a successful close.
1 open issue
2 resolved since last review
Review effort: Balanced
python/pom.xml's clean plugin deletes tsfile/*.h to drop generated SDK headers, with an exception only for the read bridge header. CI runs clean verify, so the write header was removed before build_ext. Add it to the excludes.
mvn clean used to delete every *.h there, which removed the hand-written bridge headers (each needed its own exception) along with the generated Cython public-API headers. Switch to an explicit list: compiled extensions, generated .cpp, and the two Cython-generated headers; the bridge headers now survive clean without per-file exceptions.
Review pass over the write bridge: - get_position() no longer asks the target for tell() after creation. TsFileIOWriter records chunk offsets from it after every flush, and a stream whose tell() becomes unavailable mid-write would have silently produced -1 offsets; track bytes written internally instead (updated by seek_to_end too). - write_file_new_memory() now runs the same lazy init_tsfile_config() as write_file_new(), so a writer built on a memory handle in a fresh process does not read zero-initialized global config. - Document writable file objects in python/README (reset semantics, forward-only streams, FileSyncError), and add C-API and Python tests: memory-handle round trip through tsfile_writer_new, memory_threshold propagation, and a target whose tell() fails after the first call.
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
3 open issues
MixingPyGILState_Ensure/ReleasewithPyEval_SaveThread/PyEval_RestoreThreadis unsupported and… · Newos.fsencoderequiresosto be imported in this module at runtime. Please ensureimport osis… · New Clearing the Python exception (PyErr_Clear()) here discards the underlying failure reason (e.g.,… · New
1 resolved since last review
Review effort: Lite
- The sink guard mixed PyGILState_Ensure/Release with PyEval_SaveThread/RestoreThread. Every access already happens while the GIL is held, so the GIL alone serializes target_/position_ access; use a plain GIL guard and drop the mutex. - A failing write()/flush() no longer discards the underlying exception: its text is stored per thread and passed as the context of the raised FileSyncError, so e.g. the OSError message survives. Test asserts it.
Second review pass: - The per-thread error text is cleared at the start of every sink operation, so a previous failure cannot be attached to an unrelated one; truncate/seek failures now record their message too. - libtsfile_init() runs before the target is rewound/truncated, so a failed initialization no longer destroys the caller's file content. - The C API memory round-trip test uses a per-process unique file name so parallel CTest processes cannot collide.
Third review pass: - The bridge destructor skips close() once the interpreter is finalized, where PyGILState_Ensure is undefined. - seek_to_end() restores the previous offset when tell() cannot report the end position, so a fallback append does not start at the wrong place. - The error text is read only after the failing close/flush call, and only for writers created from a file object, so a path writer cannot inherit an unrelated sink message. - write_file_get_buffer() documents E_OVERFLOW and has a test for the non-memory (RET_NOT_SUPPORT) path.
Fourth review pass: - TsFileWriter::init(unique_ptr) deleted nothing when a writer that already owned a sink was initialized again, leaking the old sink (and its Python target); release it once the new init succeeds. - Tests: tree-model write (register_timeseries + write_row_record) to a file object, and an explicit flush() mid-write followed by close and a successful read-back.
A text stream would fail on the first bytes with an opaque TypeError; check the target's mode (when it exposes one) and raise ValueError before touching the file. Added a test with a "w"-mode file.
The new C API tests use _getpid() under _WIN32 but the file only included io.h, breaking the MSVC build (the py-msvc jobs build the C++ tests too).
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #1107 +/- ##
===========================================
+ Coverage 66.89% 67.34% +0.44%
===========================================
Files 788 788
Lines 57630 57812 +182
Branches 9310 9360 +50
===========================================
+ Hits 38551 38931 +380
+ Misses 17399 17199 -200
- Partials 1680 1682 +2 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
It forwards its target to TsFileWriter, so the new sink support already worked; update the annotation/docstring, the README and add a round-trip test through TsFileTableWriter.
There was a problem hiding this comment.
🔵 Needs a closer look
Human assessment is needed because repeated writer initialization can leak resources and retain prior state, while Python target truncation and flush behavior introduce additional data-integrity and error-reporting risks.
No open issues 🎉
3 resolved since last review
Review effort: Balanced


Problem
The C++ write stack already accepts any
RandomAccessWriteFilesink (merged in #1094), and Python can read from file-like objects — butTsFileWriter(pathname)only took a path. Writing a TsFile to an object store (for example throughfsspec/s3fs) therefore required a local temp file and a copy.Change
write_file_new_memory()/write_file_get_buffer()on top ofMemoryRandomAccessWriteFile, so a C caller can serialize into memory and upload the bytes.TsFileWriter::init(std::unique_ptr<RandomAccessWriteFile>): hand sink ownership to the writer so the Python bridge cannot leak it.free_write_file()and the writer factories now cast the opaque handle toRandomAccessWriteFile(its destructor is virtual); before, they assumedWriteFileand would have been wrong for an in-memory sink.python_random_access_write_file.{h,cc}implementsRandomAccessWriteFileon top ofwrite/tell(and optionalseek/flush/truncate);TsFileWriter(pathname)dispatches on the target type —str/bytes/PathLikekeep the old path, anything else must be a writable binary file object. Targets are reset to an empty file before writing, and a failingflush()is reported instead of ignored.python/tests/test_write_sources.py):BytesIOround trip, path-vs-object byte equality, bytes paths, thepathname=keyword, an existing buffer being reset, and flush-failure propagation.Usage
Testing
spotless:checkclean.pytest python/tests).