Lazy rows - #2666
Open
norberttech wants to merge 17 commits into
Open
Lazy rows#2666norberttech wants to merge 17 commits into
norberttech wants to merge 17 commits into
Conversation
- Rows holds a Schema plus one Column per definition; Row is a (Rows, index) view - Rows::of(Schema, Row ...) and Rows::fromColumns() replace the constructor, trusted() and conformed() - new Column / ColumnBuilder / Backend contracts with a PHP backend, Config::backend() replaces hydrator() - FloeEngine, NativeFloeEncoder, adaptive/native hydrators and extension gates removed - core consumers ported to column operations (gather, slice, project, withColumns) - a date is midnight at UTC; join collisions suggest a prefix for every join shape
- array_to_row(s) refuse keys the schema does not declare - extractors project declared columns through DeclaredColumns - every row() call moves to a declared schema - docs, upgrading entry, completers and playground phar updated
- remove Hydrator, TypedRowValues, Row\Encoder and Row\RowsBuffer; add Rows\RowsBuffer - extractors build through RowsBuilder with the context backend, encoders take Rows - split adapter encoders into decoders and encoders - Floe takes a Backend instead of a hydrator - parquet reads column chunks via readColumns() / ParquetFile::columns() - Parquet and Floe refuse withSchema(), the footer is the schema
- columnar BATCH frame (FrameEncoder/FrameDecoder, BufferLayout), codec per buffer - row format removed: PhpFloeEncoder, Encoding/Decoding, ROW frames - read batches are frame-sized, batchSize an upper bound; bucket reads coalesce - NOT NULL refusal moves to Rows::fromColumns, nested nulls refused on decode - oversized batches split in halves on OffsetOverflow/FrameTooLarge - Rows serialize as a BATCH frame; golden fixtures and docs regenerated
- DefaultBackend, NativeColumn, NativeColumnBuilder over Arrow arrays - flow-batch-frame crate for column buffers and BATCH bodies - CSV reads straight into native columns via nextColumns() - Backend gains adopt() and allocatedBytes(); pipelines build with the configured backend - RawRowValues, row hydrator and ROW-frame encoder removed - untagged builds report the next minor as -dev; Composer conflict on older extensions
- ScalarFunction evaluates a whole batch into one Column - sort, join, group-by, window, constraints and aggregates on columns - uuid/json/date-time values compare by value; one sort order - spilling steps stop at a process-wide memory limit - group-by and window stay in memory under the limit - BucketingProcessor and BucketingStrategy removed
- Rust Parquet core: row groups straight into native columns, offset/limit reads only the row groups needed - from_parquet()/to_parquet() use flow_php when loaded and no engine is given - Rows cast to the writer schema once, in the loader, for every engine - Flow float written as Parquet DOUBLE; invalid UTF-8 refused on write - PHP engine: nested columns read page by page, DELTA_BINARY_PACKED follows the spec, RLE booleans in v2 pages, required structs with null children
…-ext deprecated - ParquetEngine::openForRead() returns a ParquetFileReader, replaces readColumns() - flow_php lib surface: footer read and decoded once in Rust, PHP engine values and write acceptance - column chunks fetched once per row group instead of per page header - PhpParquetEngine reads unsigned ints and INT96 exactly, refuses values it wrote wrongly - from_parquet opens each file once and reads ENUM columns
- TextValues: one per-type text rule for CSV, JSON, XML and Text - CSV / JSON: one sink per adapter, encoder contracts with PHP and native implementations - flow_php: native CSV and JSON writers - Parquet: writeColumns() on the PHP, native and arrow writers - XML: column-wise encoding, XMLValueNodes - FloatType casts NAN, INF and -INF
- flow_php casts nested values in one walk into the arrow builders - container casts stop re-validating, identity physicals for plain nested types, bulk column builders - XML node documents are UTF-8 and read as text, string cast without re-parsing - date / numeric-string / xml_element cast fixes, NAN and INF handled without PHP coercion - just benchmark runs again, comparators registered through a PHPUnit extension
…column - element bytes carry the owner document, the element path and its markup - older element bytes still read - element columns compare by the element markup
…columns - from_json() / from_json_lines() read natively with flow_php through JsonSourceOpener - strict JSON on the native lane: malformed input refused with file and record - batch assembly shared by the CSV and JSON readers, one-value append shared with NativeColumnBuilder - a scalar JSON record throws RuntimeException instead of TypeError
…hrough the Arrow C Data Interface - Parquet core and lib surface moved from flow_php into arrow-ext, old Reader/Writer removed - ArrowParquetEngine runs on arrow-ext again, deprecation undone - flow_php ETL reader/writer exchange batches with arrow-ext via C Data, imports counted in allocatedBytes() - native ETL lane needs both extensions, flow_php alone uses the PHP engine - lib writer on typed arrow-rs builders, rows checked whole before appending
- extensions register the package interfaces; Rust classes implement them - Adaptive classes are the only extension_loaded() pick per contract - CSV, JSON and Parquet open sources move into Rust, iterating as Generators - openers, Default*, Native* and Arrow* classes removed - ext-php-rs 0.16 in both extensions, rustfmt at width 120 - tools updated, rust-toolchain action repinned
- one file read loop and one write frame for every file format, per-read state out of extractors and loaders - the configured backend reaches every batch: extractor batches adopted at the read boundary, Rows::concat/matchTo/project, functions, joins, caches, Floe - no hidden default backend: Floe readers, merger, serializer, Config and caches take it explicitly - one record decoder for CSV, Excel and Google Sheet; one JSON extractor and loader - file plumbing moved to Extractor\File, Extractor\Memory and Loader\File; Column\Php split into Physical and Layout - dead API removed: Rows::merge and ten other Rows methods, Column::builder, RowsBuffer, XMLReaderExtractor
…yground - flow_php and arrow-ext declare an ABI; a mismatched build is refused instead of failing on a missing class - local Path::uri() keeps the root slash (file:///...) - Parquet Option::BYTE_ARRAY_TO_STRING removed - WASM: ucontext shim over Emscripten fibers, so sinks running in a Fiber (to_branch()) work; undefined symbols now fail the link - published examples aligned with the columnar API; examples for the column backend, forEach() and CSV header/absent cells - docs updated for lazy-rows, new column-backend page
… its deprecated schema API - doctrine/dbal raised to ^4.5, dependencies updated - tables built through TableEditor instead of deprecated Table::addPrimaryKeyConstraint/addIndex/addUniqueIndex - Column::getType() replaced by getTypeName()
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## 1.x #2666 +/- ##
============================================
- Coverage 88.66% 86.24% -2.43%
- Complexity 0 5340 +5340
============================================
Files 2044 2203 +159
Lines 78135 91198 +13063
============================================
+ Hits 69276 78650 +9374
- Misses 8859 12548 +3689
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves: #xxx
Change Log
Added
flow-php/etl- columnarRowsbatch built by a configurable columnBackend(PhpBackend,RustBackend,AdaptiveBackend)flow-php/etl-FileReadLoop/FileBatchesandFileWriteFrame/FileSinksshared by every file formatflow-php/etl-Rows::select(),Rows::concat(),RowsJoinandColumnBuilder::appendPhysicals()flow-php/flow-php-ext- native Arrow column backend, CSV, JSON and Parquet read and write as native columnsflow-php/arrow-ext- new Parquet core exchanging batches with flow_php through the Arrow C Data Interfaceflow-php/etl-adapter-http-withBatchSize()onfrom_static_http_requests()flow-php/etl- Column Backend documentation page and new playground examplesFixed
flow-php/etl- group-by keys stay stable when a batch lacks a nullable key columnflow-php/etl- date and datetime keys at the same instant join and group togetherflow-php/etl-matchTo()reports the first bad rowflow-php/etl- html and xml elements keep their parent through a columnflow-php/etl-adapter-excel- html, html_element and xml_element cells are written instead of emptyflow-php/etl-adapter-seal- html values no longer leak internal bytesflow-php/parquet-DELTA_BINARY_PACKED, unsigned integers and INT96 follow the specflow-php/filesystem-Path::uri()of a local file keeps the root slashflow-php/etl-to_branch()runs in the WASM playgroundChanged
flow-php/etl-Rowremoved, every reader, writer, function and processor works on columnsflow-php/etl- the configured backend builds every batch, function result, join, sort, group-by and cache readflow-php/etl-Rows::empty(),matchTo(),project(),concat()andResultColumntake aBackendflow-php/etl-FloeSerializerand the caches take the serializer and backend explicitlyflow-php/etl- file plumbing moved toExtractor\FileandLoader\File, column internals toColumn\PhysicalandColumn\Layoutflow-php/etl- Floe writes one BATCH frame per batchflow-php/etl- a mismatched flow_php or arrow extension build is refused with a clear messageflow-php/etl-adapter-csv- header names, absent cells and blank leading rows follow one rule shared with Excel and Google Sheetflow-php/etl-adapter-json-from_json_lines()/to_json_lines()returnJsonExtractor/JsonLoader,*JSONEncoderrenamed*JsonEncoderflow-php/etl-adapter-excel-CellStylerreceives the written cellflow-php/etl-adapter-doctrine- markup is bound as rendered textflow-php/etl-adapter-postgresql-InsertQueryBuilder::build()takes columns, markup is bound as rendered textflow-php/parquet-ParquetEnginefollows thePhp/Rust/Adaptivepattern, arrow-ext owns ParquetRemoved
flow-php/etl-row(),InferredBatch,DataFrame::getEach(),RowsBuffer,RowEquality,DeclaredColumnsand elevenRowshelpersflow-php/etl-adapter-json-JsonLinesExtractorandJsonLinesLoaderflow-php/etl-adapter-xml-XMLReaderExtractorandXMLLoader::write()flow-php/arrow-ext-RandomAccessFileandOutputStreamflow-php/parquet-ArrowParquetEngine, thearrow()factories andOption::BYTE_ARRAY_TO_STRINGDeprecated
Security