Repository navigation
Spark FileFormat connector (upstream PR 9658), integration branch - #35
Draft
gitRasheed wants to merge 144 commits into
Draft
gitRasheed wants to merge 144 commits into
gitRasheed wants to merge 144 commits into
Conversation
…vortex-data#10414) Previously any Polars predicate that polars_to_vortex could not convert raised NotImplementedError inside _io_source, failing the whole collect(). Instead, decompose the predicate into its top-level AND conjuncts, push the convertible conjuncts into the Vortex scan, and evaluate the residual conjuncts with Polars on each yielded batch. Since Polars does not re-apply a predicate handed to an IO source, both parts are always applied, and a row limit is only pushed into the scan when there is no residual filter (otherwise it is enforced after filtering each batch).
…ortex-data#10920) Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
…vortex-data#10896) `Binary::reduce` short-circuits `AND`/`OR` when one operand is a constant, but it ignored the nullability of the result: - `x AND false` and `x OR true` became a constant built with `bool::into()`, which is always non-nullable. - `x AND true` and `x OR false` became `x` as is, even when `x` is non-nullable and the constant is nullable. In both cases the binary expression is `bool?` while its rewrite is `bool`.
## Summary Upgrade fsst-rs from 0.6.0 to 0.7.0 and onpair from 0.2.1 to 0.3.0. FSST compression now uses the new bulk API. On ClickBench string columns, it is 1.5–1.85x faster. ## Changes - **FSST compression (`encodings/fsst/src/compress.rs`)**: replace the per-row `compress_into` loop with batched `Compressor::compress_bulk_into::<LANES>`. The library gives the same output for each row as `compress`. - Rows go into batches of 256 KiB or 16 Ki rows. Each batch is compressed into a reused scratch buffer, then copied into buffers from the execution context's allocator. - The codes `VarBin` is built directly from these buffers and the input mask. This removes `VarBinBuilder` and the per-row validity bits. - Lanes: 8 on Arm, measured best on Apple Silicon (2, 4, 6, 10, 12 and 16 are slower). x86-64 keeps the upstream default of 4, which is not measured. - **onpair (`encodings/onpair/src/array.rs`)**: 0.3.0 makes `DictionaryStorage` an `unsafe trait`. `OnPairDictionaryStorage` is now an `unsafe impl`. Its buffers are immutable and reference-counted, so the contract holds. - **Golden snapshot**: in `string_fsst_structured`, the symbol count changes from 223 to 216 and the size from 151,382 to 151,313 bytes. fsst-rs 0.7.0 breaks training ties on symbol content, which makes training deterministic. --------- Signed-off-by: Robert Kruszewski <github@robertk.io>
Always fuzz core, preview and unstable editions Signed-off-by: Robert Kruszewski <github@robertk.io>
…0925) Signed-off-by: Claude <noreply@anthropic.com>
- Support building and linking vortex-duckdb from Windows. - Run vortex-sqllogictest and vortex-duckdb tests in CI for Windows-x64. - Use url.to_file_path() instead of url.path() which produces a leading slash (/C:/...) on Windows. - Remove -fno-gnu-unique since it's obsolete for duckdb 2.0's C++17. - Fix sqllogic tests error messages to be working for Windows - Replace back slashes with forward slashes in sqllogic test runner as duckdb produces paths with forward slashes in output. - Fix spatial tests concurrently loading spatial extension, run tests sequentially --------- Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
We have fast in place decompression path, pass the array by value and use them. --------- Signed-off-by: Robert Kruszewski <github@robertk.io>
…ex-data#10609) ## Summary - Tracking Issue: vortex-data#10605 This is the first step towards Elias-Fano: the `Unary` encoding, which will hold the bucket counts of EF's upper bits. It is deliberately naive. There is no index, no slice kernel and no compute kernel, so the PR is mostly boilerplate, and the benchmarks set the baselines that later work is measured against. It is in no file edition and `vortex-file` does not register it, so the default writers reject it. , so the bits can stay on a device Next up is slicing for `Unary`, then the naive `EliasFano` parent with `upper` and `lower` children. ## Changes **Layout.** One bit buffer, stored the way Bool stores its bits: a `BufferHandle` with a `BitBufferMeta` that holds the bit offset and length. Each value `v` is written as `v` set bits, then one unset bit that closes it. The array takes `len + num_ones` bits, where `num_ones` is the sum of the values. The value at index `k` is the number of set bits between the `k`-th unset bit and the one before it. For `[2, 0, 3]`, the bits are `110 0 1110`. The metadata holds two fields. `num_ones` gives the exact bit length together with the array's length. The bit offset, below 8, lets a slice share its parent's buffer. The dtype must be a non-nullable unsigned integer, and the input need not be sorted. Equality and hashing compare the buffer and the bit offset, as Bool's do. **Validation.** `validate` checks the dtype and the bit length. When the buffer is on the host, it also checks three things about the bits: - they end with an unset bit; - they hold exactly `num_ones` set bits; - each value fits the dtype. This check runs only when the values could sum past the type's maximum. Bits on a device can't be read there, so they aren't checked, as in VarBin and FSST. **As EF's upper child.** `Unary` will hold one count per bucket, so its length is the number of buckets, not the parent's length. EF will read those bits through an extension trait, the way FoR reads BitPacked. --------- Signed-off-by: rapour <reza.apour@gmail.com>
Follow-up to vortex-data#10920, which added five Hugging Face inputs to `string-bench`. CI's runner script, `scripts/string-split.py`, launches one `string-bench` process per input name from a hard-coded list. That list still only had `URL` and `l_comment`, so CI never benchmarked the new inputs. --------- Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
…10897) ## Summary - Tracking Issue: vortex-data#10701 Every decimal precision now has one canonical storage width, `DecimalDType::canonical_values_type()`, and the Arrow and DuckDB imports produce arrays at that width. Before this, the Arrow import kept whatever width Arrow used (a `Decimal128` column at precision 10 arrived as `i128`), and the DuckDB import narrowed precision 1 and 2 with a lossy `as` cast. This doesn't make the canonical width an invariant yet. `DecimalArray` still accepts any storage width, and every kernel, writer and exporter already handles mixed widths, so only the two imports change behavior. Enforcing the width in `try_new` comes later in vortex-data#10701, once DecimalByteParts decoding, the read path for old files and the compressor stop producing other widths. ## Changes `canonical_values_type()` replaces `DecimalType::smallest_decimal_value_type` with the same precision-to-width table, so the migrated call sites behave as before. Two shared helpers in `vortex_array::arrays::decimal` do the copies that the imports need now and that the later steps reuse. `copy_to_canonical_width` copies values into the canonical width, and a narrowing copy fails on a valid value outside the precision and writes zeros in null slots. `check_precision` checks values in place. The Arrow import stays zero-copy when the Arrow width already matches and copies otherwise. Either way, every valid value is checked against the precision, so a value outside it now fails at import instead of reaching `scalar_at` or `min_max`, which can panic on it (vortex-data#10749). The error points at arrow-rs `PrimitiveArray::null_if_overflow_precision` for callers that would rather get nulls. The dtype is built with `DecimalDType::try_new`, so an invalid precision or scale from FFI is an error rather than a panic. The DuckDB import goes through the same helpers. This overlaps with vortex-data#10305, which touches many of the same files. For review, read the commits one at a time. ## API Changes `DecimalType::smallest_decimal_value_type` is removed in favor of `DecimalDType::canonical_values_type()`. Arrow and DuckDB decimal imports return arrays at the canonical width and fail on a valid value outside the declared precision. --------- Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
## Summary
`vx.scalar(decimal.Decimal("9999999999999999999"))` fails with an
overflow error, and so does a JNI decimal literal at precision 19 with a
value above `i64::MAX`. Both factories picked the value width from
`DecimalDType::required_bit_width()`, which counts magnitude bits
without a sign bit. At precision 19 that is 64, so they chose `i64`,
which can't hold the largest 19-digit values.
## Changes
Both factories now match on `DecimalDType::canonical_values_type()` from
vortex-data#10897, which gives `i128` at precision 19. Every other precision keeps
its width, because the sign bit still fits there. The Python precision
inference test and a new JNI unit test cover the largest and smallest
values at precision 19.
This is stacked on vortex-data#10897 and retargets to `develop` after that merges.
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Allow serializing filters, projections, and aggregates in table function state. This allows duckdb to merge identical plans into one --------- Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
…ta#10622) The filter of bit-packed, sequence and run-end arrays, run-end decoding and constant byte views allocated their output buffers from the global allocator. Allocate them through the allocator of the execution context instead, so a session can choose where decoded buffers come from. Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
vortex-data#10891) Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
…10911) The new `deploy-vercel` workflow deploys a built static site to Vercel. `web.yml` uses it to deploy explore to the new `vortex-explorer` project. The Cloudflare Pages deploy continues, so a DNS change can roll back the move. The workflow does not build. It downloads the site artifact, adds a Build Output API `config.json`, and runs `vercel deploy --prebuilt`. The token is scoped to one Vercel project and is in the `vercel-explorer` environment. A lockfile pins the Vercel CLI to 64.0.2, and the install runs with `--ignore-scripts`. Pushes to develop deploy to production. Same-repo PRs get a preview. Fork PRs do not deploy. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Will Manning <will@willmanning.io> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ortex-data#10241) `StatsSetRef::transfer_from` copies the entries under one read lock on the source and one write lock on the target, returns early when the source is empty (the common case for freshly decoded children) or when the source is the target. `finalize_done` and the single-step `DynArrayData::execute` use it.
…0143) The gather indexed the byte buffer with a bounds check per element and did not exempt null indices, so a garbage index under a null bit could panic. It now runs `try_map_into` or `try_map_masked_into`. --------- Signed-off-by: Robert Kruszewski <github@robertk.io>
This PR deploys the docs to the new `vortex-docs` Vercel project. The deploy uses the `deploy-vercel` workflow from vortex-data#10911, so merge that PR first. The Cloudflare Pages deploy continues, so a DNS change can roll back the move. The Sphinx links are relative. Thus the docs need the Pages URL rules: `page.html` is at `/page` and `dir/index.html` is at `/dir/`. The new `pages-html-routes.py` script makes the same routes from each build. `docs/vercel-output-config.json` copies the headers from `docs/_headers`. I compared 598 URL forms against docs.vortex.dev. 597 are the same. An unknown path now returns 404, not the home page. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Will Manning <will@willmanning.io> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
- `in_list_of` removes equal elements after it sorts them. One null element stays, so the SQL semantics do not change. - The DataFusion converter converts equal literals once. - For a predicate, the converter reads the type and nullability, and does not ask for the return field. DataFusion names the return field by formatting the whole expression. For a large IN list, the formatting costs more than the conversion. - `contiguous_integer_bounds` checks the dtype before it views a scalar as a primitive, because a failed view builds an error. Signed-off-by: Robert Kruszewski <github@robertk.io>
…rtex-data#10913) Docs and explore are on Vercel now. This PR removes the Cloudflare Pages deploy jobs from `web.yml`, `docs.yml` and `docs-preview.yml`. It deletes `docs/_headers`, because `docs/vercel-output-config.json` has the same headers. It also updates the preview text in CONTRIBUTING and a stale Pages comment in `Cargo.toml`. It also updates comments that use the old benchmarks-website paths, because bench moves to vortex.dev in vortex-data/vortex.dev#112. Do not merge until these are done: 1. vortex-data#10912 merges and deploys docs to Vercel. 2. docs.vortex.dev points at Vercel. 3. vortex-data/vortex.dev#112 merges. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Will Manning <will@willmanning.io> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…#10935) ## Summary `scripts/capture-file-sizes.py` measured every directory under `vortex-bench/data` whose name started with the benchmark's (`clickbench` matched `clickbench_partitioned`, and anything else named `clickbench_*`), so data sitting next to a benchmark's own could leak into its file-size report. The script now takes `--dataset-dir` and measures exactly that directory. ## Changes Each CI matrix entry carries a `dataset_dir`, which defaults to the benchmark's subcommand; ClickBench sets `clickbench_partitioned`. The workflow passes it through. For every benchmark the CI captures today, the script's output is byte-identical to before. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Martin Prammer <martin@spiraldb.com> Co-authored-by: Claude <noreply@anthropic.com>
## Summary Adds `--opt files=N` to the partitioned ClickBench benchmark, which runs the first N of its 100 files (about 1M rows each) from a data directory of its own, such as `clickbench_partitioned_10`. This gives ClickBench sizes to compare across, the way TPC-H has scale factors, and a small run for quick local checks. Running a subset today fails, because the hard-coded per-query row counts hold only for the full dataset: on one file, DataFusion panics at Q7 (expected 18 rows, got 5) and the run ends there. ## Changes Row counts are checked only at 100 files, and the default stays at 100, so existing runs, their data paths and their result identities are unchanged. A subset's results carry its file count, as `files` on the `clickbench` dataset in JSON output and as `dataset_variant` `files=N` in v3 records, so they never compare against full runs. `ClickBenchBenchmark::new` and `Flavor::download` take the file count; `files` with the single-file flavor is an error. Checked locally: `data-gen clickbench --opt files=1` downloads one file and converts it, and `datafusion-bench clickbench --opt files=1 --formats parquet,vortex` runs all 43 queries on both formats. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Martin Prammer <martin@spiraldb.com> Co-authored-by: Claude <noreply@anthropic.com>
## Summary - Tracking Issue: vortex-data#10701 Decimal `Widen`, legacy decimal reads and DecimalByteParts splitting need raw extrema before choosing or checking precision. The existing decimal min/max implementation constructs checked scalars, so callers cannot use it directly for values outside the declared precision. ## Changes Expose the raw range from the existing pairwise scanner in `aggregate_fn/fns/min_max/decimal.rs`. Both the aggregate and new raw-range callers use that implementation. Scalar construction stays in the aggregate layer, and the raw helper ignores null slots without allocating or checking precision. Tests cover all six storage widths, native extrema, null payloads, empty and all-null inputs, and aggregate result dtypes. Benchmarks add wide decimal cases to the existing min/max target. Checks are left to CI. ## API Changes Add public `vortex_array::arrays::decimal::valid_range`, re-exported from the existing decimal min/max implementation. --------- Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
…10951) This PR fixes four small problems on the docs and explore sites: - explore had no security headers. It now sends a CSP. The CSP allows the wasm module, Google Fonts, the worker, and the https URLs that users open. - On preview deployments, the Vercel toolbar was blocked. Previews now allow its sources. The production domains keep the strict policy. - Docs search did not finish. The theme now marks its main content with `role="main"`, which Sphinx search requires. - Javadoc pages requested a font file that JDK 17 does not make. The docs build now removes that request. I tested each change on preview deploys in a browser. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Will Manning <will@willmanning.io> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 29, 2026
A stable write stores each list chunk as one segment, so reading one field of the structs in an array reads all of them. The list layout stores offsets, validity and elements apart and splits struct elements into one column per field, but only the unstable edition permits it. `vortex.listLayout` (default false) writes it, through `VortexWriter.Builder.listLayout`. Enabled editions are shared by every clone of a session, so enabling the unstable edition on the JVM-wide session would change every later write. The JNI writer instead writes through a session of its own with that edition enabled. The guide says these files may hold other unstable encodings and that future versions may not read them. Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
Reading `arr.a` from an `array<struct<...>>` read the whole array: the scan builder restored each array's full table type, and the reader projected only through structs. Pruning now recurses through arrays. Each array whose elements are pruned is read through `list_transform(arr, e -> ...)`, whose body keeps the element fields Spark reads in the same way a struct is pruned: `select` when only leaves are dropped, `pack` when a child is pruned too, masked with the element's own nulls when the elements are nullable. Fields resolve against each file's schema as before, which now keeps array shapes as well as struct shapes, and a nullable element field the file lacks reads as null. A file in the list layout then reads only the offsets, validity and element columns the query uses. A stable file still reads the whole array. Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
A list-layout write ran in a new default session, so it lost every encoding, layout or function registered on the session the caller supplied. It now runs in a fork of that session. `VortexSession::fork` copies the session's variables into a store of its own, so a variable registered on the fork is not seen by the original. The enabled editions are then replaced with a new set holding the session's editions plus the unstable one; the copied set would still share its contents with the original. Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
Spark prunes fields inside arrays only on the V2 path, and keeps whole arrays after `inline` or `transform` and the inner arrays' fields after `explode`. Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
`NullFilledArrayVector.getArray` read the length of the wrapped column's array without checking the row, so a null array threw a NullPointerException where Spark's `ColumnVector` contract returns null. Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
A supplied schema, or vortex.mergeSchema=false, can ask for a column in a type other than the one a file stores. The reader wraps the file's vectors without comparing the two, so a BIGINT read as TIMESTAMP comes back as that many microseconds instead of failing, and so does a struct, array or map leaf. Each case writes a column of one type to Parquet and Vortex and reads it back as another, through the V2 and V1 paths. Where Parquet fails, Vortex must fail and name the file, the column and the type read; where both return rows, the rows must match. A case may let Vortex refuse a widening Parquet makes, since Parquet's rules differ between Spark 3.5 and 4. Vortex-only tests read files that differ only in ways the reader already normalizes (nullability, missing nullable fields, Utf8 and Utf8View, timestamp units and zones, decimal precision) with filters on each, check that mergeSchema=false fails on a file of another type, and check that a millisecond timestamp is not read as BIGINT, since Parquet would return milliseconds and the vector returns microseconds. Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
Before scanning a file, the reader now compares each column it returns, down to struct fields and array and map elements, with the type the file stores. Filter columns are covered too, since Spark reads them to apply the filter. A mismatch fails with the file, the column path and both types. The conversions allowed are the ones Spark's Parquet source makes on the running Spark version that the column vectors already return correctly: dates and ints as each other, microsecond timestamps as BIGINT, timestamps without a time zone as timestamps with one and, on Spark 4, the reverse, and decimals that keep every digit, with scale widening only on Spark 4. Spark 3.5's Parquet reader also refuses some same-scale precision widenings, such as DECIMAL(9,2) to DECIMAL(10,2), because it stores the two in different physical types. Vortex reads those correctly and allows them. Parquet also widens integers and floats, reads dates as TIMESTAMP_NTZ, and reads binary and strings as each other. The vectors convert none of these, so the reader refuses them instead of returning wrong values. Filters are now pushed only on columns stored in the type the filter literal is built in. A timestamp filter on a file storing another unit or time zone failed in the native scan. Filters on a column read through one of the conversions above, such as a wider decimal, are left to Spark for the same reason. Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
The type check skips structs inside arrays and maps, so an array or map of structs whose BIGINT field is read as TIMESTAMP returns that many microseconds where Parquet fails. A map whose struct fields are read in another order, or with a field the file lacks, fails with an UnsupportedOperationException or ArrayIndexOutOfBoundsException from the column vectors instead of a type error, since they return a map's structs by position. Add V1 and V2 cases compared with Parquet for a struct field in an array, in a map value and in an array inside a map value, and for map struct fields in another order or missing from the file. An array of structs read with its fields in another order must match Parquet, since the reader resolves those by name. Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
Struct fields are now checked the way the reader resolves them. Inside arrays they are matched by name, as the projection prunes and reorders them there. Inside a map, where the vectors return keys and values as the file stores them, each field read must be the file's field at the same position under the same name, so a reordered or added field fails with the file, the column path and both types instead of returning the wrong field or failing in the vector. File fields of wide structs are looked up through a map rather than a scan per field read. Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
The reader matched the whole table schema against every file it opened to decide which filters to push, even with no filters. It now skips this without filters and otherwise follows only the column paths the filters reference. Null checks are pushed again on a column stored in another type than filter literals are built in, such as a millisecond timestamp, since they compare no literal. Such a column is given NullType, which no literal is built for, so comparisons on it stay in Spark. Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
VortexOutputWriter copied every row into a list and converted the list to Arrow per value when the batch was full. Each row is now appended straight into the Arrow vectors through one typed field writer per column, so strings are copied once and decimals of up to 18 digits are written from the unscaled long. Vectors start at 64 slots and grow, so 1-row batches no longer allocate Arrow's default capacity. The schema is exported once per writer and released on close; each batch used to export it again and never release it. A write that fails part-way through a row marks the writer failed: later writes throw, and close releases everything and throws instead of finishing the misaligned batch. Memory still allocated when the allocator closes now fails close instead of being logged at debug. Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
…ailing The previous commit made close fail when the writer's allocator still held memory. Native code can release a batch after close has finished the file, and failing then would fail a task whose output is complete. Close now logs a warning and leaves the allocator open, so a late release is still freed: Arrow marks an allocator closed even when its close throws, and with assertions on, releasing into it throws and keeps the memory. Tests still assert that nothing is left, through the allocator() accessor. Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
Each batch used to export the schema again and never release it. The writer now exports it once and releases it on close, so the test passes; it reads the allocator through the package-private accessor instead of reflection. Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
The native writer holds one-row batches until it has enough bytes to write; with 10,000 rows that was every batch, about 1.2M Arrow buffer ledgers, and the 512 MB test JVM ran out of heap. 2,000 rows still cover every null pattern, and a batch size of 300 still crosses batch boundaries and grows the vectors past their first 64 slots. Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
Spark's concurrent partition writer closes a task's writers one after another when the task aborts and stops at the first close that throws. close() rethrew an earlier write failure, so the writers after the failed one were never closed and kept their Arrow memory, streams and native writers. After a write failure, close() now releases everything and returns, logging any cleanup failure. The truncated file still can't be committed: Spark commits only after every write returned, and write() refuses every row once one has failed. Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
The compat suites compiled only test/java, so the Scala format the abort test writes through was missing and the write failed to find its data source before opening any writer. With the format present, Spark 4.2 hung instead: Arrow's out-of-memory exception doesn't serialize, and the SerializationDebugger Spark falls back to needs java.io opened, which Spark's launcher does but the test JVM didn't. Its failed initialization killed the executor thread before it reported the task failure. The test now reports the exception it caught when no writer opened. Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
Check IN and NOT IN on strings, ints, longs, dates, timestamps and decimals, with nulls, duplicates, empty lists and lists of more than 16 values, through the V1 and V2 read paths. Add an opt-in benchmark of IN over rows sorted by a 31-character key, run with -Dvortex.benchmark=true, which the Spark test tasks now forward to the test JVM. Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
An IN filter became an OR of one equality per value, and each value cost a pass over the column. With 1000 keys spread over 4M rows sorted by a 31-character key, the pushed filter took 67 s; Spark evaluating it itself took 0.7 s. Up to 16 values the OR stays: it is cheap at that size and prunes zones value by value. Beyond that, push min <= col <= max AND list_contains over a list literal, which probes one hashed set per row. The bounds are spelled out because the scan reduces a list literal to a prepared set before zone pruning, and the set's stats rewrite only reads a literal, so list_contains alone prunes nothing. On the benchmark, pushed IN of 1000 keys spread over the table goes from 67.4 s to 1.3 s, 1000 keys from one stretch from 2.8 s to 0.12 s, and 10 keys stay at 0.13 s (0.17 s before). Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
Catalog tables (saveAsTable, CREATE TABLE ... USING vortex, bucketBy) are read through the V1 HadoopFsRelation and VortexFileFormat. Spark's SchemaPruning prunes nested fields there only for Parquet and ORC, so these tables read every field of a struct or array of structs even when the query uses one. VortexSchemaPruning applies the same rewrite to V1 Vortex relations, just before Spark's rule, which still prunes _metadata. It keeps the field ordinals consistent with what the reader returns: struct fields are pruned, also inside arrays, while maps keep their table type, as on the V2 path. Bucket and sort columns keep their table type. VortexSparkSessionExtensions registers the rule. Spark loads it through ServiceLoader, and it can also be listed in spark.sql.extensions; it registers the rule once per SparkSessionExtensions either way. Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
Spark 3.5 loads SparkSessionExtensionsProvider classes only in SparkSession.Builder.getOrCreate. The SparkSession(SparkContext, options) constructor applies spark.sql.extensions alone, and PySpark 3.5 builds every session with it unless a JVM session already exists. Those sessions never get VortexSchemaPruning, so V1 Vortex reads keep whole structs. Spark 4.0 loads providers in that constructor too. A connector cannot register the rule after the session exists: the only hooks are spark.experimental.extraOptimizations, which runs after Spark's own pruning, or the session's internal extensions. So the user guide now says to set spark.sql.extensions on PySpark 3.5, and VortexPySparkSessionTest builds sessions through the PySpark constructor to check which versions register the rule. Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
The footer cache weighed each entry by its footer and layout flatbuffer bytes only. A parsed footer also holds its dtype flatbuffer, the dtype and layout trees a scan materializes, the segment specs and the file statistics, so a wide footer of about 2,400 leaves weighed 495 KB while holding 8.5 MB, and a 100 MB cache grew to about 2.9 GB. Each footer now carries an estimate of what it holds, computed once when it is parsed. DType::approx_resident_bytes walks the serialized dtype without parsing fields, so opening a file still resolves only the fields it reads, and charges supplied and shared dtypes once per field slot and backing buffer. MultiFileSession::new takes the cache limit in bytes, and footers under 1 KiB weigh at least 1 KiB. Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
The list layout test compared bytes read for `SELECT id, arr.a` against `SELECT id, arr`. `id` sits at offset 0 and the array's offsets about 976 KB in, within the 1 MiB coalescing distance Java readables use. When the read driver saw both requests at once it merged them into one read that also pulled in all of `s` between them, so `arr.a` read 1,972,664 bytes instead of 18,300 on a loaded full-suite run. Pruning itself was intact. The measured queries now leave `id` out. The array's offsets and the `a` elements are 1.9 MB apart, beyond the coalescing distance, so the count no longer depends on timing. Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
gitRasheed
force-pushed
the
rasheed/spark-fileformat
branch
from
October 31, 2026 02:04
241b2b6 to
ca6c961
Compare
Signed-off-by: Christopher Wirt <cwirt@quantcast.com> (cherry picked from commit 4bb52b0; the old connector's VortexDataWriter change is ported separately)
VortexOutputWriter now hands each batch to VortexWriter.writeBatch(root, allocator), so the native writer retains each Java allocation on its own and frees columns it is done with before the file finishes. The writer no longer exports the schema itself. Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
The cherry-picked test called the create overload without the listLayout argument, which this branch added. Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
Writes 64 batches with a nullable struct column and checks that fewer than half of them are still allocated before close. Before the writer retained Java buffers independently, every batch stayed allocated until the file finished. Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
The writer pipeline holds up to about 25 batches of this schema at once, which left the 64-batch version a thin margin under its half-the-batches limit. With 128 batches the limit is 64, against the 18 to 24 batches measured at peak. Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
Each batch handed to the native writer costs a fixed amount on top of its rows: exporting the Arrow tree, importing and converting it again on the native side, and allocating its buffers. On a wide nested schema (about 2,400 leaves, mostly null) that is ~32 ms per batch. On that schema at 10% density, with the default four native workers, writer CPU for 100k rows went from 15.0 to 10.0 s (median of three). The writer holds one batch at a time, so the cost is one larger batch: peak direct memory per writer went from 120-124 to 184-188 MiB, and from 280-284 to 328-360 MiB fully dense. Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
A null struct used to write a null into every vector below it, which on a wide, mostly null nested schema was most of the Java-side write time. Now a null slot is left as allocated, and a struct catches its children's counts up to its own slot when it next gets a value. This relies on the writer allocating fresh vectors for every batch. In Arrow 19.0.0, allocateNew zeroes the validity and offset buffers of fixed-width, variable-width, list, map and struct vectors, every reallocation zeroes the part it adds, and clear() resets lastSet. The offsets of skipped slots are filled by the next set on a variable-width or list vector, or by setValueCount, which a struct passes to its children and a list to its elements. On the same schema and batch size, writer CPU for 100k rows went from 10.0 to 6.6 s, and task-thread CPU from 6.2 to 3.3 s (median of three). The round-trip tests now cover null structs holding decimals, lists, maps and nested structs, nulls in runs across batch boundaries, and the Parquet parity case in small batches. Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Our working Spark FileFormat connector: upstream PR 9658 (Robert Kruszewski's FileFormat rewrite), plus our fixes as one commit each, so each can go upstream as its own PR once 9658 merges. It's a draft and won't be merged as is.
As of 2026-10-31 the branch sits on 9658's latest head, which is on a develop that includes upstream's nested struct pruning (upstream PR 10957) and the nullable-columns change. Earlier pieces merged here as #33, #34, #36, #37 and #39 to #45 were re-applied on that head. One difference from develop: develop now rejects timestamps outside jiff's range, and this branch keeps accepting every i64 timestamp, as Arrow, Parquet and Spark do; three of develop's tests are inverted to match.
Changes on top of 9658
VortexWriter.build()releases its schema export;list_transformexpressions.vortex.listLayout=truefiles); nested pruning for catalog and bucketed tables; largeINlists pushed as a set lookup; large files split into row ranges;Utf8Viewread in place.vortex.*session defaults, driver-side validation.Testing
Full Gradle
checkpasses on Spark 3.5/2.12, 3.5/2.13, 4.0, and the 4.1 and 4.2 compat runs (about 480 tests per suite). One bytes-read assertion in the catalog-pruning tests failed once in five runs on 3.5/2.13; results were correct in every run; under investigation.I used AI tools to help write and review this change.