[Spark] Evaluate the FileFormat rewrite (vortex-data/vortex#9658) against our findings #31

Activity

1. gitRasheed commented on Oct 29, 2026

   

   gitRasheed

   on Oct 29, 2026

   CollaboratorAuthor

   More actions

   Results for write lifecycle (A) and partition layout (B). Astra wrote each scenario as a JUnit test through the DataFrame API and SQL, on Spark 4.0.4 and 4.2.0 (B also 4.1.2). The tests are kept on the evaluation worktrees and will go onto rasheed/spark-fileformat as regression tests.

   vortex-data#9658 fixes:

   • fix(spark): delete a failed attempt's files and harden writer cleanup #22: a mid-task retry gives exactly-once rows, including paths with spaces and Côte. Failed partition closes and task commits leave no files.
   • [Spark] Dynamic partition overwrite deletes every partition #26: dynamic partition overwrite keeps untouched partitions, through .save() and SQL INSERT OVERWRITE.
   • [Spark] Writes land at their final paths before the job commits #27: a concurrent reader sees only committed data, and a failed job or a failed job commit leaves no files and no _SUCCESS.
   • [Spark] Appends from separate applications overwrite each other's files #30: appends from two applications keep both runs.
   • fix(spark): skip hidden files when listing a Vortex table directory #20: _SUCCESS, .crc, _temporary/0/..., .hidden and _metadata are ignored.
   • [Spark] Writer, listFiles and delete disagree on URL encoding #21 and fix(spark): name partition directories the way Spark does #24: partition directory names are Spark's for every special value tested, for bare paths and URIs, and they read back correctly. Parquet and Vortex read each other's layouts.
   • [Spark] Date and timestamp partitions are written as epoch counts #28: new date and whole-second timestamp partitions get readable directories, the right inferred types, and pruning.

   Still broken, and the same with Parquet (Spark's FileOutputCommitter, not Vortex):

   • A driver killed mid-job leaves its committed tasks under _temporary/0, and the next append publishes them: 257 rows instead of 129.
   • A job commit that fails halfway leaves the partitions it already published.
   • With committer algorithm 2, a failed job keeps the files of tasks that finished.
   • Timestamps with six-digit fractions infer as strings; reading with an explicit schema works.
   • A percent-encoded URI root (file:///tmp/a%20b) is taken literally, as Spark does for Parquet.

   Only matters for data written by the current connector: old p=with+space, p=C%C3%B4te and epoch-count date partitions read differently. Nobody has production Vortex tables yet, so I don't think this needs a migration path.

   Conclusion: for #20, #21, #22, #26, #27, #28 and #30, fixing the current connector is wasted once vortex-data#9658 lands. Open questions now are reads (C), write performance (D) and configuration/ops (E), which are running.
2. gitRasheed commented on Oct 29, 2026

   

   gitRasheed

   on Oct 29, 2026

   CollaboratorAuthor

   More actions

   Results for reads (C) and configuration/ops (E). C ran on Spark 4.0.4; E ran on 3.5.9 (Scala 2.12 and 2.13), 4.0.4, 4.1.2 and 4.2.0 with the same results on each.

   Fixed by vortex-data#9658: count(*) reads only footers (#19); float filters with NaN and signed zero match Spark; Kryo with registrationRequired works (#25); mixed-case options are recognized; catalog SQL (CREATE TABLE, INSERT, INSERT OVERWRITE, saveAsTable, CTAS) works; S3 works once hadoop-aws is on the classpath.

   Fork features vortex-data#9658 drops:

   • Nested schema pruning ([Spark] Push nested schema pruning down to the native scan #9/feat(spark): push nested struct pruning down to the native scan #10): reading s.a or s.t.x.q reads all of s, 10.8 MB either way. VortexScanBuilder leaves nested pruning off, and VortexPartitionReader projects top-level names only.
   • File splitting ([Spark] Distribute scans of a large Vortex file across Spark input partitions #2/perf(spark): optional row-range splitting of large Vortex files #8): isSplitable returns false, so two 3.6 MB files with maxPartitionBytes=1 MiB give 2 partitions, and vortex.read.splitFiles is ignored.
   • String views ([Spark] Avoid per-value byte-array copies when reading Arrow UTF8 views #4/perf(spark): read Utf8View values in place instead of copying each one #6): StringViewAccessor still copies every value (UTF8String.fromBytes).
   • Scan options overriding table options in another case (perf(spark): optional row-range splitting of large Vortex files #8): VortexTable.scala:85 merges case-sensitive maps first, so the winner depends on iteration order.

   Still broken:

   • Struct fields inside arrays (spark: prune struct fields inside arrays (blocked on list layout) #16) read whole elements, as before. Still blocked on the list layout.
   • Timestamps at year 9999 panic on write, and timestamp literals fail as predicates. fix(array): accept timestamps beyond jiff's range vortex-data/vortex#10953 fixes the native side.
   • The native over-read below nullable structs (perf(layout): read a path below a nullable struct's field without the rest of the field vortex-data/vortex#10952) still applies once nested pruning is back.
   • vortex.* settings in the Spark session conf are ignored.
   • Malformed batch.size or session-provider values fail on executors after overwrite has deleted the old files; out-of-range batch sizes fall back to the default silently.
   • The S3 mock tests fail: no hadoop-aws on the test classpath. Deployments need it too, and that should be documented.
   • The docs list Spark 4.0 and 4.1 only; 4.2 passes the same tests.

   So vortex-data#9658 is the right base, but it loses the three read optimizations the fork added. Next step: port them, plus the fixes above, as PRs into rasheed/spark-fileformat.
3. gitRasheed commented on Oct 29, 2026

   

   gitRasheed

   on Oct 29, 2026

   CollaboratorAuthor

   More actions

   Results for write performance and resources (D), Spark 4.0.4, local[4], 4M rows, median of three rounds (shared machine, so directional).

   Shape	Old connector, 4 tasks / 1 task	vortex-data#9658, 4 tasks / 1 task
   6 columns, 4 workers	11.3 s / 17.4 s	11.9 s / 21.5 s
   nested, 4 workers	14.5 s / 20.7 s	15.1 s / 22.5 s
   100 columns, 4 workers	130.9 s / 154.6 s	123.2 s / 158.1 s

   Fixed by vortex-data#9658: writes configure the native worker pool (default 4), so #23 isn't needed there.

   Regressed:

   • Row copies are back: VortexOutputWriter calls row.copy() into a list and converts per value, which perf(spark): append rows straight into the Arrow batch #7 removed on the fork.
   • Every batch exports the Arrow schema and never releases it: 1,728 bytes after a five-batch write, and 2,400 leaked allocators (4.2 MB) across three jobs of small partitioned writers. The allocator-close failure is swallowed (VortexOutputWriter.java:372-376). Repeated 100-column writes ran out of a 1 GiB heap in round two, where the old connector finished all three; the leak is the likely cause, not yet confirmed.
   • The JVM doesn't exit after a default write: Vortex spark integration is a fileformat not a tableprovider vortex-data/vortex#9658 writes through Java streams (HadoopWritable), so native workers make Java upcalls and attach as non-daemon threads ([JNI] Native workers attach as non-daemon JVM threads #29). main returned after spark.stop(), and the JVM was still alive at 45 seconds. With zero workers it exits.

   Still broken:

   • The schema export leak in VortexWriter.build() (fix(jni): release the schema export in VortexWriter.build() #18), 288 bytes per writer; fixed by fix: native fixes the FileFormat connector needs (nested over-read, timestamp range, schema export leak) #34.
   • After one batch with a 16 MiB string, later batches keep that capacity (same on both connectors).

   Next on rasheed/spark-fileformat: port #7's direct Arrow writers with a single schema export per writer, and fix #29.