Fuzzing Crash: VortexError in array_ops

[github-actions]

Fuzzing Crash Report

Analysis

Crash Location: vortex-compressor/src/builtins/dict/float.rs:140:dictionary_encode

Error Message:

Assertion failed error: this must be present since `DictScheme` declared that we need distinct values

Stack Trace

stack backtrace:
   0: __rustc::rust_begin_unwind
             at /rustc/db3e99bbab28c6ca778b13222becdea54533d908/library/std/src/panicking.rs:689:5
   1: core::panicking::panic_fmt
             at /rustc/db3e99bbab28c6ca778b13222becdea54533d908/library/core/src/panicking.rs:80:14
   2: panic_display<vortex_error::VortexError>
             at /rustc/db3e99bbab28c6ca778b13222becdea54533d908/library/core/src/panicking.rs:259:5
   3: {closure#0}<&vortex_compressor::stats::float::DistinctInfo<f64>>
             at ./vortex-error/src/lib.rs:500:9
   4: unwrap_or_else<&vortex_compressor::stats::float::DistinctInfo<f64>, vortex_error::{impl#12}::vortex_expect::{closure_env#0}<&vortex_compressor::stats::float::DistinctInfo<f64>>>
             at /rustc/db3e99bbab28c6ca778b13222becdea54533d908/library/core/src/option.rs:1067:21
   5: vortex_expect<&vortex_compressor::stats::float::DistinctInfo<f64>>
             at ./vortex-error/src/lib.rs:349:14
   6: dictionary_encode
             at ./vortex-compressor/src/builtins/dict/float.rs:140:42
   7: compress
             at ./vortex-compressor/src/builtins/dict/float.rs:110:20
   8: estimate_compression_ratio_with_sampling<dyn vortex_compressor::scheme::Scheme>
             at ./vortex-compressor/src/estimate.rs:122:10
   9: choose_best_scheme
             at ./vortex-compressor/src/compressor.rs:359:40
  10: choose_and_compress
             at ./vortex-compressor/src/compressor.rs:314:36
  11: compress_canonical
             at ./vortex-compressor/src/compressor.rs:187:22
  12: compress
             at ./vortex-compressor/src/compressor.rs:136:14
  13: compress_canonical
             at ./vortex-compressor/src/compressor.rs:215:45
  14: compress
             at ./vortex-compressor/src/compressor.rs:136:14
  15: compress_canonical
             at ./vortex-compressor/src/compressor.rs:215:45
  16: compress
             at ./vortex-compressor/src/compressor.rs:136:14
  17: compress_array
             at ./fuzz/src/array/mod.rs:542:14
  18: run_fuzz_action
             at ./fuzz/src/array/mod.rs:586:33
  19: __libfuzzer_sys_run
   ... (25 more frames truncated)

Summary

• Target: array_ops
• Crash File: crash-b0077fc81fb9dcb92a6b73cdcf6f39b02ac2317b
• Branch: develop
• Commit: b9c47cf
• Crash Artifact: https://github.com/vortex-data/vortex/actions/runs/24118262281/artifacts/6320972461

Reproduce

cargo +nightly fuzz run -D --sanitizer=none array_ops ./fuzz/artifacts/array_ops/crash-b0077fc81fb9dcb92a6b73cdcf6f39b02ac2317b -- -rss_limit_mb=0

Reproduction Steps

1. Download the crash artifact: https://github.com/vortex-data/vortex/actions/runs/24118262281/artifacts/6320972461

2. Assuming you download the zipfile to ~/Downloads, and your working directory is the repository root:

# Create the artifacts directory if you haven't already.
mkdir -p ./fuzz/artifacts

# Move the zipfile.
mv ~/Downloads/array_ops-crash-artifacts.zip ./fuzz/artifacts/

# Unzip the zipfile.
unzip ./fuzz/artifacts/array_ops-crash-artifacts.zip -d ./fuzz/artifacts/

# You can remove the zipfile now if you want to.
rm ./fuzz/artifacts/array_ops-crash-artifacts.zip

3. Reproduce the crash:

cargo +nightly fuzz run -D --sanitizer=none array_ops ./fuzz/artifacts/array_ops/crash-b0077fc81fb9dcb92a6b73cdcf6f39b02ac2317b -- -rss_limit_mb=0

If you want a backtrace:

RUST_BACKTRACE=1 cargo +nightly fuzz run -D --sanitizer=none array_ops ./fuzz/artifacts/array_ops/crash-b0077fc81fb9dcb92a6b73cdcf6f39b02ac2317b -- -rss_limit_mb=0

RUST_BACKTRACE=full cargo +nightly fuzz run -D --sanitizer=none array_ops ./fuzz/artifacts/array_ops/crash-b0077fc81fb9dcb92a6b73cdcf6f39b02ac2317b -- -rss_limit_mb=0

Single command to get a backtrace

mkdir -p ./fuzz/artifacts
mv ~/Downloads/array_ops-crash-artifacts.zip ./fuzz/artifacts/
unzip ./fuzz/artifacts/array_ops-crash-artifacts.zip -d ./fuzz/artifacts/
rm ./fuzz/artifacts/array_ops-crash-artifacts.zip
RUST_BACKTRACE=1 cargo +nightly fuzz run -D --sanitizer=none array_ops ./fuzz/artifacts/array_ops/crash-b0077fc81fb9dcb92a6b73cdcf6f39b02ac2317b -- -rss_limit_mb=0

---

Auto-created by fuzzing workflow

[connortsui20]

I thought this had been fixed with #7241 (see #7227) but I guess it wasn't?

[connortsui20]

Just following the backtrace something seems very wrong here, if we are sampling float dict then we should have generated those stats literally in the line right before the crash. Ill have to take a deeper look later.

[robert3005]

The sample ends up being all null, you have an array stats like

FloatStats { null_count: 1936, value_count: 8, average_run_length: 1, erased: F64(TypedStats { distinct: Some(DistinctInfo { distinct_values: {NativeValue(-1.9490628022799996e289), NativeValue(3.1315130625140196e-294)}, distinct_count: 2 }) }) }

you then sample it and end up with

FloatStats { null_count: 1024, value_count: 0, average_run_length: 0, erased: F64(TypedStats { distinct: None }) }

[connortsui20]

thats not good, that shouldnt happen because the float dict scheme declares this above which should trigger the cache to generate them:

impl Scheme for FloatDictScheme {
...
    fn stats_options(&self) -> GenerateStatsOptions {
        GenerateStatsOptions {
            count_distinct_values: true,
        }
    }

And so when we do this in the estimate with sampling function:

    //                                                     vvvvvvvvvvvvvvvvvvvvvvvv
    let mut sample_data = ArrayAndStats::new(sample_array, scheme.stats_options());
    let sample_ctx = ctx.with_sampling();

    let after = scheme
        .compress(compressor, &mut sample_data, sample_ctx)?
        .nbytes();

That should work?

[connortsui20]

Note that we know what the issue is now, it is because if the sampled array is all null the distinct count info is not generated

[github-actions]

Crash No Longer Reproduces

This crash was retested against the latest main branch and the fuzzer completed successfully (exit code 0).

The underlying bug appears to have been fixed. Closing this issue.

If the crash reappears, the fuzzer will automatically open a new issue.

---

Auto-checked by weekly fuzzer issue cleanup