Skip to content

Feature/snowflake benchmarking - #4627

Open
ness-david-dedu wants to merge 8 commits into
redpanda-data:mainfrom
ness-david-dedu:feature/snowflake-benchmarking
Open

Feature/snowflake benchmarking#4627
ness-david-dedu wants to merge 8 commits into
redpanda-data:mainfrom
ness-david-dedu:feature/snowflake-benchmarking

Conversation

@ness-david-dedu

Copy link
Copy Markdown
Contributor
  • Adds a benchmark suite for both Snowflake write connectors under internal/impl/snowflake/bench/ — Taskfiles, setup
    SQL, and configs for snowflake_streaming (Snowpipe Streaming API) and snowflake_put (staged PUT + Snowpipe). Includes
    both one-dimension-at-a-time sweeps and full cross-product matrix tasks (bench:streaming:matrix:full,
    bench:bulk:matrix) for exploring BATCH/PARALLELISM/MAX_IN_FLIGHT/CHUNK_SIZE/UPLOAD_THREADS combinations.
    • Publishes results to docs/benchmark-results/snowflake.md (merged streaming + bulk into one file, two sections,
      condensed with collapsible raw sweep data), linked from docs/benchmarking.md and SUMMARY.md.
    • Fixes a concurrency bug in snowflake_put found while benchmarking: WriteBatch uploaded per-file PUTs serially and
      accessed s.db without synchronization outside Connect/Close.
      • connMut is now an RWMutex guarding only the *sql.DB pointer swap, not held across the whole batch.
      • Files in a batch upload concurrently via errgroup, bounded by upload_parallel_threads.
      • This raised bulk throughput from 4,661 msg/sec (serialized) to 36,166 msg/sec best-found config (BATCH=20000
        MAX_IN_FLIGHT=64 UPLOAD_THREADS=32), closing most of the gap to snowflake_streaming's 40,442 msg/sec ceiling.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant