Skip to content

Support log payload compression in IoTConsensus V1 - #19244

Open
hongzhi-gao wants to merge 6 commits into
apache:masterfrom
hongzhi-gao:research/iotv1-log-compression
Open

hongzhi-gao wants to merge 6 commits into
apache:masterfrom
hongzhi-gao:research/iotv1-log-compression

Conversation

@hongzhi-gao

@hongzhi-gao hongzhi-gao commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Problem

IoTConsensus V1 replicates uncompressed logs through syncLogEntries. Large payloads increase inter-node traffic and can cause replication backlog.

This PR adds configurable log payload compression while retaining existing batching, RPCs, application, and synchronization-index advancement. Deployment requires all nodes to run the same supporting version and compression configuration. Rolling-upgrade negotiation and per-peer fallback are outside this PR's scope.

Design and implementation

Preserve batching; encode individual buffers

The compression unit is each ByteBuffer in TLogEntry.data; the existing Batch policy is unchanged. A queued consensus request normally produces one buffer. WAL backfill may collect multiple records associated with the same consensus index into one entry, so an entry can contain multiple buffers.

Each buffer carries its actual codec and uncompressed length. Compressed and raw buffers may coexist; entries containing only raw buffers omit compression metadata.

flowchart LR
    A[Queue / WAL backfill] --> B[Existing batching]
    B --> C[Encode individual buffers and retain original Batch]
    C --> D[Existing syncLogEntries RPC]
    D --> E[Validate and decode the entire request]
    E --> F[Existing deserialization, application and ACK]
    F --> G[Advance the acknowledged batch prefix]
Loading

Per-buffer encoding preserves existing boundaries, avoids whole-request concatenation and splitting, and supports individual benefit checks and workspace reservations. Its tradeoffs are descriptor overhead and the inability to exploit cross-buffer redundancy. This round did not compare two native implementations and does not establish that per-buffer compression always outperforms whole-request compression.

Algorithms, configuration, and raw fallback

New startup setting, disabled by default:

data_region_iot_log_compression=NONE

Supported values are NONE, LZ4, SNAPPY, and ZSTD. LZ4 is the suggested initial choice when enabling compression:

data_region_iot_log_compression=LZ4

Internal configuration reuses TsFile CompressionType and its codec factories, with explicit mapping to independent RPC codec IDs. Zstd uses TsFile's default level, currently the maximum level in the pinned dependency. There is no specialized Zstd-1 path or direct consensus-module dependency on Zstd JNI. GZIP and LZMA2 are not exposed because their output-bound handling and internal decompression allocations require further adaptation.

Buffers are sent raw when:

  • They are smaller than the fixed internal 4 KiB threshold.
  • The buffer or its entire entry has no net size benefit after descriptor overhead.
  • Temporary encoding memory cannot be reserved.

For example, a 2 KiB buffer is sent unchanged even with LZ4 enabled. The 4 KiB cutoff controls whether compression is attempted; the post-encoding check separately requires net size reduction. No public minimum-size parameter or fixed minimum-savings percentage is added.

Memory, retries, and errors

The sender creates an encoded copy without changing the original Batch or shared ByteBuffer positions. Workspace is reserved per buffer; only the actual encoded results remain retained. RPC callbacks and synchronous client-borrow/send failures release the reservation. Retries encode again from the original Batch.

Original backlog retains its existing memorySize accounting. Encoded copies use the consensus memory pool; native/internal codec workspace is not separately charged to that Java pool.

Before constructing consensus requests or deserializing them, the receiver validates descriptors, codecs, lengths, and total decoded size, then decodes the entire request. Only a fully decoded request enters the original application path. Compression failures return logCompressionError and cannot complete the Batch or advance the synchronization index. Later ACKs cannot advance past the failed prefix.

A Batch that has sent compressed payloads no longer takes the existing TApplicationException skip path, preventing protocol errors from incorrectly acknowledging it.

The decoded-size limit reuses the existing RPC frame configuration as a per-request bound, not a global receive-memory quota. Decoding remains available when local sending is configured as NONE. There is no independent CRC: modified content that still decodes to the declared length may not be detected by this layer.

WAL, TsFile, and snapshot formats are unchanged. Optional Thrift fields do not make compressed payloads safe for old receivers; version consistency remains an operational requirement.

Performance evaluation

Environment and method

Tests used iot-benchmark 62e0dc1 and database revision 434292b65c5, with CRC removed and all codecs using TsFile defaults. ZSTD is not Zstd-1. The subsequent master merge was not rebenchmarked.

Item Configuration
Hardware Three Intel Core i7-10700 servers, each with 8 physical cores, 16 logical CPUs, 32 GiB RAM and 1 Gbit/s Ethernet
Storage Seagate ST10000NM001G 10 TB HDD, ext4; binaries, data and WAL on HDD
OS / JVM Ubuntu 20.04.6, Linux 5.4.0-195; Java 17.0.12 on the first receiver and 17.0.16 on the other two hosts
Topologies Two replicas: one writer and one receiver; three replicas: one writer and two receivers; driver co-located with the writer
DataNode memory 1536 MiB heap and 512 MiB direct memory
Replication Default 16 MiB batch limit, up to five in-flight Batches per peer, internal 4 KiB compression threshold
Writes Tree model, aligned writes, one DataRegion; 8 devices × 24 measurements, four clients, 256 rows per request
Data per phase 2048 requests and 12,582,912 points
Profiles Mixed six types, random DOUBLE with eight decimal places, and TEXT of length 64

Each fresh cluster received a separate 512-request mixed warm-up. Two topologies × four modes × three profiles × three repetitions produced 72 measured phases. Mode and profile order were randomized; other jobs remained running on the shared servers. DataNodes ran native binaries without an experimental agent; a client-side observer recorded supplementary replication timings.

Official iot-benchmark results

The primary metrics come directly from the benchmark's reported throughput and request latency: throughput counts points/second, while latency measures a write request in milliseconds.

All completed successful repetitions are included, without excluding official scores because of WAL waits. Values are three-run arithmetic means with [minimum, maximum]. P99 summaries are means of individual run P99 values, not percentiles recomputed from pooled requests.

Write throughput: million points/second

Replicas Profile NONE LZ4 SNAPPY ZSTD
2 Mixed six types 3.26 [1.06, 4.68] 4.65 [4.15, 5.61] 3.43 [0.84, 5.69] 3.33 [1.65, 5.61]
2 DOUBLE 4.29 [1.70, 6.44] 3.95 [3.50, 4.71] 3.65 [1.15, 6.45] 3.76 [1.94, 6.52]
2 TEXT-64 1.26 [0.51, 1.65] 1.74 [1.65, 1.87] 1.63 [1.22, 1.94] 1.54 [1.13, 1.98]
3 Mixed six types 3.90 [1.93, 5.50] 4.24 [3.03, 5.35] 4.68 [3.47, 5.31] 4.21 [3.31, 5.51]
3 DOUBLE 4.56 [3.45, 6.30] 4.60 [3.45, 6.06] 5.12 [4.61, 6.11] 4.67 [2.96, 6.21]
3 TEXT-64 1.68 [1.50, 1.83] 1.72 [1.59, 1.87] 1.81 [1.69, 1.91] 1.78 [1.64, 1.93]

Request P99: milliseconds

Replicas Profile NONE LZ4 SNAPPY ZSTD
2 Mixed six types 9.21 [2.90, 17.44] 9.23 [4.22, 13.27] 6.87 [2.28, 12.48] 7.93 [2.52, 18.27]
2 DOUBLE 5.62 [2.33, 10.79] 4.26 [2.16, 8.26] 3.21 [1.27, 6.37] 3.68 [1.28, 7.63]
2 TEXT-64 93.71 [45.57, 137.40] 108.52 [104.83, 113.62] 110.30 [97.73, 128.85] 95.20 [90.65, 99.41]
3 Mixed six types 10.57 [2.13, 17.27] 11.92 [7.75, 15.02] 15.55 [14.05, 16.86] 10.95 [9.92, 12.10]
3 DOUBLE 5.13 [1.75, 7.18] 3.83 [2.21, 5.95] 8.96 [7.03, 12.64] 4.20 [1.75, 8.14]
3 TEXT-64 95.86 [92.05, 99.08] 112.07 [105.09, 116.45] 107.52 [92.00, 115.72] 104.34 [100.83, 109.49]

Server load and replication payload

These supplementary measurements cover the first client write through replica catch-up, rather than the official benchmark timing window. CPU-s is the three-run mean cumulative CPU consumption of all DataNodes. RSS is the largest sampled value for any individual DataNode across the three runs.

Replicas Profile NONE CPU-s LZ4 CPU-s NONE max RSS MiB LZ4 max RSS MiB LZ4 encoded/original payload
2 Mixed six types 12.3 11.1 1210 1069 13.0%
2 DOUBLE 6.7 8.0 2201 2205 24.8%
2 TEXT-64 32.3 33.8 2148 2270 23.3%
3 Mixed six types 17.0 18.0 1711 1083 13.0%
3 DOUBLE 9.8 9.8 2194 2141 24.8%
3 TEXT-64 48.1 46.6 2233 2283 25.3%

Payload ratios include raw fallbacks but exclude Thrift/TCP overhead. Lower RSS samples do not establish that compression saves memory; backlog and GC timing also contribute. Resource sampling and clock alignment introduce uncertainty, so small CPU differences are not treated as established improvements.

Findings

  • LZ4 reduced replication payload by approximately 75%–87%, supporting it as an initial choice when enabling compression. Throughput and P99 did not improve uniformly.
  • Snappy had the highest mean throughput in all three-replica profiles, but its mixed and DOUBLE P99 values were higher than LZ4.
  • TsFile-default Zstd reduced payload further, but mixed-profile CPU consumption from writing through replica catch-up was approximately three times LZ4's. Client-visible write scores do not fully reflect this deferred replication cost.

Validation and limitations

  • The current merge commit passed 63 focused tests and full-reactor English/Chinese test-compile, covering codec round trips, real bidirectional asynchronous Thrift, whole-request rejection on invalid input, length limits, per-buffer memory fallback, cleanup on failures, and synchronization-prefix protection.
  • All 72 measured phases had zero failed client writes, caught-up native synchronization frontiers, and 288 passing query checks, including queries after stopping the writer.
  • One pre-measurement metrics-scrape failure was retained and repeated using a fresh cluster. No incomplete attempt was admitted.
  • Queries may route to another surviving replica, so these checks are not per-replica, per-point checksums.
  • The benchmark uses synthetic data and WORKLOAD_BUFFER_SIZE=100 input reuse. Short phases and shared-host variability make these fixed-workload regression comparisons, not sustained maximum-throughput measurements or universal production guarantees.
  • The internal 4 KiB cutoff is conservative; it has not been retuned for the current implementation.
  • Power-loss fault injection, table-model performance, and single-buffer stress near the frame limit were not tested.

@hongzhi-gao

Copy link
Copy Markdown
Contributor Author

@Caideyipi @jt2594838 PTAL

@Caideyipi Caideyipi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed commit 960f564. No blocking issues found. The 56 focused consensus tests passed.

Comment on lines +25 to +26
import org.apache.iotdb.consensus.i18n.IoTConsensusMessages;
import org.apache.iotdb.consensus.iot.thrift.TLogCompressionType;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not reuse the CompressionType from TsFile?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 434292b. Internal configuration now uses TsFile CompressionType, with NONE mapped to UNCOMPRESSED. The wire enum remains separate because its numeric IDs differ from TsFile's; an explicit mapping preserves the RPC contract. Unsupported TsFile codecs are rejected by the consensus configuration builder, with regression coverage.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are the numeric IDs?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codec TLogCompressionType.getValue() TsFile CompressionType.serialize()
NONE / UNCOMPRESSED 0 0
LZ4 1 7
SNAPPY 2 1
ZSTD 3 8

Comment thread iotdb-core/consensus/pom.xml Resolved
Comment on lines +1876 to +1877
# IoTConsensus V1 log payload codec: NONE, LZ4, SNAPPY, ZSTD (level 1).
# IoTConsensus V1 log payload codec: NONE, LZ4, SNAPPY, ZSTD.
# Compression uses the TsFile implementation and its default codec settings.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are the others not supported?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reusing the TsFile interfaces does not mean that every codec can be enabled directly. For log replication, we also need lossless round trips, controlled compression buffers and decompressed payload sizes, error handling that cannot incorrectly acknowledge a batch, and validation of RPCs, memory cleanup, retries, and performance.

This PR currently supports NONE and the three codecs that have been adapted and validated: LZ4, Snappy, and Zstd. The remaining two are not enabled yet:

  • GZIP: its maximum compressed size is an estimate, so buffer overflow can occur before the existing size-benefit check can fall back to raw data. Its decompressor also creates a complete intermediate result array that is not bounded by the caller-provided output length.
  • LZMA2: its decompressor likewise creates a complete intermediate result array. The compression output bound and additional temporary allocations also need further adaptation and validation.

This PR therefore starts with these three compression codecs. GZIP and LZMA2 can be added after their resource-bound handling and tests are in place.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants