Repository navigation
Support log payload compression in IoTConsensus V1 - #19244
hongzhi-gao wants to merge 6 commits into
Conversation
|
@Caideyipi @jt2594838 PTAL |
| import org.apache.iotdb.consensus.i18n.IoTConsensusMessages; | ||
| import org.apache.iotdb.consensus.iot.thrift.TLogCompressionType; |
There was a problem hiding this comment.
Why not reuse the CompressionType from TsFile?
There was a problem hiding this comment.
Addressed in 434292b. Internal configuration now uses TsFile CompressionType, with NONE mapped to UNCOMPRESSED. The wire enum remains separate because its numeric IDs differ from TsFile's; an explicit mapping preserves the RPC contract. Unsupported TsFile codecs are rejected by the consensus configuration builder, with regression coverage.
There was a problem hiding this comment.
What are the numeric IDs?
There was a problem hiding this comment.
| Codec | TLogCompressionType.getValue() | TsFile CompressionType.serialize() |
|---|---|---|
| NONE / UNCOMPRESSED | 0 | 0 |
| LZ4 | 1 | 7 |
| SNAPPY | 2 | 1 |
| ZSTD | 3 | 8 |
| # IoTConsensus V1 log payload codec: NONE, LZ4, SNAPPY, ZSTD (level 1). | ||
| # IoTConsensus V1 log payload codec: NONE, LZ4, SNAPPY, ZSTD. | ||
| # Compression uses the TsFile implementation and its default codec settings. |
There was a problem hiding this comment.
Why are the others not supported?
There was a problem hiding this comment.
Reusing the TsFile interfaces does not mean that every codec can be enabled directly. For log replication, we also need lossless round trips, controlled compression buffers and decompressed payload sizes, error handling that cannot incorrectly acknowledge a batch, and validation of RPCs, memory cleanup, retries, and performance.
This PR currently supports NONE and the three codecs that have been adapted and validated: LZ4, Snappy, and Zstd. The remaining two are not enabled yet:
- GZIP: its maximum compressed size is an estimate, so buffer overflow can occur before the existing size-benefit check can fall back to raw data. Its decompressor also creates a complete intermediate result array that is not bounded by the caller-provided output length.
- LZMA2: its decompressor likewise creates a complete intermediate result array. The compression output bound and additional temporary allocations also need further adaptation and validation.
This PR therefore starts with these three compression codecs. GZIP and LZMA2 can be added after their resource-bound handling and tests are in place.
Problem
IoTConsensus V1 replicates uncompressed logs through
syncLogEntries. Large payloads increase inter-node traffic and can cause replication backlog.This PR adds configurable log payload compression while retaining existing batching, RPCs, application, and synchronization-index advancement. Deployment requires all nodes to run the same supporting version and compression configuration. Rolling-upgrade negotiation and per-peer fallback are outside this PR's scope.
Design and implementation
Preserve batching; encode individual buffers
The compression unit is each ByteBuffer in
TLogEntry.data; the existing Batch policy is unchanged. A queued consensus request normally produces one buffer. WAL backfill may collect multiple records associated with the same consensus index into one entry, so an entry can contain multiple buffers.Each buffer carries its actual codec and uncompressed length. Compressed and raw buffers may coexist; entries containing only raw buffers omit compression metadata.
flowchart LR A[Queue / WAL backfill] --> B[Existing batching] B --> C[Encode individual buffers and retain original Batch] C --> D[Existing syncLogEntries RPC] D --> E[Validate and decode the entire request] E --> F[Existing deserialization, application and ACK] F --> G[Advance the acknowledged batch prefix]Per-buffer encoding preserves existing boundaries, avoids whole-request concatenation and splitting, and supports individual benefit checks and workspace reservations. Its tradeoffs are descriptor overhead and the inability to exploit cross-buffer redundancy. This round did not compare two native implementations and does not establish that per-buffer compression always outperforms whole-request compression.
Algorithms, configuration, and raw fallback
New startup setting, disabled by default:
data_region_iot_log_compression=NONESupported values are
NONE,LZ4,SNAPPY, andZSTD. LZ4 is the suggested initial choice when enabling compression:data_region_iot_log_compression=LZ4Internal configuration reuses TsFile
CompressionTypeand its codec factories, with explicit mapping to independent RPC codec IDs. Zstd uses TsFile's default level, currently the maximum level in the pinned dependency. There is no specialized Zstd-1 path or direct consensus-module dependency on Zstd JNI. GZIP and LZMA2 are not exposed because their output-bound handling and internal decompression allocations require further adaptation.Buffers are sent raw when:
For example, a 2 KiB buffer is sent unchanged even with LZ4 enabled. The 4 KiB cutoff controls whether compression is attempted; the post-encoding check separately requires net size reduction. No public minimum-size parameter or fixed minimum-savings percentage is added.
Memory, retries, and errors
The sender creates an encoded copy without changing the original Batch or shared ByteBuffer positions. Workspace is reserved per buffer; only the actual encoded results remain retained. RPC callbacks and synchronous client-borrow/send failures release the reservation. Retries encode again from the original Batch.
Original backlog retains its existing
memorySizeaccounting. Encoded copies use the consensus memory pool; native/internal codec workspace is not separately charged to that Java pool.Before constructing consensus requests or deserializing them, the receiver validates descriptors, codecs, lengths, and total decoded size, then decodes the entire request. Only a fully decoded request enters the original application path. Compression failures return
logCompressionErrorand cannot complete the Batch or advance the synchronization index. Later ACKs cannot advance past the failed prefix.A Batch that has sent compressed payloads no longer takes the existing
TApplicationExceptionskip path, preventing protocol errors from incorrectly acknowledging it.The decoded-size limit reuses the existing RPC frame configuration as a per-request bound, not a global receive-memory quota. Decoding remains available when local sending is configured as NONE. There is no independent CRC: modified content that still decodes to the declared length may not be detected by this layer.
WAL, TsFile, and snapshot formats are unchanged. Optional Thrift fields do not make compressed payloads safe for old receivers; version consistency remains an operational requirement.
Performance evaluation
Environment and method
Tests used iot-benchmark
62e0dc1and database revision434292b65c5, with CRC removed and all codecs using TsFile defaults. ZSTD is not Zstd-1. The subsequent master merge was not rebenchmarked.Each fresh cluster received a separate 512-request mixed warm-up. Two topologies × four modes × three profiles × three repetitions produced 72 measured phases. Mode and profile order were randomized; other jobs remained running on the shared servers. DataNodes ran native binaries without an experimental agent; a client-side observer recorded supplementary replication timings.
Official iot-benchmark results
The primary metrics come directly from the benchmark's reported throughput and request latency: throughput counts points/second, while latency measures a write request in milliseconds.
All completed successful repetitions are included, without excluding official scores because of WAL waits. Values are three-run arithmetic means with
[minimum, maximum]. P99 summaries are means of individual run P99 values, not percentiles recomputed from pooled requests.Write throughput: million points/second
Request P99: milliseconds
Server load and replication payload
These supplementary measurements cover the first client write through replica catch-up, rather than the official benchmark timing window. CPU-s is the three-run mean cumulative CPU consumption of all DataNodes. RSS is the largest sampled value for any individual DataNode across the three runs.
Payload ratios include raw fallbacks but exclude Thrift/TCP overhead. Lower RSS samples do not establish that compression saves memory; backlog and GC timing also contribute. Resource sampling and clock alignment introduce uncertainty, so small CPU differences are not treated as established improvements.
Findings
Validation and limitations
test-compile, covering codec round trips, real bidirectional asynchronous Thrift, whole-request rejection on invalid input, length limits, per-buffer memory fallback, cleanup on failures, and synchronization-prefix protection.WORKLOAD_BUFFER_SIZE=100input reuse. Short phases and shared-host variability make these fixed-workload regression comparisons, not sustained maximum-throughput measurements or universal production guarantees.