Add Transpose filter (method 0C) for fixed-size records - #245
Open
plokijuter wants to merge 4 commits into
Open
Conversation
Groups byte i of each R-byte record together so that homogeneous columns reach the next coder. Aimed at fixed-size records: sensor logs, struct arrays, PCM audio, numeric tables -- data where LZ and context models see interleaved, unrelated byte streams. Measured with PPMd:o=16 behind the filter: int32 ascending array 98450 -> 18411 (+81.2%) 16-byte struct records 68722 -> 22215 (+67.6%) fixed-record .dat file 76019 -> 25319 (+66.6%) 4x float32 sensor log 399919 -> 221368 (+44.6%) random data 205808 -> 205824 (identity) text 3862 -> 3878 (identity) R is auto-detected by default (-m0=Transpose); -m0=Transpose:15 forces it. Detection asks whether transposing HELPS, not whether the data is periodic: it compares the mean absolute difference between bytes R apart against that of adjacent bytes, and stays at R=1 (identity) unless a column is clearly more homogeneous. An autocorrelation-based detector was tried first and rejected -- it found periods everywhere (harmonic sidebands) and degraded 7 of 10 test files, one from 107607 to 137309 bytes. The block size is a fixed 64 KiB constant, deliberately independent of the caller's buffer size: 7-Zip does not use the same buffer sizes when compressing and decompressing, and a buffer-dependent block makes the transform irreversible. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018CeqaL4dXjGa9pUc6qRnof
The last partial block of a stream is never transposed: a filter is not told which call is the final one, so the tail is written through unfiltered. With a fixed 64 KiB block that tail cost a lot on small and medium files -- on a 450 KB test file, 56509 bytes (12.6%) reached PPMd as raw interleaved data and cost 7020 bytes, the whole gap against transposing the tail by hand. The block size is now chosen at encoding time from kExpectedDataSize (aim for at most 1/32 of the stream, clamped to 4 KiB..64 KiB) and written into the coder properties, so encoder and decoder agree regardless of their buffer sizes. Properties grow to 2 bytes; a 1-byte property is still read and implies the previous 64 KiB block. With PPMd:o=16 behind the filter: 16-byte struct records 22215 -> 3211 int32 ascending array 18411 -> 2442 fixed-record .dat file 25319 -> 20975 4x float32 sensor log 221368 -> 215478 48 MB sensor log unchanged (large streams keep the 64 KiB block) 65535-byte file 10305 -> 3055 (the filter now engages from 4 KiB) Zero-padding the last block through the AES-CBC path was considered and rejected: that protocol is only safe because a 16-byte block divides the FilterCoder buffer, so a full buffer never presents a partial block. A block that must be a multiple of R has no such property, and an exactly-full buffer would spin in CFilterCoder::Code. Verified: 60 real files, forced R from 2 to 256, sizes 0/1/100/4095/4096/ 65535/65536/200000, a 48 MB stream, and archives written by the previous 1-byte-property build -- all extract byte-identical. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018CeqaL4dXjGa9pUc6qRnof
The heuristic that picks R from the mean absolute difference between bytes R apart is not safe as a default. Measured over a 40-file corpus it degrades 13 files, the worst from 15026 to 352332 bytes (+2245%): a tiled RGB image whose tile repeats across the whole file, which a context model exploits directly and which block-wise transposition destroys. Adds an opt-in mode that measures instead of guessing: it transposes a sample with each of the most promising R values, compresses each, and keeps the winner. R=1 is always in the running, so measuring cannot pick worse than not filtering -- as far as the sample is representative. -m0=Transpose heuristic, fast -m0=Transpose:a=1 measure, probing with PPMd -m0=Transpose:a=2 measure, probing with LZMA The probe must be the coder that actually follows. LZMA and PPMd do not prefer the same R, and probing with the wrong one produces confident nonsense: on a 1.6 MB stereo sine, an LZMA probe picked R=4 (17374 bytes) where PPMd wanted R=1 (9420). Three cases fixed by probing correctly: 1.6 MB stereo sine 17374 -> 9436 mono sine, R=2 optimal 384966 -> 186871 256-byte repeating tile 16939 -> 885 The measurement sample also had to grow from 256 KB to 4 MB. The verdict flips with sample size, because the transposed form costs linearly in the data while the raw form stays nearly flat when it has long-range redundancy: on a 3.5 MB sawtooth, R=12 wins on any prefix up to 2 MB (7295 vs 9644) and loses on the whole file (19520 vs 10170). A short sample concludes backwards, with confidence. Known limitation: a filter never sees more than the FilterCoder buffer (2 MB), so on files a few MB and larger the sample is still not representative and the measure mode gets 3 of 15 test files wrong. Deciding reliably needs a pass over the whole input, which does not belong in a stream filter. Extrapolating the slope between two samples and requiring a margin was tried and rejected: tuned on 15 files it looked exact, and on the 40-file corpus it left +11.7% on the table at every threshold. Also adds mingw-shim/README.md: cross-compiling from Linux needs symlinks for the headers the source includes with Windows casing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018CeqaL4dXjGa9pUc6qRnof
…lter
A stream filter cannot choose R. It never sees more than the FilterCoder
buffer (2 MB), and the verdict flips with how much it sees: on a 3.5 MB
sawtooth, R=12 wins on every prefix up to 2 MB (7295 vs 9644 bytes) and loses
on the whole file (19520 vs 10170). The transposed form costs linearly in the
data while the raw form stays nearly flat when it has long-range redundancy,
so a short sample concludes backwards, with confidence.
Adds -m0=Transpose:a=3, resolved in Update.cpp before the coders are built,
where the input files are known. Four steps, cheapest first:
1. one reference compression at R=1, needed anyway. If the file already
compresses more than 10x, stop: measured over 55 files, no gain is ever
missed above that ratio, and every trap sits above it.
2. rank by mean absolute difference over all R in 2..256 -- integer
subtraction, essentially free. Not a grid: the optimum of one test file
is R=15 and of another R=21, and a grid that skips them costs 27% and
40x respectively.
3. rank by a fast LZMA pass over a curated grid, which sees repetitions the
statistical criterion is blind to.
4. compress for real at R=1 and at the (at most two) candidates, keep the
smallest.
Step 4 is what matters: R=1 is always in the race and the result is a minimum
over real compressed sizes, so degrading is impossible by construction, not
merely rare. Every earlier attempt tried to predict well; this one stops
predicting. When R=1 wins, the filter is dropped from the chain entirely, so
it does not even cost the 16-byte coder record.
Because degradation is impossible, the trigger thresholds can be generous: a
candidate too many costs time, never correctness. A tight 0.30 threshold left
85% and 139% of gain on the table on float sensor logs whose ratio is 0.52.
Probing with PPMd, whatever coder follows. The obvious rule -- probe with the
coder that will actually run -- was implemented and then reverted on
measurement: for an LZMA2 chain, an LZMA probe missed 5.8x on struct records
and 11x on an int32 array, while the PPMd probe found the optimum on 4 of 5.
A context model ranks column homogeneity better than a match finder does.
Measured with -m0=Transpose:a=3 -m1=PPMd:o=16, against the best R found by
exhaustive search:
fixed-record .dat 76019 -> 20975 optimum
16-byte struct records 68722 -> 3211 optimum
4x float32 sensor log 399919 -> 215478 optimum
periodic PCM (85% trap) 9420 -> 9420 correctly abstains
256-byte tile (19x trap) 869 -> 869 correctly abstains
RGB tile16 (13.4x trap) 2280 -> 2280 correctly abstains
70 files of assorted types: 0 degradations, 0 round-trip failures. Sizes
0/1/100/1000/100000, multiple files per archive, PPMd and LZMA2 chains all
round-trip byte-identical.
Above TRANSPOSE_FULL_LIMIT (64 MB) the decision is made on a prefix and the
guarantee no longer holds, so a clear margin is required before accepting the
transposition there.
The pass lives in Update.cpp, which compiles into the executables rather than
7z.dll, so UI/Console needs the encoder objects to link -- 7z.exe grows from
576 KB to 814 KB. 7zFM.exe and 7zG.exe are unchanged and therefore still
lack a=3; they keep the older in-filter modes a=1 and a=2.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018CeqaL4dXjGa9pUc6qRnof
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a byte-transposition filter for data made of fixed-size records: it groups byte i of each R-byte record together, so that homogeneous columns reach the next coder instead of interleaved, unrelated byte streams.
This is the one structure LZ and context models handle poorly today. Sensor logs, struct arrays, PCM audio and numeric tables all have it, and the existing Delta filter only covers the special case R = the stride of a single scalar series.
Measured, with
PPMd:o=16behind the filterOn the 48 MB sensor log that is also 39% smaller than
xz -9eand 66% smaller thanzstd -19.Choosing R
-m0=Transposedetects R automatically;-m0=Transpose:15forces it.Detection asks whether transposing helps, not whether the data is periodic. It compares the mean absolute difference between bytes R apart against that of adjacent bytes, and stays at R=1 (identity) unless a column is clearly more homogeneous, so data with no record structure is passed through untouched.
An autocorrelation detector was written first and rejected on measurement: it found periods everywhere (harmonic sidebands clear a 60%-of-peak threshold) and degraded 7 of 10 test files, one from 107607 to 137309 bytes. Periodicity of a signal says nothing about the homogeneity of its columns.
Block size
The transform is applied on fixed-size blocks whose size is deliberately independent of the caller's buffer, since 7-Zip does not use the same buffer sizes when compressing and decompressing. The size is chosen at encoding time from
kExpectedDataSize(at most 1/32 of the stream, clamped to 4 KiB..64 KiB) and recorded in the coder properties, so both sides agree whatever their buffers are.The reason it is not simply a constant: a filter is never told which call is the final one, so the last partial block of a stream is written through unfiltered. With a fixed 64 KiB block that tail reached 12.6% of a 450 KB file and cost 7020 bytes.
Zero-padding the final block through the AES-CBC path in
CFilterCoder::Codewas considered and rejected: that protocol is only safe because a 16-byte block divides the FilterCoder buffer, so a full buffer never presents a partial block. A block that must be a multiple of R has no such property, and an exactly-full buffer would spin.Interface
0C(free slot;0BRISCV is the last assigned,21is LZMA2)R-1thenlog2of the block size. A 1-byte property is still accepted and implies a 64 KiB block.C/Transpose.{c,h}plusCPP/7zip/Compress/TransposeFilter.cpp, registered withREGISTER_FILTER_E. No existing file is modified beyond the build hooks andDOC/Methods.txt.Verification
60 real files of assorted types, forced R from 2 to 256, sizes 0/1/100/4095/4096/65535/65536/200000, a 48 MB stream, and archives written by an earlier 1-byte-property build: all extract byte-identical. Compiles clean under
-Werror -Wall -Wextra.Happy to adjust the method ID, the property layout, or the detection heuristic if you would rather they were shaped differently.