numcodecs.LZ4 not compatible with lz4.fram and cramjam.lz4

[Rachit931]

While comparing LZ4 implementations, it was found that lz4.frame and cramjam.lz4 are compatible with each other, but numcodecs.LZ4 is not compatible with them.

From the tests :

1. Data compressed with lz4.frame can be decompressed by cramjam.lz4 and vice versa.
2. Data compressed with lz4.frame or cramjam.lz4 cannot be decompressed using numcodecs.LZ4.
3. Data compressed with numcodecs.LZ4 cannot be decompressed using lz4.frame or cramjam.lz4.

import cramjam
import numcodecs
import os
import lz4.frame

initial_data = os.urandom(128)

compressed = {}

compressed["lz4"] = lz4.frame.compress(initial_data)
compressed["cramjam"] = bytes(cramjam.lz4.compress(initial_data))
compressed["numcodecs"] = numcodecs.LZ4().encode(initial_data)

libraries = ["lz4", "cramjam", "numcodecs"]

for compressor in libraries:
    for decompressor in libraries:
        print(f"Decompressing data compressed by {compressor} using {decompressor}... ", end="")
        
        try:
            if decompressor == "lz4":
                decompressed = lz4.frame.decompress(compressed[compressor])
            elif decompressor == "cramjam":
                decompressed = bytes(cramjam.lz4.decompress(compressed[compressor]))
            elif decompressor == "numcodecs":
                decompressed = numcodecs.LZ4().decode(compressed[compressor])
            assert initial_data == decompressed, f"Decompression failed for {decompressor} with data compressed by {compressor}"
            print("Success!")
        except Exception as e:
            print(f"Failed: {e}")

Outputs :

Decompressing data compressed by lz4 using cramjam... Success!
Decompressing data compressed by lz4 using numcodecs... Failed: LZ4 decompression error: -13
Decompressing data compressed by cramjam using lz4... Success!
Decompressing data compressed by cramjam using cramjam... Success!
Decompressing data compressed by cramjam using numcodecs... Failed: LZ4 decompression error: -9
Decompressing data compressed by numcodecs using lz4... Failed: LZ4F_getFrameInfo failed with code: ERROR_frameType_unknown
Decompressing data compressed by numcodecs using cramjam... Failed: LZ4 error: ERROR_frameType_unknown
Decompressing data compressed by numcodecs using numcodecs... Success!

[ariostas]

So it turns out that the issue is that the headers are different. numcodecs just has a 4-byte header for the uncompressed size

numcodecs/numcodecs/lz4.pyx

Lines 61 to 64 in 408a367

	Notes
	-----
	The compressed output includes a 4-byte header storing the original size of the decompressed
	data as a little-endian 32-bit integer.

whereas other libraries have an LZ4Frame which has more metadata. To use it, one needs lz4frame.h and lz4frame.c, but blosc doesn't include those (see here).

If anyone wants to work on this, here's a vibe-coded fix that you can use as inspiration ariostas@3195c4c

[Rachit931]

I would like to take this issue
Will update as soon as I make progress.