[client-v2] useHttpCompression(true) requests Accept-Encoding: lz4 without http_zlib_compression_level, so the server compresses responses with LZ4HC (level 3) #3201

Description

When useHttpCompression(true) is enabled together with compressServerResponse(true), client-v2 asks the server for HTTP-level LZ4:

• HttpAPIClientHelper.addRequestParams (client-v2/src/main/java/com/clickhouse/client/api/internal/HttpAPIClientHelper.java:1000-1006) sends enable_http_compression=1
• HttpAPIClientHelper.addHeaders (client-v2/src/main/java/com/clickhouse/client/api/internal/HttpAPIClientHelper.java:954-957) sends Accept-Encoding: lz4

Neither sends http_zlib_compression_level. That setting controls the level handed to Lz4DeflatingWriteBuffer on the server, and its default is 3. LZ4F switches to its HC compressor at level 2 and above, so every HTTP-compressed response is compressed with LZ4HC instead of fast LZ4. The server spends several times more CPU producing the body.

This does not affect the default configuration: ClientConfigProperties.USE_HTTP_COMPRESSION defaults to false (ClientConfigProperties.java:100), so by default client-v2 sends compress=1 and gets native blocks with fast LZ4. client-v1 (ClickHouseHttpConnection) also still uses compress=1 for LZ4 and is unaffected. The issue is scoped to the useHttpCompression(true) path.

ClickHouse server version

26.9.15.5 (local, defaults: enable_http_compression = 1, http_zlib_compression_level = 3). Verified against this running server, not by inspection only.

Server-side evidence (curl, output to /dev/null, so there is no client cost)

Query: SELECT number, toString(number), rand(), rand64() FROM numbers(8000000) FORMAT RowBinary

Level 3 costs ~5x the CPU of level 1 and, on this data, is not even smaller.

Reproduction (client-v2)

client-v2/src/test/java/com/clickhouse/client/api/Lz4HcReproTest.java:

package com.clickhouse.client.api;

import com.clickhouse.client.api.query.QueryResponse;
import com.clickhouse.client.api.query.QuerySettings;
import com.clickhouse.data.ClickHouseFormat;
import org.testng.annotations.Test;

import java.io.InputStream;

public class Lz4HcReproTest {

    static final String Q = "SELECT number, toString(number), rand(), rand64() FROM numbers(8000000)";

    @Test
    public void repro() throws Exception {
        run("default (compress=1)", false, null);
        run("useHttpCompression(true)", true, null);
        run("useHttpCompression(true) + level=1", true, "1");
    }

    static void run(String label, boolean httpCompression, String level) throws Exception {
        Client.Builder b = new Client.Builder()
                .addEndpoint("http://localhost:8123")
                .setUsername("default")
                .setPassword("")
                .compressServerResponse(true)
                .useHttpCompression(httpCompression);
        if (level != null) {
            b.serverSetting("http_zlib_compression_level", level);
        }
        try (Client client = b.build()) {
            query(client); // warm-up
            long best = Long.MAX_VALUE, bytes = 0;
            for (int i = 0; i < 3; i++) {
                long t = System.nanoTime();
                bytes = query(client);
                long ms = (System.nanoTime() - t) / 1_000_000;
                if (ms < best) best = ms;
            }
            System.out.printf("REPRO %-50s best=%5d ms, decoded=%d bytes%n", label, best, bytes);
        }
    }

    static long query(Client client) throws Exception {
        QuerySettings s = new QuerySettings().setFormat(ClickHouseFormat.RowBinary);
        try (QueryResponse r = client.query(Q, s).get()) {
            InputStream in = r.getInputStream();
            byte[] buf = new byte[65536];
            long n = 0; int read;
            while ((read = in.read(buf)) > 0) n += read;
            return n;
        }
    }
}

mvn -pl client-v2 -Dtest=Lz4HcReproTest test against a stock 26.9.15.5 server:

REPRO default (compress=1)                               best=  935 ms, decoded=222888890 bytes
REPRO useHttpCompression(true)                           best=10198 ms, decoded=222888890 bytes
REPRO useHttpCompression(true) + level=1                 best= 9205 ms, decoded=222888890 bytes

Matching system.query_log rows (Settings['http_zlib_compression_level'], query_duration_ms, NetworkSendBytes):

(unset)  9730 / 9739 / 9832 / 10421 ms   137.8 MB
1        8693 / 8657 / 8867 / 9133  ms   138.0 MB
(compress=1 runs, for contrast)  946 / 932 / 942 / 1066 ms   114.3 MB

Expected: the useHttpCompression(true) numbers should match the level=1 numbers (fast LZ4). Actual: ~1 s / ~11% extra per query, purely server-side CPU spent on LZ4HC.

Two notes on magnitude:

• The absolute penalty is the ~1.7 s shown by the curl table; in the Java run it is partly hidden because the client is the bottleneck.
• Independently of this bug, the useHttpCompression(true) path is ~10x slower end-to-end than the default compress=1 path here (~9 s vs ~0.9 s for the same 222 MB), i.e. the framed-LZ4 read side reaches only ~17 MB/s. That is a separate problem, but it is why fixing the level alone only recovers part of the gap.

Suggested fix

In HttpAPIClientHelper.addRequestParams, when useHttpCompression and server compression are on and the negotiated algorithm is lz4, also send http_zlib_compression_level with a value below 2 (e.g. 1), unless the user already set it via serverSetting(...).

Constraint to keep in mind (same one the Rust client hit): http_zlib_compression_level is a server setting, and a readonly = 1 user can only send it with its current value — sending 1 makes every such query fail with Code: 164 ... Cannot modify 'http_zlib_compression_level' setting in readonly mode. So this needs to be opt-out-able, or skipped when the client knows it is talking as a readonly user.

Link

Same root cause as ClickHouse/clickhouse-rs#490 (ClickHouse/clickhouse-rs#490), found while checking whether that regression reaches other clients.