Fix client-v2: detect the codec of a compressed response instead of assuming LZ4

CHANGELOG.md

@@ -5,6 +5,20 @@
 
 ### Breaking Changes
 
+- **[client-v2] The algorithm of a compressed response is now requested with `Accept-Encoding` and is no longer the
+  one the server picks.** A response was requested with the `compress=1` framing, whose codec the server chooses on
+  its own: ClickHouse `26.9` changed that codec from `LZ4` to `ZSTD(3)`, the framed output follows the built-in
+  default and no setting overrides it, so the client could not keep reading a response it asked for. A response is
+  now requested with the content coding of the new `client.compression_algorithm` property
+  (`Client.Builder#compressionAlgorithm`), which defaults to `LZ4` and keeps the algorithm of a compressed body the
+  same on every server version. Set the property to `ZSTD`, `GZIP` or `NONE` to select another algorithm; `ZSTD`
+  needs `com.github.luben:zstd-jni` on the classpath, which the client does not bring - the dependency of
+  `clickhouse-jdbc` stays `provided`, so packaging is unchanged and an application that selects `ZSTD` declares the
+  dependency itself. A client that reads a
+  compressed response now also sends `enable_http_compression=1`, which a user profile that forbids setting changes
+  (`readonly = 1`) rejects - such a profile has to use `readonly = 2` or `client.compression_algorithm = NONE`.
+  (https://github.com/ClickHouse/clickhouse-java/issues/3105)
+
 - **[client-v2]** `com.clickhouse.client.api.metrics.OperationMetrics` now has a single constructor,
   `OperationMetrics(ClientStatisticsHolder, OperationType)`; the constructor without an operation type was removed.
   Metrics are created by the client, which always knows the kind of the operation it runs, and the constructor takes
@@ -153,6 +167,13 @@
 
 ### Bug Fixes 
 
+- **[client-v2]** Fixed every compressed read failing with `Invalid LZ4 magic byte: '-112'` against ClickHouse `26.9`
+  and later. The server chooses the codec of the `compress=1` framing the client requested and switched that codec to
+  `ZSTD(3)`, while the response reader asserted the LZ4 method byte of every block, so any query answered with a
+  compressed body died before the first row was parsed. The algorithm of a response is now requested with
+  `Accept-Encoding`, so the client reads the algorithm it asked for; see the breaking-changes entry above.
+  (https://github.com/ClickHouse/clickhouse-java/issues/3105)
+
 - **[jdbc-v2, client-v2]** Fixes issue with `FORMAT` in query unable to override format set by client when used with
   ClickHouse 26.8+. Default format is `RowBinaryWithNamesAndTypes` set at client level. For JDBC, recommend using
   `format=JSONEachRow` to query JSON. Setting `format=` (empty or `null`) omits the format request header so explicit

client-v2/src/main/java/com/clickhouse/client/api/Client.java

@@ -11,6 +11,7 @@
 import com.clickhouse.client.api.data_formats.internal.BinaryStreamReader;
 import com.clickhouse.client.api.data_formats.internal.MapBackedRecord;
 import com.clickhouse.client.api.data_formats.internal.ProcessParser;
+import com.clickhouse.client.api.enums.CompressionAlgorithm;
 import com.clickhouse.client.api.enums.Protocol;
 import com.clickhouse.client.api.enums.ProxyType;
 import com.clickhouse.client.api.enums.SSLMode;
@@ -659,10 +660,32 @@ public Builder compressClientRequest(boolean enabled) {
             return this;
         }
 
+        /**
+         * Algorithm of a compressed request or response body. The algorithm is requested with the HTTP
+         * content-coding of the operation, so a compressed body always uses the algorithm set here and
+         * never one the server picks on its own. {@link CompressionAlgorithm#NONE} disables compression.
+         * Default is {@link CompressionAlgorithm#LZ4}.
+         * <p>
+         * {@link CompressionAlgorithm#ZSTD} needs {@code com.github.luben:zstd-jni} on the classpath, which the
+         * client does not bring: an application that selects the algorithm declares the dependency itself.
+         * <p>
+         * A request body follows this algorithm only together with {@link #useHttpCompression(boolean)};
+         * the ClickHouse framing of a request compressed without it is always LZ4.
+         *
+         * @param algorithm - algorithm of a compressed body
+         * @return same instance of the builder
+         */
+        public Builder compressionAlgorithm(CompressionAlgorithm algorithm) {
+            ValidationUtils.checkNotNull(algorithm, "algorithm");
+            this.configuration.put(ClientConfigProperties.COMPRESSION_ALGORITHM.getKey(), algorithm.name());
+            return this;
+        }
+
         /**
          * Configures the client to use HTTP compression. In this case compression is controlled by
-         * http headers. Client compression will set {@code Content-Encoding: lz4} header and server
-         * compression will set {@code Accept-Encoding: lz4} header. Default is false.
+         * http headers. Client compression will set the {@code Content-Encoding} header and server
+         * compression will set the {@code Accept-Encoding} header, both to the content coding of
+         * {@link #compressionAlgorithm(CompressionAlgorithm)}. Default is false.
          *
          * @param enabled - indicates if http compression is enabled
          * @return

client-v2/src/main/java/com/clickhouse/client/api/ClientConfigProperties.java

@@ -2,6 +2,7 @@
 
 import com.clickhouse.client.api.data_formats.ClickHouseFormatReader;
 import com.clickhouse.client.api.data_formats.internal.AbstractBinaryFormatReader;
+import com.clickhouse.client.api.enums.CompressionAlgorithm;
 import com.clickhouse.client.api.enums.SSLMode;
 import com.clickhouse.client.api.internal.ClickHouseLZ4OutputStream;
 import com.clickhouse.data.ClickHouseDataType;
@@ -245,6 +246,26 @@ public Object parseValue(String value) {
                     .collect(Collectors.toList());
         }
     },
+
+    /**
+     * Algorithm of a compressed request or response body. The algorithm is requested with the HTTP
+     * content-coding of the operation ({@code Accept-Encoding} for a response, {@code Content-Encoding}
+     * for a request), so a compressed body always uses the algorithm the client asked for and never one
+     * the server picks on its own. {@link CompressionAlgorithm#NONE} disables compression of both
+     * directions.
+     * <p>
+     * The name of an algorithm and its content-coding token are both accepted, in any case.
+     * <p>
+     * Appended at the end of the enum on purpose: adding a constant in the middle would shift the ordinal
+     * of every following constant (see {@code docs/changes_checklist.md}).
+     */
+    COMPRESSION_ALGORITHM("client.compression_algorithm", CompressionAlgorithm.class,
+            CompressionAlgorithm.LZ4.name()) {
+        @Override
+        public Object parseValue(String value) {
+            return value == null ? null : CompressionAlgorithm.fromValue(value);
+        }
+    },
     ;
 
     private static final Logger LOG = LoggerFactory.getLogger(ClientConfigProperties.class);

client-v2/src/main/java/com/clickhouse/client/api/enums/CompressionAlgorithm.java

@@ -0,0 +1,71 @@
+package com.clickhouse.client.api.enums;
+
+/**
+ * Enumerates the compression algorithms the client can ask the server for and can apply itself.
+ *
+ * <p>The algorithm is requested with the HTTP content-coding of the operation - {@code Accept-Encoding}
+ * for a response and {@code Content-Encoding} for a request - so a compressed body always uses the
+ * algorithm of the request and never one the server picks on its own.</p>
+ *
+ * <ul>
+ *     <li>{@link #LZ4} - default. Needs {@code org.lz4:lz4-java}, which the client depends on.</li>
+ *     <li>{@link #ZSTD} - needs {@code com.github.luben:zstd-jni} on the classpath, which the client does not
+ *     bring: an application that selects this algorithm declares the dependency itself.</li>
+ *     <li>{@link #GZIP} - supported by the JDK, so it needs no additional dependency.</li>
+ *     <li>{@link #NONE} - no compression, whatever {@code compress}/{@code decompress} are set to.</li>
+ * </ul>
+ */
+public enum CompressionAlgorithm {
+
+    /**
+     * ClickHouse LZ4. Default algorithm.
+     */
+    LZ4("lz4"),
+
+    /**
+     * Zstandard. Requires {@code com.github.luben:zstd-jni} on the classpath.
+     */
+    ZSTD("zstd"),
+
+    /**
+     * gzip. Supported by the JDK.
+     */
+    GZIP("gzip"),
+
+    /**
+     * No compression.
+     */
+    NONE("none");
+
+    private final String httpContentCoding;
+
+    CompressionAlgorithm(String httpContentCoding) {
+        this.httpContentCoding = httpContentCoding;
+    }
+
+    /**
+     * Returns the HTTP content-coding token of the algorithm, as used in the {@code Accept-Encoding}
+     * and {@code Content-Encoding} headers.
+     *
+     * @return content-coding token
+     */
+    public String getHttpContentCoding() {
+        return httpContentCoding;
+    }
+
+    /**
+     * Case-insensitive variant of {@link #valueOf(String)} that also accepts the content-coding token.
+     *
+     * @param value algorithm name or content-coding token in any case
+     * @return matching algorithm
+     * @throws IllegalArgumentException when the value does not match any algorithm
+     */
+    public static CompressionAlgorithm fromValue(String value) {
+        for (CompressionAlgorithm algorithm : values()) {
+            if (algorithm.name().equalsIgnoreCase(value) || algorithm.httpContentCoding.equalsIgnoreCase(value)) {
+                return algorithm;
+            }
+        }
+        throw new IllegalArgumentException("Unknown compression algorithm '" + value + "'");
+    }
+}

client-v2/src/main/java/com/clickhouse/client/api/insert/InsertSettings.java

@@ -2,6 +2,7 @@
 
 import com.clickhouse.client.api.Client;
 import com.clickhouse.client.api.ClientConfigProperties;
+import com.clickhouse.client.api.enums.CompressionAlgorithm;
 import com.clickhouse.client.api.Session;
 import com.clickhouse.client.api.internal.CommonSettings;
 import org.apache.hc.core5.http.HttpHeaders;
@@ -213,6 +214,19 @@ public InsertSettings compressClientRequest(boolean enabled) {
         return this;
     }
 
+    /**
+     * Algorithm of a compressed request or response body of this operation. The algorithm is requested with
+     * the HTTP content coding of the operation, so a compressed body always uses the algorithm set here.
+     * {@code CompressionAlgorithm.NONE} disables compression. Defaults to the algorithm of the client.
+     *
+     * @param algorithm - algorithm of a compressed body
+     * @return same instance of the settings
+     */
+    public InsertSettings compressionAlgorithm(CompressionAlgorithm algorithm) {
+        settings.setOption(ClientConfigProperties.COMPRESSION_ALGORITHM.getKey(), algorithm);
+        return this;
+    }
+
     public InsertSettings useHttpCompression(boolean enabled) {
         settings.setOption(ClientConfigProperties.USE_HTTP_COMPRESSION.getKey(), enabled);
         return this;