diff --git a/CHANGELOG.md b/CHANGELOG.md index 7b575c5a5..6757cafad 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,20 @@ ### Breaking Changes +- **[client-v2] The algorithm of a compressed response is now requested with `Accept-Encoding` and is no longer the + one the server picks.** A response was requested with the `compress=1` framing, whose codec the server chooses on + its own: ClickHouse `26.9` changed that codec from `LZ4` to `ZSTD(3)`, the framed output follows the built-in + default and no setting overrides it, so the client could not keep reading a response it asked for. A response is + now requested with the content coding of the new `client.compression_algorithm` property + (`Client.Builder#compressionAlgorithm`), which defaults to `LZ4` and keeps the algorithm of a compressed body the + same on every server version. Set the property to `ZSTD`, `GZIP` or `NONE` to select another algorithm; `ZSTD` + needs `com.github.luben:zstd-jni` on the classpath, which the client does not bring - the dependency of + `clickhouse-jdbc` stays `provided`, so packaging is unchanged and an application that selects `ZSTD` declares the + dependency itself. A client that reads a + compressed response now also sends `enable_http_compression=1`, which a user profile that forbids setting changes + (`readonly = 1`) rejects - such a profile has to use `readonly = 2` or `client.compression_algorithm = NONE`. + (https://github.com/ClickHouse/clickhouse-java/issues/3105) + - **[client-v2]** `com.clickhouse.client.api.metrics.OperationMetrics` now has a single constructor, `OperationMetrics(ClientStatisticsHolder, OperationType)`; the constructor without an operation type was removed. Metrics are created by the client, which always knows the kind of the operation it runs, and the constructor takes @@ -153,6 +167,13 @@ ### Bug Fixes +- **[client-v2]** Fixed every compressed read failing with `Invalid LZ4 magic byte: '-112'` against ClickHouse `26.9` + and later. The server chooses the codec of the `compress=1` framing the client requested and switched that codec to + `ZSTD(3)`, while the response reader asserted the LZ4 method byte of every block, so any query answered with a + compressed body died before the first row was parsed. The algorithm of a response is now requested with + `Accept-Encoding`, so the client reads the algorithm it asked for; see the breaking-changes entry above. + (https://github.com/ClickHouse/clickhouse-java/issues/3105) + - **[jdbc-v2]** Added the non-reserved keywords `AGGREGATE`, `BOUNDED`, `EXTEND`, `HANDLER`, `IDLE`, `PROTOCOL`, `RECENT`, `TIMEOUT` and `UNORDERED` (ClickHouse `26.8+`; `IDLE`, `TIMEOUT` and `RECENT` come from the multi-word keywords `IDLE TIMEOUT` and `RECENT SAMPLES`) to the list of keywords allowed in identifier positions. The server diff --git a/client-v2/src/main/java/com/clickhouse/client/api/Client.java b/client-v2/src/main/java/com/clickhouse/client/api/Client.java index 22b949e88..92b641209 100644 --- a/client-v2/src/main/java/com/clickhouse/client/api/Client.java +++ b/client-v2/src/main/java/com/clickhouse/client/api/Client.java @@ -11,6 +11,7 @@ import com.clickhouse.client.api.data_formats.internal.BinaryStreamReader; import com.clickhouse.client.api.data_formats.internal.MapBackedRecord; import com.clickhouse.client.api.data_formats.internal.ProcessParser; +import com.clickhouse.client.api.enums.CompressionAlgorithm; import com.clickhouse.client.api.enums.Protocol; import com.clickhouse.client.api.enums.ProxyType; import com.clickhouse.client.api.enums.SSLMode; @@ -659,10 +660,32 @@ public Builder compressClientRequest(boolean enabled) { return this; } + /** + * Algorithm of a compressed request or response body. The algorithm is requested with the HTTP + * content-coding of the operation, so a compressed body always uses the algorithm set here and + * never one the server picks on its own. {@link CompressionAlgorithm#NONE} disables compression. + * Default is {@link CompressionAlgorithm#LZ4}. + *

+ * {@link CompressionAlgorithm#ZSTD} needs {@code com.github.luben:zstd-jni} on the classpath, which the + * client does not bring: an application that selects the algorithm declares the dependency itself. + *

+ * A request body follows this algorithm only together with {@link #useHttpCompression(boolean)}; + * the ClickHouse framing of a request compressed without it is always LZ4. + * + * @param algorithm - algorithm of a compressed body + * @return same instance of the builder + */ + public Builder compressionAlgorithm(CompressionAlgorithm algorithm) { + ValidationUtils.checkNotNull(algorithm, "algorithm"); + this.configuration.put(ClientConfigProperties.COMPRESSION_ALGORITHM.getKey(), algorithm.name()); + return this; + } + /** * Configures the client to use HTTP compression. In this case compression is controlled by - * http headers. Client compression will set {@code Content-Encoding: lz4} header and server - * compression will set {@code Accept-Encoding: lz4} header. Default is false. + * http headers. Client compression will set the {@code Content-Encoding} header and server + * compression will set the {@code Accept-Encoding} header, both to the content coding of + * {@link #compressionAlgorithm(CompressionAlgorithm)}. Default is false. * * @param enabled - indicates if http compression is enabled * @return diff --git a/client-v2/src/main/java/com/clickhouse/client/api/ClientConfigProperties.java b/client-v2/src/main/java/com/clickhouse/client/api/ClientConfigProperties.java index db286dee6..daede437d 100644 --- a/client-v2/src/main/java/com/clickhouse/client/api/ClientConfigProperties.java +++ b/client-v2/src/main/java/com/clickhouse/client/api/ClientConfigProperties.java @@ -2,6 +2,7 @@ import com.clickhouse.client.api.data_formats.ClickHouseFormatReader; import com.clickhouse.client.api.data_formats.internal.AbstractBinaryFormatReader; +import com.clickhouse.client.api.enums.CompressionAlgorithm; import com.clickhouse.client.api.enums.SSLMode; import com.clickhouse.client.api.internal.ClickHouseLZ4OutputStream; import com.clickhouse.data.ClickHouseDataType; @@ -245,6 +246,26 @@ public Object parseValue(String value) { .collect(Collectors.toList()); } }, + + /** + * Algorithm of a compressed request or response body. The algorithm is requested with the HTTP + * content-coding of the operation ({@code Accept-Encoding} for a response, {@code Content-Encoding} + * for a request), so a compressed body always uses the algorithm the client asked for and never one + * the server picks on its own. {@link CompressionAlgorithm#NONE} disables compression of both + * directions. + *

+ * The name of an algorithm and its content-coding token are both accepted, in any case. + *

+ * Appended at the end of the enum on purpose: adding a constant in the middle would shift the ordinal + * of every following constant (see {@code docs/changes_checklist.md}). + */ + COMPRESSION_ALGORITHM("client.compression_algorithm", CompressionAlgorithm.class, + CompressionAlgorithm.LZ4.name()) { + @Override + public Object parseValue(String value) { + return value == null ? null : CompressionAlgorithm.fromValue(value); + } + }, ; private static final Logger LOG = LoggerFactory.getLogger(ClientConfigProperties.class); diff --git a/client-v2/src/main/java/com/clickhouse/client/api/enums/CompressionAlgorithm.java b/client-v2/src/main/java/com/clickhouse/client/api/enums/CompressionAlgorithm.java new file mode 100644 index 000000000..d9df4f26d --- /dev/null +++ b/client-v2/src/main/java/com/clickhouse/client/api/enums/CompressionAlgorithm.java @@ -0,0 +1,71 @@ +package com.clickhouse.client.api.enums; + +/** + * Enumerates the compression algorithms the client can ask the server for and can apply itself. + * + *

The algorithm is requested with the HTTP content-coding of the operation - {@code Accept-Encoding} + * for a response and {@code Content-Encoding} for a request - so a compressed body always uses the + * algorithm of the request and never one the server picks on its own.

+ * + * + */ +public enum CompressionAlgorithm { + + /** + * ClickHouse LZ4. Default algorithm. + */ + LZ4("lz4"), + + /** + * Zstandard. Requires {@code com.github.luben:zstd-jni} on the classpath. + */ + ZSTD("zstd"), + + /** + * gzip. Supported by the JDK. + */ + GZIP("gzip"), + + /** + * No compression. + */ + NONE("none"); + + private final String httpContentCoding; + + CompressionAlgorithm(String httpContentCoding) { + this.httpContentCoding = httpContentCoding; + } + + /** + * Returns the HTTP content-coding token of the algorithm, as used in the {@code Accept-Encoding} + * and {@code Content-Encoding} headers. + * + * @return content-coding token + */ + public String getHttpContentCoding() { + return httpContentCoding; + } + + /** + * Case-insensitive variant of {@link #valueOf(String)} that also accepts the content-coding token. + * + * @param value algorithm name or content-coding token in any case + * @return matching algorithm + * @throws IllegalArgumentException when the value does not match any algorithm + */ + public static CompressionAlgorithm fromValue(String value) { + for (CompressionAlgorithm algorithm : values()) { + if (algorithm.name().equalsIgnoreCase(value) || algorithm.httpContentCoding.equalsIgnoreCase(value)) { + return algorithm; + } + } + throw new IllegalArgumentException("Unknown compression algorithm '" + value + "'"); + } +} diff --git a/client-v2/src/main/java/com/clickhouse/client/api/insert/InsertSettings.java b/client-v2/src/main/java/com/clickhouse/client/api/insert/InsertSettings.java index 078019b17..0434bf25c 100644 --- a/client-v2/src/main/java/com/clickhouse/client/api/insert/InsertSettings.java +++ b/client-v2/src/main/java/com/clickhouse/client/api/insert/InsertSettings.java @@ -2,6 +2,7 @@ import com.clickhouse.client.api.Client; import com.clickhouse.client.api.ClientConfigProperties; +import com.clickhouse.client.api.enums.CompressionAlgorithm; import com.clickhouse.client.api.Session; import com.clickhouse.client.api.internal.CommonSettings; import org.apache.hc.core5.http.HttpHeaders; @@ -213,6 +214,19 @@ public InsertSettings compressClientRequest(boolean enabled) { return this; } + /** + * Algorithm of a compressed request or response body of this operation. The algorithm is requested with + * the HTTP content coding of the operation, so a compressed body always uses the algorithm set here. + * {@code CompressionAlgorithm.NONE} disables compression. Defaults to the algorithm of the client. + * + * @param algorithm - algorithm of a compressed body + * @return same instance of the settings + */ + public InsertSettings compressionAlgorithm(CompressionAlgorithm algorithm) { + settings.setOption(ClientConfigProperties.COMPRESSION_ALGORITHM.getKey(), algorithm); + return this; + } + public InsertSettings useHttpCompression(boolean enabled) { settings.setOption(ClientConfigProperties.USE_HTTP_COMPRESSION.getKey(), enabled); return this; diff --git a/client-v2/src/main/java/com/clickhouse/client/api/internal/HttpAPIClientHelper.java b/client-v2/src/main/java/com/clickhouse/client/api/internal/HttpAPIClientHelper.java index 9dab47a39..5050fc746 100644 --- a/client-v2/src/main/java/com/clickhouse/client/api/internal/HttpAPIClientHelper.java +++ b/client-v2/src/main/java/com/clickhouse/client/api/internal/HttpAPIClientHelper.java @@ -11,6 +11,7 @@ import com.clickhouse.client.api.DataTransferException; import com.clickhouse.client.api.ServerException; import com.clickhouse.client.api.TransportException; +import com.clickhouse.client.api.enums.CompressionAlgorithm; import com.clickhouse.client.api.enums.ProxyType; import com.clickhouse.client.api.enums.SSLMode; import com.clickhouse.client.api.http.ClickHouseHttpProto; @@ -123,7 +124,6 @@ public class HttpAPIClientHelper { private static final int ERROR_BODY_BUFFER_SIZE = 1024; // Error messages are usually small - private final String DEFAULT_HTTP_COMPRESSION_ALGO = "lz4"; private static final Pattern PATTERN_HEADER_VALUE_ASCII = Pattern.compile( "\\p{Graph}+(?:[ ]\\p{Graph}+)*"); @@ -163,7 +163,14 @@ public HttpAPIClientHelper(Map configuration, Object metricsRegi boolean usingServerCompression = ClientConfigProperties.COMPRESS_SERVER_RESPONSE.getOrDefault(configuration); boolean useHttpCompression = ClientConfigProperties.USE_HTTP_COMPRESSION.getOrDefault(configuration); - LOG.debug("client compression: {}, server compression: {}, http compression: {}", usingClientCompression, usingServerCompression, useHttpCompression); + CompressionAlgorithm algorithm = compressionAlgorithm(configuration); + LOG.debug("client compression: {}, server compression: {}, http compression: {}, algorithm: {}", + usingClientCompression, usingServerCompression, useHttpCompression, algorithm); + if (usingClientCompression && !useHttpCompression + && !(algorithm == CompressionAlgorithm.LZ4 || algorithm == CompressionAlgorithm.NONE)) { + LOG.warn("Request compression uses LZ4 instead of {}: the ClickHouse framing of a request is LZ4 " + + "unless http compression is used", algorithm); + } defaultRetryCauses = new HashSet<>(ClientConfigProperties.CLIENT_RETRY_ON_FAILURE.getOrDefault(configuration)); if (defaultRetryCauses.contains(ClientFaultCause.None)) { @@ -779,9 +786,7 @@ private TransportResponse doExecuteRequest(TransportRequest transportRequest, Sp spanRecorder.recordHttpStatus(requestSpan, httpResponse.getCode()); } - httpResponse.setEntity(wrapResponseEntity(httpResponse.getEntity(), - httpResponse.getCode(), - requestConfig)); + httpResponse.setEntity(wrapResponseEntity(httpResponse.getEntity())); if (httpResponse.containsHeader(ClickHouseHttpProto.HEADER_EXCEPTION_CODE)) { throw readError(req, httpResponse); @@ -936,14 +941,17 @@ private void addHeaders(HttpPost req, Map requestConfig) { boolean serverCompression = ClientConfigProperties.COMPRESS_SERVER_RESPONSE.getOrDefault(requestConfig); boolean useHttpCompression = ClientConfigProperties.USE_HTTP_COMPRESSION.getOrDefault(requestConfig); boolean appCompressedData = ClientConfigProperties.APP_COMPRESSED_DATA.getOrDefault(requestConfig); + CompressionAlgorithm algorithm = compressionAlgorithm(requestConfig); - if (useHttpCompression) { + if (algorithm != CompressionAlgorithm.NONE) { if (serverCompression) { - setHeader(req, HttpHeaders.ACCEPT_ENCODING, DEFAULT_HTTP_COMPRESSION_ALGO); + // the codec of a compressed response is the one requested here: the server picks its own + // default codec for the compress=1 framing and does not let a client select it + setHeader(req, HttpHeaders.ACCEPT_ENCODING, algorithm.getHttpContentCoding()); } - if (clientCompression && !appCompressedData) { - setHeader(req, HttpHeaders.CONTENT_ENCODING, DEFAULT_HTTP_COMPRESSION_ALGO); + if (useHttpCompression && clientCompression && !appCompressedData) { + setHeader(req, HttpHeaders.CONTENT_ENCODING, algorithm.getHttpContentCoding()); } } @@ -982,17 +990,24 @@ private void addRequestParams(Map requestConfig, BiConsumer requestConfig, BiConsumer requestConfig) { + Object value = requestConfig.get(ClientConfigProperties.COMPRESSION_ALGORITHM.getKey()); + if (value == null) { + return ClientConfigProperties.COMPRESSION_ALGORITHM.getDefObjVal(); + } + return value instanceof CompressionAlgorithm + ? (CompressionAlgorithm) value + : CompressionAlgorithm.fromValue(String.valueOf(value)); + } + private HttpEntity wrapRequestEntity(HttpEntity httpEntity, Map requestConfig) { boolean clientCompression = ClientConfigProperties.COMPRESS_CLIENT_REQUEST.getOrDefault(requestConfig); boolean useHttpCompression = ClientConfigProperties.USE_HTTP_COMPRESSION.getOrDefault(requestConfig); boolean appCompressedData = ClientConfigProperties.APP_COMPRESSED_DATA.getOrDefault(requestConfig); + CompressionAlgorithm algorithm = compressionAlgorithm(requestConfig); if (httpEntity.getContentEncoding() != null && !appCompressedData) { // http header is set and data is not compressed return new CompressedEntity(httpEntity, false, CompressorStreamFactory.getSingleton()); - } else if (clientCompression && !appCompressedData) { + } else if (clientCompression && !appCompressedData && algorithm != CompressionAlgorithm.NONE) { int buffSize = ClientConfigProperties.COMPRESSION_LZ4_UNCOMPRESSED_BUF_SIZE.getOrDefault(requestConfig); return new LZ4Entity(httpEntity, useHttpCompression, false, true, buffSize, false, lz4Factory); @@ -1036,21 +1066,14 @@ private HttpEntity wrapRequestEntity(HttpEntity httpEntity, Map } } - private HttpEntity wrapResponseEntity(HttpEntity httpEntity, int httpStatus, Map requestConfig) { - boolean serverCompression = ClientConfigProperties.COMPRESS_SERVER_RESPONSE.getOrDefault(requestConfig); - boolean useHttpCompression = ClientConfigProperties.USE_HTTP_COMPRESSION.getOrDefault(requestConfig); - + private HttpEntity wrapResponseEntity(HttpEntity httpEntity) { if (httpEntity.getContentEncoding() != null) { - // http compressed response + // the algorithm of a compressed response is the one the request asked for return new CompressedEntity(httpEntity, true, CompressorStreamFactory.getSingleton()); } - // data compression - if (serverCompression && !(httpStatus == HttpStatus.SC_FORBIDDEN || httpStatus == HttpStatus.SC_UNAUTHORIZED)) { - int buffSize = ClientConfigProperties.COMPRESSION_LZ4_UNCOMPRESSED_BUF_SIZE.getOrDefault(requestConfig); - return new LZ4Entity(httpEntity, useHttpCompression, true, false, buffSize, true, lz4Factory); - } - + // a response without a content coding is not compressed: the server answers an unsupported + // Accept-Encoding, and a request that asks for no compression, with a plain body return httpEntity; } diff --git a/client-v2/src/main/java/com/clickhouse/client/api/query/QuerySettings.java b/client-v2/src/main/java/com/clickhouse/client/api/query/QuerySettings.java index 9df39f407..100694dac 100644 --- a/client-v2/src/main/java/com/clickhouse/client/api/query/QuerySettings.java +++ b/client-v2/src/main/java/com/clickhouse/client/api/query/QuerySettings.java @@ -4,6 +4,7 @@ import com.clickhouse.client.api.Client; import com.clickhouse.client.api.ClientConfigProperties; import com.clickhouse.client.api.Session; +import com.clickhouse.client.api.enums.CompressionAlgorithm; import com.clickhouse.client.api.internal.CommonSettings; import com.clickhouse.client.api.internal.ServerSettings; import com.clickhouse.client.api.internal.ValidationUtils; @@ -232,6 +233,19 @@ public TimeZone getServerTimeZone() { return (TimeZone) settings.getOption(ClientConfigProperties.SERVER_TIMEZONE.getKey()); } + /** + * Algorithm of a compressed response body of this operation. The algorithm is requested with the HTTP + * content coding of the operation, so a compressed body always uses the algorithm set here. + * {@link CompressionAlgorithm#NONE} disables compression. Defaults to the algorithm of the client. + * + * @param algorithm - algorithm of a compressed body + * @return same instance of the settings + */ + public QuerySettings compressionAlgorithm(CompressionAlgorithm algorithm) { + settings.setOption(ClientConfigProperties.COMPRESSION_ALGORITHM.getKey(), algorithm); + return this; + } + /** * Defines list of headers that should be sent with current request. The Client will use a header value * defined in {@code headers} instead of any other. diff --git a/client-v2/src/test/java/com/clickhouse/client/ClientTests.java b/client-v2/src/test/java/com/clickhouse/client/ClientTests.java index aef988818..45feadd70 100644 --- a/client-v2/src/test/java/com/clickhouse/client/ClientTests.java +++ b/client-v2/src/test/java/com/clickhouse/client/ClientTests.java @@ -333,7 +333,7 @@ public void testDefaultSettings() { Assert.assertEquals(config.get(p.getKey()), p.getDefaultValue(), "Default value doesn't match"); } } - Assert.assertEquals(config.size(), 38); // to check everything is set. Increment when new added. + Assert.assertEquals(config.size(), 39); // to check everything is set. Increment when new added. } try (Client client = new Client.Builder() @@ -368,7 +368,7 @@ public void testDefaultSettings() { .queryFormat(ClickHouseFormat.CSV.name()) .build()) { Map config = client.getConfiguration(); - Assert.assertEquals(config.size(), 39); // to check everything is set. Increment when new added. + Assert.assertEquals(config.size(), 40); // to check everything is set. Increment when new added. Assert.assertEquals(config.get(ClientConfigProperties.DATABASE.getKey()), "mydb"); Assert.assertEquals(config.get(ClientConfigProperties.MAX_EXECUTION_TIME.getKey()), "10"); Assert.assertEquals(config.get(ClientConfigProperties.COMPRESSION_LZ4_UNCOMPRESSED_BUF_SIZE.getKey()), "300000"); @@ -438,7 +438,7 @@ public void testWithOldDefaults() { Assert.assertEquals(config.get(p.getKey()), p.getDefaultValue(), "Default value doesn't match"); } } - Assert.assertEquals(config.size(), 38); // to check everything is set. Increment when new added. + Assert.assertEquals(config.size(), 39); // to check everything is set. Increment when new added. } } diff --git a/client-v2/src/test/java/com/clickhouse/client/api/ClientConfigPropertiesTest.java b/client-v2/src/test/java/com/clickhouse/client/api/ClientConfigPropertiesTest.java index 9c23ca522..43991c051 100644 --- a/client-v2/src/test/java/com/clickhouse/client/api/ClientConfigPropertiesTest.java +++ b/client-v2/src/test/java/com/clickhouse/client/api/ClientConfigPropertiesTest.java @@ -1,6 +1,7 @@ package com.clickhouse.client.api; +import com.clickhouse.client.api.enums.CompressionAlgorithm; import org.testng.Assert; import org.testng.annotations.DataProvider; import org.testng.annotations.Test; @@ -76,4 +77,42 @@ public void testParseConfigMapSanitizesSslCipherSuites() { Assert.assertEquals(parsed.get(ClientConfigProperties.SSL_CIPHER_SUITES.getKey()), Arrays.asList("TLS_AES_256_GCM_SHA384", "TLS_AES_128_GCM_SHA256")); } -} \ No newline at end of file + + @DataProvider(name = "compressionAlgorithms") + public static Object[][] compressionAlgorithms() { + return new Object[][]{ + // raw client.compression_algorithm value -> expected algorithm + {"LZ4", CompressionAlgorithm.LZ4}, + {"lz4", CompressionAlgorithm.LZ4}, + {"ZSTD", CompressionAlgorithm.ZSTD}, + {"zstd", CompressionAlgorithm.ZSTD}, + {"GZIP", CompressionAlgorithm.GZIP}, + {"gzip", CompressionAlgorithm.GZIP}, + {"NONE", CompressionAlgorithm.NONE}, + {"none", CompressionAlgorithm.NONE}, + }; + } + + @Test(groups = {"unit"}, dataProvider = "compressionAlgorithms") + public void testCompressionAlgorithmParsed(String raw, CompressionAlgorithm expected) { + Assert.assertEquals(ClientConfigProperties.COMPRESSION_ALGORITHM.parseValue(raw), expected); + + Map config = new HashMap<>(); + config.put(ClientConfigProperties.COMPRESSION_ALGORITHM.getKey(), raw); + Assert.assertEquals(ClientConfigProperties.parseConfigMap(config) + .get(ClientConfigProperties.COMPRESSION_ALGORITHM.getKey()), expected); + } + + @Test(groups = {"unit"}) + public void testCompressionAlgorithmDefaultsToLz4() { + Assert.assertEquals(ClientConfigProperties.COMPRESSION_ALGORITHM.getDefObjVal(), CompressionAlgorithm.LZ4); + Assert.assertEquals( + ClientConfigProperties.COMPRESSION_ALGORITHM.getOrDefault(Collections.emptyMap()), + CompressionAlgorithm.LZ4); + } + + @Test(groups = {"unit"}, expectedExceptions = IllegalArgumentException.class) + public void testUnknownCompressionAlgorithmRejected() { + ClientConfigProperties.COMPRESSION_ALGORITHM.parseValue("snappy"); + } +} diff --git a/client-v2/src/test/java/com/clickhouse/client/api/CompressionRequestUnitTest.java b/client-v2/src/test/java/com/clickhouse/client/api/CompressionRequestUnitTest.java new file mode 100644 index 000000000..a722e1693 --- /dev/null +++ b/client-v2/src/test/java/com/clickhouse/client/api/CompressionRequestUnitTest.java @@ -0,0 +1,179 @@ +package com.clickhouse.client.api; + +import com.clickhouse.client.api.enums.CompressionAlgorithm; +import com.clickhouse.client.api.enums.Protocol; +import com.clickhouse.client.api.insert.InsertSettings; +import com.clickhouse.client.api.query.QuerySettings; +import com.clickhouse.data.ClickHouseFormat; +import com.github.tomakehurst.wiremock.WireMockServer; +import com.github.tomakehurst.wiremock.client.WireMock; +import com.github.tomakehurst.wiremock.common.ConsoleNotifier; +import com.github.tomakehurst.wiremock.core.WireMockConfiguration; +import com.github.tomakehurst.wiremock.http.Request; +import com.github.tomakehurst.wiremock.verification.LoggedRequest; +import org.testng.Assert; +import org.testng.annotations.AfterClass; +import org.testng.annotations.BeforeClass; +import org.testng.annotations.DataProvider; +import org.testng.annotations.Test; + +import java.io.ByteArrayInputStream; +import java.nio.charset.StandardCharsets; +import java.util.List; +import java.util.function.UnaryOperator; + +public class CompressionRequestUnitTest { + + private WireMockServer server; + + @BeforeClass(groups = {"unit"}) + public void startServer() { + server = new WireMockServer(WireMockConfiguration.options() + .dynamicPort() + .notifier(new ConsoleNotifier(false))); + server.start(); + server.addStubMapping(WireMock.post(WireMock.anyUrl()) + .willReturn(WireMock.aResponse().withBody("")).build()); + } + + @AfterClass(groups = {"unit"}) + public void stopServer() { + if (server != null) { + server.stop(); + } + } + + @DataProvider(name = "responseCompressionRequests") + public static Object[][] responseCompressionRequests() { + return new Object[][]{ + // algorithm -> content coding the response is requested with (null: no compression requested) + {CompressionAlgorithm.LZ4, "lz4"}, + {CompressionAlgorithm.ZSTD, "zstd"}, + {CompressionAlgorithm.GZIP, "gzip"}, + {CompressionAlgorithm.NONE, null}, + }; + } + + @Test(groups = {"unit"}, dataProvider = "responseCompressionRequests") + public void testResponseCompressionRequestedWithContentCoding(CompressionAlgorithm algorithm, String coding) { + LoggedRequest request = runQuery(builder -> builder.compressionAlgorithm(algorithm), null); + + Assert.assertEquals(header(request, "Accept-Encoding"), coding); + // the codec of the compress=1 framing is the one of the server, so it is never requested + Assert.assertFalse(request.queryParameter("compress").isPresent(), + "compress=1 must not be requested"); + Assert.assertEquals(request.queryParameter("enable_http_compression").isPresent(), coding != null); + } + + @Test(groups = {"unit"}) + public void testAlgorithmDefaultsToLz4() { + LoggedRequest request = runQuery(builder -> builder, null); + + Assert.assertEquals(header(request, "Accept-Encoding"), "lz4"); + Assert.assertFalse(request.queryParameter("compress").isPresent()); + } + + @Test(groups = {"unit"}) + public void testOperationOverridesClientAlgorithm() { + LoggedRequest request = runQuery(builder -> builder.compressionAlgorithm(CompressionAlgorithm.LZ4), + new QuerySettings().compressionAlgorithm(CompressionAlgorithm.GZIP)); + + Assert.assertEquals(header(request, "Accept-Encoding"), "gzip"); + } + + @Test(groups = {"unit"}) + public void testOperationAcceptsAlgorithmName() { + LoggedRequest request = runQuery(builder -> builder, + (QuerySettings) new QuerySettings() + .setOption(ClientConfigProperties.COMPRESSION_ALGORITHM.getKey(), "zstd")); + + Assert.assertEquals(header(request, "Accept-Encoding"), "zstd"); + } + + @Test(groups = {"unit"}) + public void testRequestCompressedWithContentCodingOfAlgorithm() { + LoggedRequest request = runQuery(builder -> builder + .compressionAlgorithm(CompressionAlgorithm.GZIP) + .compressClientRequest(true) + .useHttpCompression(true), null); + + Assert.assertEquals(header(request, "Content-Encoding"), "gzip"); + } + + @Test(groups = {"unit"}) + public void testRequestFramingStaysLz4WithoutHttpCompression() { + LoggedRequest request = runQuery(builder -> builder + .compressionAlgorithm(CompressionAlgorithm.GZIP) + .compressClientRequest(true) + .useHttpCompression(false), null); + + // the ClickHouse framing of a request is LZ4, so the algorithm applies to the response only + Assert.assertNull(header(request, "Content-Encoding")); + Assert.assertEquals(header(request, "Accept-Encoding"), "gzip"); + Assert.assertTrue(request.queryParameter("enable_http_compression").isPresent(), + "the response is compressed with the requested content coding"); + Assert.assertFalse(request.queryParameter("compress").isPresent(), + "compress=1 must not be requested"); + Assert.assertTrue(request.queryParameter("decompress").isPresent(), + "a request compressed without http compression keeps the ClickHouse framing"); + } + + @Test(groups = {"unit"}) + public void testInsertOperationOverridesClientAlgorithm() { + server.resetRequests(); + try (Client client = newBuilder() + .compressionAlgorithm(CompressionAlgorithm.LZ4) + .compressClientRequest(true) + .useHttpCompression(true) + .build()) { + try { + client.insert("some_table", + new ByteArrayInputStream("1\n".getBytes(StandardCharsets.UTF_8)), + ClickHouseFormat.TSV, + new InsertSettings().compressionAlgorithm(CompressionAlgorithm.GZIP)).get(); + } catch (Exception e) { + // the stub answers an empty body, so only the request itself is of interest here + } + } + + LoggedRequest request = lastRequest(); + Assert.assertEquals(header(request, "Content-Encoding"), "gzip"); + Assert.assertEquals(header(request, "Accept-Encoding"), "gzip"); + } + + private LoggedRequest runQuery(UnaryOperator configure, + QuerySettings settings) { + server.resetRequests(); + try (Client client = configure.apply(newBuilder()).build()) { + try { + if (settings == null) { + client.query("SELECT 1").get(); + } else { + client.query("SELECT 1", settings).get(); + } + } catch (Exception e) { + // the stub answers an empty body, so only the request itself is of interest here + } + } + + return lastRequest(); + } + + private Client.Builder newBuilder() { + return new Client.Builder() + .addEndpoint(Protocol.HTTP, "localhost", server.port(), false) + .setUsername("default") + .setPassword("") + .retryOnFailures(); + } + + private LoggedRequest lastRequest() { + List requests = server.findAll(WireMock.postRequestedFor(WireMock.anyUrl())); + Assert.assertEquals(requests.size(), 1, "expected exactly one request"); + return requests.get(0); + } + + private static String header(Request request, String name) { + return request.containsHeader(name) ? request.getHeader(name) : null; + } +} diff --git a/client-v2/src/test/java/com/clickhouse/client/api/internal/HttpAPIClientHelperTest.java b/client-v2/src/test/java/com/clickhouse/client/api/internal/HttpAPIClientHelperTest.java index 0c7fa375a..519e2c297 100644 --- a/client-v2/src/test/java/com/clickhouse/client/api/internal/HttpAPIClientHelperTest.java +++ b/client-v2/src/test/java/com/clickhouse/client/api/internal/HttpAPIClientHelperTest.java @@ -332,28 +332,34 @@ public void testShouldRetryUsesServerExceptionFromCause(Throwable ex, boolean ex /** * A multipart body (statement parameters sent as form data) is never compressed, so the request must not * declare a content encoding - the server would try to decompress the plain body and fail with - * LZ4_DECODER_FAILED. A request that is not multipart, and response compression, keep their signalling. + * LZ4_DECODER_FAILED. A request that is not multipart, and response compression, keep their signalling: + * a compressed response is always asked for with a content coding, so its codec is the one the client + * selects, whatever the form of the request body is. */ @DataProvider(name = "requestCompressionSignalling") public static Object[][] requestCompressionSignalling() { return new Object[][] { - // clientCompression, useHttpCompression, sendParamsInBody, withParams, - // contentEncoding, acceptEncoding, decompressParam - {true, true, true, true, null, "lz4", false}, - {true, true, true, false, "lz4", "lz4", false}, // no parameters -> not a multipart request - {true, true, false, true, "lz4", "lz4", false}, - {false, true, true, true, null, "lz4", false}, - {true, false, true, true, null, null, false}, - {true, false, false, true, null, null, true}, + // clientCompression, useHttpCompression, serverCompression, sendParamsInBody, withParams, + // contentEncoding, acceptEncoding, decompressParam, httpCompressionParam + {true, true, true, true, true, null, "lz4", false, true}, + {true, true, true, true, false, "lz4", "lz4", false, true}, // no parameters -> not a multipart request + {true, true, true, false, true, "lz4", "lz4", false, true}, + {false, true, true, true, true, null, "lz4", false, true}, + {true, false, true, true, true, null, "lz4", false, true}, + {true, false, true, false, true, null, "lz4", true, true}, + // no response compression -> nothing is signalled for it; the request body keeps its own + {true, false, false, false, true, null, null, true, false}, }; } @Test(dataProvider = "requestCompressionSignalling") public void testRequestCompressionSignalling(boolean clientCompression, boolean useHttpCompression, - boolean sendParamsInBody, boolean withParams, - String expectedContentEncoding, String expectedAcceptEncoding, - boolean expectDecompressParam) { + boolean serverCompression, boolean sendParamsInBody, + boolean withParams, String expectedContentEncoding, + String expectedAcceptEncoding, boolean expectDecompressParam, + boolean expectHttpCompressionParam) { Map reqConfig = compressionConfig(clientCompression, useHttpCompression, sendParamsInBody); + reqConfig.put(ClientConfigProperties.COMPRESS_SERVER_RESPONSE.getKey(), serverCompression); if (withParams) { reqConfig.put(HttpAPIClientHelper.KEY_STATEMENT_PARAMS, Collections.singletonMap("p1", "1")); } @@ -362,18 +368,20 @@ public void testRequestCompressionSignalling(boolean clientCompression, boolean "SELECT {p1:Int32}").getDelegate(); String setup = "clientCompression=" + clientCompression + ", useHttpCompression=" + useHttpCompression - + ", sendParamsInBody=" + sendParamsInBody + ", withParams=" + withParams; + + ", serverCompression=" + serverCompression + ", sendParamsInBody=" + sendParamsInBody + + ", withParams=" + withParams; assertEquals(headerValue(req, HttpHeaders.CONTENT_ENCODING), expectedContentEncoding, "unexpected " + HttpHeaders.CONTENT_ENCODING + " for " + setup); assertEquals(req.getEntity().getContentEncoding(), expectedContentEncoding, "the request body entity must declare the same encoding as the request for " + setup); assertEquals(headerValue(req, HttpHeaders.ACCEPT_ENCODING), expectedAcceptEncoding, - "response compression signalling must not depend on the request body form"); + "response compression signalling must not depend on the request body form, for " + setup); String query = req.getRequestUri(); assertEquals(query.contains(ClickHouseHttpProto.QPARAM_DECOMPRESS + "=1"), expectDecompressParam, "unexpected " + ClickHouseHttpProto.QPARAM_DECOMPRESS + " parameter in " + query); - assertEquals(query.contains(ClickHouseHttpProto.QPARAM_ENABLE_HTTP_COMPRESSION + "=1"), useHttpCompression, + assertEquals(query.contains(ClickHouseHttpProto.QPARAM_ENABLE_HTTP_COMPRESSION + "=1"), + expectHttpCompressionParam, "unexpected " + ClickHouseHttpProto.QPARAM_ENABLE_HTTP_COMPRESSION + " parameter in " + query); } diff --git a/client-v2/src/test/java/com/clickhouse/client/query/QueryServerContentCompressionTests.java b/client-v2/src/test/java/com/clickhouse/client/query/QueryServerContentCompressionTests.java index 1001b08be..9dc385318 100644 --- a/client-v2/src/test/java/com/clickhouse/client/query/QueryServerContentCompressionTests.java +++ b/client-v2/src/test/java/com/clickhouse/client/query/QueryServerContentCompressionTests.java @@ -1,8 +1,40 @@ -package com.clickhouse.client.query; - -public class QueryServerContentCompressionTests extends QueryTests { - - QueryServerContentCompressionTests() { - super(true, false); - } -} +package com.clickhouse.client.query; + +import com.clickhouse.client.api.Client; +import com.clickhouse.client.api.enums.CompressionAlgorithm; +import com.clickhouse.client.api.query.GenericRecord; +import org.testng.Assert; +import org.testng.annotations.DataProvider; +import org.testng.annotations.Test; + +import java.util.List; + +public class QueryServerContentCompressionTests extends QueryTests { + + QueryServerContentCompressionTests() { + super(true, false); + } + + @Test(groups = {"integration"}, dataProvider = "compressionAlgorithms") + public void testQueryWithCompressionAlgorithm(CompressionAlgorithm algorithm) throws Exception { + try (Client client = newClient().compressionAlgorithm(algorithm).build()) { + List records = client.queryAll("SELECT number, toString(number) AS str " + + "FROM system.numbers LIMIT 1000"); + + Assert.assertEquals(records.size(), 1000); + Assert.assertEquals(records.get(0).getLong("number"), 0); + Assert.assertEquals(records.get(999).getLong("number"), 999); + Assert.assertEquals(records.get(999).getString("str"), "999"); + } + } + + @DataProvider(name = "compressionAlgorithms") + public Object[][] compressionAlgorithms() { + return new Object[][]{ + {CompressionAlgorithm.LZ4}, + {CompressionAlgorithm.ZSTD}, + {CompressionAlgorithm.GZIP}, + {CompressionAlgorithm.NONE}, + }; + } +} diff --git a/docs/features.md b/docs/features.md index 78df8b01d..69112495f 100644 --- a/docs/features.md +++ b/docs/features.md @@ -32,7 +32,7 @@ This document lists stable, user-visible behavior in `client-v2` and `jdbc-v2` t - Session handling: Supports client-wide and per-operation HTTP sessions, operation-level session overrides, runtime updates of client `session_id`, and server-side session validation through `session_check`. - Metadata discovery: Loads table schemas from table names or queries and allows schema registration for typed read/write operations. - Server information loading: Can refresh server version, current user, and server time zone information. -- Compression support: Supports response compression, ClickHouse LZ4 request/response compression, HTTP content compression, and caller-supplied precompressed insert bodies. +- Compression support: Supports response compression, ClickHouse LZ4 request compression, HTTP content compression, and caller-supplied precompressed insert bodies. The algorithm of a compressed body is selected with `client.compression_algorithm` (`LZ4` by default, also `ZSTD`, `GZIP` and `NONE`) and is requested with the HTTP content coding of the operation, so a compressed response uses the algorithm the client asked for on every server version. - Retry behavior: Can retry failed operations for configured failure causes and retry limits. - Client-side request cancellation: `Client.cancelTransportRequest(String queryId)` aborts the in-flight HTTP request and its IO for the operation started with the given query id. It requires the caller to set the query id in operation settings, is best-effort (it cancels client-side IO but the result is not guaranteed), and does not issue a server-side `KILL QUERY` - the server stops the query on its own once the client disconnects. A cancelled operation that is being retried stops instead of issuing another request, also when the cancellation lands between two attempts (for example from `DataStreamWriter#onRetry()`). - Metrics and observability: Exposes client/server operation metrics and optionally integrates connection-pool gauges with Micrometer. diff --git a/docs/releases/0_11_0.md b/docs/releases/0_11_0.md index 23d7ff473..31d4b0006 100644 --- a/docs/releases/0_11_0.md +++ b/docs/releases/0_11_0.md @@ -2,6 +2,38 @@ # Migration Guide +## CLIENT-V2: A Compressed Response Uses the Algorithm the Client Asks For + +A compressed response was requested with the `compress=1` framing of the HTTP interface, whose codec the server +chooses on its own. ClickHouse `26.9` changed that codec from `LZ4` to `ZSTD(3)`; the framed output follows the +built-in default codec and no setting overrides it, so a client that expects `LZ4` cannot read the response of a +`26.9` server at all. + +The client now requests a response with the HTTP content coding of the algorithm it will decode, so the algorithm of +a compressed body is always the one the client asked for: + +- The new property `client.compression_algorithm` (builder method `Client.Builder#compressionAlgorithm`) selects the + algorithm out of `LZ4` (default), `ZSTD`, `GZIP` and `NONE`. Both the name and the content-coding token of an + algorithm are accepted, in any case. +- The default is `LZ4`, so an application that does not set the property keeps reading `LZ4` on every server version. +- `ZSTD` needs `com.github.luben:zstd-jni` on the classpath, which the client does not bring: the dependency of + `clickhouse-jdbc` stays `provided`, so packaging is unchanged and an application that selects `ZSTD` declares the + dependency itself. +- `NONE` disables compression of the request and the response, whatever `compress` and `decompress` are set to. + +What to check in an application: + +- **A user profile that forbids setting changes.** A client that reads a compressed response now also sends + `enable_http_compression=1`, which the server rejects for a profile with `readonly = 1`. Use `readonly = 2`, which + allows setting changes, or set `client.compression_algorithm` to `NONE`. +- **Code that inspects the response encoding.** A compressed response now carries `Content-Encoding` with the + requested coding instead of the `compress=1` framing of ClickHouse. + +The algorithm of a compressed *request* body is unchanged: it follows `client.compression_algorithm` only together +with `Client.Builder#useHttpCompression`, and the ClickHouse framing of a request compressed without it is always +`LZ4`. + + ## CLIENT-V2: `OperationMetrics` Has a Single Constructor `com.clickhouse.client.api.metrics.OperationMetrics` now has one constructor,