SSL Exception From HTTP Apache Client Library

[meilz381]

Describe the bug
A clear and concise description of what the bug is.
There is an issue with the HTTP client inside the client library where the request fails every time when we try to retrieve certain elements from ADX (paged response from a stored query result) when it is done with the same parameters (the same pages fail).

To Reproduce
Steps to reproduce the behavior:
Create a big stored query result (~1GB) and retrieve the result paged. What seems to appear random, for certain requests the EntityUtils.toString(entity) method invocation in the file "com.microsoft.azure.kusto.data.Utils" throws an IOException, because of an SSL exception (see screenshot) about wrong padding, that comes from the HTTP Apache Client library that is used. In the end, it results in a DataClientException. This request did/will never succeed even with retries.

Expected behavior
With other HTTP clients (OkHttpClient, AsyncHttpClient), cURL and tools like Postman or Insomnia I'm able to do the same request and get the result, without any exception (see screenshot). So maybe a change to another HTTP client library could be a solution.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: MacOS/Linux
• SDK Version: 5.0.2/3

Additional context
Add any other context about the problem here.

[yogilad]

Hi,

This is a known issue that started with the introduction of TLS 1.3.
Please open a support ticket with the above description.
Ops team can temporarily disable TLS 1.3.

[andreassiegel]

Ops team can temporarily disable TLS 1.3.

@yogilad Is temporarily disabling TLS 1.3 the final solution?

Are there any plans to update the library so that it will be successfully working with TLS 1.3, i.e., is the known issue going to be fixed?

[mar1u50]

I'm using SDK Version: 4.0.4 and I'm hit by the same problem.
From time to time I get this exception and if I retry sometimes it works.

Error in post request:Incorrect inner plaintext: no content type
com.microsoft.azure.kusto.data.exceptions.DataClientException: Error in post request:Incorrect inner plaintext: no content type
	at com.microsoft.azure.kusto.data.Utils.post(Utils.java:100)
	at com.microsoft.azure.kusto.data.ClientImpl.executeToJsonResult(ClientImpl.java:156)
	at com.microsoft.azure.kusto.data.ClientImpl.execute(ClientImpl.java:105)

....
Caused by: javax.net.ssl.SSLException: Incorrect inner plaintext: no content type
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:133)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:378)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:316)
	at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:134)
	at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1513)
	at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1484)
	at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1069)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
	at org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:261)
	at org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:222)
	at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:183)
	at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135)
	at java.base/java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:242)
	at java.base/java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
	at java.base/java.util.zip.GZIPInputStream.read(GZIPInputStream.java:117)
	at org.apache.http.client.entity.LazyDecompressingInputStream.read(LazyDecompressingInputStream.java:70)
	at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:270)
	at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:313)
	at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:188)
	at java.base/java.io.InputStreamReader.read(InputStreamReader.java:177)
	at java.base/java.io.Reader.read(Reader.java:250)
	at org.apache.http.util.EntityUtils.toString(EntityUtils.java:227)
	at org.apache.http.util.EntityUtils.toString(EntityUtils.java:308)
	at com.microsoft.azure.kusto.data.Utils.post(Utils.java:87)
	... 20 common frames omitted
Caused by: javax.crypto.BadPaddingException: Incorrect inner plaintext: no content type
	at java.base/sun.security.ssl.SSLCipher$T13GcmReadCipherGenerator$GcmReadCipher.decrypt(SSLCipher.java:1954)
	at java.base/sun.security.ssl.SSLSocketInputRecord.decodeInputRecord(SSLSocketInputRecord.java:264)
	at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:181)
	at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111)
	... 42 common frames omitted

Temporarily disabling TLS 1.3 is the only solution?

Would upgrading to the latest SDK (5.0.5) fix the issue?

[mar1u50]

I upgraded kusto to 5.0.5 and the exception slightly changed:

com.microsoft.azure.kusto.data.exceptions.DataServiceException: IOException in post request:Incorrect inner plaintext: no content type
	at com.microsoft.azure.kusto.data.http.HttpPostUtils.post(HttpPostUtils.java:77)
	at com.microsoft.azure.kusto.data.ClientImpl.lambda$executeToJsonResult$1(ClientImpl.java:224)
	at com.microsoft.azure.kusto.data.instrumentation.MonitoredActivity.invoke(MonitoredActivity.java:33)
	at com.microsoft.azure.kusto.data.instrumentation.MonitoredActivity.invoke(MonitoredActivity.java:26)
	at com.microsoft.azure.kusto.data.ClientImpl.executeToJsonResult(ClientImpl.java:223)
	at com.microsoft.azure.kusto.data.ClientImpl.executeImpl(ClientImpl.java:173)
	at com.microsoft.azure.kusto.data.ClientImpl.lambda$execute$0(ClientImpl.java:122)
	at com.microsoft.azure.kusto.data.instrumentation.MonitoredActivity.invoke(MonitoredActivity.java:33)
	at com.microsoft.azure.kusto.data.ClientImpl.execute(ClientImpl.java:121)
	at com.microsoft.azure.kusto.data.ClientImpl.execute(ClientImpl.java:116)

I don't see the SSLException anymore but I still get "Incorrect inner plaintext: no content".

Is temporarily disabling TLS 1.3 the final solution?

[andreassiegel]

Is temporarily disabling TLS 1.3 the final solution?

@mar1u50 It seems to be. We never heard anything else.

Also be aware that disabling TLS 1.3 was everything but straightforward. It took the Azure support a quite long time. We didn’t have the impression it’s as simple as indicated by @yogilad.

In particular, the whole process did not feel as if the support is familiar with such requests. For us, it involved some back and forth and discussions about the used JDK version as this was supposed to be the reason for the issue (it wasn’t), for example.

Overall this wasn’t a good experience, and not getting any further comment or reply here is really disappointing.

[mar1u50]

I was able to fix the problem on the client side. I run my application with -Djdk.tls.client.protocols=TLSv1.2

[mar1u50]

@yogilad, is there a reason not to keep this issue open? I tested with Kusto 5.0.5, and the problem still exists. I don't think it is good to keep using TLS 1.2 long-term.

[AsafMah]

Right now, downgrading to TLS v1.2 is the recommended mitigation.

We will keep this issue open and update when a we'll have a permanent resolution.

[The-Funk]

Since we're getting rid of the Apache entity references in the next version, I think that should resolve this bug.

[georgebanasios]

Just an update on this:

The issue seems to have been fixed with the removal of Apache references. I tested a few large queries (over the query limits) in the latest version and no longer encountered the SSL exception that occurred previously.

[ohadbitt]

Just an update on this:

The issue seems to have been fixed with the removal of Apache references. I tested a few large queries (over the query limits) in the latest version and no longer encountered the SSL exception that occurred previously.

We cannot really say that for certain - its not easily reproduced ... did you reproduce the problem before (with old client) and see that now its gone? I myself cannot reproduce it anymore locally regardless of the SDK version

[georgebanasios]

You are right, I tested it with the Azure client with a query that used to fail with the old client. If I manage to reproduce it with the old client I'll give an update.

[andreassiegel]

@ohadbitt @georgebanasios I know it's hard to prove the actual absence of a bug, however, I can state what we did in the very beginning when the issue was observed:

As described by @meilz381, we extracted the plain HTTP request that was sent by the Kusto SDK via the built-in Apache HTTP client. The request worked fine with all the clients we tested. We also tried the Apache HTTP client independently and ran into an issue directly.

Thus, if the Apache HTTP client has now been replaced in the SDK, we could be quite confident to assume the bug is also gone now.