[Note: I originally posted this problem/solution to Stackoverflow.com]
Problem:
I am getting the following error when connecting to a SQL Server database using version the Microsoft JDBC Driver:
|
1 |
com.microsoft.sqlserver.jdbc.SQLServerException: The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption. Error: "SQL Server returned an incomplete response. The connection has been closed. ClientConnectionId:98d0b6f4-f3ca-4683-939e-7c0a0fca5931". |
We recently upgraded our applications from Java 6 & Java 7 to Java 8. All systems running Java are running SUSE Linux Enterprise Server 11 (x86_64), VERSION = 11, PATCHLEVEL = 3.
Here are the facts I have collected with a Java program that I wrote which simply sequentially opens and closes 1,000 database connections.
- Connections are dropped with this error about 5%-10% of the time. The error DOES NOT occur on every connection.
- The problem ONLY occurs with Java 8. I ran the same program on Java 7 and the problem is not reproducible. This is consistent with our experience in production prior to upgrading. We’ve had zero problems running under Java 7 in production.
- The problem DOES NOT occur on all of our Linux servers running Java 8, it only occurs on some of them. This is perplexing to me, but when I run the same test program on the same version of the Linux JVM (1.8.0_60, 64 bit) on different Linux instances, the problem does not occur on one of the Linux instances, but the problem does occur on others. The Linux instances are running the same version of SUSE and they are at the same patch level.
- The problem occurs when connecting to BOTH SQL Server 2008 and SQL Server 2014 servers/databases.
- The problem occurs regardless if I am using the 4.0 version of the SQL Server JDBC driver or the newer 4.1 version of the driver.
The thing that makes my observations unique on this compared to others on the web is that although the problem happens ONLY on Java 8, I cannot get the problem to occur on one of the seemingly identical Linux servers that is running the same Java 8 JVM. Other folks have seen this problem on earlier versions of Java as well, but that has not been our experience.
Any input, suggestions, or observations you may have are appreciated.
Solution
I turned on SSL logging in the Java 8 JVM on a Linux instance which reproduces the problem. SSL logging is turned on using -Djavax.net.debug=ssl:handshake:verbose. This revealed some useful information.
The workaround that we are using in production and has proven to work for us is to set this parameter on the JVM:
-Djdk.tls.client.protocols=TLSv1
If you want more details, please read on.
On a server where the problem can be reproduced (again, only 5-10% of the time), I observed the following:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 |
*** ClientHello, TLSv1.2 --- 8<-- SNIP ----- main, WRITE: TLSv1.2 Handshake, length = 195 main, READ: TLSv1.2 Handshake, length = 1130 *** ServerHello, TLSv1.2 --- 8<-- SNIP ----- %% Initialized: [Session-79, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256] ** TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 --- 8<-- SNIP ----- Algorithm: [SHA1withRSA] --- 8<-- SNIP ----- *** Diffie-Hellman ServerKeyExchange --- 8<-- SNIP ----- *** ServerHelloDone *** ClientKeyExchange, DH --- 8<-- SNIP ----- main, WRITE: TLSv1.2 Handshake, length = 133 --- 8<-- SNIP ----- main, WRITE: TLSv1.2 Change Cipher Spec, length = 1 *** Finished verify_data: { 108, 116, 29, 115, 13, 26, 154, 198, 17, 125, 114, 166 } *** main, WRITE: TLSv1.2 Handshake, length = 40 main, called close() main, called closeInternal(true) main, SEND TLSv1.2 ALERT: warning, description = close_notify main, WRITE: TLSv1.2 Alert, length = 26 main, called closeSocket(true) main, waiting for close_notify or alert: state 5 main, received EOFException: ignored main, called closeInternal(false) main, close invoked again; state = 5 main, handling exception: java.io.IOException: SQL Server returned an incomplete response. |
Notice that TLSv1.2 is selected by the database server and used in this exchange. I’ve observed that, when connections fail from the problematic linux service, TLSv1.2 is ALWAYS the level which was selected. However, connections do not ALWAYS fail when TLSv1.2 is used. They only fail 5-10% of the time.
Now here is an exchange from a server that does NOT have the problem. Everything else is equal. I.e., connecting to the same database, same version of the JVM (Java 1.8.0_60), same JDBC driver, etc. Notice that, here, TLSv1 is selected by the database server instead of TLSv1.2 as in the faulty server’s case.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
*** ClientHello, TLSv1.2 --- 8<-- SNIP ----- main, WRITE: TLSv1.2 Handshake, length = 207 main, READ: TLSv1 Handshake, length = 604 *** ServerHello, TLSv1 --- 8<-- SNIP ----- Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA --- 8<-- SNIP ----- %% Initialized: [Session-79, TLS_RSA_WITH_AES_128_CBC_SHA] ** TLS_RSA_WITH_AES_128_CBC_SHA --- 8<-- SNIP ----- Algorithm: [SHA1withRSA] --- 8<-- SNIP ----- *** *** ServerHelloDone *** ClientKeyExchange, RSA PreMasterSecret, TLSv1 --- 8<-- SNIP ----- main, WRITE: TLSv1 Handshake, length = 134 main, WRITE: TLSv1 Change Cipher Spec, length = 1 *** Finished verify_data: { 26, 155, 166, 89, 229, 193, 126, 39, 103, 206, 126, 21 } *** main, WRITE: TLSv1 Handshake, length = 48 main, READ: TLSv1 Change Cipher Spec, length = 1 main, READ: TLSv1 Handshake, length = 48 *** Finished |
So, when TLSv1 is negotiated between the Linux JVM and the SQL Server, connections are ALWAYS successful. When TLSv1.2 is negotiated, we get sporadic connection failures.
(Note: Java 7 (1.7.0_51) always negotiates TLSv1, which is why the problem never occurred for us with a Java 7 JVM.)
The open questions we still have are:
- WHY is that the same Java 8 JVM run from 2 different Linux servers will always negotiate TLSv1, but when connecting from another Linux server it always negotiates TLSv1.2.
- And also why are TLSv1.2 negotiated connections successful most, but not all, of the time on that server?
Resources:
Java 8 , JCE Unlimited Strength Policy and SSL Handshake over TLS
http://blogs.msdn.com/b/saponsqlserver/archive/2013/05/10/analyzing-jdbc-connection-issues.aspx
https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html#descPhase2
https://blogs.oracle.com/java-platform-group/entry/java_8_will_use_tls