We Keep Coding

Spring Boot application (using mybatis) stopped connecting to mysql after introducing CA-signed certs

About a week ago, a spring boot application in a dev environment (AWS EC2 instance), which had until that point successfully connected to MySql using SSL, suddenly stopped connecting. Here's a selection from the unhelpful stack trace:
### Error querying database. Cause: com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
### The error may exist in com/domain/app/data/mybatis/mappers/SomeMapper.xml
### The error may involve com.domain.app.data.mybatis.mappers.SomeMapper.getUserByUsername
### The error occurred while executing a query
### Cause: com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:149)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:140)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectOne(DefaultSqlSession.java:76)
at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:87)
at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:93)
at com.sun.proxy.$Proxy91.getUserByUsername(Unknown Source)
at com.domain.app.security.SomeService.loadUserByUsername(SomeService.java:32)
at org.springframework.security.authentication.dao.DaoAuthenticationProvider.retrieveUser(DaoAuthenticationProvider.java:108)
... 96 more
Caused by: com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure
...
Caused by: java.net.ConnectException: Connection timed out (Connection timed out)
Some facts:
The exact same app build works when run from my local machine (with identical db connection info). Misconfigured application code presumably ruled out.
mysql resides on a separate EC2 instance, and I can connect to mysql from a CLI on the 'faulty' EC2 instance. Security/ACL problems presumably ruled out.
Connection string is of this form: jdbc:mysql://x.x.x.x:3306/db_name?useSSL=true&verifyServerCertificate=true&requireSSL=true&autoReconnect=true,{db username},{db password}. Again, this string works when used from my own machine.
About two weeks ago, CA-signed ssl certs were added into the spring boot app within a java keystore. The same certs were not added to mysql, which is using self-generated and self-signed certs. There's a slight, if barely believable chance, that the app hasn't been tested in the faulty environment since - though I'm pretty sure I have used the environment since then.
My current thinking is that maybe the new certs in the dev application are somehow colliding with mysql's self-generated and self-signed certs, in a way that might not happen locally on my machine. But I don't know where to begin validating or testing this theory.
I've been troubleshooting this for about 7 hours, including googling the heck out of it, and reading many SO responses to related questions. Apologies if this has been answered in some way that I don't understand.
Can any of you point me either to some answers or troubleshooting pointers?

MySQLNonTransientConnectionException/CommunicationsException when validation is true in domain.xml?

Setup: Glassfish 3x, Java EE, EclipseLink.
jdbc connection pool config (from domain.xml):
<jdbc-connection-pool validation-table-name="dummy" pool-resize-quantity="5" max-pool-size="300" datasource-classname="com.mysql.jdbc.jdbc2.optional.MysqlXADataSource" wrap-jdbc-objects="false" res-type="javax.sql.XADataSource" steady-pool-size="25" name="mysql_bazaarify_rootPool" is-connection-validation-required="true" non-transactional-connections="true" validate-atmost-once-period-in-seconds="60">
<property name="URL" value="jdbc:mysql://demodb.mydb.com:3306/database?zeroDateTimeBehavior=convertToNull&useUnicode=yes"></property>
Getting CommunicationsException infrequently for different queries.
I have configured for connections to be validated before use with 60 seconds of validity.
wait_time on sql server is 8 hours.
Still I am seeing below error in my logs (~50 times a day)
Exception Trace:
Internal Exception:
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No operations allowed after connection closed.
Error Code: 0
....
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
The last packet successfully received from the server was 56,688,227 milliseconds ago.
The last packet sent successfully to the server was 56,688,227 milliseconds ago. is longer than the server configured value of 'wait_timeout'.
You should consider either expiring and/or testing connection validity before use in your application,
increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
....
Caused by: java.net.SocketException: Connection timed out
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3832)
... 140 more
My question is, when I have set for connection to be validated (every 60 seconds), is it possible that I get a stale/broken connection from connection pool which says that the last communication with mysql was (>60seconds) ago?
Shouldn't the connection be validated until a healthy one is returned from the pool? Are these exceptions normal with the settings I have configured?
P.S.
I observed that in all such failures, the milliseconds ago value reported is always >8 hours. Does that help deduce anything?

Java exception connecting to MySQL

SOLVED
The latest version of MySQL JDBC driver evidently attempts to create an SSL connection. The app which WORKED did NOT define javax.net.ssl.keystore (and other) values, so (I assume) it reverts to a non-SSL connection. The app which FAILED did define all of the keystore/certificate parameters and then the connection attempt failed - still not sure of exactly why, if someone could enlighten would appreciate. The exception thrown could also be more informative.
Anyway, adding "&useSSL=false&allowPublicKeyRetrieval=true" to the connection parameters works for both applications. (Although not a very good setup for any production app, can use for this POC!)
============= Original Posting Follows =========
This has had me stumped for a few days! AWS server (86_64), Fedora 29, Java 8 (1.8.0_201-b09), MySQL - all upgraded to the latest release. MySQL is running on the same server. Driver: com.mysql.cj.jdbc.Driver
Connection params: jdbc:mysql://172.31.10.222:3306?user=xxxx&password=xxxx. (IP is the AWS private IP for this server.)
Have 2 different utilities - separate 'main' routines, but all in the same JAR file. The first one works perfectly, the second one aborts with the stack pasted below. This is consistent over dozens of trials with minor variations.
The different main routines both use the 'connectToDB' method shown in the stack trace. Instrumented this to log all parameters and the class name of the driver in both cases and verified that they are identical.
Have already tried/verified:
Multiple Google searches of the same error
MySql is running and OK (connect thru another terminal window w no problem),
Same JAR used with all local code
Classpath is identical
Running under the same Linux user, using the same MySQL user/password
Verified the IP above (172.31.10.222)
Also changed the IP to 'localhost', 127.0.0.1, and a symbolic name from /etc/hosts
Added -Xmx2G to make sure not an OOM issue
These are 2 different 'main' methods, but don't do much more than load some properties (from the same property files) before trying to connect to the DB. Then they execute the same connectToDB routine.
Something in the context must be different! Hoping that someone here can at least provide some hints on where to look!
Thanks
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at com.mysql.cj.jdbc.exceptions.SQLError.createCommunicationsException(SQLError.java:174)
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:64)
at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:835)
at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:455)
at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:240)
at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:207)
at spc.tools.spcConnection.connectToDB(spcConnection.java:692)
at spc.tools.spcConnection.connectToLocalDB(spcConnection.java:1418)
at spc.tools.spcConnection.loadAllConnectionsOK(spcConnection.java:1112)
at spc.tools.spcConnection.initOneTime(spcConnection.java:1012)
at spc.tools.spcConnection.init(spcConnection.java:194)
at spc.ops.spcServerBase.environmentOK(spcServerBase.java:367)
at spc.ops.spcServerBase.environmentOK(spcServerBase.java:342)
at spc.run.daemon.spcDaemon.main(spcDaemon.java:1082)
Caused by: com.mysql.cj.exceptions.CJCommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:61)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:105)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:151)
at com.mysql.cj.exceptions.ExceptionFactory.createCommunicationsException(ExceptionFactory.java:167)
at com.mysql.cj.protocol.a.NativeProtocol.negotiateSSLConnection(NativeProtocol.java:351)
at com.mysql.cj.protocol.a.NativeAuthenticationProvider.negotiateSSLConnection(NativeAuthenticationProvider.java:777)
at com.mysql.cj.protocol.a.NativeAuthenticationProvider.proceedHandshakeWithPluggableAuthentication(NativeAuthenticationProvider.java:486)
at com.mysql.cj.protocol.a.NativeAuthenticationProvider.connect(NativeAuthenticationProvider.java:202)
at com.mysql.cj.protocol.a.NativeProtocol.connect(NativeProtocol.java:1442)
at com.mysql.cj.NativeSession.connect(NativeSession.java:165)
at com.mysql.cj.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:955)
at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:825)
... 11 more
Caused by: java.net.SocketException: Broken pipe (Write failed)
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
at sun.security.ssl.OutputRecord.writeBuffer(OutputRecord.java:431)
at sun.security.ssl.OutputRecord.write(OutputRecord.java:417)
at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:879)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:850)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:720)
at sun.security.ssl.Handshaker.sendChangeCipherSpec(Handshaker.java:1144)
at sun.security.ssl.ClientHandshaker.sendChangeCipherAndFinish(ClientHandshaker.java:1280)
at sun.security.ssl.ClientHandshaker.serverHelloDone(ClientHandshaker.java:1190)
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:369)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
at sun.security.ssl.Handshaker.process_record(Handshaker.java:965)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
at com.mysql.cj.protocol.ExportControlled.performTlsHandshake(ExportControlled.java:315)
====================================================== New Information =====
Here are the key lines from the connectToDB method (responding to comment below):
Driver dvr = (Driver)Class.forName(info.sDBDriver).newInstance();
if(dvr!=null) conn = dvr.connect(accessString(), null);
verified the expected driver and that accessString() returns the 'jdbc:mysql:...' string as above. The exception is thrown executing the 2nd line above.
Since the original post, put in code to run a 'netstat -ln' immediately before these lines and post the result to the log. This shows an active listener for port 3306.

It would be great if you can post connectToDB() method.
According to this sentence " The first one works perfectly, the second one aborts with the stack pasted below.", you must have closed the db connection in the first connectToDB() call.
So that second call is broken as the DB connection was terminated by first call.
The good way to setup is to use dependency injection for database object.
Hope this helps!

Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException

when i run my code, i get this exception:
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1121)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:357)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2482)
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2519)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2304)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:346)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriverManager(DriverManagerDataSource.java:153)
at org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriver(DriverManagerDataSource.java:144)
at org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnectionFromDriver(AbstractDriverBasedDataSource.java:155)
at org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnection(AbstractDriverBasedDataSource.java:120)
at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:139)
at org.hibernate.tool.hbm2ddl.SuppliedConnectionProviderConnectionHelper.prepare(SuppliedConnectionProviderConnectionHelper.java:51)
at org.hibernate.tool.hbm2ddl.DatabaseExporter.<init>(DatabaseExporter.java:52)
at org.hibernate.tool.hbm2ddl.SchemaExport.execute(SchemaExport.java:367)
at org.hibernate.tool.hbm2ddl.SchemaExport.create(SchemaExport.java:304)
at org.hibernate.tool.hbm2ddl.SchemaExport.create(SchemaExport.java:293)
at org.hibernate.internal.SessionFactoryImpl.<init>(SessionFactoryImpl.java:517)
at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1857)
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
Aug 25, 2015 8:23:45 PM com.sun.jersey.spi.container.ContainerResponse mapMappableContainerException
SCHWERWIEGEND: The RuntimeException could not be mapped to a response, re-throwing to the HTTP container
org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException: org.hibernate.exception.JDBCConnectionException: Could not open connection
at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:431)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:457)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:276)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
at com.sun.proxy.$Proxy76.create(Unknown Source)
I don't know where is the problem. But yesterday i worked fine.
my database.properties:
jdbc.driverClassName=com.mysql.jdbc.Driver
jdbc.url=jdbc:mysql://localhost:3306/streaming
jdbc.username=root
jdbc.password=password
hibernate.dialect=org.hibernate.dialect.MySQLDialect
hibernate.showsql=true
hibernate.hbm2ddl.auto=create
It is possible for me to run mysql directly from command line. Where is the problem? Was is wrong.
I use spring 4 and hibernate with mysql as database language.
Has someone any ideas?
Thank.

There is CommunicationsException: Communications link failure.
If you get a SQLException: Connection refused or Connection timed out or a MySQL specific
CommunicationsException: Communications link failure
then it means that the DB isn't reachable at all. This can have one or more of the following causes:
IP address or hostname in JDBC URL is wrong.
Hostname in JDBC URL is not recognized by local DNS server.
Port number is missing or wrong in JDBC URL.
DB server is down.
DB server doesn't accept TCP/IP connections.
DB server has run out of connections.
Something in between Java and DB is blocking connections, e.g. a firewall or proxy.
To solve the one or the other, follow the following advices:
Verify and test them with ping.
Refresh DNS or use IP address in JDBC URL instead.
Verify it based on my.cnf of MySQL DB.
Start the DB.
Verify if mysqld is started without the --skip-networking option.
Restart the DB and fix your code accordingly that it closes connections in finally.
Disable firewall and/or configure firewall/proxy to allow/forward the port.
Source: More Details

I see that #Amit Tamrakar have good answer but I want to add:
if you make a lot of requests to db, (I mean connections), may be that application (some app thread) just wait for use free connection from app db connection pool, but throws timeout exception.
.h.engine.jdbc.spi.SqlExceptionHelper : HikariPool-1 - Connection is not available, request timed out after 30005ms.
Could not open JPA EntityManager for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
You can try to increase db connection pool in application config file, for example:
spring.datasource.hikari.maximum-pool-size=20
You can set size which you need. This config in app by default is 8.
And we need remember that each db have own free db connection size which we don't need exceed.
I mean if db have 5 free connection and you want to use 10 connections in your app then we will receive so exceptions.

DB is down.
You must start the DB server and check connectivity with either telnet or mysql command line client first.

I was not unable to figure out the exact cause, but I realized that it was because of some thread pools since I was able to connect to the db using the terminal.
for checking the connectivity you can use the below command.
psql -h hostname -U username -d database
so restarting the server worked for me.

Cannot create PoolableConnectionFactory error while starting application in Bluemix

Once I deploy my Java Application to IBM Bluemix, it can't create the initial database connection due to this error:
Building new Hibernate SessionFactory 2015-04-11 14:24:04 [main] ERROR
o.h.util.JDBCExceptionReporter - Cannot create
PoolableConnectionFactory (Communications link failure The last packet
sent successfully to the server was 0 milliseconds ago. The driver has
not received any packets from the server.)
I tried to set the "hibernate.current_session_context_class" to "thread" (which solved the problem when I deployed the same app to Google Application Engine) but that didn't help with Websphere Liberty Profile:
<prop key="hibernate.current_session_context_class" >
thread
</prop>
Any ideas?
UPDATE 1 --- This is the actual root cause. Bluemix offers the ClearDB MySQL service but its pipeline cannot connect to it :(
Building new Hibernate SessionFactory 2015-04-11 15:04:49 [main] ERROR
o.h.util.JDBCExceptionReporter - Communications link failure
The last packet sent successfully to the server was 0 milliseconds
ago. The driver has not received any packets from the server.
UPDATE 2 --- Even the Experimental MySQL DB is also off limits if there's any attempt to connect from the pipeline machines:
o.h.util.JDBCExceptionReporter - Cannot create
PoolableConnectionFactory (Communications link failure
The last packet sent successfully to the server was 0 milliseconds
ago. The driver has not received any packets from the server.)
org.hibernate.exception.GenericJDBCException: Cannot open connection
at
org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:140)
at org.hibe at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create
PoolableConnectionFactory (Communications link failure
The last packet sent successfully to the server was 0 milliseconds
ago. The driver has not received any packets from the server.) at
org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1549)
at
org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1388)
at
org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
at
org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.getConnection(LocalDataSourceConnectionProvider.java:81)
at
org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
... 40 more Caused by:
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
Communications link failure
Please let me know if you have any ideas.

Ah ok. So this is happening because the Liberty buildpack autowires in JDBC connections for you during deployment to Bluemix. For the unit tests you will need to provide this connection info to your app.