We Keep Coding

Spring Boot application (using mybatis) stopped connecting to mysql after introducing CA-signed certs

About a week ago, a spring boot application in a dev environment (AWS EC2 instance), which had until that point successfully connected to MySql using SSL, suddenly stopped connecting. Here's a selection from the unhelpful stack trace:
### Error querying database. Cause: com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
### The error may exist in com/domain/app/data/mybatis/mappers/SomeMapper.xml
### The error may involve com.domain.app.data.mybatis.mappers.SomeMapper.getUserByUsername
### The error occurred while executing a query
### Cause: com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:149)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:140)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectOne(DefaultSqlSession.java:76)
at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:87)
at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:93)
at com.sun.proxy.$Proxy91.getUserByUsername(Unknown Source)
at com.domain.app.security.SomeService.loadUserByUsername(SomeService.java:32)
at org.springframework.security.authentication.dao.DaoAuthenticationProvider.retrieveUser(DaoAuthenticationProvider.java:108)
... 96 more
Caused by: com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure
...
Caused by: java.net.ConnectException: Connection timed out (Connection timed out)
Some facts:
The exact same app build works when run from my local machine (with identical db connection info). Misconfigured application code presumably ruled out.
mysql resides on a separate EC2 instance, and I can connect to mysql from a CLI on the 'faulty' EC2 instance. Security/ACL problems presumably ruled out.
Connection string is of this form: jdbc:mysql://x.x.x.x:3306/db_name?useSSL=true&verifyServerCertificate=true&requireSSL=true&autoReconnect=true,{db username},{db password}. Again, this string works when used from my own machine.
About two weeks ago, CA-signed ssl certs were added into the spring boot app within a java keystore. The same certs were not added to mysql, which is using self-generated and self-signed certs. There's a slight, if barely believable chance, that the app hasn't been tested in the faulty environment since - though I'm pretty sure I have used the environment since then.
My current thinking is that maybe the new certs in the dev application are somehow colliding with mysql's self-generated and self-signed certs, in a way that might not happen locally on my machine. But I don't know where to begin validating or testing this theory.
I've been troubleshooting this for about 7 hours, including googling the heck out of it, and reading many SO responses to related questions. Apologies if this has been answered in some way that I don't understand.
Can any of you point me either to some answers or troubleshooting pointers?

Communications link failure issue

I hosted my first Java EE web site but my mySQL connections are timing out after few minutes and I am getting this error message:
HTTP Status 500 - com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
type Exception report
message com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
description The server encountered an internal error that prevented it from fulfilling this request.
exception
dao.DAOException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
Last packet sent to the server was 0 ms ago.
dao.OffreDaoImpl.lister(OffreDaoImpl.java:95)
filters.PrechargementFilter.doFilter(PrechargementFilter.java:96)
org.apache.catalina.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:108)
root cause
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
Last packet sent to the server was 0 ms ago.
sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:525)
com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1074)
com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2985)
com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2871)
com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3414)
com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1936)
com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2060)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2542)
com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1734)
com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1885)
com.jolbox.bonecp.PreparedStatementHandle.executeQuery(PreparedStatementHandle.java:172)
dao.OffreDaoImpl.lister(OffreDaoImpl.java:90)
filters.PrechargementFilter.doFilter(PrechargementFilter.java:96)
org.apache.catalina.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:108)
root cause
java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:2431)
com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2882)
com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2871)
com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3414)
com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1936)
com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2060)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2542)
com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1734)
com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1885)
com.jolbox.bonecp.PreparedStatementHandle.executeQuery(PreparedStatementHandle.java:172)
dao.OffreDaoImpl.lister(OffreDaoImpl.java:90)
filters.PrechargementFilter.doFilter(PrechargementFilter.java:96)
org.apache.catalina.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:108)
I asked my hoster to increase the timeout value and here is his response:
Please accept our sincerest apologizes for misunderstanding.
Unfortunately due to shared hosting environment your request could not be >accomplished - it will affect other customers, that share server resources >for you. (10 min is max value for this parameter).
Since you strongly need some features - we can only suggest you considering >to move to VPS or dedicated server where you will have full admin >privileges and can do whatever you want without affecting of another >customers.
As a solution, we would like to suggest you to add in your jdbc connection >string following command:
&autoReconnect=true&failOverReadOnly=false&maxReconnects=10
In this case, after timeout, you will be reconnected again.
So I edited my config.properties file:
url = jdbc:mysql://localhost:3306/myDatabase&autoReconnect=true&failOverReadOnly=false&maxReconnects=10
driver = com.mysql.jdbc.Driver
username = xxxxxx
password = xxxxxx
But I am still facing the same issue.

Yes I am using localhost even when I host the website. But I tried to replace localhost string by the server address IP in the URL connection and I got this error message:
Caused by: java.sql.SQLException: Unable to open a test connection to the given database. JDBC url = jdbc:mysql://xxx.xxx.xxx.xx:3306/database_db?autoReconnect=true&failOverReadOnly=false&maxReconnects=10, username = user_name. Terminating connection pool. Original Exception: ------
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server. Attempted reconnect 10 times. Giving up.
So I noted the issue to my hoster and here is his reply:
Dear Customer,
Kindly use localhost instead IP address, as for your string, we actually asking you to not maintain connections as long as you wish. The thing is we have hard limit on opened connections overall = 300 and 50 per user.
And as this is shared server these connections are being allocated on demand to each user, and if your account consumes 50 connections it is 1/6 of all connections available.

Migrating from MySql: MariaDB server closing client connections unexpectedly

We in the process of migrating from MySql to MariaDB due to licensing/commercial usage reasons.
We have successfully replaced the MySql connector jar with MariaDB client jar (first change) and are now trying to replace MySql server with MariaDB server without changing the data files.
All our applications run perfectly for about 8-12 hours after which we see the following exception:
org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Cannot open connection
Caused by:
org.hibernate.exception.JDBCConnectionException: Cannot open connection
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:74)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:29)
at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:426)
at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:144)
at org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:119)
at org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:57)
at org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1326)
at org.springframework.orm.hibernate3.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:494)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:315)
at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:257)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:102)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:209)
at $Proxy4.getMessageCountByStatus(Unknown Source)
at com.onmobile.cmfweb.monitoring.CmfMessagesMonitor.getMessageCounts(CmfMessagesMonitor.java:56)
at sun.reflect.GeneratedMethodAccessor625.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:270)
at com.onmobile.cmfshare.MethodInvockingBean.invoke(MethodInvockingBean.java:28)
at org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean$MethodInvokingJob.executeInternal(MethodInvokingJobDetailFactoryBean.java:212)
at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:79)
at org.quartz.core.JobRunShell.run(JobRunShell.java:203)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)
Caused by: java.sql.SQLNonTransientConnectionException: Could not connect to localhost:3306: unexpected end of stream, read 0bytes from 4
at org.mariadb.jdbc.internal.SQLExceptionMapper.get(SQLExceptionMapper.java:136)
at org.mariadb.jdbc.internal.SQLExceptionMapper.throwException(SQLExceptionMapper.java:106)
at org.mariadb.jdbc.Driver.connect(Driver.java:114)
at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:37)
at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:290)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:840)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95)
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:544)
at org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.getConnection(LocalDataSourceConnectionProvider.java:81)
at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:423)
... 21 more
Caused by: org.mariadb.jdbc.internal.common.QueryException: Could not connect to localhost:3306: unexpected end of stream, read 0bytes from 4
at org.mariadb.jdbc.internal.mysql.MySQLProtocol.connect(MySQLProtocol.java:509)
at org.mariadb.jdbc.internal.mysql.MySQLProtocol.connect(MySQLProtocol.java:669)
at org.mariadb.jdbc.internal.mysql.MySQLProtocol.<init>(MySQLProtocol.java:264)
at org.mariadb.jdbc.Driver.connect(Driver.java:110)
... 28 more
Caused by: java.io.EOFException: unexpected end of stream, read 0bytes from 4
at org.mariadb.jdbc.internal.common.packet.buffer.ReadUtil.readFully(ReadUtil.java:84)
at org.mariadb.jdbc.internal.common.packet.buffer.ReadUtil.readFully(ReadUtil.java:92)
at org.mariadb.jdbc.internal.common.packet.RawPacket.nextPacket(RawPacket.java:77)
at org.mariadb.jdbc.internal.common.packet.SyncPacketFetcher.getRawPacket(SyncPacketFetcher.java:67)
at org.mariadb.jdbc.internal.mysql.MySQLProtocol.connect(MySQLProtocol.java:467)
... 31 more
The exceptions disappear if MySql server is used again.
From what I have debugged so far MariaDB server is closing the client connections for some reason. I initially suspected of idle connections but we are using Hibernate configurations like 'testOnBorrow' so it shouldn't be the case.
Can anyone help us to find out the cause for this and help us fix the problem? Is there any specific configuration in MariaDB that I should be doing?
I've even run our applications by increasing the value of the param 'interactive_timeout' in the MariaDB server but it did not help.
Btw, we are using Spring-Hibernate and pooling our connections using Commons-pool jar.
Any kind of help will be deeply appreciated.

We encountered the same problem recently, due to a combination of incorrect parameters.
This error is caused by your web instance trying to use a connection that is no longer a valid one.
This can be resolved by making sure the following parameters are correct:
You have a validationQuery configured for your database connection. i.e. validationQuery="SELECT 1" in case of mariadb, in the server configuration.
You have the wait_timeout set to a reasonable value. 8 hours keepAlive seems a bit optimistic, we are now using wait_timeout=180
Make sure the validationInterval, which you can also set in the server configuration (server.xml in case of a tomcat setup), is set to a value that is lower than the wait_timeout value. I reduced the wait_timeout to 15s in one case, with the validationInterval value being higher than that, which caused the error to still appear at times.
Now it is set to validationInterval=60, combined with the wait_timeout=180 which should catch any broken connections in time.

You can try to increase 'wait_timeout' instead of 'interactive_timeout'. 8-12 hours until error correlates quite well with the default value of wait_timeout which is 10 hours. JDBC driver does not set interactive client flag when connecting, so change to 'interactive_timeout' might have no effect. I also replied to the cross-post in Maria KB ;)

According to the findings in this article you could get away with two options:
You should try turning innodb_file_per_table off
or...
Increasing the open file limits, this tells you the current value on yout machine $ ulimit -n, should you be running on high spec machines

I also had an error like this one Could not connect to localhost:3306: unexpected end of stream, read 0 bytes from 4
My architecture was an webapp of Tomcat that was inserting data into MariaDB.
I was using the connector version 1.5.1RC!
I switched to the 1.3.1 connector version and now runs fine!

I also faced this issue. The issue goes away with lowering the wait_timeout.
SHOW SESSION VARIABLES LIKE 'wait_timeout'; ->> would be 8 hours. which is huge.
SET session wait_timeout=300; ->> this should fix the issue.

Look at
SHOW VARIABLES LIKE 'max_connections';
SHOW GLOBAL STATUS LIKE '%connect%';
Compare the my.cnf values between the two systems; you may see a difference in things like above that make a difference. If the problem is still not obvious, please provide those values for further discussion.

I think that the problem is with MariaDB Connector/J. Try to use a stable one like MariaDB Connector/J 1.2.3 for example. Also, study Failover behaviour with Basic failover when autoReconnect is set to true or Standard failover.

Java OracleDB connection taking too long the first time

I'm having a problem when connecting to an Oracle database, it takes a long time (about ~5 minutes) and it sends the below shown exception. Most of the time, after the first error, the next connections for the same process work correctly.
It is a RHEL 6 machine, with two different network interfaces and ip addresses.
NOTE:
I am not using an url like: "jdbc:oracle:thin:#xxxx:yyy, it is actually: "jdbc:oracle:thin:#xxxx:yyyy:zzz.
The SID is not missing, sorry for that :(
This is roughly what I've isolated:
bin/java -classpath ojdbc6_g.jar -Djavax.net.debug=all
-Djava.util.logging.config.file=logging.properties
Class.forName ("oracle.jdbc.OracleDriver")
DriverManager.getConnection("jdbc:oracle:thin:#xxxx:yyyy", "aaaa", "bbbb")
Error StackTrace:
java.sql.SQLRecoverableException: IO Error: Connection reset
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:533)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:557)
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:233)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:29)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:556)
at java.sql.DriverManager.getConnection(DriverManager.java:579)
at java.sql.DriverManager.getConnection(DriverManager.java:221)
at test.jdbc.Main(Test.java:120)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
at oracle.net.ns.DataPacket.send(DataPacket.java:248)
at oracle.net.ns.NetOutputStream.flush(NetOutputStream.java:227)
at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:309)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:257)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:182)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:99)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:121)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:77)
at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1173)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:309)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:200)
at oracle.jdbc.driver.T4CTTIoauthenticate.doOSESSKEY(T4CTTIoauthenticate.java:404)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:430)
... 35 more
There's a very verbose log of what happens over here: http://pastebin.com/MMFKU26z
The line that says GET STUCK HERE represents the 5 minute waiting time

You are probably running into an issue with the Oracle JDBC driver which uses a blocking random number generator by default on Linux. Try running Java with the following argument:
-Djava.security.egd=file:/dev/./urandom
Alternatively you could start up a daemon to feed the random number generator. The Linux "rngd" daemon is an example.
Sources:
http://www.usn-it.de/index.php/2009/02/20/oracle-11g-jdbc-driver-hangs-blocked-by-devrandom-entropy-pool-empty/
http://bugs.java.com/view_bug.do;?bug_id=6521844
https://bugs.openjdk.java.net/browse/JDK-6202721
https://community.oracle.com/message/3701989
Oracle JDBC intermittent Connection Issue
Oracle getConnection slow
How to solve performance problem with Java SecureRandom?

Looking at your connection string, it seems that Oracle service name or SID is missing. The connection string should look like "jdbc:oracle:thin:#xxxx:yyyy/zzz", where zzz is the SID. You may also want to have a look at the answer to this question.

IOException: Network adapter could not establish the connection [duplicate]

This question already has answers here:
The Network Adapter could not establish the connection when connecting with Oracle DB
(4 answers)
Closed 6 years ago.
We started to have this "Network adapter could not establish the connection" problem in our production environment from time to time, I did a lot of google and went through some questions here but still didn't get the solution.
Here are some information about our environment:
We are using RAC with 2 oracle instances (version 10.2.0.4).
We have several app servers (JBoss AS5) running on top of the RAC.
The connection url in oracle-ds.xml is "jdbc:oracle:thin:#(DESCRIPTION =(ADDRESS_LIST =(ADDRESS = (PROTOCOL = TCP)(HOST = db1_vip)(PORT = 1521))(ADDRESS = (PROTOCOL = TCP)(HOST=db2_vip)(PORT = 1521))(LOAD_BALANCE = yes))(CONNECT_DATA=(SERVER = DEDICATED)(SERVICE_NAME = G1db)(FAILOVER_MODE =(TYPE =SELECT)(METHOD = BASIC)(RETRIES = 120)(DELAY = 5))))"
We added db1_vip/db2_vip as well the actual ip of 2 oracle instances to the /etc/hosts of both app servers
There IS firewall between oracle instances and app servers, but we opened the 1521 port on both oracle instances.
We've run following tests to check if there is any problem on network or oracle TNS listener:
run ping from app servers to both oracle instances, everything works fine without any packet loss.
telnet the 1521 port from app servers to both oracle instances, nothing wrong.
tnsping also works fine.
We checked listener log but found nothing valuable.
The most weird thing is that this error happens around 10 times an hour on one app server, but only happens 1 or 2 times a day on another app server.
Can anyone shed some light on this error?
Thanks
[EDIT4]: We found timeout problem when tnsping one oracle instance, so we changed the jdbc url to connect only one instance and found the IOException never happened again, so we think the problem is related with database and the DBA team will continue look into this.
[EDIT3]: We did following tries:
disabled the firewall between app servers and database servers
Use ip instead of host name
use wireshark to check if there is any tcp packet problem.
none of them succeed, someone please help~
[EDIT]: the error stack trace:
java.sql.SQLException: Io Exception: The Network Adapter could not establish the connection
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:146)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:255)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:387)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:414)
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:165)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:35)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:801)
at org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory.getLocalManagedConnection(LocalManagedConnectionFactory.java:207)
... 5 more

The error
java.sql.SQLException: Io Exception: The Network Adapter could not
establish the connection
has nothing to do with Java/JDBC.
This is the error you get when there's something wrong with your network setup. My guess, firewall problem. Can you disable it and see if the error is still there?

edit the connection and test it although your tnsping found!, if use virtualbox (in adapter check permit all), in my case, the error was when i use the ip, instead of use localhost