We Keep Coding

Spring Boot application (using mybatis) stopped connecting to mysql after introducing CA-signed certs

About a week ago, a spring boot application in a dev environment (AWS EC2 instance), which had until that point successfully connected to MySql using SSL, suddenly stopped connecting. Here's a selection from the unhelpful stack trace:
### Error querying database. Cause: com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
### The error may exist in com/domain/app/data/mybatis/mappers/SomeMapper.xml
### The error may involve com.domain.app.data.mybatis.mappers.SomeMapper.getUserByUsername
### The error occurred while executing a query
### Cause: com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:149)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:140)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectOne(DefaultSqlSession.java:76)
at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:87)
at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:93)
at com.sun.proxy.$Proxy91.getUserByUsername(Unknown Source)
at com.domain.app.security.SomeService.loadUserByUsername(SomeService.java:32)
at org.springframework.security.authentication.dao.DaoAuthenticationProvider.retrieveUser(DaoAuthenticationProvider.java:108)
... 96 more
Caused by: com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure
...
Caused by: java.net.ConnectException: Connection timed out (Connection timed out)
Some facts:
The exact same app build works when run from my local machine (with identical db connection info). Misconfigured application code presumably ruled out.
mysql resides on a separate EC2 instance, and I can connect to mysql from a CLI on the 'faulty' EC2 instance. Security/ACL problems presumably ruled out.
Connection string is of this form: jdbc:mysql://x.x.x.x:3306/db_name?useSSL=true&verifyServerCertificate=true&requireSSL=true&autoReconnect=true,{db username},{db password}. Again, this string works when used from my own machine.
About two weeks ago, CA-signed ssl certs were added into the spring boot app within a java keystore. The same certs were not added to mysql, which is using self-generated and self-signed certs. There's a slight, if barely believable chance, that the app hasn't been tested in the faulty environment since - though I'm pretty sure I have used the environment since then.
My current thinking is that maybe the new certs in the dev application are somehow colliding with mysql's self-generated and self-signed certs, in a way that might not happen locally on my machine. But I don't know where to begin validating or testing this theory.
I've been troubleshooting this for about 7 hours, including googling the heck out of it, and reading many SO responses to related questions. Apologies if this has been answered in some way that I don't understand.
Can any of you point me either to some answers or troubleshooting pointers?

How to Grant a Java Web App Access to a Digital Ocean Droplet

I'm hosting mysql server on a Digital Ocean droplet, and I am trying to figure out how to grant a Spring Boot web application access to the droplet, so that it can connect to the database. I configured the droplet so that it can only be reached via an ssh tunnel (i.e.: I disabled password authentication), but the database server itself can be connected with a username and password.
I know how to connect to the database using Connector/J, configuring datasources, and so forth. However, the extra security layer of Digital Ocean is new to me, and I'm not sure how to approach this problem.
EDIT: When I run the application and try to hit an endpoint, I get the following error:
The error occurred while executing a query
### Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is java.sql.SQLException: null, message from server: "Host '<my network hostname>' is not allowed to connect to this MySQL server"] with root cause

So, I was mistaken in thinking that the authentication issue was with the Digital Ocean droplet. As the error message (see question above) indicates, the Spring Boot application was able to get to the remote mysql server, and the authentication issue occurred there. It turns out that the mysql user I was attempting to connect with could only be used on localhost.
So, I created a new mysql user and tied it to my local machine's public IP address, and that solved the issue. For details on how to accomplish this, please read the following answer:
Host 'xxx.xx.xxx.xxx' is not allowed to connect to this MySQL server

Unable to acquire JDBC Connection on integration test when using Docker bridge network

When I run maven test locally is passed. But got this error when I run it on CI server.
Error Message
Could not open JPA EntityManager for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
Stacktrace
org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
Caused by: org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
Caused by: java.net.UnknownHostException: mysql
When running local test, they all passed, maven test default setting provided by IntelliJ IDEA is used.
Since the error complains about database connection, so I checked by Jenkins Audit to Database Plugin. Connection Successful!
The connection parameter in my application.properties also corresponds to this
spring.datasource.url=jdbc:mysql://mysql:3306/database?useLegacyDatetimeCode=false&serverTimezone=Asia/Shanghai
spring.datasource.username=root
spring.datasource.password=password
spring.datasource.maxActive=5
The MySQL in the URL is the MySQL docker container name. If change it with localhost or private IP in docker container inspect mysql the error message is the same, while the Stacktrace is a little different on last two lines.
for localhost
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
Caused by: java.net.ConnectException: Connection refused (Connection refused)
for private IP
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
Caused by: java.net.SocketTimeoutException: connect timed out
The different I think is the host in URL, localhost is used for the local test.
While the Jenkins server used Docker bridge network.
The container status is:
docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
NAMES
51ea7c7864a4 mysql:5.7 "docker-entrypoint.s…" 19 hours ago Up 19 hours 0.0.0.0:3306->3306/tcp mysql
de364f7b5eaf maven:3-jdk-8 "/usr/local/bin/mvn-…" 21 hours ago Up 21 hours
optimistic_stallman
a6545591e358 jenkinsci/blueocean "/sbin/tini -- /usr/…" 43 hours ago Up 43 hours 0.0.0.0:50000->50000/tcp, 0.0.0.0:2048->8080/tcp frosty_cray
When I run the JUnit test in IntelliJ, it fails sometimes on the local environment. The error log is like:
Caused by: org.h2.jdbc.JdbcSQLException: Schema "DATABASE" not found; SQL statement:
TRUNCATE TABLE database.data_log
I have searched the issue, it's said h2 database use upper case by default.
After run maven test, this issue will go if run JUnit test in IDE again. But this should be not related to the root cause.
Search on the error message, find some similar question but with different nested exception:
Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException
SpingREST: Could not open JPA EntityManager for transaction; nested exception is org.hiberna
Could not open JPA EntityManager for transaction; org.hibernate.exception.GenericJDBCException: Could not open connection
Could not open JPA EntityManager for transaction in spring
All of them is about nested exception is javax.persistence.PersistenceException
But nested exception is org.hibernate.exception.JDBCConnectionException: is my situation.
Read Connect Java to a MySQL database
however since that plugin connects OK, means the connection from Jenkins container to MySQL container is fine.
Summarise:
1. local test with maven passed
2. Jenkins plugin connect to MySQL success
3. Integration test fails when run from Jenkins
4. local test environment is WIN10 64bit; Jenkins run in docker container on Ubuntu 16.04 64bit server, with MySQL 5.7 container connects to the same bridge network.

Thanks to #rohit-thomas. We narrow down the question to something related to the host of the URL.
The simple answer is changing the host of the JDBC URL in the spring boot application.properties to the docker host IP address. From
spring.datasource.url=jdbc:mysql://mysql:3306/database?
to
spring.datasource.url=jdbc:mysql://172.17.0.1:3306/database?
From inside of a Docker container, how do I connect to the localhost of the machine?
This post also helps as the final solution.
ip addr show docker0
4: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
...
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
...
My conclusion is:
The Jenkins container builds from an image is able to communicate with the MySQL container with its container name or private address on docker bridge network. However, as the application built by the Jenkins is not able to do that.
Since the MySQL container port has bound to the host machine, the application could communicate with MySQL container through the host port.
If the conclusion is wrong, comments are welcome.

Some things you can check that might help you resolve this issue.
In application.properties try to use docker host IP address.
from
spring.datasource.url = jdbc:mysql://mysql:3306/DATABASE_URI_PATH
to
spring.datasource.url = jdbc:mysql://192.168.99.100:33060/DATABASE_URI_PATH
Note:
You will need to map your IP and port when you do docker run or ports in docker file.And use the same docker network among your containers.
Verify if you server app can reach your mysql or vice versa. Go inside the docker container and try to ping.

JDBC Connection Error - Cannot create PoolableConnectionFactory

I have a jar that connects to a DB. The first time I connect to the machine, the jar runs fine, connects to the DB and queries the tables properly etc. However, when I try a second and third time etc, the jar begins to run but gets stuck when connecting to the DB and then I receive the following error:
org.springframework.jdbc.CannotGetJdbcConnectionException: Could not
get JDBC Connection; nested exception is java.sql.SQLException: Cannot
create PoolableConnectionFactory (IO Error: Connection reset)
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:571)
at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88)
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:331)
I checked and there is no problem with the firewall. The strange part is that it works once but then stops working.

Make sure your database server is accepting connections from the Confluence server TCP address and user is authorized to connect.

Mysql communication error in tomcat even with out using any mysql in application

I am Getting following error when ever i am starting my tomcat
com.mysql.jdbc.CommunicationsException: Communications link failure due to underlying exception:
** BEGIN NESTED EXCEPTION **
java.net.ConnectException
MESSAGE: Connection refused
I am getting this error even with out using any application in my tomcat.
Could you please help me why tomcat is looking for mysql connect at the time of start
Regards

May be you have configured a Tomcat Datasource Resource in your server.xml or context.xml in your ${CATALINA_BASE}/conf directory. So, whenever tomcat starts it tries to create the connection pool for your datasource and it is not able to communicate to the db server host. This is highly possible if you don't use mysql in your application