I keep receiving connectivity issues when connecting my Spring project to a MySQL server. I have tried various tricks that I have found online but to no avail. I need help in order to escape this quandary. I'm placing my Spring settings below and the stack trace as well.
Spring Settings
spring.datasource.tomcat.initialize=true
spring.dataSource.tomcat.url=jdbc:mysql://nope/canttellyou?
autoReconnect=true&useSSL=false&zeroDateTimeBehavior=convertToNull
spring.dataSource.tomcat.username=nope
spring.dataSource.tomcat.password=sorry
spring.dataSource.tomcat.driver-class-name=com.mysql.jdbc.Driver
spring.datasource.tomcat.test-on-borrow=true
spring.datasource.tomcat.validation-query=SELECT 1
spring.datasource.tomcat.log-validation-errors=true
StackTrace
stackTrace: org.springframework.dao.RecoverableDataAccessException:
PreparedStatementCallback; SQL [select catname from animals]; The last
packet successfully received from the server was 52,962,310
milliseconds ago. The last packet sent successfully to the server was
52,962,310 milliseconds ago. is longer than the server configured value
of 'wait_timeout'. You should consider either expiring and/or testing
connection validity before use in your application, increasing the
server configured values for client timeouts, or using the Connector/J
connection property 'autoReconnect=true' to avoid this problem.; nested
exception is com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
The last packet successfully received from the server was 52,962,310
milliseconds ago. The last packet sent successfully to the server was
52,962,310 milliseconds ago. is longer than the server configured value
of 'wait_timeout'. You should consider either expiring and/or testing
connection validity before use in your application, increasing the
server configured values for client timeouts, or using the Connector/J
connection property 'autoReconnect=true' to avoid this problem.
I've been having the same problem. I think the configuration setting we want to turn on is:
testWhileIdle
Here's the related tomcat connection pool docs
I won't know until tomorrow whether or not this worked for me.
Related
I am using Dropwizard with the Hibernate bundle. After letting it run for some hours/days, any attempt to access the database will result in a
com.mysql.cj.exceptions.CJCommunicationsException: The last packet successfully received from the server was 33.304.281 milliseconds ago. Th
e last packet sent successfully to the server was 33.304.283 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should
consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeout
s, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
[...]
Causing: java.sql.SQLNonTransientConnectionException: No operations allowed after connection closed.
The config looks something like this:
database:
# the name of your JDBC driver
driverClass: com.mysql.cj.jdbc.Driver
# the username
user: ...
# the password
password: ...
# the JDBC URL
url: jdbc:mysql://localhost/my_database
# any properties specific to your JDBC driver:
properties:
charSet: utf8mb4
hibernate.hbm2ddl.auto: update
# the maximum amount of time to wait on an empty pool before throwing an exception
maxWaitForConnection: 1s
# the SQL query to run when validating a connection s liveness
validationQuery: "/* MyService Health Check */ SELECT 1"
# the timeout before a connection validation queries fail
validationQueryTimeout: 3s
# the minimum number of connections to keep open
minSize: 8
# the maximum number of connections to keep open
maxSize: 32
# whether or not idle connections should be validated
checkConnectionWhileIdle: true
checkConnectionOnBorrow: true
# the amount of time to sleep between runs of the idle connection validation, abandoned cleaner and idle pool resizing
evictionInterval: 10s
# the minimum amount of time an connection must sit idle in the pool before it is eligible for eviction
minIdleTime: 1 minute
maxConnectionAge: 14400s
The hibernate bundle is instantiated from the configuration just like in the DW docs: https://www.dropwizard.io/en/latest/manual/hibernate.html#configuration
The DB wait_timeout is 28800s. I don't understand how the connections do not get either removed due to old age (the time since last package suggested by the error messages is greater than 14400s) or after validation because of
checkConnectionOnBorrow: true
dropwizard Version: 2.0.18
mysql-connector-java Version: 8.0.23
I am out of ideas on how to approach this please help.
The problem should be fixed if you change the database.url property to contain autoReconnect=true.
So in your configuration put:
database:
url: jdbc:mysql://localhost/my_database?autoReconnect=true
In documentation of Connector/J:
autoReconnect
Should the driver try to re-establish stale and/or dead connections?
If enabled the driver will throw an exception for queries issued on a
stale or dead connection, which belong to the current transaction, but
will attempt reconnect before the next query issued on the connection
in a new transaction. The use of this feature is not recommended,
because it has side effects related to session state and data
consistency when applications don't handle SQLExceptions properly, and
is only designed to be used when you are unable to configure your
application to handle SQLExceptions resulting from dead and stale
connections properly. Alternatively, as a last option, investigate
setting the MySQL server variable 'wait_timeout' to a high value,
rather than the default of 8 hours.
Reference: https://dev.mysql.com/doc/connector-j/8.0/en/connector-j-connp-props-high-availability-and-clustering.html
About a week ago, a spring boot application in a dev environment (AWS EC2 instance), which had until that point successfully connected to MySql using SSL, suddenly stopped connecting. Here's a selection from the unhelpful stack trace:
### Error querying database. Cause: com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
### The error may exist in com/domain/app/data/mybatis/mappers/SomeMapper.xml
### The error may involve com.domain.app.data.mybatis.mappers.SomeMapper.getUserByUsername
### The error occurred while executing a query
### Cause: com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:149)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:140)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectOne(DefaultSqlSession.java:76)
at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:87)
at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:93)
at com.sun.proxy.$Proxy91.getUserByUsername(Unknown Source)
at com.domain.app.security.SomeService.loadUserByUsername(SomeService.java:32)
at org.springframework.security.authentication.dao.DaoAuthenticationProvider.retrieveUser(DaoAuthenticationProvider.java:108)
... 96 more
Caused by: com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure
...
Caused by: java.net.ConnectException: Connection timed out (Connection timed out)
Some facts:
The exact same app build works when run from my local machine (with identical db connection info). Misconfigured application code presumably ruled out.
mysql resides on a separate EC2 instance, and I can connect to mysql from a CLI on the 'faulty' EC2 instance. Security/ACL problems presumably ruled out.
Connection string is of this form: jdbc:mysql://x.x.x.x:3306/db_name?useSSL=true&verifyServerCertificate=true&requireSSL=true&autoReconnect=true,{db username},{db password}. Again, this string works when used from my own machine.
About two weeks ago, CA-signed ssl certs were added into the spring boot app within a java keystore. The same certs were not added to mysql, which is using self-generated and self-signed certs. There's a slight, if barely believable chance, that the app hasn't been tested in the faulty environment since - though I'm pretty sure I have used the environment since then.
My current thinking is that maybe the new certs in the dev application are somehow colliding with mysql's self-generated and self-signed certs, in a way that might not happen locally on my machine. But I don't know where to begin validating or testing this theory.
I've been troubleshooting this for about 7 hours, including googling the heck out of it, and reading many SO responses to related questions. Apologies if this has been answered in some way that I don't understand.
Can any of you point me either to some answers or troubleshooting pointers?
{
"message": "Internal Server Error: Database error: The last packet successfully received from the server was 89,722,963 milliseconds ago. The last packet sent successfully to the server was 89,722,964 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem."
}
Setup: Glassfish 3x, Java EE, EclipseLink.
jdbc connection pool config (from domain.xml):
<jdbc-connection-pool validation-table-name="dummy" pool-resize-quantity="5" max-pool-size="300" datasource-classname="com.mysql.jdbc.jdbc2.optional.MysqlXADataSource" wrap-jdbc-objects="false" res-type="javax.sql.XADataSource" steady-pool-size="25" name="mysql_bazaarify_rootPool" is-connection-validation-required="true" non-transactional-connections="true" validate-atmost-once-period-in-seconds="60">
<property name="URL" value="jdbc:mysql://demodb.mydb.com:3306/database?zeroDateTimeBehavior=convertToNull&useUnicode=yes"></property>
Getting CommunicationsException infrequently for different queries.
I have configured for connections to be validated before use with 60 seconds of validity.
wait_time on sql server is 8 hours.
Still I am seeing below error in my logs (~50 times a day)
Exception Trace:
Internal Exception:
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No operations allowed after connection closed.
Error Code: 0
....
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
The last packet successfully received from the server was 56,688,227 milliseconds ago.
The last packet sent successfully to the server was 56,688,227 milliseconds ago. is longer than the server configured value of 'wait_timeout'.
You should consider either expiring and/or testing connection validity before use in your application,
increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
....
Caused by: java.net.SocketException: Connection timed out
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3832)
... 140 more
My question is, when I have set for connection to be validated (every 60 seconds), is it possible that I get a stale/broken connection from connection pool which says that the last communication with mysql was (>60seconds) ago?
Shouldn't the connection be validated until a healthy one is returned from the pool? Are these exceptions normal with the settings I have configured?
P.S.
I observed that in all such failures, the milliseconds ago value reported is always >8 hours. Does that help deduce anything?
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
The last packet successfully received from the server was 38,160,127
milliseconds ago. The last packet sent successfully to the server was
38,160,127 milliseconds ago. is longer than the server configured
value of 'wait_timeout'. You should consider either expiring and/or
testing connection validity before use in your application, increasing
the server configured values for client timeouts, or using the
Connector/J connection property 'autoReconnect=true' to avoid this
problem.
1) As per the solutions given by TOMCAT I have increased the value of wait_timeout but, it is not working.
2) How to set autoReconnect=true in my URL I don't know. I tried with all the possibilities but it does not work.
URL:
<property
name="url"
value="jdbc:mysql://127.0.01:3306/PaxWeb3?zeroDateTimeBehavior" />
I have tried using:
value="jdbc:mysql://127.0.01:3306/PaxWeb3?zeroDateTimeBehavior&autoReconnect=true"
and
value="jdbc:mysql://127.0.01:3306/PaxWeb3?zeroDateTimeBehavior;&autoReconnect=true"
but they won't work either.