I have a Spring MVC project deployed on a TomCat server and I keep getting an issue with the connection to the database. After several hours, when a user tries to login they will be met with a 500 error and this message:
HTTP Status 500 - Request processing failed; nested exception is org.springframework.dao.RecoverableDataAccessException: The last packet successfully received from the server was 75,026,904 milliseconds ago. The last packet sent successfully to the server was 75,031,521 milliseconds ago. is longer than the server configured value of 'wait_timeout'.
I believe the issue is because TomCat thinks the connection to MySQL is still open, however MySQL closes the connection after 8 hours. Once you refresh the page after getting this error, everything works fine.
My question is, is there a way to have the NamedParameterJdbcTemplate, that I am using to query the database, open and close a connection each time it is used rather than maintaining one persistent connection? Or is there a better way to solve this problem?
Perhaps setting autoReconnect=true? I am trying this now, but it will take several hours before I know if it worked.
Assuming you're using Tomcat's connection pooling, you can set your pool to testOnBorrow="true" and validationQuery="SELECT 1 FROM my_test_table". This way the connection will be checked before it's checked out by a thread. If it's closed, another one would be opened and given to the thread servicing your request. You can read the documentation of the options provided by Tomcat here.
Related
I have been getting these exceptions for a number of weeks with no solution...
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
The last packet successfully received from the server was 179,695,604 milliseconds ago.
The last packet sent successfully to the server was 179,695,604 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem
So I have altered my application's Context.xml to have the autoReconnect=true tag set with my databases for connection pooling in Tomcat 7. I have even set wait_timeout to infinity in the context file.
What am I missing? Is this a common problem? It seems to have a small amount of information around on the net, but when I follow these guides, the same thing happens the next day after a period of inactivity.
The more I use the server, the less this happens. I think it is expiration of the pool connections but how can I stop them expiring if wait_timeout is failing? Any ideas on how to diagnose the problem or config files?
I was facing a similar problem, autoReconnect=true throws the CommunicationsException exception, but then creates a new connection to mysql. So the next request would succeed. This behavior would continue to occur and the first request after an idle time would fail.
To add to what Alex has already answered, I added the following params to my JDBC connection string and I don't see the error any more.
testOnBorrow="true" validationQuery="SELECT 1" validationInterval="60000"
The description of testOnBorrow sufficiently explains it. And the good thing is I do not have to make any changes in my code.
References:
https://tomcat.apache.org/tomcat-7.0-doc/jdbc-pool.html
http://www.tomcatexpert.com/blog/2010/04/01/configuring-jdbc-pool-high-concurrency
The MySQL Connector/J documentation says about autoReconnect:
If enabled the driver will throw an exception for a queries issued on
a stale or dead connection, which belong to the current transaction,
but will attempt reconnect before the next query issued on the
connection in a new transaction.
Meaning you will still get exceptions.
Connectors such as JDBI get around this by adding optional test queries on idle, borrow from pool and return to pool. Perhaps you could add something to your own JDBC connection wrapper to do the same. Alternatively, follow the documentation for autoReconnect and properly catch SQLExceptions arising from dead/stale connections.
There are further useful references on this answer using DBCP and c3p0
I run a java programm and try to use it with a hosted DB, I run the Vserver ubuntu 12.04 with Mysql myself, full root-access.
I changed the my.cnf to have more resources.
When I start the application it is fast, hardly any difference to a local database.
My problem is that after a while of inactivity the program freezes probably because the connection dropped.
There is no entry in any errorlog. If I kill the application and restart it, it is working again, nothing else but kill works, it is a linuxPC.
I used ?autoReconnect=true but I am not sure this is correct, the tables are innodb.
Does anyone have an idea how to avoid the connection to drop or how to make sure a reconnection is made?
PS [17.12.2015]
?autoReconnect=true was removed
today I got some details after a long wait
com.openbravo.basic.BasicException:
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet successfully received from the server was 2.435.471 milliseconds ago. The last packet sent successfully to the server was 959.832 milliseconds ago.
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
Communications link failure
The last packet successfully received from the server was 2.435.471 milliseconds ago. The last packet sent successfully to the server was 959.832 milliseconds ago.
java.net.SocketException:
Is this what happens when wait_timeout is too short?
After the "error" the program worked again!
Your question lacks specific info, but I guess what is happen. MySQL Server has a parameter called wait_timeout (See official doc).
When you have a connection that exceed that timeout, MySQL will close it, and if you don't manage SQLExceptions properly, your application will have problems.
You can try to increase wait_timeout or review your connection code to manage exceptions, but both are workarounds.
I wouldn't recommend you to relay on autoReconnect as an alternative for any problem, instead it would be better to encapsulate connection management into business logic in order to manage open/close connection every time you need. Maybe connection pooling can help you.
I mean, when you call a business method from you UI (it doesn't mind if it is web, ws, desktop or whatever) you have to manage open connection and start transaction, (and other cross cutting concerns as authorization, audit, log, ...). During all businesses logic, control possible exceptions, commit or rollback and free resources.
If you post some code, you will get more specific answers.
Hope it helps!
autoReconnect is dangerous for InnoDB. When the connection is lost in the middle of a transaction the previous actions in the transaction are rolled back. But the code proceeds to run as if the transaction continues. This can lead to subsequent writes not being consistent with the rolled back data.
You would be better off recognizing the lost connection and restarting the transaction.
I know this sort of question has been asked for a few times, I read some of them but did not get any smarter.
My Java application is connecting to a database server via JDBC through a SSH Tunnel. The tunnel is opened once at beginning. When starting I opened the database connection everytime it is used. Due to changes in the app I needed the connection opened on startup and decided to keep it open until my application is closed. When I close the app I sometimes, not always, get following error:
- Could not retrieve transation read-only status server
java.sql.SQLException: Could not retrieve transation read-only status server
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet successfully received from the server was 369.901 milliseconds ago. The last packet sent successfully to the server was 8 milliseconds ago.
Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
Could this be because of the always open database connection? I test the application only for short times. It will run on 4 computers all day long. Can I expect this error more often then? The connection is used every few minutes, so it should be more performant to keep it open, but maybe during break it is not used for half an hour.
What would you recommend me to do? Always reopen the connection or keep it like it is and find a workaround when I get this error? Do you maybe have another idea why this error appears?
Just ask if you need more error log, database code or whatever you need.
Thanks!
I'm going to say yes, your issue is probably due to the fact that you have a persistent connection open.
I took over a website a while ago and the guy before me had the same idea: Open a single connection, send the queries through it when needed, and never close it. A month after I took over the site, the database wouldn't return any more query results just like this.
As a general rule of thumb and for good programming practice, always clean up after yourself. If you're not using a variable, set it to null and delete it. Not using a connection? Terminate it. This is way less prominent in Java than it is in C++ as Java does all the cleaning up for you for the most part.
I am trying to insert the tens of millions of data into oracle database by through java application but after 850k data inserted I am getting the error:
java.sql.exception: IO Exception :Connection reset by peer socket
write error.
I have a similar issue in my aplication.
In my case, after a few minutes in which the connection has not used, oracle server resets the connection and when I had to use it again, this exception was thrown.
I found that the problem was that oracle sets the timeout for connection at 10 minutes in sqlora.net file.
Maybe you can try to increase this time interval or check if the connection is still valid instead to check if it is closed (a connection can be not valid even if it is not closed).
The last option works fine for me.
I hope this helps.
I have an apache proxy between browser and my custom web server. So after starting the webserver, I can access my web application for like 4-5 hours. After that I get this error
Proxy Error The proxy server received an invalid response from an
upstream server. The proxy server could not handle the request GET /.
Reason: Error reading from remote server
To access the web application again, I have to restart the web server. I tried to check the log I got from my web server and only error in that log , I am finding is this
com.mysql.jdbc.CommunicationsException: Communications link failure
due to underlying exception:
** BEGIN NESTED EXCEPTION **
java.net.SocketException MESSAGE: Connection timed out
STACKTRACE:
java.net.SocketException: Connection timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:146)
at com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:113)
at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:160)
at com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:188)
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1910)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2304)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2803)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1573)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1665)
at com.mysql.jdbc.Connection.execSQL(Connection.java:3170)
at com.mysql.jdbc.Connection.execSQL(Connection.java:3099)
at com.mysql.jdbc.Statement.execute(Statement.java:695)
at hra.database.Pool.getConnection(Pool.java:62)
** END NESTED EXCEPTION **
Last packet sent to the server was 925624 ms ago.
And I tried to fix this database error, but I still get the above mentioned proxy error, so definitely the log information is not useful.
This is what I am getting from Apache Proxy log
(70007)The timeout specified has expired: proxy: error reading status
line from remote server
Does anyone know what can be the issue or any helpful pointers to fix this issue?
apache proxy fails because customwebserver does not answer. your
customwebserver logs a crash waiting for mysql to answer. rebooting
customwebserver fixes it, so mysql is not the problem. it looks like the problem the way your customwebserver handles its connections to mysql.
can we see your jdbc config? do you have autoReconnect=true ?
What is likely happening is that your up-stream server is shutting itself down because of inactivity. Find the setting on the server that controls that and turn it up.
I can see that you see two connection timeout errors
Apache Proxy Web server timing out while connecting to custom web server
Custom web server timing out while connecting to database
If the timeout set on proxy server is less than timeout set for database connection, before the database connection times out, proxy server would time out as well.
As a first thing resolve all DB connectivity issues and test your application by directly connecting to custom web server (not via proxy).
Observe and identify normal response time and max response time.
Set proxy time out ProxyTimeout to be a value that is more than (max response time) + (network delay)
If possible you may share proxy server configurations to help you further.
I think like stated by the previous answer that you encounter a timeout raised by one of the members of your infrastructure:
server
network device (routers are clever beasts but firewalls can be clever too)
Did you setup a real connection pool ? Using heart beat mechanism to keep alive connections ?
This induces some network traffic but avoids such problems ?
Try to use a network capture tool to have many logs to analyze
HTH
jerome