I'm new to java and tomcat. I'm developing a website in java using spring mvc. It's deployed to a linux server that's running Tomcat 8. Everything works fine when I deploy, it connects to the database great. The issue is that the site seems to go idle very quickly. I haven't been able to time it exactly, but it seems like it only takes about a minute of inactivity for the entire site to go idle. Then the next request is extremely slow, loading in all my classes. I'm losing my sessions as well.
Is this a common occurrence? Does it sound like I'm doing something wrong in java? Tomcat? Both?
EDIT: In light of StuPointerException's comment, I've updated my database connection management. I'm now using Apache dbcp. I will update if this resolves the problem. I want to give my QA tester ample time to hit my site some more.
It's difficult to answer your question directly without more information about your server setup.
For what it's worth though, every time I see this kind of behaviour it's down to a mis-configured database connection pool. There can be a significant overhead in creating new database connections.
If you don't use a connection pool or you're allowing connections in the pool to die (due to missing validation queries/checks) then you will start to see performance problems over time due to connection timeouts.
Related
I have a Java application requesting about 2.4 million records from a Microsoft SQL Server (Microsoft SQL Server 2008 R2 (SP3))
The application runs fine on all hosts, except one. On this host, the application is able to retrieve data on some occasions. But on some others, it hangs.
Monitoring the MS Sql server indicates that the SPID associated with the query is in an ASYNC_NETWORK_IO wait state.
There are a few links online that talk about it.
https://blogs.msdn.microsoft.com/joesack/2009/01/08/troubleshooting-async_network_io-networkio/
https://social.msdn.microsoft.com/Forums/sqlserver/en-US/6db233d5-8892-4f8a-88c7-b72d0fc59ca9/very-high-asyncnetworkio?forum=sqldatabaseengine
https://social.msdn.microsoft.com/Forums/sqlserver/en-US/1df2cab8-33ca-4870-9daf-ed333a64630c/network-packet-size-and-delay-by-sql-server-sending-data-to-client?forum=sqldatabaseengine
Based on the above, the ASYNC_NETWORK_IO means 2 things:
1. Application is slow to process the results
2. Network between application and DB has some issues.
For #1 above, We analyzed using tcpdumps and found that in the cases where the query goes into ASYNC_NETWORK_IO state, the application server's tcp connection has a window size that oscillates between 0 and a small number, and eventually remains stuck at 0. Based on some more analysis, aspects related to firewalls between DB and application have also been mostly ruled out.
So I am staring at #2, unable to understand what could possibly go wrong. All the more baffling because the same code has been running under similar data loads for more than a year now. And it also runs fine on other hosts.
The JDBC driver being used is sqljdbc4-4.0.jar.
This by default has an adaptive buffering feature, which does things under the hood to reduce application resources.
We use the default fetch size of 128 (which i believe is not a good one).
So i am going to experiment overriding the default adaptive buffering behavior, though the MS docs suggest that it is good to have adaptive buffering for large result sets.
I will change the connection setting to use selectMethod=cursor.
And also change the fetchSize to 1024.
Now if it does not work:
What are some aspects of the problem that are worth investigating.
Assuming its still an issue with the client, what other connection settings, network settings should be inspected/changed to make progress?
If it does work consistently, what is the impact of making the connection setting change to selectMethod=cursor
On the application side?
Database side?
Update: I tested the application adding the selectMethod=cursor to the connection. However, it results in the same issue as above.
Based on discussions with other administrators in the team - at this point the issue may be in the jdbc driver, or on the OS (when it tries to handle the data on the network).
After a good amount of discussions with the System admin, Network Admin and Database admin - it was agreed that somewhere in the OS -> Application stack, the data from network wasn't handled. In the meantime, we tested out a solution where we broke down the query to return smaller sized results. So we broke it down to 5 queries, each returning about 500k records.
Now when we ran these queries sequentially, we still ran into the same issue.
However, when we ran the queries in parallel, it always was successful.
Given that the solution worked always we haven't bothered getting to the root cause of the problem anymore.
On another note, the hardware and software running the application was also outdated. It was running Red Hat 5. So, it could well have to do something with that.
I have an application developed with Java using MySQL and C3P0 for connection pooling. The application works perfectly fine in localhost, the connection management was superb. However, I uploaded this application to Daily Razor "private JVM", and here we go, there were lot of MySQL connections than the application will ever make! Normally the application will make max 10 connections, but when I hosted there I can see 30 or more.
Apart from that, I always had number of mysql processors running in my localhost, but when uploaded to the online server I can only see 2. It is like upside down. The application works fine but there were number of times that I had to restart the server due to slow connection issue.
What is making this kind of thing? Anyway pls don't ask for code, because I don't know where the issue is
We have an application which we run inside the JBoss EJB Container. This application makes connections to mysql and runs stored procedures on mysql. We have observed that after a point in time Jboss stops responding to web connections to web application hosted on it. So after investigating we have found the following issues.
The number of socket connections from jboss keeps on increasing and once it goes above thousand we observe that jboss stops working completely because of the limit on a process for the number of socket connections(i.e 1024), we have cross checked the code for socket connections, but we feel it makes socket connections only to mysql, so either this is a problem or something else is doing this, can't find the actual cause. We have tried using netstat, lsof on linux, any other suggestions to finding the root cause of the connection issue would be of great help.
We also checked the show processlist of mysql, but it shows only 8 to 10 active connections at any point in time. So no luck here.
There is also another interesting thing, we had reduced the timeout for connections from our application from 86400 seconds to 30 seconds, and we have reduced the wait timeout for mysql database to 50 seconds, so there is a gap of 20 seconds. We have again and again cross checked the database for any issues with this, but this hardly affects it. But any suggestions in this would also be helpful. We plan to reduce the difference to 5 seconds.
Update : We have subsequently changed the connection timeout from 30 to 170 and also mysql waittimeout to 180
we have tried making changes according to jboss forums where it says cache connection manager tag, we have to enable an attribute called debug=true, we have tried this solution, but what happens is if there are transactions, this causes them to drop off, which is causing havoc in our application, we subsequently reverted the changes, and are running it just like that, but the application is still on the verge of a disaster. We are still running clueless, JBOSS seems to be at the core of our issues, still no solution :(
First of all, we are running a Java Web application running on WAS 5.1. Behind that, we use an Oracle data base. The problem that we're faced to is really simple, but after a couple of hours of Google search, I decided to ask you.
We have an application that is running on WAS. When we start the server, WAS sets his DataSource so that it points to the data base. Everything works fine, expect when the DBAs have to reboot the data base server. When they do, the data source is no longer valid and we have to manually restart all server and we are currently trying to correct that, if possible. We need to find a way to do it because we have 3 pre-production environnement for for our application, and there are two servers associated with it, one for the application and the other is a report generator web service. So, when the DBAs wants to reboot the server (and they usually don't tell us!) we have to reboot six servers. I was wondering if in Java, there was a way to reset the data source so that we don't need to restart the servers.
For you information, WebSphere is v5.1 and Oracle is 9g with Java 1.4.2.17.
We also use RAD:
Version: 6.0.1
Build id: 20050725_1800
You should configure your application server to always test the connection before leasing it out to a client. I'm not familiar with Websphere that much, but in WebLogic, you can set a jdbc sql statement such as select 1 from dual and the container removes stale connections from the connection pool.
Here is a link on how to do it in Websphere
http://www-01.ibm.com/support/docview.wss?uid=swg21439688
Based on what i read from your note, you should receive Stale connection exception as WAS has stale handles (in its pool) as the DB has been restarted.
The Data Source configuration can be configured to purge the entire pool once a stale connection is detected. The default policy is to purge the individual connection.
Adopting this would prevent you from restarting your WAS Servers.
There are a number of resources in this space
http://www-01.ibm.com/support/docview.wss?uid=swg21063645
HTH
Manglu
I have made a simple HTTP client, which downloads a set of URLs parsed from a webpage.
My problem is that the download is slow, compared to a real browser (IE, Firefox, Chrome). Especially if the page contains many objects.
I noticed (with wireshark) that many times the real browsers will setup 5-10 TCP connections within the same millisecond instantly after starting the load of a page. Those conections will then live concurrently for a period of time.
My client will also setup concurrent TCP-connections (and it will reuse TCP connections), but not at all this aggressively. I'm guessing that this is one of the reasons my client is slower.
I have tried creating several URLConnections before reading from the input stream, but this does not work for me. I am inexperienced though, so I probably do it wrong.
Does anyone know of way to do this (mimic what the browsers are doing in terms of TCP connection setup) with URLConnection?
I recommend using HttpClient:
http://hc.apache.org/httpcomponents-client-ga/
It has support for internal connection management, pooling etc.
Browsers tend to have this sort of stuff.
Things may have changed since I last used it, but UrlConnection didn't work well for production apps. Ex. it didn't have a clean way to shut it down.
I would also recommend using a high performance networking library, like Apache Mina. This will automatically create thread pool for you and save you a lot of time.