We have a basic Java EE app that runs under tomcat and maintains a connection pool to a SQL server database. We were having some data issues showing up only in production, so I created a testing tool that would simulate different numbers of users going through the system on different paths.
I've worked on this a bit and so the problem's evolved as I chased it. Now the problem is this.
Ten user threads works perfectly. Twenty user threads and the log record that gets created when the user logs into the system never gets inserted for any of the 20 users. In fact, Hibernate 3.3 goes through the motions of inserting the record, but when I use the show_sql setting, the insert statement never shows up in the dump. Again this works perfectly with 10 users. And more puzzling, every once in a while it will work for one of the 20 users. :(
I'm using the JTDS driver, btw, to avoid the problems we kept finding with the MS one.
I am running SQL Server Express 2008 R2 on my local box with tomcat and running my test app in my eclipse IDE. Has anyone seen anything like this? Any ideas as to why hibernate might be locking after 10 users?
I believe the problem is that you cannot open enough sessions as you need (Because they are pooled)
How do you open the session ?
What size does your connection pool have?
Do you always close the sessions?
Related
If you have any problem with the comment, download an application with java and netbeans using a mysql database, there are 2 options:
1.When I work with the java application consuming my database from my own machine, it goes smoothly in that way:
does not slow down when showing the data from the database among other points, such as when showing a list of products in a table
2.when I work with the java application consuming an external database such as hostinger, which is in my case, it goes slow in such cases, an example: when opening a table with a list of products it takes almost 2 minutes to display and it should not be so really
Well based on all this I have used pool connections both with the classes: ComboPooledDataSource and BasicDataSource but anyway it is still the same or capable I am missing some value to add, I cannot find the solution to this problem, 2 words come to mind which are connections or network latency, I would really appreciate if you would support me with this ..
There are a java web application which does use Hibernate for interacting with database. Recently, the web app started working slowly and its getting worse for some of its operations and after debugging I've found out that it's a database issue. One of the tables, responding for specific data got about 3mil rows and its increasing, so it takes about to 5-10 sec to add one record.
I was thinking about creating different tables for each year, so the data of a year 2018 for example would be stored only in table_xxx_2018. One solution would be to manually, beforehand create all tables and classes and map them in .hbm.xml files, but I don't know if its good solution or no. Also, it doesn't seem to be sustainable. So, I was looking forward to see if its possible to create dynamic tables using hibernate and map them with responding classes in java.
I also googled some tweaks and improvements for PostgreSQL but it didn't help. The main problem is increased flow of the data which is coming to the app, and it seems that Postgres started choking at 3mil+ rows per table.
These are the server specs, only Postgres is running on this server and nothing else:
CPU Intel Xeon E5-2630 #2.30Ghz
RAM 32 GB and SSD drives in RAID
Any suggestions are welcome
EDIT1:
An example of the piece of code
#Override
public void saveOrUpdateCitizen(Citizen citizen) {
Session session = sessionFactory.getCurrentSession();
session.saveOrUpdate(citizen);
}
I think I have a leak in one program of mine, one is a desktop application that using multi threading quartz scheduler jobs to connect to the database and make some checks and actions and the other is a Tomcat web application tool to do some transactions on the database, so I need to know where the leak is cause my database admin complained of so many connections caused by one of those application to the database user and they both connecting to the same user so I couldn't know which one.
I have searched and found http://java-monitor.com/forum/livedemo.php but it doesn't even download.
so is there any code or something I can do to know if there is no any tool ?
Note that : the programs are on the same machine and i have doubled check the closing connection also for the result-sets and prepared statement too
Try the tool yourkit.
https://www.yourkit.com/java/profiler/
It have a graphical memory debugger in it.
There is a 30 day trial available.
My Java web app running Tomcat (7.0.28) periodically becomes unresponsive. I'm hoping for some suggestions of possible culprits (synchronization?), as well as maybe some recommended tools for gathering more information about what's occurring during a crash. Some facts that I have accumulated:
When the web app freezes up, tomcat continues to feed request threads into the app, but the app does not release them. The thread pool fills up to the maximum (currently 250), and then subsequent requests immediately fail. During normal operation, there is never more than 2 or 3 active threads.
There are no errors or exceptions of any kind logged to any of our tomcat or web app logs when the problem occurs.
Doing a "Stop" and then a "Start" on our application via the tomcat management web app immediately fixes this problem (until today).
Lately the frequency has been two or three times a day, though today was much worse, probably 20 times, and sometimes not coming back to life immediately.
The problem occurs only during business hours
The problem does not occur on our staging system
When the problem occurs, processor and memory usage on the server remains flat (and fairly low). Tomcat reports plenty of free memory.
Tomcat continues to be responsive when the problem occurs. The management web app works perfectly well, and tomcat continues sending requests into our app until all threads in the pool are filled.
Our database server remains responsive when the problem occurs. We use Spring framework for data access and injection.
Problem generally occurs when usage is high, but there is never an unusually high spike in usage.
Problem history: something similar occurred about a year and a half ago. After many server config and code changes, the problem disappeared until about a month ago. Within the past few weeks it has occurred much more frequently, an average of 2 or 3 times a day, sometimes several times in a row.
I identified some server code today that may not have been threadsafe, and I put a fix in for that, but the problem is still happening (though less frequently). Is this the sort of problem that un-threadsafe code can cause?
UPDATE: With several posts suggesting database connection pool exhaustion, I did some searching in that direction and found this other Stackoverflow question which explains almost all of the problems I'm experiencing.
Apparently, the default values for maxActive and maxIdle connections in Apache's BasicDataSource implementation are each 8. Also, maxWait is set to -1, so when the pool is exhausted and a new request for a connection comes in, it will wait forever without logging any sort of exception. I'm still going to wait for this problem to happen again and perform a jstack dump on the JVM so that I can analyze that information, but it's looking like this is the problem. The only thing it doesn't explain is why the app sometimes doesn't recover from this problem. I suppose the requests just pile up sometimes and once it gets behind it can never catch up.
UPDATE II: I ran a jstack during a crash and found about 250 (max threads) of the following:
"http-nio-443-exec-294" daemon prio=10 tid=0x00002aaabd4ed800 nid=0x5a5d in Object.wait() [0x00000000579e2000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1118)
- locked <0x0000000743116b30> (a org.apache.commons.pool.impl.GenericObjectPool$Latch)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:573)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:637)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:666)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:674)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:718)
To my untrained eye, this looks fairly conclusive. It looks like the database connection pool has hit its cap. I configured a maxWait of three seconds without modifying the maxActive and maxIdle just to ensure that we begin to see exceptions logged when the pool fills up. Then I'll set those values to something appropriate and monitor.
UPDATE III: After configuring maxWait, I began to see these in my logs, as expected:
org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77)
I've set maxActive to -1 (infinite) and maxIdle to 10. I will monitor for a while, but my guess is that this is the end of the problem.
From experience, you may want to look at your database connection pool implementation. It could be that your database has plenty of capacity, but the connection pool in your application is limited to a small number of connections. I can't remember the details, but I seem to recall having a similar problem, which was one of the reasons I switched to using BoneCP, which I've found to be very fast and reliable under load tests.
After trying the debugging suggested below, try increasing the number of connection available in the pool and see if that has any impact.
I identified some server code today that may not have been threadsafe,
and I put a fix in for that, but the problem is still happening
(though less frequently). Is this the sort of problem that
un-threadsafe code can cause?
It depends what you mean by thread-safe. It sounds to me as though your application is causing threads to deadlock. You might want to run your production environment with the JVM configured to allow a debugger to connect, and then use JVisualVM, JConsole or another profiling tool (YourKit is excellent IMO) to have a peek at what threads you've got, and what they're waiting on.
doing profiling on an java application running websphere 7 and DB2 we can see that we spend most of our time in the com.ibm.ws.rsadapter.jdbc package handling connections to and from the database.
How can we tune our jdbc performance?
What other strategies exist when database performance is a bottleneck?
Thanks
You should check your websphere manual for how you configure a connection pool.
Update 2021
Here is an introduction inculding code samples
Update 2021
One cause of slow connect times is a deactivated database, which does not open its files and allocate its memory buffers and heaps until the first application attempts to connect to it. Ask your DBA to confirm that the database is active before running your tests. The LIST ACTIVE DATABASES command (run from the local DB2 server or over a remote attachment) should show your database in its output. If the database is not activated, have your DBA activate it explicitly with ACTIVATE DATABASE yourDBname. That will ensure that the database files and memory structures remain available even when the last user disconnects from the database.
Use GET MONITOR SWITCHES to ensure all your monitor switches are enabled for your database, otherwise you'll miss out on some potentially revealing performance details. The additional overhead of tracking the data associated with those monitor switches is minimal, while the value of the performance data is significant.
If the database is always active and things still seem slow, there are detailed DB2 traces called event monitors that log everything they encounter to a file, pipe, or DB2 table. The statement event monitor is one I turn to fairly often to analyze SQL statement efficiency and UOW hygiene. I also prefer taking the extra hit to log the event monitor records to a table rather than a file, so I can use SQL to search the data for all sorts of patterns. The db2evtbl utility makes it fairly easy to define the event monitor you want and create the tables to store its output. The SET EVENT MONITOR STATE command is how you start and stop the event monitor you've created.
In my experience what you are seeing is pretty common. The question to ask is what exactly is the DB2 connection doing...
The first thing to do is to try and isolate the performance issue down to a section of the website - i.e. is there one part of the application that see poor performance, when you find that you can increase the trace logging to see if you can see the query causing issues.
Additionally, if you chat to your DBA's they may be able to run some analysis on the database to tell you what queries are taking the time to return values, this may also help in your troubleshooting.
Good luck!
Connection pooling
Caching
DBAs