Is it possible to call a stored procedure asynchronously using Hibernate? The used connection should be released after triggering the stored procedure while the query continues to run in the background on PostgreSQL. The stored procedure should then write a result in a dedicated table from which we can collect it later. Main motive is to prevent connection exhaustion of c3p0 connection pool.
I'm not sure you can do that with Hibernate. If you call the procedure the flow control from your program will pause until it finishes. However you could try to make the stored procedure async. That way when you make the call with hibernate, it will ask the DB to start an async job and your connection will be immediately released. You'll have to make also some queries to know if your stored procedure has finished.
There is no support within JDBC for asynchronous execution, so Hibernate does not offer such a feature either. If you want to return directly from a long running stored procedure call, you want to make this call in a separate thread, maybe even using a ThreadPoolExecutor. This will let you proceed with the thread which originally triggered the call. In this case you can most likely also process the result of the stored procedure in the executing thread, instead of writing into a temp table and then reading from there. However, this will depend on your usecase.
You will, however, need a connection for that, so it won't help you with running out of connections in the pool. Is this an actual problem? Have you configured the c3po pool correctly as well as the underlying database?
Related
I am using ORMLite (Java) with HikariCP and I have a lot of code that's called asynchronously i.e. CompletableFuture.supplyAsync(Supplier). Inside a Supplier I have various calls to different ORMLite Dao objects inside these tasks. I ran into a deadlock scenario and found that increasing the size of the connection pool seemed to have fixed the issue.
After some research, I found that this is a well-documented problem. Having looked at the information provided here:
"Each thread is waiting for a second database connection, and none would become available since all threads are blocked."
This caught my eye. When looking at the ORMLite source code, I found that for each Dao method called, a connection is acquired. In my case, I am calling multiple Dao methods in each asynchronous task I create via CompletableFuture which brings me to my question.
My understanding is that I should only acquire a single connection in each thread so is there a way to do this with ORMLite? Bearing in mind I have calls to different Dao objects and having done some further research, Dao#callBatchTasks(Callable) caught my eye but I am unsure whether I can have calls to different Dao objects inside the Callable. Or is there a flaw in my logic or something fundamental I am missing in my write-up?
I found that for each Dao method called, a connection is acquired
If ORMLite needs to go to the database, it needs to get a connection, right? Is there something that it is not doing right?
My understanding is that I should only acquire a single connection in each thread so is there a way to do this with ORMLite?
I don't think that there is anything that says that a thread can't open multiple database connections. As long as they are all closed when you are done with te operation, then I think this is fine. If you are querying different DAOs that all live on the same database then you can actually reuse the connection each time as long as you are done with it. I.e. if you are paging across the results using an open connection then in the middle query another table, then this might screw up the cursor. However, if you do a dao1.queryAll() and then a dao2.queryAll(), this can use the same connection since the first query will have finished.
But if you are using a connection pool, the pool should do the connection management for you with help from ORMLite's ConnectionSource. The pool is supposed to create a new one for you if needed and otherwise reuse old connections. It's a bit strange to have a limit on your database connections.
Dao#callBatchTasks(Callable) caught my eye but I am unsure whether I can have calls to different Dao objects inside the Callable.
You can call different dao objects as long as they are all on the same database. You actually could call a dao with connections to a different database, but you'd have no guarantee around transactions between the databases.
For example if you did something like the following.
You call dao1.callBatchTasks(callable)
In your Callable, you call dao2.delete(entity1) which works.
Still in the Callable, you call dao2.delete(entity2) which throws.
The dao1 transaction will be rolled back, but the entity1 would still be deleted.
The scenario you quoted in the statement
Each thread is waiting for a second database connection, and none
would become available since all threads are blocked.
would result in a deadlock only when a thread is holding a lock to connection A AND trying to acquire Connection B, which might be held by another thread which might be trying to get hold of Connection A. As long as you are not "holding" one resource while trying to "grab" the other one, you should be good.
When an external event occurs (incoming measurement data) an event handler in my Java code is being called. The data should be written to a MySQL database. Due to the high frequency of these calls (>1000 per sec) I'd like to handle the inserts efficiently. Unfortunately I'm not a professional developer and an idiot with databases.
Neglecting the efficiency aspect my code would look roughly like this:
public class X {
public void eventHandler(data) {
connection = DriverManager.getConnection()
statement = connection.prepareStatement("insert …")
statement.setString(1, data)
statement.executeUpdate()
statement.close()
connection.close()
}
}
My understanding is that by calling addBatch() and executeBatch() on statement I could limit the physical disk access to let's say every 1000th insert. However as you can see in my code sketch above the statement object is newly instantiated with every call of eventHandler(). Therefore my impression is that the batch mechanism won't be helpful in this context. Same for turning off auto-commit and then calling commit() on the connection object since that one is closed after every insert.
I could turn connection and statement from local variables into class members and reuse them during the whole runtime of the program. But wouldn't it be bad style to keep the database connection open at all time?
A solution would be to buffer the data manually and then write to the database only after collecting a proper batch. But so far I still hope that you smart guys will tell me how to let the database do the buffering for me.
I could turn connection and statement from local variables into class
members and reuse them during the whole runtime of the program. But
wouldn't it be bad style to keep the database connection open at all
time?
Considering that most (database-)connection pools are usually configured to keep at least one or more connections open at all times, I wouldn't call it "bad style". This is to avoid the overhead of starting a new connection on each database operation (unless necessary, if all the already opened connections are in use and the pool allows for more).
I'd probably go with some form of batching in this case (but of course I don't know all your requirements/environment etc). If the data doesn't need to be immediately available somewhere else, you could build some form of a job queue for writing the data, push the incoming data there and let other thread(s) take care of writing it to database in N size batches. Take a look what classes are available in the java.util.concurrent-package.
I would suggest you use a LinkedList<> to buffer the data(like a queue), then store the data into the dbms as and when required in a separate thread, executed at regular intervals(maybe every 2 seconds?)
See how to construct a queue using linkedlist in java
In my Java webapp, each instance is checking on startup if the database is up-to-date via a JDBC connection. If the DB is not up-to-date, it performs an update routine by executing SQL scripts.
I can't control when instances get startet. Therefore, I need to ensure that only a single instance is performing a database update at the same time. Ideally, I would need to lock the complete database, but according to
http://www.postgresql.org/docs/8.4/static/explicit-locking.html
and
http://wiki.postgresql.org/wiki/Lock_database
PostgreSQL doesn't support it (I'm still using version 8.4).
What other options do I have?
If you control the code for all the instances, then you can create a table in the database where each instance that starts, looks in this table for a record with a timestamp. Lets call it your "lock" record.
If a process finds that the lock record does not exist, then it inserts the record and processes the data you require.
If a process finds that the lock record does exist then you can assume that another process has created it and do nothing, busy wait, or what ever.
With this design you are effectively creating a "lock" in the database to synchronize your processes with. You code it, so all processes know they have to adhere to the logic of the lock record.
Once the first process that has the lock, has completed processing, it should clear the lock record so the next restart behaves correctly. You also need to think about the situation where the lock has not been cleared due to a server error, or execution erorr. Typically, if the lock is older than n minutes you can consider it to be "stale", therefore delete it, and create it again (or just update it).
When dealing with the "lock" record be sure to utilise the Serializable isolation level on your DB connection in order to guarantee atomicity.
The Service layer of your Java code can enforce with your locking strategy prior to calling your Data Access layer. It won't matter whether you use Hibernate or not, as it's just application logic.
Ideally, I would need to lock the complete database.
Does it really matter what your lock applies to, as long as you're effectively serializing access? Just acquire an exclusive lock on any table, or row for that matter.
Imagine a simple connection pool for java.sql.Connection. After the connection has been released back to the pool we have no idea if any transactions are open, any temporary tables created, etc.
Rather than manually checking if getAutoCommit() is false, and then seeing if we need to roll back, calling rollback(), etc I was hoping there would be a reset() function that does something similar to SQL Severs sp_resetconnection stored procedure but is not DBMS dependent. However looking at Connection's API it seems like there is not.
Does such a function exist?
There is not. In fact, even the SQL Server connection pool datasource class does not invoke the sp_resetconnection call, as it depends upon the application server (or other application managing the connection pool) to return the connection to a known state. See http://msdn.microsoft.com/en-us/library/ms378484(v=sql.90).aspx.
Various drivers, servers, etc. may have their own features, but I'm relatively certain that there is no non-proprietary or cross-database analogs to a Java method that would do what the sp_resetconnection proc does.
I don't believe such a function exists. I looked to solve this about a year ago, and ended up having to call the DBMS specific functions.
I have a Java servlet that runs a database query that may take several minutes to run before attempting to write to the response stream. During the database query the user may have disconnected (thus making the query useless).
Since I can't kill it in the database thread, I'm trying to kill it from another thread. Do any of the servlet containers provide a recommended way of "canceling" or "killing" a request thread? If I carry a reference to the Thread around, can I force an interrupt or similar?
Tour question is not about java threads. It is about killing database query into the database. I say it because as far as I understand your question what happens is that client sends HTTP request to servlet that performs JDBC connection and runs query that takes a lot of time. So, java does not work this time. The DB does. This means that you have to kill the DB query into the DB. How to do this? This depends on your database. MySql (for example) has a kind of command line shell that allows retrieving the list of current queries and terminating the queries. So this is what you can do. Your second servelet may connect to MySql, retrieve running queries, identify which one should be killed (this is application specific functionality) and kill it. I believe that once you do this the first servlet will get JDBCException and can exit.
This is the way to show list of running queries:
http://www.electrictoolbox.com/show-running-queries-mysql/
Here is how to kill query:
http://dev.mysql.com/doc/refman/5.0/en/kill.html
And the last note that probably should be the first. Check why is your query taking so long time? IMHO in most cases it means that your schema is not optimal or some index is missing. Generally, if your query takes more than 0.1 seconds check your DB schema.
If you are running a hour long DB query , you should not in first
place call from a servlet ,as you response stream will timeout, you will
get 504.
May i know what this query is doing, something involving
calculation and large updates or
inserts.
you should try placing this query in DB JOBS.
You can java.sql.Statement.cancel() you will have to have the running statements registered somewhere (ServletContext or whatever structure you find fit) and unregistered upon completion.
The JDBC driver must support this method (cancel()) as well, I don't know if PostgreSQL supports it, though