If query execution takes a long time then the result is not coming to the Java layer. if we run the query in Oracle SQL developer there the query getting timed out. If anybody can help with this then its great.
Two things can happen,
1) In your java layer it takes too much time to process those db results to actual pojos. You might think like results are not coming but most of the time it takes too long to process those data. Because same query get executed really fast in oracle dev tools.
2) Oracle has locks and maybe some other thread or a request from a web application is holding on to a database record so the entire process waits for the table or row to get finished.
Related
I try load 50000 rows in table with 200 columns. I do executeBatch() every 1000 rows. And I get lock for this table. The same code works for MS SQL and Oracle but with postgresql I get this issue. When I decrease executeBatch number from 1000 to 75 all works correctly.
Is there any param in config file witch responding for batch buffer size?
Same issue http://www.postgresql.org/message-id/c44a1bc0-dcb1-4b57-8106-e50f9303b7d1#79g2000hsk.googlegroups.com
When I execute insert statements in batch for tables with a large number of columns occurs hanging, when call statement.executeBatch().
It is specific for postgresql jdbc driver.
To avoid this issue we should increase buffer size params(SO_SNDBUF, SO_RCVBUF) for socket.
For Windows we have to create such params in register:
[HKEY_LOCAL_MACHINE \SYSTEM \CurrentControlSet \Services \Afd \Parameters]
DefaultReceiveWindow(type=DWORD) = 1640960(Decimal)
DefaultSendWindow(type=DWORD) = 1640960(Decimal)
This number(1640960) I get from internet as general recommendation!
And it works for me.
Generally you need to look for the following things.
Are you able to actually get a lock for the table?
Do you have other java code locks that you are waiting for?
In general the first place to check is the pg_stat_activity system view which will show you the query being executed and whether it is active, idle, waiting, etc. Then if it is waiting (i.e. waiting is t), then you will want to check the pg_locks view and see what else may have a lock on anything in the relation.
If waiting is not true, then you are better off checking your java code for client-side locks but since this works for MySQL and Oracle, I assume this is less of an issue.
There is one other thing to be aware of. I am not sure about MySQL and Oracle here but PostgreSQL limits you to one query at a time per connection. You might have some locking there?
I'm having problems in generating a report the result reaches more than 500,000 lines. Believe me, this result is already filter.
The query (DB2) runs almost instantly, but the the interaction in resultSet is absurdly slow.
I'm doing several tests to try to improve this process but so far without success.
- At first was converting the direct data for the bean (used for report generation), but is very slow and the database gives timeout.
- I tried to turn into a simpler process for testing (resultSet to HashMap) unsuccessfully
- Used the setFetchSize configuration (2000) for the statement
- I looked on the possibility of using thread safe, but does not support resultSet
Already modified the timeout of the bank to increase the processing time, but my problem was not resolved.
Anyway, already tried several possibilities. Does anyone have any tips or solution to my problem?
First of all let me clear,
Reporting, Report Generation task should never be done on application DB.
Application DB, Transactional DBs are designed for fast transactions which doesnt involve heavy result fetching, processing. Those tasks should be handled on DW server or standby replicas.
Second,
Reporting application logic should be processed in less crowded hours (when system is not used by users i.e. nights)
If possible put your processing logic on DB side in form of procedures (maths part) with efficient queries to improve the performance in terms of processing and data transfer.
Try to collect reports periodically using triggers/scheduled jobs etc. and while creating reports use those intermediate reports instead of DB (As you said your query execution is not a problem, but this will save iterating over a large set.) You can use values from intermediate reports thus iterating frequency will be less.
I've a typical scenario & need to understand best possible way to handle this, so here it goes -
I'm developing a solution that will retrieve data from a remote SOAP based web service & will then push this data to an Oracle database on network.
Also, this will be a scheduled task that will execute every 15 minutes.
I've event queues on remote service that contains the INSERT/UPDATE/DELETE operations that have been done since last retrieval, & once I retrieve the events for last 15 minutes, it again add events for next retrieval.
Now, its just pushing data to Oracle so all my interactions are INSERT & UPDATE statements.
There are around 60 tables on Oracle with some of them having 100+ columns. Moreover, for every 15 minutes cycle there would be around 60-70 Inserts, 100+ Updates & 10-20 Deletes.
This will be an executable jar file that will terminate after operation & will again start on next 15 minutes cycle.
So, I need to understand how should I handle WRITE operations (best practices) to improve performance for this application as whole ?
Current Test Code (on every cycle) -
Connects to remote service to get events.
Creates a connection with DB (single connection object).
Identifies the type of operation (INSERT/UPDATE/DELETE) & table on which it is done.
After above, calls the respective method based on type of operation & table.
Uses Preparedstatement with positional parameters, & retrieves each column value from remote service & assigns that to statement parameters.
Commits the statement & returns to get event class to process next event.
Above is repeated till all the retrieved events are processed after which program closes & then starts on next cycle & everything repeats again.
Thanks for help !
If you are inserting or updating one row at a time,You can consider executing a batch Insert or a batch Update. It has been proven that if you are attempting to update or insert rows after a certain quantity, you get much better performance.
The number of DB operations you are talking about (200 every 15 minutes) is tiny and will be easy to finish in less than 15 minutes. Some concrete suggestions:
You should profile your application to understand where it is spending its time. If you don't do this, then you don't know what to optimize next and you don't know if something you did helped or hurt.
If possible, try to get all of the events in one round-trip to the remote server.
You should reuse the connection to the remote service (probably by using a library that supports connection persistence and reuse).
You should reuse the DB connections by using a connection pooling library rather than creating a new connection for each insert/update/delete. Believe it or not, creating the connection probably takes 100+ times as long as doing your DB operation once you have the connection in hand.
You should consider doing multiple (or all) of the database operations in the same transaction rather than creating a new transaction for each row that is changed. However, you should carefully consider your failure modes such that you don't lose any events (if that is an important consideration).
You should consider utilizing prepared statement caching. This may help, but maybe not if Oracle is configured properly.
You should consider trying to analyze your operations to find any that can be batched together. This can be a lot faster if you have some "hot" operations that get done often.
"I've a typical scenario"
No you haven't. You have a bespoke architecture, with a unique data model, unique data and unique business requirements. That's not a bad thing, it's the state of pretty much every computer system that's not been bought off-the-shelf (and even some of them).
So, it's an experiment and you must approach it as such. There is no "best practice". Try various things and see what works best.
"need to understand best possible way to handle this"
You will improve your chances of success enormously by hiring somebody who understands Oracle databases.
I am executing a series of sql statements using a JDBC connection on a DB2 server. On the last execute() of the simple sql: DELETE FROM MYTABLE, the thread gets hung for a long period of time even if the table somply contains a single record.
The application server I am using is WAS. I wonder if this is an issue specific to WAS and DB2 combination as the same code works on other environments.
Does anybody have any idea what is going on here?
Have you issue the command directly from the CLP? It could be other problem such as:
Transaction log problem: There are a lot of rows to delete, and this takes time. Also, the transaction logs have reached the limit, and the database does not do a rollback but waits for empty log freed by other transactions.
Lock problem (concurrency): some of the rows your are trying to delete have locks in other transactions, and the applications has to wait to release them (lock wait)
Also, try to do frequent commits.
Deleting rows in a database can be a terrible work: don't forget the database server will log all the data of the table in case of a ROLLBACK. Then I assume the problem is coming from the database especially if the table has many rows.
Have you tried to run manually all the SQL requests yourself in an interactive environment?
I have a Java servlet that runs a database query that may take several minutes to run before attempting to write to the response stream. During the database query the user may have disconnected (thus making the query useless).
Since I can't kill it in the database thread, I'm trying to kill it from another thread. Do any of the servlet containers provide a recommended way of "canceling" or "killing" a request thread? If I carry a reference to the Thread around, can I force an interrupt or similar?
Tour question is not about java threads. It is about killing database query into the database. I say it because as far as I understand your question what happens is that client sends HTTP request to servlet that performs JDBC connection and runs query that takes a lot of time. So, java does not work this time. The DB does. This means that you have to kill the DB query into the DB. How to do this? This depends on your database. MySql (for example) has a kind of command line shell that allows retrieving the list of current queries and terminating the queries. So this is what you can do. Your second servelet may connect to MySql, retrieve running queries, identify which one should be killed (this is application specific functionality) and kill it. I believe that once you do this the first servlet will get JDBCException and can exit.
This is the way to show list of running queries:
http://www.electrictoolbox.com/show-running-queries-mysql/
Here is how to kill query:
http://dev.mysql.com/doc/refman/5.0/en/kill.html
And the last note that probably should be the first. Check why is your query taking so long time? IMHO in most cases it means that your schema is not optimal or some index is missing. Generally, if your query takes more than 0.1 seconds check your DB schema.
If you are running a hour long DB query , you should not in first
place call from a servlet ,as you response stream will timeout, you will
get 504.
May i know what this query is doing, something involving
calculation and large updates or
inserts.
you should try placing this query in DB JOBS.
You can java.sql.Statement.cancel() you will have to have the running statements registered somewhere (ServletContext or whatever structure you find fit) and unregistered upon completion.
The JDBC driver must support this method (cancel()) as well, I don't know if PostgreSQL supports it, though