Im using sun.jdbc.odbc.JdbcOdbcDriver to connect to an oracle database.I know I would be probably be better off using the thin driver but I want the app to work without specifying the db server name and port no.My connection string is like jdbc:odbc:DSN.
The queries that I execute in my application may return millons of rows.All the data is critical so I cannot limit them within the query.My concern is for my java app running into memory issues.
When I check the fetch size of the statement it is set to 1.This seems extremely sub-optimal(A query retrieving 45K rows took abt 13 mins)to me and I would like to have a fetch size of atleast 500 so as to improve performance.
My understanding is that when my query is executed(I'm using Statement) the statement object points to the results on the database which I iterate using a ResultSet. The resultSet will hit the database to fetch n no of rows (where n is fetch size) each time I do a resultSet.next(). Is this interpretation correct? If so does it mean that my app will never face any out of memory issues until my fetch size is so large that the JVM gets swamped?
When I do a stmt.setFetchSize() after creating a statement I get an invalid fetch size sql exception.I am able to avoid this exception if I set the stmt.setMaxRows() to a larger value than the fetch size.But
1. I dont want my results to be limited to the MaxRows value.
2. Tried setting max rows to a huge value and tried with fetch size of 500 but saw no improvement in time taken.
Please help me figure out how I can set a valid fetch size and get some improvement.Any other optimization suggestions for the same driver would be appreciated.
Thanks,
Fell
I haven't used a JDBC-ODBC bridge in quite a few years, but I would expect that the ODBC driver's fetch size would be controlling. What ODBC driver are you using and what version of the ODBC driver are you using? What is the fetch size specified in the DSN?
As a separate issue, I would seriously question your decision to use a JDBC-ODBC bridge in this day and age. A JDBC driver can use a TNS alias rather than explicitly specifying a host and port which is no harder to configure than an ODBC DSN. And getting rid of the requirement to install, configure, and maintain an ODBC driver and DSN vastly improves performance an maintainability.
Are you sure the fetch size makes any difference in Sun's ODBC-JDBC driver? We tried it a few years ago with Java 5. The value is set, even validated but never used. I suspect it's still the case.
Use JDBC. There is no advantage in using ODBC bridge. In PreparedStatement or CallableStatement you can call setFetchSize() to restrict the rowset size.
Related
MySQL Connector-J Documentation (here) mentions two ways in which the JDBC retrieves results from the MySQL database. One is the default operation, in which the entire result set is loaded into memory and made accessible in the code. The second is row by row streaming.
I would like to know whether the latest versions of MySQL/MySQL JDBC support server side cursors. Specifically, I would like to know whether the options useCursorFetch=True and defaultFetchSize>0 can be used to ensure that the result set is retrieved from the database in batches of certain size (fetch size). MySQL describes server side cursors in its C API (here), and I would like to know whether similar support is there with MySQL JDBC.
If this support exists, what are the constraints of such an operation? I understand that a temporary table would be created in the server's memory from which results would be fetched. But what are the other things to look out for (such as table/row locks, restrictions on update/insertions, and result set/connection closing)?
The most recent version of the documentation you linked to has this note:
By default, ResultSets are completely retrieved and stored in memory. [...] you can tell the driver to stream the results back one row at a time.
To enable this functionality, create a Statement instance in the following manner:
stmt = conn.createStatement(java.sql.ResultSet.TYPE_FORWARD_ONLY,
java.sql.ResultSet.CONCUR_READ_ONLY);
stmt.setFetchSize(Integer.MIN_VALUE);
This sounds like what you're looking for.
I have a performance issue with ResultSet.next() while retrieving data from an sqlite database. I am using org.sqlite.JDBC driver. I tried setting up statement/ResultSet.fetchSize(), but that did not work either.
The operation lasts around 25 seconds when I use an SQL statement with a string paramater in the WHERE clause (with a non-string parameter in the WHERE, it lasts just 4 seconds).
I found that with some drivers it is possible to set up send string parameter as unicode = false in the URL, which should improve performance. However, I was not able to find way how to set it up with SQLite connection.
Can some one help me with this issue?
In SQLite, result rows are computed on demand. (SQLite is not a client/server database, so there is no communication overhead.)
So when next() is slow, this just means that your query is slow, and that the database must do much I/O and/or many computations until it can return the next row.
To speed it up, you have to improve the database design, or the query, or the hardware.
I'm trying to render a list of records in a jsf page with a query that returns many records, I'm using weblogic 10.3.0.0 and sql server driver 4 from Microsoft to connect a sql server database, when I run this jsf page, this consumes a lot of memory because the query returns many record and therefore a OutOfMemoryError is occurring. I've seen that with setFetchSize you can limit the results, but here:
What does Statement.setFetchSize(nSize) method really do in SQL Server JDBC driver?
the microsoft's sql server driver not limit this, I used jDTS driver as the above post has suggested, but the same problem ocurre, I've also tried to use this:
http://msdn.microsoft.com/en-us/library/bb879937.aspx
for use adaptive buffering with the driver, but my driver version is 4 so this by default has adaptive buffering, but apparently no, I've tried this:
statement = connectionDB.createStatement();
SQLServerStatement SQLstmt = (SQLServerStatement) statement;
SQLstmt.setResponseBuffering("adaptive");
but this no returns results, I've put this in the connection properties also, but the problem still occure, I understand that the problem is the query have big results and the driver no execute it with chunks, and therefore the memory is decreasing, and I believe that this is the problem. I don't know who workaround use, if do the manual pagination with the query, if use another driver, etc., please help me to find a workaround, whatever info is well received, sorry for my poor english
I got this doubt when I was modifying a code for doing batch update for MySQL retrieval using Java.
My understanding is that fetch size is the maximum number of rows in a ResultSet object and Batch Limit is the number of select/insert/update queries that can be added to a batch, for batch execution. Can anyone correct me if I am wrong here.
Thanks.
You are almost correct. However to add to it javadoc of Statement#setFetchSize()
Gives the JDBC driver a hint as to the number of rows that should be
fetched from the database
Whereas the batch limit is something which is related to how many rows you can insert or update something related to max_allowed_packet
On a side note:
You may also check the JDBC API implementation notes as a good read
ResultSet
By default, ResultSets are completely retrieved and stored in memory.
In most cases this is the most efficient way to operate, and due to
the design of the MySQL network protocol is easier to implement. If
you are working with ResultSets that have a large number of rows or
large values, and cannot allocate heap space in your JVM for the
memory required, you can tell the driver to stream the results back
one row at a time.
I am using Java to read from a SQL RDBMS and return the results to the user. The problem is that the database table has 155 Million rows, which make the wait time really long.
I wanted to know if it is possible to retrieve results as they come from the database and present them incrementaly to the user (in batches).
My query is a simple SELECT * FROM Table_Name query.
Is there a mechanism or technology that can give me callbacks of DB records, in batches until the SELECT query finishes?
The RDBMS that is used is MS SQL Server 2008.
Thanks in advance.
Methods Statement#setFetchSize and Statement#getMoreResults are supposed to allow you to manage incremental fetches from the database. Unfortunately, this is the interface spec and vendors may or may not implement these. Memory management during a fetch is really down to the vendor (which is why I wouldn't strictly say that "JDBC just works like this").
From the JDBC documentation on Statement :
setFetchSize(int rows)
Gives the JDBC driver a hint as to the number of rows that should be
fetched from the database when more rows are needed for ResultSet
objects genrated by this Statement.
getMoreResults()
Moves to this Statement object's next result, returns true if it is a
ResultSet object, and implicitly closes any current ResultSet object(s)
obtained with the method getResultSet.
getMoreResults(int current)
Moves to this Statement object's next result, deals with any current
ResultSet object(s) according to the instructions specified by the given
flag, and returns true if the next result is a ResultSet object.
current param indicates Keep or close current ResultSet?
Also, this SO response answers about the use of setFetchSize with regards to SQLServer 2005 and how it doesn't seem to manage batched fetches. The recommendation is to test this using the 2008 driver or moreover, to use the jTDS driver (which gets thumbs up in the comments)
This response to the same SO post may also be useful as it contains a link to SQLServer driver settings on MSDN.
There's also some good info on the MS technet website but relating more to SQLServer 2005. Couldn't find the 2008 specific version in my cursory review. Anyway, it recommends creating the Statement with:
com.microsoft.sqlserver.jdbc.SQLServerResultSet.TYPE_SS_SERVER_CURSOR_FORWARD_ONLY (2004) scrollability for forward-only, read-only access, and then use the setFetchSize method to tune performance
Using pagination (LIMIT pageno, rows / TOP) might create holes and duplicates, but might be used in combination with checking the last row ID (WHERE id > ? ORDER BY id LIMIT 0, 100).
You may use TYPE_FORWARD_ONLY or FETCH_FORWARD_ONLY.
This is exactly how is JDBC driver supposed to work (I remember the bug in old PostgreSQL driver, that caused all fetched records to be stored in memory).
However, it enables you to read record when the query starts to fetch them. This is where I would start to search.
For example, Oracle optimizes SELECT * queries for fetching the whole set. It means it can take a lot of time before first results will appear. You can give hints to optimize for fetching first results, so you can show first rows to your user quite fast, but the whole query can take longer to execute.
You should test your query on console first, to check when it starts to fetch results. Then try with JDBC and monitor the memory usage while you iterate through ResultSet. If the memory usage grows fast, check if you have opened ResultSet in forward-only and read-only mode, if necessary update driver.
If such solution is not feasible because of memory usage, you can still use cursors manually and fetch N rows (say, 100) in each query.
Cursor documentation for MSSQL: for example here: http://msdn.microsoft.com/en-us/library/ms180152.aspx