My database has millions of records and that is being used on a portal to display to the user and pagination is done using offset and also the data is sorted by some column, Is there any alternate solution to offset and using any index column such as auto increment field or unique field, it does not works if I have to sort with some different column, also even when the row is deleted it does not return expected results.
I am running sql queries in my java application, I've tried a way by only adding limit to my queries. So it works like the offset will always be zero and the limit will be number of (limit + offset) as per pagination logic
Ex : User requests for 10 records per page and navigates to 51 page
Alternate Logic : limit 10 offset 500 -> limit = 510
Queries look like
select * from history limit 510 order by log_date;
so with the help of absolute method of ResultSet I navigate to the row number as per the specified offset and fetch the results after that row
-- No of rows returned 510
rs.absolute(500);
while(rs.next()){
//store data in the object
}
but even by this way I m telling data base to return 510 records and if user navigates to the last page it will fetch all the rows which will be very inefficient.
So you are hitting the DB on every new page request - with this out of the confusion :
In the sample you gave, you are fetching all of the result set into java application and then filtering. Let the Database do the filtering and give you the result. Send the PAGE_NUMBER you want into the db query itself.
select * from history limit 510 order by log_date
OFFSET PAGE_NUMBER*MAX_ROWS_TO_SELECT ROWS FETCH NEXT MAX_ROWS_TO_SELCET ROWS ONLY;
As mentioned in the above sample, you have to get only 10 records. This is the efficient way for you to get the required page content in optimized DB resultset size.
My solution is only applicable to phoenix 4.7 (hdp 2.5) and the way I found was to sort the data in ascending order with the primary key and if there is the composite key data was sorted as per the first column of the key and then the offset and limit works properly.
Related
I am working on a trading application which is deployed on Weblogic and has a limitation that any request that takes more than one minute in processing is killed automatically. The restriction is set at the kernel level and is not application dependent, and is not in our control to change.
There is a search functionality in my application which fails when queried for more than 100 records during a given time frame, and I was assigned a task to see the possible solutions.
The first approach that I suggested was to use pagination instead of querying for all records at the same time. I was told that it won't help as on the database side it would any ways fetch all records at the same time. This was new for me as I had the understanding until now that this is handled on the database side and the query fetches only given number of records per page, and with each previous and next it handles it reducing the overall response time.
I did search a lot before posting this query on how pagination works and how it helps reduce the response time but did not get a concrete answer. So would be really great if somebody can help me explain this. Thanks in advance!!!
The first approach that I suggested was to use pagination instead of querying for all records at the same time. I was told that it won't help as on the database side it would any ways fetch all records at the same time
This is true if you are using LIMIT and OFFSET clause in your query for pagination. In this case, database loads the matched records(matched with WHERE clause) from disk and then applies OFFSET and LIMIT clause. Since databases use B-tree for indexing, it doesn't know to jump to OFFSET record directly without loading matched records to memory.
To load only the page size records, you need to use key based pagination. In this approach we avoid OFFSET clause, instead we use the key of record and LIMIT clause.
Example for key-based pagination:
Let's say you want to paginate the users
Request for first 10 records:
select * from user where userid > 0 order by userid asc limit 10
Let's say last userid in above query is 10.
Request for next 10 records:
select * from user where userid > 10 order by userid asc limit 10
We want to programmably copy all records from one table to another periodically.
Now I use SELECT * FROM users LIMIT 2 OFFSET <offset> for fetch records.
The table records like below:
user_1
user_2
user_3
user_4
user_5
user_6
When I fetched the first page (user_1, user_2), then the record "user_2" was be deleted at the source table.
And now I fetched the second page is (user_4, user_5), the third page is (user_6).
This lead to I lost the records "user_3" at the destination table.
And the real source table may be has 1000 000 records, How can I resolve the problem effectively?
First you should use an unique index on the source table and use it in an order clause to make sure that the order or the rows is consistent over time. Next you do not use offsets but start after the last element fetched.
Something like:
SELECT * FROM users ORDER BY id LIMIT 2;
for the first time, and then
SELECT * FROM users WHERE ID > last_recieved_id ORDER BY id LIMIT 2;
for the next ones.
This will be immune to asynchronous deletions.
I you have no unique index but have a non unique one in your table, you can still apply the above solution with a non-strict comparison operator. You will consistently re-get the last rows and it would certainly break with a limit 2, but it could work for reasonable values.
If you have no index - which is known to cause different other problems - the only reliable way is to have one single big select and use the SQL cursor to page.
I need to implement a web service for a feed of videos and consume it with an Android client.
By the way my implementation was a method getVideos(offset,quantity) with a MySQL table that returns the result of the query SELECT * FROM videos ORDER BY id DESC LIMIT offset,quantity where the id is an auto-incremental value.
But, since it is a very active feed I've detected the following erroneous case:
The database have the videos 1,2,3...10.
The Android client request the videos offset=0 , quantity=5 so the items 10,9,8,7,6 are returned. The user start to play some videos and in the meanwhile 5 new videos are published, so now the table contains the items 1,2,3...15 now. Then the user continues scrolling and, when the user reach the end of the list, the client attempts to request the next bundle: offset=5, quantity=5, but the same items are returned, appearing duplicates (or adding nothing) into the ListView.
What if the best approach for this problem?
If you don't want data to repeat then don't use OFFSET, use a where clause on id instead.
Check what's the last id you were given and then run a query like:
SELECT * FROM videos WHERE id<:last_id ORDER BY id DESC LIMIT 0,:quantity
Not only this guarantees the results will not repeat but also it should actually be faster since the db won't have to calculate all the offset rows.
UPDATE
How about getting a maximum value of id column when you make the first query and then adding to WHERE that all the results have to be lower or equal to that original value? That way you won't ever get duplicates unless you update some position. Better yet if you add a modification time column to your rows and use time of the first query. That way you won't show edited rows but at least they won't break the order.
SELECT *
FROM videos
WHERE mtime < :original_query_time
ORDER BY id DESC
LIMIT :offset, :quantitiy;
Can I limit the number of rows in result set?
My table contains some 800000 rows, if I fetch them in result set, this will definitely lead to OOM exception. each row has 40 columns.
I do not want to want work on them at the same time, but each row is to be filtered out for some data.
Thank you in advance.
Something like following should be a SQL solution but albeit rather ineffective, since each time you will have to fetch the increasing amount of rows.
Assuming that you have your ORDER BY is based on unique int and
that you will be fetching 1000 rows at a time.
SET currenttop = 0;
SET cuurentid = 0;
SELECT * FROM YourTable t
WHERE t1.ID > #currentID AND (#currentid := t1.ID) IS NOT NULL;
LIMIT (#currenttop:=#currenttop+1000);
Of course you can choose to handle variable from your Java code.
You could use JDBC fetch size to limit the result in the result set. It is better than the SQL LIMIT as it will work for other database as well without changing the query. Jdbc diriver will not read the whole result from the database. Each time it will retrieve the records specified by the fetch size and there will be no memory issue anymore.
You can use limit keyword with sql query, see following
SELECT * FROM tbl LIMIT 5,10; # Retrieve rows 6-15
You can read more about using limit here
Cheers !!
You have a couple options. First, add a limit to the SQL query. Second, you could use JDBCTemplate.query() with a RowCallbackHandler to process one row at a time. The template will handle memory issues with a large result set.
I have to retrieve the more than 100 records from the Database table using JSP. So, i want to control those database records into 10 records per page and each page will have the next and previous buttons contains link for these Pagination.
So, is there any tool available for pagination in JSP?
Please Guide me get out of this issue?
Have a look at http://www.ittreats.com/sun/java/pagination-using-jsp.html. This is a custom taglib that may help.
pagination is about the ability of database engine to limit how many records to be returned in one time query execution.
for example, in mySQL, you should try to execute query with 'LIMIT' and 'OFFSET' clauses in it, like:
"SELECT * FROM post LIMIT 10 OFFSET 0" means, "retrieves 10 records from table post started from the very first record (index number 0)",.
for full information about the LIMIT, see http://dev.mysql.com/doc/refman/5.0/en/limit-optimization.html
then you also need to display it in a convenience view, in a link like www.siteame.com/posts.jsp?p=2,.
if user access that link, your servlet should process it this way:
i assume that the number of posts is 100, the posts to be retrieved is 10, then the OFFSET should be (10 x (2 - 1)) (the p=2, or the second page). as the result your query should be "SELECT * FROM post LIMIT 10 OFFSET 10", make sure that the OFFSET value is less than the number of your posts,.