We Keep Coding

What is the best way for pagination on mongodb using java

I am trying to create a simple pagination in mongodb by below code.
collection.find().skip(n).limit(n);
but doesn't it looks like there will be a performance issue if we see that in java terms first find will return all the records consider i have 2 million records, then it will pass it to skip method then it will be passed to limit method. it means every time this query will be fetching all the db records or mongodb drivers work differently, what i missed?

When talking about pagination in MongoDB, it is easily to write this code:
collection.find().skip(pageSize*(pageNum-1)).limit(pageSize);
Above is the native solution supported by MongoDB, but this is not efficient if there are huge documents in the collection. Suppose you have 100M documents, and you want to get the data from the middle offset(50Mth). MongoDB has to build up the full dataset and walk from the beginning to the specified offset, this will be low performance. As your offset increases, the performance keeps degrade.
The root cause is the skip() command which is not efficient and can not take big benifit from index.
Below is another solution to improve performance on large data pagination:
The typical usage scenario of pagination is that there is a table or list to show data of specified page, and also a 'Previous Page' & 'Next Page' button to load data of previous or next page.
If you got the '_id' of the last document in current page, you can use find() instead of skip(). Use _id > currentPage_LastDocument._id as one of the criteria to find next page data. Here is pseudocode:
//Page 1
collection.find().limit(pageSize);
//Get the _id of the last document in this page
last_id = ...
//Page 2
users = collection.find({'_id': {$gt: last_id}}).limit(pageSize);
//Update the last id with the _id of the last document in this page
last_id = ...
This will avoid MongoDB to walk through large data when using skip().

Jpa paging with numbers and next, previous

I apologize for asking this question if someone already asked it and for asking a perhaps dumb question but I am new to this and you are experts and I would love to learn from your expert advice and experience.
I want to add paging to an application displaying 200 records per page. The bottom of the page should have numbers. If there are 1050, the first page will display 100 and the bottom of the page will show numbers 1,2,3,4,5 & 6.
What is the general logic to accomplish this?
I know that the database must select 200 every time and I must keep track of the first record.
Is there a way to know how many records will be returned total so that I can know how many numbers to display on the bottom of the page? Does it require selecting a count() statement or something else?
For the 1050 records, The numbers 1,2,3,4,5 & 6 will display and clicking each one requires a call to the server. Is there a way to know how many records will be returned in the next call to the server? Does it require selecting a count() statement or something else?

You can use the JPA Criteria API to accomplish this. Assuming TypedQuery, you would use setFirstResult and setLastResult to limit the records returned from the database. Of course the values for these methods will be dependent on what page was requested, and how many records are getting displayed per page.
first = (page - 1) * page size;
last = (page * size) - 1;
Note: This assumes the first page is 1 (as opposed to zero).
To get a record count you execute a standard Criteria query. As an example:
final CriteriaBuilder builder = entityManager.getCriteriaBuilder();
final CriteriaQuery<Long> countQuery = builder.createQuery(Long.class);
countQuery.select(builder.count(countQuery.from(MyEntity.class)));
final Long count = entityManager.createQuery(countQuery)
.getSingleResult();
You can customize the above code to perform the count relative to another query as well. Finally, you need some way of communicating the total count back to the client. One way of doing this is wrapping the result set of your query in another object that contains an attribute for the count, or returns said object as a result of your DAO call. Or alternatively, you could store the count attribute in an object in your request scope.
public class ResultListWrapper<T> {
private Long count;
private Collection<T> results;
// constructor, getters, setters
}

You will usually perform the same query except using count int the select list instead of the columns prior running the actual query to find how many pages there are. Getting a specific page in hibernate can then be done with something like:
int maxRecords = // page size ...
int startRow = // page number * page size
Query query = session.createQuery('...');
query.setMaxResults(maxRecords);
query.setFirstResult(startRow);
If performing the extra query is too expensive an operation then you could consider just providing next/previous links or alternatively providing extra pages as needed (e.g. As the last of the loaded data comes into view) via ajax.
For displaying paged results including showing page number links I would suggest (assuming you are using jsp pages to display this data) using the display tag JSP tag library. Display tag handles paging and sorting display of tabular data assuming you can get it to the JSP.

Yes, you have to do a select count before getting the rows themselves.

Have a look at Spring Data JPA. It's basically working how Perception suggested earlier, just that you don't have to do it, the framework does that for you. You get paging and sorting by default (you still need to do filtering yourself if you don't want to use Query DSL).
They have a 1h video on the main page that's showing a quick intro, but you'll only need to watch about 15 mins to get an idea of what id does.

way(client side or server side) to go for pagination /sortable columns?

I have 3000 records in an employee table which I have fetched from my database with a single query. I can show 20 records per page. So there will be 150 pages for with each page showing 20 records. I have two questions on pagination and sortable column approach:
1) If I implement a simple pagination without sortable columns, should I send all 3000 records to client and do the pagination client side using javascript or jquery. So if user clicks second page, call will not go to server side and it will be faster. Though I am not sure what will be the impact of sending 3000 or more records on browser/client side? So what is the best approach either sending all the records to client in single go and do the sorting there or on click of page send the call to server side and then just return that specific page results?
2) In this scenario, I need to provide the pagination along with sortable columns (6 columns). So here user can click any column like employee name or department name, then names should be arranged in ascending or descending order. Again I want to know the best approach in terms of time response/memory?

Sending data to your client is almost certainly going to your bottleneck (especially for mobile clients), so you should always strive to send as little data as possible. With that said, it is almost definitely better to do your pagination on the server side. This is a much more scalable solution. It is likely that the amount of data will grow, so it's a safer bet for the future to just do the pagination on the server.
Also, remember that it is fairly unlikely that any user will actually bother looking through hundreds of result pages, so transferring all the data is likely wasteful as well. This may be a relevant read for you.

I assume you have a bean class representing records in this table, with instances loaded from whatever ORM you have in place.
If you haven't already, you should implement caching of these beans in your application. This can be done locally, perhaps using Guava's CacheBuilder, or remotely using calls to Memcached for example (the latter would be necessary for multiple app servers/load balancing). The cache for these beans should be keyed on a unique id, most likely the mapping to the primary key column of the corresponding table.
Getting to the pagination: simply write your queries to return only IDs of the selected records. Include LIMIT and OFFSET or your DB language's equivalent to paginate. The caller of the query can also filter/sort at will using WHERE, ORDER BY etc.
Back in the Java layer, iterate through the resulting IDs of these queries and build your List of beans by calling the cache. If the cache misses, it will call your ORM to individually query and load that bean. Once the List is built, it can be processed/serialized and sent to the UI.

I know this doesn't directly answer the client vs server side pagination, but I would recommend using DataTables.net to both display and paginate your data. It provides a very nice display, allows for sorting and pagination, built in search function, and a lot more. The first time I used it was for the first web project I worked on, and I, as a complete noobie, was able to get it to work. The forums also provide very good information/help, and the creator will answer your questions.
DataTables can be used both client-side and server-side, and can support thousands of rows.
As for speed, I only had a few hundred rows, but used the client-side processing and never noticed a delay.

USE SERVER PAGINATION!
Sure, you could probably get away with sending down a JSON array of 3000 elements and using JavaScript to page/sort on the client. But a good web programmer should know how to page and sort records on the server. (They should really know a couple ways). So, think of it as good practice :)
If you want a slick user interface, consider using a JavaScript grid component that uses AJAX to fetch data. Typically, these components pass back the following parameters (or some variant of them):
Start Record Index
Number of Records to Return
Sort Column
Sort Direction
Columns to Fetch (sometimes)
It is up to the developer to implement a handler or interface that returns a result set based on these input parameters.

ZK : Design Pattern : Paging through large amounts of data?

I would like some advice on how to tackle paging and sorting through large amounts of data.
At the moment, I retrieve all our user data and stick it in a listbox. This is great for now, but when we have 100 000 users, I don't want to populate a listbox with 100 000 users, nor have a collection on the server with 100 000 users on it.
What would you recommend is a good approach to do this?
Should I keep a ConnectedRecordset and pass that to the Listbox. Does ZK have the capability to manage the data and the connection?
~ OR ~
Would it be better to manually look for page events, and repopulate the listbox with the second set of data, only pulling say 10 rows at a time?
Also, when I want to allow a user to sort by a column, then surely one needs to requery the data from the database.
What is the correct way to challenge this? Please offer advice or links to articles.
Thanks,
Mark

You can take a look at the tutorial of Sorting huge data using ZK and Handling huge data using ZK

Display 1000's of records in jsp page

We are getting 1000 of records from services we need to display all records in jsp page. We have set the data to object and stored in java collections. How can get that collections in java script using Ajax and need to display 10 records in every time based on scroll we can load another 10 records upto completion.
Please suggest the compatible technology.
At this time We are using the struts2 and jquery.

It sounds like you want something along the lines of SlickGrid. It is very fast, and is the data grid that powers SEDE result tables.
Another option, which I have used before with great results, is a YUI DataTable with pagination (server-side or client-side). With client-side pagination — which is typically faster, since all the data is already in the browser — I've created YUI data tables that work with more data than the browser can parse at once, with minimal performance degradation.

You can try implementing a simple pagination technique
int totalRecords;
int maxRecordsPerPage;
int totalPages = ( totalRecords / maxRecordsPerPage );
int displayRecordFrom;
int displayRecordTo;
Total Records : Number of records fetched.
Max Records Per page :
Total Pages : this is optional, either you can display total pages some thing like google or just put next button or link
Display Record From And To : As you are storing records in collection, it can be fetched used get(index)
After fetching results use Jettison or any other Java JSON library to output results into JSON. Instead of working from scratch, Its better to use pre-tested third party Javascript components using JQuery or other library.