I currently have a Richfaces dataTable bound to a backing bean that contains all the data. This table is bound to a Richfaces data scroller.
Users can then enter scroll between the data.
There is also a text box that dynamically updates the data table results displayed to them, based on text they enter into it. It reRenders the dataTable and datascroller on the keyUp event.
The backing bean it’s bound to first pulls all the data from a number of database tables. This data is pulled when the user submits a normal post request to the server, and it’s the results of this request that are used for all subsequent Ajax related queries (Results saved as list of objects, no more database calls made).
I have a problem in that the dataset can be huge at times, 100,000s of records.
This causes the initial request to the server to take a very long time.
A possible solution I'm looking at is pulling back only a small amount of the data in one thread for the initial user port request. This data can then be displayed in the data table while the main thread works in the background pulling the bulk of data back.
Is this feasible? Would it be possible to update my datatable/datascoller as the main thread pulls back new data? Would it be difficult?
Database and query optimization has been looked at, so no more improvements of any significance can be made there.
Thanks in advance (I know this probably is not an easy question to answer)
Implement SerializableDataModel to hold your data for sorting and paginating.
Seems like you need exactly what Seam Application Framework's Query object is providing.
If you don't want to use Seam, you can either view the source, and copy how they are doing it and just steal the idea.
Basically what you need to do is fetch a given result set for each time the user press next, previous, first, last etc
Related
Our team is building a small application wherein a UI has about 10 drop-down list boxes. ( DDLB ).
These list boxes will be populated by selecting data from different tables.
Our JAVA person feels that making separate database call for each list will be very expensive and wants to make a single database call for all lists.
I feel it is impractical to populate all lists in one database call due to following reason
a. Imagine an end user chooses state = 'NY' from one DDLB.
b. The next drop down should be populated with values from ZIP_CODES table for STATE='NY'
Unless we know ahead of time what state a user will be choosing - our only choice is to populate a java structure with all values from ZIP_CODES table. And after the user has selected the state - parse this structure for NY zipcodes.
And imagine doing this for all the DDLB in the form. This will not only be practical but also resource intensive.
Any thoughts ?
If there are not many items in those lists and memory amount allows you could load all values for all drop boxes into memory at application startup and then filter data in memory. It will be better then execute SQL query for every action user makes with those drop boxes.
You could also use some cache engines (like EhCache) that could offload data to disk and store only some fraction in memory.
You can run some timings to see, but I suspect you're sweating something that might take 100th of a second to execute. UI design wise I never put zip codes in selection menus because the list is too long and people already know it well enough to just punch in. When they leave the zip code field I will query the city and state and pre-fill those fields if they're not already set.
I have a Spring MVC Webapplication, there is one form from which I am taking some input,based on those inputs, I am creating one unique id (by checking it in the database,logic in Java side not database for creating unique id). Everything is fine till now.
Now I want to give a bulk upload option through CSV file. I parsed whole data from csv file and I tried to insert data (8000 records) in the database using TaskExecutionService. But still this whole process is not much fast that I get a immediate response, so I want to show the progress of whole task on jsp page , so that It looks interactive and also on completion of all threads, I want to display a link to download csv file with all the unique ids.
I don't know how to update page as threads are running behind the scenes?
Any other approach for this process is also welcome, if compatible with spring MVC.
This can be done using ajax, where you can poll the progress percentage from your back end and use js/jquery or other plugins to show the progress.Any how you have to design the logic to get the percentage of work done by your threads.May be you can count the number of rows inserted and get a logic out of it to calculate the percentage of work done. so there should be a method which returns the % of work done and ajax calls to this method can help to display the progress.
refer jquery progress bar
Progressbar.js
I have 3000 records in an employee table which I have fetched from my database with a single query. I can show 20 records per page. So there will be 150 pages for with each page showing 20 records. I have two questions on pagination and sortable column approach:
1) If I implement a simple pagination without sortable columns, should I send all 3000 records to client and do the pagination client side using javascript or jquery. So if user clicks second page, call will not go to server side and it will be faster. Though I am not sure what will be the impact of sending 3000 or more records on browser/client side? So what is the best approach either sending all the records to client in single go and do the sorting there or on click of page send the call to server side and then just return that specific page results?
2) In this scenario, I need to provide the pagination along with sortable columns (6 columns). So here user can click any column like employee name or department name, then names should be arranged in ascending or descending order. Again I want to know the best approach in terms of time response/memory?
Sending data to your client is almost certainly going to your bottleneck (especially for mobile clients), so you should always strive to send as little data as possible. With that said, it is almost definitely better to do your pagination on the server side. This is a much more scalable solution. It is likely that the amount of data will grow, so it's a safer bet for the future to just do the pagination on the server.
Also, remember that it is fairly unlikely that any user will actually bother looking through hundreds of result pages, so transferring all the data is likely wasteful as well. This may be a relevant read for you.
I assume you have a bean class representing records in this table, with instances loaded from whatever ORM you have in place.
If you haven't already, you should implement caching of these beans in your application. This can be done locally, perhaps using Guava's CacheBuilder, or remotely using calls to Memcached for example (the latter would be necessary for multiple app servers/load balancing). The cache for these beans should be keyed on a unique id, most likely the mapping to the primary key column of the corresponding table.
Getting to the pagination: simply write your queries to return only IDs of the selected records. Include LIMIT and OFFSET or your DB language's equivalent to paginate. The caller of the query can also filter/sort at will using WHERE, ORDER BY etc.
Back in the Java layer, iterate through the resulting IDs of these queries and build your List of beans by calling the cache. If the cache misses, it will call your ORM to individually query and load that bean. Once the List is built, it can be processed/serialized and sent to the UI.
I know this doesn't directly answer the client vs server side pagination, but I would recommend using DataTables.net to both display and paginate your data. It provides a very nice display, allows for sorting and pagination, built in search function, and a lot more. The first time I used it was for the first web project I worked on, and I, as a complete noobie, was able to get it to work. The forums also provide very good information/help, and the creator will answer your questions.
DataTables can be used both client-side and server-side, and can support thousands of rows.
As for speed, I only had a few hundred rows, but used the client-side processing and never noticed a delay.
USE SERVER PAGINATION!
Sure, you could probably get away with sending down a JSON array of 3000 elements and using JavaScript to page/sort on the client. But a good web programmer should know how to page and sort records on the server. (They should really know a couple ways). So, think of it as good practice :)
If you want a slick user interface, consider using a JavaScript grid component that uses AJAX to fetch data. Typically, these components pass back the following parameters (or some variant of them):
Start Record Index
Number of Records to Return
Sort Column
Sort Direction
Columns to Fetch (sometimes)
It is up to the developer to implement a handler or interface that returns a result set based on these input parameters.
I have a customer with a very small set of data and records that I'd normally just serialize to a data file and be done but they want to run extra reports and have expandability down the road to do things their own way. The MySQL database came up and so I'm adapting their Java POS (point of sale) system to work with it.
I've done this before and here was my approach in a nutshell for one of the tables, say Customers:
I setup a loop to store the primary key into an arraylist then setup a form to go from one record to the next running SQL queries based on the PK. The query would pull down the fname, lname, address, etc. and fill in the fields on the screen.
I thought it might be a little clunky running a SQL query each time they click Next. So I'm looking for another approach to this problem. Any help is appreciated! I don't need exact code or anything, just some concepts will do fine
Thanks!
I would say the solution you suggest yourself is not very good not only because you run SQL query every time a button is pressed, but also because you are iterating over primary keys, which probably are not sorted in any meaningful order...
What you want is to retrieve a certain number of records which are sorted sensibly (by first/last name or something) and keep them as a kind of cache in your ArrayList or something similar... This can be done quite easily with SQL. When the user starts iterating over the results by pressing "Next", you can in the background start loading more records.
The key to keep usability is to load some records before the user actually request them to keep latency small, but keeping in mind that you also don't want to load the whole database at once....
Take a look at indexing your database. http://www.informit.com/articles/article.aspx?p=377652
Use JPA with the built in Hibernate provider. If you are not familiar with one or both, then download NetBeans - it includes a very easy to follow tutorial you can use to get up to speed. Managing lists of objects is trivial with the new JPA and you won't find yourself reinventing the wheel.
the key concept here is pagination.
Let's say you set your page size to 10. This means you select 10 records from the database, in a certain order, so your query should have an order by clause and a limit clause at the end. You use this resultset to display the form while the users navigates with Previous/Next buttons.
When the user navigates out of the page then you fetch an other page.
https://www.google.com/search?q=java+sql+pagination
Say you retrieve 100 records, and display them on a page. The user only updates 2 of the records on the page. Now you want to update only the two records, and not the other 98.
Is it best to have one submit on the page, then somehow know which 2 are updated, then send only those two to the db for an update?
What does the "somehow" look like?
Or, would you have an update-submit button for each row, and have it only update the record its tied to?
Of course there are different ways you could do this. In general, you can save yourself some trouble and server-side processing by using Javascript to assemble your POST data for only the records that have changed. Two thoughts on how this might work:
1) Go the ajax route and do live-editing. So records are presented in a table and appear to be non-editable. When a user clicks a particular row, that row becomes editable by using Javascript to create the appropriate html form on the fly. Then have either a submit button or some other handler (say, moving focus to another table row) which will trigger the POST which updates the DB (asynchronously via your preferred ajax method). Happily the mainstream Javascript frameworks can help a lot in this area.
2) Checkboxes - whenever a row is edited, its checkbox becomes checked. When the submit button is clicked, use javascript to post the POST data by grabbing everything in row whose checkbox is checked. A user can un-check a box to cancel changes to that row before submitting.
Ajax it using jQuery or some other JavaScript library and put and update button on each row.
There are many answers to this question and to some extent they depend upon your development tools and the "feel" of the site.
If you were implementing Ajax calls to do the updates on a line by line basis then this would logically seem right to have a button per line and then update it with an Ajax call when a line was changed.
This is also just the scenario that disconnected data sets were designed to solve and ADO.net handles these very well.
So as ever, the answer is "It Depends!"
You can use JavaScript to mark each field as changed when a user changes an input field. Create a hidden fields that has the id of the row you are updating, and dirty flag. (like is_dirty_$id) In JavaScript, create an onChange handler that sets the hidden field as dirty. when any input is changed.
Alternatively, you can create hidden fields for each real field you display. the hidden field would contain the initial values. check each field on the server side to determine what has changed.
You probably want to store a last_modified date as a hidden field for each record. This way if another user updates the same record, you can display an error message saying "this record has been updated by another user" or similar.
One submit button. I could foresee case I might use more then one, but in the general case just one. (Note, this looks like a web page question to me, so I'm answering with that assumption.)
There are 3 ways that come to mind which you could handle the tracking changes:
JavaScript: Put a onChange() function on the controls that update a hidden field. If that hidden has a value, then update the associated record. Requires JS on the browser, and doesn't tell you which fields to update, just which records.
Lots of form fields: Put a hidden field out with each control and compare them all when they come back. This would be ugly, but it would allow you to know which fields to update (not just the record). It would also allow you to know if someone undid a change that started.
Sessions: You could place the original values in session variables, then do the comparison when the values come back. This would be a little more elegant then lots of hidden fields, and less open to people playing with the posted back data (since you should never trust anything that comes back, even in hidden fields). Requires cookies on the browser and sessions on the server technology.