We Keep Coding

Strategies to build real-time data grid using Oracle, Java & AngularJS

This is my first stack-overflow post, so please ignore/forgive if I am not being specific enough.I'm sure I will learn the process gradually.
I have built a JSON to be displayed in angular data grid. This JSON comes from a complex query over materialized view.My thought to refresh the JSON as underlying data changes is as follows:
a) Register query for Oracle CQRN (Oracle Continuous Query Result Change Notification) at application startup
b) When the Underlying data changes, Oracle Database Change Listener in Java side gets invoked and Ire-query the data (with change) and push it to socket end-point. That way the JSON gets changed with latest data.
This works fine with simple query.
Issues are:
a) In my case the query is very complex and involves multiple materialized views with UNION ALL and complex JOINS. CQRN does not support materialized view registration for query result change.
b) The query I am registering at start-up, for query result change notification, is pretty static. It does not meet the requirement of various different parameterized queries behind the data-grid.
Can anyone suggest any other alternative for example cache the grid data in the middle-tire and and refresh cache with updated data whenever the underlying grid data changes. I should be notified when underlying grid data changes so I will re-query & send the updated data to socket end-point, which will refresh the grid.
I have to show the grid-data changes in real-time, so I have used Java WebSocket (JSR 356)
Technology stack:
UI: Javascaipt/AngularJS
Middle-tier: Java 1.7
Server: Jetty 9.2
Database: Oracle 11g R2
Build Platform: Maven 3.3
Suggestion for any other suitable approach also will be much appreciated.
Thanks & regards,
- Joy

While not directly answering your question we just implemented a real time data grid involving multiple data sources and CQRN. This built in is based on a table changing. Our technique was:
add on insert trigger (data feed was real time, no deletes, no updates) to the base tables
call a stored procedure to manipulate the data. You would use the logic in your materialized view. The procedure inserts data into a destination table. That has a trigger to call the CQRN.
often with realtime you need to delete old data so everything stays fast

Java Data base query/update for large amount of data

What is the best way to implement the following scenario?
I need to call/query a data base table containing millions of records from a java application. Then for each records in the table, my application should call a third party API and get a status field as response. Then my application should again update each row in the table with the information (status) from the API.
Note - I am trying to figure out a method to do this in the best possible way. I understand that querying all the records together is not the best way forward.

Do not try to eat the elephant in one bite. Chunk it. Heard of pagination? Use it. See here: MySQL pagination without double-querying?

you can use oracle feature such as SQL loader, Data pumping Called via JDBC or script..

Databases are not designed to update millions of records via Java API repeatedly. This can take many minutes. If this is not enough, you may need to use a dataset embedded in Java (either caching or replacing your database)

way(client side or server side) to go for pagination /sortable columns?

I have 3000 records in an employee table which I have fetched from my database with a single query. I can show 20 records per page. So there will be 150 pages for with each page showing 20 records. I have two questions on pagination and sortable column approach:
1) If I implement a simple pagination without sortable columns, should I send all 3000 records to client and do the pagination client side using javascript or jquery. So if user clicks second page, call will not go to server side and it will be faster. Though I am not sure what will be the impact of sending 3000 or more records on browser/client side? So what is the best approach either sending all the records to client in single go and do the sorting there or on click of page send the call to server side and then just return that specific page results?
2) In this scenario, I need to provide the pagination along with sortable columns (6 columns). So here user can click any column like employee name or department name, then names should be arranged in ascending or descending order. Again I want to know the best approach in terms of time response/memory?

Sending data to your client is almost certainly going to your bottleneck (especially for mobile clients), so you should always strive to send as little data as possible. With that said, it is almost definitely better to do your pagination on the server side. This is a much more scalable solution. It is likely that the amount of data will grow, so it's a safer bet for the future to just do the pagination on the server.
Also, remember that it is fairly unlikely that any user will actually bother looking through hundreds of result pages, so transferring all the data is likely wasteful as well. This may be a relevant read for you.

I assume you have a bean class representing records in this table, with instances loaded from whatever ORM you have in place.
If you haven't already, you should implement caching of these beans in your application. This can be done locally, perhaps using Guava's CacheBuilder, or remotely using calls to Memcached for example (the latter would be necessary for multiple app servers/load balancing). The cache for these beans should be keyed on a unique id, most likely the mapping to the primary key column of the corresponding table.
Getting to the pagination: simply write your queries to return only IDs of the selected records. Include LIMIT and OFFSET or your DB language's equivalent to paginate. The caller of the query can also filter/sort at will using WHERE, ORDER BY etc.
Back in the Java layer, iterate through the resulting IDs of these queries and build your List of beans by calling the cache. If the cache misses, it will call your ORM to individually query and load that bean. Once the List is built, it can be processed/serialized and sent to the UI.

I know this doesn't directly answer the client vs server side pagination, but I would recommend using DataTables.net to both display and paginate your data. It provides a very nice display, allows for sorting and pagination, built in search function, and a lot more. The first time I used it was for the first web project I worked on, and I, as a complete noobie, was able to get it to work. The forums also provide very good information/help, and the creator will answer your questions.
DataTables can be used both client-side and server-side, and can support thousands of rows.
As for speed, I only had a few hundred rows, but used the client-side processing and never noticed a delay.

USE SERVER PAGINATION!
Sure, you could probably get away with sending down a JSON array of 3000 elements and using JavaScript to page/sort on the client. But a good web programmer should know how to page and sort records on the server. (They should really know a couple ways). So, think of it as good practice :)
If you want a slick user interface, consider using a JavaScript grid component that uses AJAX to fetch data. Typically, these components pass back the following parameters (or some variant of them):
Start Record Index
Number of Records to Return
Sort Column
Sort Direction
Columns to Fetch (sometimes)
It is up to the developer to implement a handler or interface that returns a result set based on these input parameters.

Pagination in SAS based software

I am developing a SAS based system using jsp, servlet and java. And i am confused in using client side pagination(getting all result at one go) or using server side(With every click).
If i use client side pagination how much data is o.k. And what is the best way to implement pagination like javascript, ajax, jquery etc.

It really depends upon volume of your data and the probability of user loading that page in his session.
If the dataset is limited to say max 100* rows or so, and each record has few columns with small data size, you could go for client side pagination. But if the max of dataset is unknown or its gonna grow gradually, its best to go for a server side pagination.
Ajax with jQuery is definitely the way to go. Every jQuery grid plugin has its own mechanism for pagination but the basic logic is similar :
You need to design your backend APIs such that they accept
maxResults and currentPage as params, along with other params.
The API which interacts with your DB fetches maximum maxResults no. of rows and the first row starts from (currentPage -1) * pageSize
I have been using jQgrid and found it very well documented and simple to implement.
Helpful post :
https://stackoverflow.com/questions/159025/jquery-grid-recommendations
NB: *100 is just an example don't talke it literally :)

I reckon js/jquery based pagination with Ajax for data fetch is great. You need to consider if you need sorting or not. I implemented jQuery based pagination 7 months back, and that time the pagination almost become really slow (1000 rows 10 rows per page) because of too much data. So, please make sure you implement Ajax based pagination.
I used this: http://tablesorter.com/docs/
and this: http://tablesorter.com/docs/example-ajax.html
Also, antoher helpful link:
http://www.xarg.org/2011/09/jquery-pagination-revised/
P.S. - Be very careful about syntax and classes for pagination implementation. One spelling mistake and you might get into circles.
Extra information: If you are thinking of using a different language, try using Ruby on Rails. You can use will_paginate or Kaminari gem for simple implementation of pagination.

I'd certainly look at a combination of server side pagination and client side pagination.
There is no sense in returning 1000 (or 10000+) rows of data, if all you are going to do is display 10 at a time.
If you are going to be displaying data in a grid, then I'd suggest you look at Datatables.net. They have some great examples including pagination and pipelining your data from the server (ie. returning a few more records than you actually display so you have fewer calls to retrieve more data).

Display tag Issues with pagination and sorting

I am using display tag to render tables in my application. When I click either pagination or sorting, it's querying the database again and then sorting/paging. In my opinion, when we are only sorting/paging, we need not query database.
I see following options to tackle this:
I can keep the list (queried from the db) in the session and check it this request is for sorting or pagination then return the list from the session
Or I can maintain a server side cache which will work the same as above
Please correct me if I am missing something I think this functionality should already have been in the display tag library?

I have wondered about this in the past, and from what I've seen, you're on the right track with having to store the data somewhere like the session if you want to use the library-provided pagination/sorting. It doesn't seem to provide a convenient framework to do the actions client-side, or a dummy servlet to that exists just to post/return the same data.
Your alternative is to use external pagination/sorting with a partial list. This requires you to go to the database, but you can return a smaller amount of data than the entire result set.
Here are a couple related posts which seem to imply the same conclusions:
DisplayTag pagination vs Hibernate Pagination
Display tag pagination problem