In my GWT application I am retrieving the XML data from a REST server. I am using Piriti XML parser https://code.google.com/p/piriti/wiki/Xml for deserializing the object and display in a table. As long as we are returning upto 1000 records everything is fine but with the big result it just hang and gives user message to stop the java script running in the back. Could someone please help me to find the best way to handle big data in GWT OR more precisely the best approach to parse big XML file in GWT.
Thanks a lot for all your suggestions.
The problem is that parsing a big XML document slows down the browser. And you need enough memory to hold the whole DOM plus your mapped objects in memory. The only solution is to avoid such situation. You have to adapt your REST service to be able to send only small chunks of data to the browser. So if you already have a paged table you only retrieve the data for the first page at the beginning. If the user wants to change the page you do another REST call to retrieve the data for the next page.
If you cannot change the the REST service itself you can create another server side service (on a server controlled by you) as a proxy. At first access you call the original REST service, store the XML at your own server and allow the client to retrieve only parts of that XML.
Related
Technology Used:
In the question below, "frontend refers to Android" and "backend to Node.js".
Constraints:
We have rural users in developing market, so the internet may be slow and/or
jittery/unstable. Due to jitter, we need a solution where we can use whatever (if not all) data is transmitted.
We have quite large data (huge list of objects) which we cannot simple transmit through
JSON (via plain REST APIs), as until the whole data is downloaded,
we get nothing (because we are using Retrofit and its onResponse is
not called).
Goal:
To convert the list of objects (in backend) to binary data. So that
when we receive data in the frontend, we are able to access
serialized data without unpacking. Achieving it through FlatBuffers.
To transmit this data through streaming when triggered from
the frontend. I want to stream the data as I want to use (show in UI in
realtime) whatever data (list of objects) user has received (Even if
user gets disconnected during transmission). I am having issues
here, as I am unable to achieve this through REST API - Retrofit
combination. Need help here about what to use for trigger based
streaming.
To reconvert the list of objects in the frontend to Java objects and
show in user's UI. I am using FlatBuffer here, as it is fast and
able to use/serialize whatever objects are transmitted. No need for
entire data transmission to complete.
I am able to successfully implement step 1 & 3 of the goal. But, I am not able to sort out step 2.
Please suggest what is a good and easy way to achieve this (stream binary data from backend to frontend). It would be better, if we can trigger and stream using Retrofit (if possible) in the frontend.
Achieved by difference method:
On the Node side, use fs.createReadStream function for streaming.
On the Android side, use URLConnection, BufferedReader, InputStreamReader for consuming stream.
P.S - Didn't get any way to do it via Retrofit.
Is it possible to send extra data attached to a http response via Java or Php?
My Website is a homework-platform: One User enters homeworks into a database, and all users can then see the homeworks on the website. The current load is very inefficient, as the browser makes two requests for eveything to load: One for the index file and one for the homeworks. For the homeworks request the client also sends settings of the user to the server, based on which the returned homeworks are generated by a Php script.
Now, I wonder, if it is possible, to combine those two requests into one? Is it maybe possible to detect the http request with Java or Php on the server, read the cookies (where the settings are saved), then get the homeworks from the database and send the data attached to the http response to the client? Or, even better, firstly only return the index file and as soon as possible and the homework data afterwards as a second response, because the client needs some time to parse the Html & build the DOM-tree when it can't show the homeworks anyway.
While browsing the web I stumbled across terms like "Server-side rendering" and "SPDY", but I don't know if those are the right starting points.
Any help is highly appreciated, as I'm personally very interested in a solution and it would greatly improve the load time of my website.
A simple solution to your problem is to initialize your data in the index file.
You would create a javascript object, and embed it right into the html, rendered by your server. You could place this object in the global namespace (such as under window.initData), so that it can be accessed by the code in your script.
<scipt>
window.initData = {
someVariable: 23,
}; // you could use json_encode if you use php, or Jackson if you use java
</script>
However, it is not a huge problem if your data is fetched in a separate server request. Especially when it takes more time to retrieve the data from the database/web services, you can provide better user experience by first fetching the static content very quickly and displaying a spinner while the (slower) data is being loaded.
We have seen lot of applications who are working with JSON file but i have a case study of which i want to get solution.
Let us see ...
a app is working with json file which gets requests from million users and every second thousands of requests has been completed.
JSON file is updated by admin panel every minute or second or specific time frame.
what will be behaviour of JSON file while request has been received to access JSON file and open for update from admin at same time (i have read it that JSON file will be fetched in readable mode.)
Let JSON file is writing using some script and its process is third of a second than what will be behaviour while 50% file has been updated.
Either file will be given with new written content when process completed or when it was partially updated?
Don't bother with locking, just use rename().
Assuming you're running on an OS where a rename() is an atomic operation, create a new file, say "/data/file/name.json.new", then when that's complete, rename the file. In C that would look like this:
rename( "/data/file/name.json.new", "/data/file/name.json" );
This way, any process opening "/data/file/name.json" will always see a consistent data file.
Practically, by what you describe, you want a service that applies operations on a file server-side.
You should though avoid taking the responsibility of Creating, Readind, Updating and Deleting (CRUD), as you will have troubles on preserving principles such as Atomicity, Consistency, Isolation and Durability (ACID), while there are systems doing that for you, the Database Management Systems.
In simple words, scenarios like what you describe should be a responsibility of a DBMS and not yours.
You probably need a NoSQL DBMS, that responsible for the CRUD operations of your database - which can be file-based in a JSON format and other forms, preserving ACID always (or almost always, but this is probably something you will learn on searching on it). MongoDB is a great example of such system.
Because you mentioned JSON, please take into consideration that it is another story to transfer the data, and another to store them. I suggest that you use the JSON format for requests & responses, but explore other options in storage. For instance, even a Relational DBMS that uses SQL can be good for you, it always depends on your needs. You might just need to form (encode & decode) the data in JSON format wherever received or sent to each client.
Take a look here for more info.
I have a PHP page that grabs a variable via GET and then pulls in some information from a database based on that variable. Once finished with the server-side stuff, there is some javascript that runs and takes the data supplied and creates a .png image using a 3rd party API and then saves that image to my server using an AJAX POST call to another PHP page.
That all works fine, but what I'd like to do now is automate some calls to that PHP page. Namely, say I have 100 such variables to go through, I want to, preferably in Java with a for loop, call that PHP page with each variable in turn.
The problem is that client-side javascript. It won't execute with the simple URLConnection in Java. It seems like I need some sort of browser replicator or some way to have java act like it's calling the PHP in a browser?
Alternatively, I could make do with having a third PHP page act in place of the Java as the controller, but I'm faced with the same problem of getting the javascript to execute.
Am I missing something easy? Is this set up not possible? I'd really prefer to do it in Java if possible to fold it into other code I already have running.
Let me try to add some more specifics without bogging it down too much. There's a PHP file getData.php that takes in an ID number via GET. So I call it like ./getData.php?id=someId
That PHP file takes the ID, goes to my DB and retrieves some data and pastes it into the HTML source. Then once the page is finished, I have some javascript within getData.php that retrieves that data, formats it into a DataTable and passes it off to Google Visualization API in order to make a SVG chart.
Then I have more JS that runs that takes that SVG object, turns it into a Canvas object, grabs the base64 image data from it and finally POSTs to saveTo.php with the following array:
{'id' : id, 'data' : imgData}
saveTo.php simply takes in that POST data, creates a file on my server based on id and pastes the imgData into it. The end result is that I can pass in an ID to getData.php and end up with saved image of a Visualization chart that I want made based on data in my DB tied to that ID.
That all works by hand. But I have ~1,000 of these IDs and I'd like to have it so this whole process is run each morning so that I can have updated images based on yesterday's data.
I should mention that I did try using the 3rd party toolkit HtmlUnit (http://htmlunit.sourceforge.net/) but just keep getting these errors:
com.gargoylesoftware.htmlunit.IncorrectnessListenerImpl notify
WARNING: Obsolete content type encountered: 'text/javascript'.
Some more searching around and hitting upon the right keywords to try finally led me to Selenium, which is exactly what I was looking for.
http://docs.seleniumhq.org/projects/webdriver/
Which are the Best practices to send huge amount of data from server to client in GWT?
Right now we are facing performance issue in GWT 2.3.0.
Our server side is sending huge xml (Size in MB/GB) to client side, our client side parses that xml and using parsed data, list of beans are formed for populating data in Celltable grid.
We are filling 1k + / 10k+ records in CellTable grid.
Is there any effective way/ Best practices followed while dealing with such a huge data?
If we parse the data at server side and formed the beans at server side, Is this good? or any alternative way..
Any help or guidance in this matter would be appreciated.
Basically you only request as much data (and a little more) than is currently viewed by the user, not the whole data set.
See Adding Paging Controls for further details.
Two practices when dealing with large data for your case:
1) Use JSON instead of xml, that way the client doesn't need to parse the data but can directly use the data. In GWT via the GWT-JSNI interface you can write data objects accessing these JavaScript objects, see: http://code.google.com/webtoolkit/doc/latest/DevGuideCodingBasicsJSNI.html. With a JSON REST library you can generate the JSON on the server instead of sending xml to the client. But you can also use GWT-RPC on both client/server, which makes programming easier because the whole data conversion to and from JSON is handled by GWT, but adds some type information to the objects send.
2) Use paging: only get data that is visible to the user and cache it in the client. If you have a table presentation it's not likely the user needs all the data at once. GWT cell panels have support out of the box (see the link Oliver mentions about Adding Paging Controls)
As in the other answers, return only the data that can usefully be used by the user and lazily fetch other data when the user requests it (or Predictive Fetch).
See AsyncDataProvider section here:
http://code.google.com/webtoolkit/doc/latest/DevGuideUiCellWidgets.html#data-provider
Use paging. GWT cell widgets support paging out-of-the-box. So implement server side paging so that each time you click 'next' a server call is made. That way, the client only deals with say 10 or 20 records at a time.
Use Javascript Overlay types as the display beans. And to populate these beans, use JSON as the transport model instead of XML. If you use XML (i.e. Async calls), then GWT does some JAXB marshalling/unmarshalling logic in the back end. If you use JSON, much of that is avoided.