What is the best practice if you are implementing web services that will send and receive large files to/from clients. Normally we are sending JSON objects, but it could be problematic if we include large data payload inside of the JSON objects. We need to provide JSON data as well as a payload, anyone have experience with something similar?
You could embed links to the raw data in your JSON responses. For example:
{
title: 'A Really Big File',
date: '2011-11-11',
file: 'http://example.com/really_big_file.xls'
}
That way you can allow clients to decide whether or not they want to dereference the big file or not.
Base64 is a very inefficient way of doing this, but universal. You could send your files using HTTP Post-request with special parameter "multipart/form-data".
Related
this is actually two questions in one but they are closely related
I have read plenty of times that you have to send either json/string or binary data over a websocket like socket.io but that you cannot mix these types. But then I was puzzled by finding the following example in the official documentation of the socket.io client java implementation
// Sending an object
JSONObject obj = new JSONObject();
obj.put("hello", "server");
obj.put("binary", new byte[42]);
socket.emit("foo", obj);
// Receiving an object
socket.on("foo", new Emitter.Listener() {
#Override
public void call(Object... args) {
JSONObject obj = (JSONObject)args[0];
}
});
where the "binary" element of that json is clearly binary as the name suggests. The documentation talks about socket.io using org.json, but i couldnt find that this library supports adding binary data to json files anywhere.
is this functionality now supported? if so, what is socket.io doing in the background? is it splitting the emit in two separate messages and then remerging it? or is it simply saving the binary data in base64?
A bit of background.
I am trying to add a private chat functionality to my app so that a user can have multiple private two-party audio-message based chat conversations with several other users. i am having issues finding out how to tell my server how to forward all the messages. if i use a json i can simply add a sender and a receiver to the json and have the server read the id of the receiver and forward the message accordingly. but i am not sure how to handle messages containing only binary data. i have no idea how to add metadata (such a sender and a receiver id) to them so that the server knows to whom they are addressed. i have heard the suggestion of sending a json with a sender id, a receiver id and an MD5 hash of the file i am trying to send, and then send the binary data alone separately and having the server match the two messages over the md5 signature, but that seems to come with problems of its own. like i dont know how having to calculate the MD5 of a ton of audio files on the server is going to affect server performance and there is also the issue of potentially receiving the audio byte array before the json specifying its destination has arrived.
there is always the alternative of encoding my audio files in base64 and sending them as json, as i have been doing so far, but i have been told this is a bad practice and should be avoided as if inflates package sizes.
i feel like there are a bunch of messaging apps already out there, and i bet at least some are based on websockets. I would like to know if there are any best practices on how to route binary data over a websocket to a specific receiving connection.
Ill upvote any answer to the questions above as well as any hint on how to tackle the problem mentioned in the background part.
Technology Used:
In the question below, "frontend refers to Android" and "backend to Node.js".
Constraints:
We have rural users in developing market, so the internet may be slow and/or
jittery/unstable. Due to jitter, we need a solution where we can use whatever (if not all) data is transmitted.
We have quite large data (huge list of objects) which we cannot simple transmit through
JSON (via plain REST APIs), as until the whole data is downloaded,
we get nothing (because we are using Retrofit and its onResponse is
not called).
Goal:
To convert the list of objects (in backend) to binary data. So that
when we receive data in the frontend, we are able to access
serialized data without unpacking. Achieving it through FlatBuffers.
To transmit this data through streaming when triggered from
the frontend. I want to stream the data as I want to use (show in UI in
realtime) whatever data (list of objects) user has received (Even if
user gets disconnected during transmission). I am having issues
here, as I am unable to achieve this through REST API - Retrofit
combination. Need help here about what to use for trigger based
streaming.
To reconvert the list of objects in the frontend to Java objects and
show in user's UI. I am using FlatBuffer here, as it is fast and
able to use/serialize whatever objects are transmitted. No need for
entire data transmission to complete.
I am able to successfully implement step 1 & 3 of the goal. But, I am not able to sort out step 2.
Please suggest what is a good and easy way to achieve this (stream binary data from backend to frontend). It would be better, if we can trigger and stream using Retrofit (if possible) in the frontend.
Achieved by difference method:
On the Node side, use fs.createReadStream function for streaming.
On the Android side, use URLConnection, BufferedReader, InputStreamReader for consuming stream.
P.S - Didn't get any way to do it via Retrofit.
I want to write an application in Java that will communicate with Google App Engine app written in Go by sending and receiving dynamic data. The data is not human readable (as in, not ASCII, Unicode or the like) and ranges from a couple bytes to about 1MB.
I am wondering if it is possible to send such data to and from GAE using Post method directly, or is it better to just encode it as a hex dump and transfer is at text (thusly increasing its size a couple times)?
Yes, this is possible, of course. Just like an HTTP response, an HTTP request can contain a payload of any sort (unless it's a GET or other method that doesn't permit a body); just set the content-type appropriately and send the data in the body of the HTTP request.
If people can POST 10mb photos to Facebook, then I don't see why you can't do that with your data :)
Which are the Best practices to send huge amount of data from server to client in GWT?
Right now we are facing performance issue in GWT 2.3.0.
Our server side is sending huge xml (Size in MB/GB) to client side, our client side parses that xml and using parsed data, list of beans are formed for populating data in Celltable grid.
We are filling 1k + / 10k+ records in CellTable grid.
Is there any effective way/ Best practices followed while dealing with such a huge data?
If we parse the data at server side and formed the beans at server side, Is this good? or any alternative way..
Any help or guidance in this matter would be appreciated.
Basically you only request as much data (and a little more) than is currently viewed by the user, not the whole data set.
See Adding Paging Controls for further details.
Two practices when dealing with large data for your case:
1) Use JSON instead of xml, that way the client doesn't need to parse the data but can directly use the data. In GWT via the GWT-JSNI interface you can write data objects accessing these JavaScript objects, see: http://code.google.com/webtoolkit/doc/latest/DevGuideCodingBasicsJSNI.html. With a JSON REST library you can generate the JSON on the server instead of sending xml to the client. But you can also use GWT-RPC on both client/server, which makes programming easier because the whole data conversion to and from JSON is handled by GWT, but adds some type information to the objects send.
2) Use paging: only get data that is visible to the user and cache it in the client. If you have a table presentation it's not likely the user needs all the data at once. GWT cell panels have support out of the box (see the link Oliver mentions about Adding Paging Controls)
As in the other answers, return only the data that can usefully be used by the user and lazily fetch other data when the user requests it (or Predictive Fetch).
See AsyncDataProvider section here:
http://code.google.com/webtoolkit/doc/latest/DevGuideUiCellWidgets.html#data-provider
Use paging. GWT cell widgets support paging out-of-the-box. So implement server side paging so that each time you click 'next' a server call is made. That way, the client only deals with say 10 or 20 records at a time.
Use Javascript Overlay types as the display beans. And to populate these beans, use JSON as the transport model instead of XML. If you use XML (i.e. Async calls), then GWT does some JAXB marshalling/unmarshalling logic in the back end. If you use JSON, much of that is avoided.
I don't know a lot about this area so please excuse me if my question is vague or stupid.
I have a webpage which uses javascript and AJAX to display live data. Every few seconds, a request is made and a JSON response is returned and the data on the webpage is updated.
What I want to do is create a program in Java that will basically capture every response and interpret the data. I have found libraries which handle the JSON format already. However, I don't know how to get the response using Java.
So for example, a live news feed. I would like to log the data as it appears.
Thanks
Basically what you need to do is make an HTTP GET request to the page that hosts the JSON. You can do this by using a Java HTTP client. The one in the link is from Apache Commons but I believe there is actually one built into Java that is relatively straight-forward to use. When you make a request, it will return a result object that you can then use to access the response data and information such as response headers, etc.