I have a large response which is a JSON object at least 88Kb, I am having problems receiving this data on the clients.
I would like to zip the content in the Java application and send back the zip, I've done this before when using PHP on the server to zip large content and the browser then unzips it.
At present the JSON string is built up in a StringBuilder object. If this idea is flawed what else could I do? The JSON object contains status information for a large system and I need frequent updates to be sent.
Edit...I've progressed the problem, if the size of the JSON is > 512 bytes then I pass the StringBuffer onto the function below:
public StringBuilder zipStringBuilder(StringBuilder sbSource) {
StringBuilder sbZipped = null;
try {
byte[] arybytSource = sbSource.toString().getBytes();
byte[] arybytBuffer = new byte[ZIP_BUFFER];
Deflater objDeflater = new Deflater();
objDeflater.setInput(arybytSource);
ByteArrayOutputStream baos = new ByteArrayOutputStream(arybytSource.length);
objDeflater.finish();
while( !objDeflater.finished() ) {
int intCount = objDeflater.deflate(arybytBuffer);
baos.write(arybytBuffer, 0, intCount);
}
baos.close();
sbZipped = new StringBuilder();
sbZipped.append(baos.toString());
} catch (IOException e) {
e.printStackTrace();
}
return sbZipped;
}
The HTTP headers are as follows:
HTTP/1.0 200
Date: Fri Nov 13 14:47:06 GMT 2015
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: POST, GET, OPTIONS, DELETE, PUT
Access-Control-Allow-Credentials: true
Keep-Alive: timeout=2, max=100
Cache-Control: no-cache
Pragma: no-cache
Content-Encoding: zip
Content-type: application/json;charset=utf-8
But I don't receive this or the browser doesn't understand it?
I've also tried:
Content-Encoding: gzip
With same result.
If your server is behind apache web server turn on the
apache inflate command
Related
I using Okhttp3 for download file from server in android application. my link is http://www.webweb.infinityfreeapp.com/lichi/download.php?path=Add.jpg it download file in firefox, chorme smoothly, while in okhttp3 response string shows
<html><body><script>document.cookie="_test=9e105a99e90025d241c180c29fad3231 ; expires=Thu, 31-Dec-37 23:55:55 GMT; path=/" ;document.location.href="http://www.webweb.infinityfreeapp.com/lichi/download.php?path=Add.jpg&i=1";</script></body></html>
but i feel response string has Add.jpg file data. so, what can i change in okhttp3 code or php code that i gather App.jpg data in response string of okhttp3
Php Code
if(isset($_GET['path']))
{
$url = $_GET['path'];
$type = "application/pdf";
$completePath = "http://www.webweb.infinityfreeapp.com/lichi/";
$visibleName = "$url";
$completePath .= $url;
// Force download
header("Content-disposition: attachment; filename=$visibleName");
header("Content-Type: application/force-download");
header("Content-Transfer-Encoding: $type\n");
// header("Content-Length: ".filesize($completePath));
header("Pragma: no-cache");
header("Cache-Control: must-revalidate, post-check=0, pre-check=0, public");
header("Expires: 0");
readfile($completePath);
die();
}
I comment Content-Length because it crash system in uncomment
Java code
OkHttpClient client = new OkHttpClient();
String url = "http://www.webweb.infinityfreeapp.com/lichi/download.php?path=Add.jpg";
Call call = client.newCall(new Request.Builder().url(url).get().build());
Response response = call.execute();
if (response.code() == 200 || response.code() == 201) {
Headers responseHeaders = response.headers();
for (int i = 0; i < responseHeaders.size(); i++)
Log.d(LOG_TAG, responseHeaders.name(i) + ": " + responseHeaders.value(i));
String str = response.body().string();
}
Here str contain above html file information instead Add.jpg file data. so please give answer
good question
Autually if we send a get request to
http://www.webweb.infinityfreeapp.com/lichi/download.php?path=Add.jpg
we get the right resutl just like
<html><body><script>document.cookie="_test=9e105a99e90025d241c180c29fad3231 ; expires=Thu, 31-D...";</script></body></html> .
we can get a file in browser, because browser can parse html ，
when browser get the string result which is a html page, it create another request with a new Header （Cookie=_test=9e105a99e90025d241c180c29fad3231), and with the Cookie, we get an image file from server.
Thanks for quick and good solution.
i just add header as:-
.header("Cookie", "_test=9e105a99e90025d241c180c29fad3231")
and send again with above code, actual result come
I need to receive multipart/form-data response
but i have no idea on how to parse this kind of response
For example
--mf8sckatxs4PpMnOLF6ltSv26ZJc5qxy9qq
Content-Disposition: form-data; name="arguments"
Content-Type: text/plain;charset=UTF-8
Content-Length: 311
[{"code":200,"message":"123"}]
Content-Disposition: form-data; name="_0"; filename="0_BODY_feature"
Content-Type: application/octet-stream
Content-Length: 407
binarydata
Since you're using Spring you already have the Apache fileupload and http classes available. MultipartStream is designed to be attached to an incoming byte stream and can provide progress updates as data arrives.
This simple example illustrates a non-streaming scenario where you've already buffered the whole incoming body.
byte[] yourResponse = ... // the whole response as a byte array
String yourContentType = ... // the Content-Type header string
ContentType contentType = ContentType.parse(yourContentType);
MultipartStream multipartStream = new MultipartStream(
new ByteArrayInputStream(yourResponse),
contentType.getParameter("boundary").getBytes(),
1024, // internal buffer size (you choose)
null); // progress indicator (none)
boolean nextPart = multipartStream.skipPreamble();
while (nextPart) {
ByteArrayOutputStream output = new ByteArrayOutputStream();
String partHeaders = multipartStream.readHeaders();
multipartStream.readBodyData(output);
// do something with the multi-line part headers
// do something with the part 'output' byte array
nextPart = multipartStream.readBoundary();
}
Add exception handling as required.
You can store and separate those values in an Array of Strings:
String[] array = "allyourinputtext".split(";");
This will separate the values after a semicolon. Then, you can access each value by doing this:
String content = array[0];
String name = array[1];
...
This doesn't solve the WHOLE problem(since not all values are separated by semicolons), but you can play with the arguments you pass to split() to separate your values.
Note: If you want to parse a String to int (the length value for example) you can use:
int length = Integer.parseInt(array[index];
Even though I set content type to text/html it ends up as application/octet-stream on S3.
ByteArrayInputStream contentsAsStream = new ByteArrayInputStream(contentAsBytes);
ObjectMetadata md = new ObjectMetadata();
md.setContentLength(contentAsBytes.length);
md.setContentType("text/html");
s3.putObject(new PutObjectRequest(ARTIST_BUCKET_NAME, artistId, contentsAsStream, md));
If however I name the file so that it ends up with .html
s3.putObject(new PutObjectRequest(ARTIST_BUCKET_NAME, artistId + ".html", contentsAsStream, md));
then it works.
Is my md object just being ignored? How can I get round this programmatically as over time I need to upload thousands of files so cannot just go into S3 UI and manually fix the contentType.
You must be doing something else in your code. I just tried your code example using the 1.9.6 S3 SDK and the file gets the "text/html" content type.
Here's the exact (Groovy) code:
class S3Test {
static void main(String[] args) {
def s3 = new AmazonS3Client()
def random = new Random()
def bucketName = "raniz-playground"
def keyName = "content-type-test"
byte[] contentAsBytes = new byte[1024]
random.nextBytes(contentAsBytes)
ByteArrayInputStream contentsAsStream = new ByteArrayInputStream(contentAsBytes);
ObjectMetadata md = new ObjectMetadata();
md.setContentLength(contentAsBytes.length);
md.setContentType("text/html");
s3.putObject(new PutObjectRequest(bucketName, keyName, contentsAsStream, md))
def object = s3.getObject(bucketName, keyName)
println(object.objectMetadata.contentType)
object.close()
}
}
The program prints
text/html
And the S3 metadata says the same:
Here are the communication sent over the net (courtesy of Apache HTTP Commons debug logging):
>> PUT /content-type-test HTTP/1.1
>> Host: raniz-playground.s3.amazonaws.com
>> Authorization: AWS <nope>
>> User-Agent: aws-sdk-java/1.9.6 Linux/3.2.0-84-generic Java_HotSpot(TM)_64-Bit_Server_VM/25.45-b02/1.8.0_45
>> Date: Fri, 12 Jun 2015 02:11:16 GMT
>> Content-Type: text/html
>> Content-Length: 1024
>> Connection: Keep-Alive
>> Expect: 100-continue
<< HTTP/1.1 200 OK
<< x-amz-id-2: mOsmhYGkW+SxipF6S2+CnmiqOhwJ62WfWUkmZk4zU3rzkWCEH9P/bT1hUz27apmO
<< x-amz-request-id: 8706AE3BE8597644
<< Date: Fri, 12 Jun 2015 02:11:23 GMT
<< ETag: "6c53debeb28f1d12f7ad388b27c9036d"
<< Content-Length: 0
<< Server: AmazonS3
>> GET /content-type-test HTTP/1.1
>> Host: raniz-playground.s3.amazonaws.com
>> Authorization: AWS <nope>
>> User-Agent: aws-sdk-java/1.9.6 Linux/3.2.0-84-generic Java_HotSpot(TM)_64-Bit_Server_VM/25.45-b02/1.8.0_45
>> Date: Fri, 12 Jun 2015 02:11:23 GMT
>> Content-Type: application/x-www-form-urlencoded; charset=utf-8
>> Connection: Keep-Alive
<< HTTP/1.1 200 OK
<< x-amz-id-2: 9U1CQ8yIYBKYyadKi4syaAsr+7BV76Q+5UAGj2w1zDiPC2qZN0NzUCQNv6pWGu7n
<< x-amz-request-id: 6777433366DB6436
<< Date: Fri, 12 Jun 2015 02:11:24 GMT
<< Last-Modified: Fri, 12 Jun 2015 02:11:23 GMT
<< ETag: "6c53debeb28f1d12f7ad388b27c9036d"
<< Accept-Ranges: bytes
<< Content-Type: text/html
<< Content-Length: 1024
<< Server: AmazonS3
And this is also the behaviour that looking at the source code shows us - if you set the content type the SDK won't override it.
Because you have to set content type at the end just before sending, using the putObject method;
ObjectMetadata md = new ObjectMetadata();
InputStream myInputStream = new ByteArrayInputStream(bFile);
md.setContentLength(bFile.length);
md.setContentType("text/html");
md.setContentEncoding("UTF-8");
s3client.putObject(new PutObjectRequest(bucketName, keyName, myInputStream, md));
And after upload, content type is set as "text/html"
Here is a working dummy code, check that out, I've just tried and it's working;
public class TestAWS {
//TEST
private static String bucketName = "whateverBucket";
public static void main(String[] args) throws Exception {
BasicAWSCredentials awsCreds = new BasicAWSCredentials("whatever", "whatever");
AmazonS3 s3client = new AmazonS3Client(awsCreds);
try
{
String uploadFileName = "D:\\try.txt";
String keyName = "newFile.txt";
System.out.println("Uploading a new object to S3 from a file\n");
File file = new File(uploadFileName);
//bFile will be the placeholder of file bytes
byte[] bFile = new byte[(int) file.length()];
FileInputStream fileInputStream=null;
//convert file into array of bytes
fileInputStream = new FileInputStream(file);
fileInputStream.read(bFile);
fileInputStream.close();
ObjectMetadata md = new ObjectMetadata();
InputStream myInputStream = new ByteArrayInputStream(bFile);
md.setContentLength(bFile.length);
md.setContentType("text/html");
md.setContentEncoding("UTF-8");
s3client.putObject(new PutObjectRequest(bucketName, keyName, myInputStream, md));
} catch (AmazonServiceException ase)
{
System.out.println("Caught an AmazonServiceException, which "
+ "means your request made it "
+ "to Amazon S3, but was rejected with an error response"
+ " for some reason.");
System.out.println("Error Message: " + ase.getMessage());
System.out.println("HTTP Status Code: " + ase.getStatusCode());
System.out.println("AWS Error Code: " + ase.getErrorCode());
System.out.println("Error Type: " + ase.getErrorType());
System.out.println("Request ID: " + ase.getRequestId());
} catch (AmazonClientException ace)
{
System.out.println("Caught an AmazonClientException, which "
+ "means the client encountered "
+ "an internal error while trying to "
+ "communicate with S3, "
+ "such as not being able to access the network.");
System.out.println("Error Message: " + ace.getMessage());
}
}
}
Hope that it helps.
It seems that
When uploading files, the AWS S3 Java client will attempt to determine
the correct content type if one hasn't been set yet. Users are
responsible for ensuring a suitable content type is set when uploading
streams. If no content type is provided and cannot be determined by
the filename, the default content type, "application/octet-stream",
will be used.
Giving the file a .html extension provides a way to set the correct type.
According to the examples I've been looking at, the code you show should be doing what you want to do. :/
I could fix this issue easily by commandline, I faced similar issue while uploading html files through aws commandline even though the file name had correct extension.
As mentioned in earlier comments, adding --content-type param fixes the issue.
Executing below command and refreshing page returned octet stream.
aws s3api put-object --bucket [BUCKETNAME] --body index.html --key index.html --profile [PROFILE] --acl public-read
Fix: add --content type text/html
aws s3api put-object --bucket [BUCKETNAME] --body index.html --key index.html --profile [PROFILE] --acl public-read --content-type text/html
If you are using the AWS SDK for Java 2.x, it is possible to add the content type within the builder pattern.
For example, uploading a Base64-encoded image as a JPEG object to S3 (assuming you have already instantiated an S3 client):
byte[] stringAsByteArray = java.util.Base64.getDecoder().decode(base64EncodedString);
s3Client.putObject(
PutObjectRequest.builder().bucket("my-bucket").key("my-key").contentType("image/jpg").build(),
RequestBody.fromBytes(stringAsByteArray)
);
Do you have any Override on the default mime content on your S3 account? Look at this link to see how to check it: How to override default Content Types.
Anyway, it looks like your S3 client fails to determine the correct mime-type by the content of the file, so it relies on the extension. octet-stream is the widely used default content mime type when a browser/servlet can't determine the mimetype: Is there any default mime type?
With a client like this
StringBody body = new StringBody("form_username", Charset.forName("UTF-8"));
multipart.addPart("username", body);
ByteArrayBody bBody = new ByteArrayBody(bs, "form_command.dat");
multipart.addPart("data", bBody);
httppost.setEntity(multipart);
How are the values supposed to be retrieved in the netty server. I already have a HttpRequestDecoder added to the pipeline. And the messageReceived handled thus
HttpRequest request = (HttpRequest) e.getMessage();
this.mRequest = request;
if (is100ContinueExpected(request)) {
send100Continue(e);
}
ChannelBuffer content = request.getContent();
if (content.readable()) {
System.out.println("Content()\n" + content.toString(CharsetUtil.UTF_8) + "\r\n");
}
Print outputs .
Content()
--Xdq2t6unVsUp191MKhpR6BXz5P7Eoo
Content-Disposition: form-data; name="username"
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
form_username
--Xdq2t6unVsUp191MKhpR6BXz5P7Eoo
Content-Disposition: form-data; name="data"; filename="form_command.dat"
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary
--Xdq2t6unVsUp191MKhpR6BXz5P7Eoo--
End of contents
You need to use the new HttpPostRequestDecoder. This is only available in the upcoming Netty 3.5 (3 branch) and Netty 4 (master branch).
Here an example usage.
If you need to use it now, you can just copy the files mentioned in this pull request into your project namespace and use it.
Hope this helps.
Here is my code:
url = paths[0];
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
int length = connection.getContentLength(); // i get negetive length
InputStream is = (InputStream) url.getContent();
byte[] imageData = new byte[length];
int buffersize = (int) Math.ceil(length / (double) 100);
int downloaded = 0;
int read;
while (downloaded < length) {
if (length < buffersize) {
read = is.read(imageData, downloaded, length);
} else if ((length - downloaded) <= buffersize) {
read = is.read(imageData, downloaded, length - downloaded);
} else {
read = is.read(imageData, downloaded, buffersize);
}
downloaded += read;
publishProgress((downloaded * 100) / length);
}
Bitmap bitmap = BitmapFactory.decodeByteArray(imageData, 0,
length);
if (bitmap != null) {
Log.i(TAG, "Bitmap created");
} else {
Log.i(TAG, "Bitmap not created");
}
is.close();
return bitmap;
I looked at this in Java documentation and the length is negative because of the following reason:
"the number of bytes of the content, or a negative number if unknown. If the content >length is known but exceeds Long.MAX_VALUE, a negative number is returned."
What might be the reason for this? I am trying to download an image. I would like to point out that this is the fourth method that I am trying to download images. The other three are mentioned here.
Edit:
As requested, here is the full method I am using.
protected Bitmap getImage(String imgurl) {
try {
URL url = new URL(imgurl);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
int length = connection.getContentLength();
InputStream is = (InputStream) url.getContent();
byte[] imageData = new byte[length];
int buffersize = (int) Math.ceil(length / (double) 100);
int downloaded = 0;
int read;
while (downloaded < length) {
if (length < buffersize) {
read = is.read(imageData, downloaded, length);
} else if ((length - downloaded) <= buffersize) {
read = is.read(imageData, downloaded, length
- downloaded);
} else {
read = is.read(imageData, downloaded, buffersize);
}
downloaded += read;
// publishProgress((downloaded * 100) / length);
}
Bitmap bitmap = BitmapFactory.decodeByteArray(imageData, 0,length);
if (bitmap != null) {
System.out.println("Bitmap created");
} else {
System.out.println("Bitmap not created");
}
is.close();
return bitmap;
} catch (MalformedURLException e) {
System.out.println(e);
} catch (IOException e) {
System.out.println(e);
} catch (Exception e) {
System.out.println(e);
}
return null;
}
By default, this implementation of HttpURLConnection requests that servers use gzip compression.
Since getContentLength() returns the number of bytes transmitted, you cannot use that method to predict how many bytes can be read from getInputStream().
Instead, read that stream until it is exhausted: when read() returns -1.
Gzip compression can be disabled by setting the acceptable encodings in the request header:
urlConnection.setRequestProperty("Accept-Encoding", "identity");
So try this:
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestProperty("Accept-Encoding", "identity"); // <--- Add this line
int length = connection.getContentLength(); // i get negetive length
Source (Performance paragraph): http://developer.android.com/reference/java/net/HttpURLConnection.html
There are two possible common explanations for this:
The content length is not known. Or more specifically, the server is not setting a "Content-Length" header in the response message.
The content length is greater than Integer.MAX_VALUE. If that happens, getContentLength() returns -1. (The javadocs recommend that getContentLengthLong() is used instead of getContentLength() to avoid that problem.)
Either way, it is better to NOT preallocate a fixed sized byte array to hold the image.
One alternative is to create a local ByteArrayOutputStream and copy bytes read from the socket to that. Then call toByteArray to grab the full byte array.
Another alternative is to save the data in a temporary file in the file system.
Apparently a common underlying cause of this is that some implementations will by default request "gzip" encoding for the response data. That forces the server to set the content length to -1. You can prevent that like this:
connection.setRequestProperty("Accept-Encoding", "identity");
... but that means that the response won't be compressed. So that is
(IMO) a substandard solution.
Your existing client-side code is broken in another respect as well. If you get an IOException or some other exception, the code block will "exit abnormally" without closing the URLConnection. This will result in the leakage of a file descriptor. Do this too many times and your application will fail due to exhaustion of file descriptors ... or local port numbers.
It is best practice to use a try / finally to ensure that URLConnections, Sockets, Streams and so on that tie down external resources are ALWAYS closed.
Preallocating a buffer based on the (purported) content length sets you up for a denial of service attack. Imagine what if the bad guys send you a lot of request with dangerously large "Content-Length" headers and then slow-send the data. OOMEs or worse.
It seems the server doesn't offer Content-Length in its response headers, did you get the Transfer-Encoding=chunked header from the response headers?
My situation is : I perform a HttpURLConnection and consider the server would response to me the "Content-Length" with positive value, but it didn't, then I turn to
the AndroidHttpClient which android HttpClient implementation, perform the same request again and got the right Content-Length.
I used Wireshark to analysis the two requests, found a little difference about request headers.
the header list that use AndroidHttpClient :
---------------------------------- request headers
Range : bytpe=0-
Host : download.game.yy.com
Connection : Keep-Alive
User-Agent : com.duowan.mobile.netroid
---------------------------------- response headers
Server : nginx
Content-Type : text/plain; charset=utf-8
ETag : "535e2578-84e350"
Cache-Control : max-age=86400
Accept-Ranges : bytes
Content-Range : bytes 0-8708943/8708944
Content-Length : 8708944
the request header list that use HttpURLConnection :
---------------------------------- request headers
Range : bytpe=0-
Host : download.game.yy.com
Connection : Keep-Alive
User-Agent : com.duowan.mobile.netroid
Accept-Encoding : gzip // difference here
---------------------------------- response headers
Server : nginx
Content-Type : text/plain; charset=utf-8
Cache-Control : max-age=86400
Transfer-Encoding : chunked
X-Android-Received-Millis : 1398861612782
X-Android-Sent-Millis : 1398861608538
The only difference with request header is Accept-Encoding which isn't added by myself, it was Android default setting for HttpURLConnection, after that, I set it to identity then perform request again, below is the full header stacks :
---------------------------------- request headers
Range : bytpe=0-
Host : download.game.yy.com
Connection : Keep-Alive
User-Agent : com.duowan.mobile.netroid
Accept-Encoding : identity
---------------------------------- response headers
Server : nginx
Content-Type : text/plain; charset=utf-8
ETag : "535e2578-84e350"
Cache-Control : max-age=86400
Accept-Ranges : bytes
Content-Range : bytes 0-8708943/8708944
Content-Length : 8708944
X-Android-Received-Millis : 1398862186902
X-Android-Sent-Millis : 1398862186619
as you can see, after I set the Accept-Encoding to "identity" replace system default value "gzip", the server provided "Content-Length" positive, that's why AndroidHttpClient could take the right value of Content-Length and HttpURLConnection not.
the gzip compression encoding may cause chunked response that consider by server-side, and if server think you can receive chunked encoding response, it may not offer the Content-Length header, try to disable gzip acceptable behavior then see what difference with that.
See here: http://download.oracle.com/javase/6/docs/api/java/net/URLConnection.html#getContentLength%28%29
It says,
Returns:
the content length of the resource that this connection's URL references, or -1 if the content length is not known.
It simply means header does not contains the content-length field. If you are having control over server code. You should set the content-lenth somehow. Something like ServletResponse#setContentLength
I also meet this problem in my Wallpaper app. The problem is because your server doesn't provide Content-Length in its http header. Here is a snapshot about a normal http header with Content-length.
I am using a share host so I can not change the server configuration. In my app, I am set progress dialog max value with an approximated value (which is bigger than my real file size) like this:
int filesize = connection.getContentLength();
if(filesize < 0) {
progressDialog.setMax(1000000);
} else {
progressDialog.setMax(filesize);
}
You can also check my full example source code here:
Android Progress Dialog Example.
I am late here but this might help someone. I was facing same issue i was always getting -1 value, when ever i was trying get the content length.
previously i was using below method to get content length.
long totalByte=connection.getContentLength();
I used below method which solved my problem .
long totalByte=Long.parseLong(connection.getHeaderField("Content-Length"));