We Keep Coding

WebResourceResponse can't read full inputstream from HttpConnection (Android)

I'm working with an ionic application(like hybrid) which can play some videos.I want to add some headers to the request so that I override the "shouldInterceptRequest".
URL myUrl = new URL(real);
HttpURLConnection connection = (HttpURLConnection) myUrl.openConnection();
for (Map.Entry < String, String > entry: headers.entrySet()) {
connection.setRequestProperty(entry.getKey(), entry.getValue());
}
InputStream in = connection.getInputStream();
WebResourceResponse response = new WebResourceResponse("video/mp4", "UTF-8", in );
for (Map.Entry < String, List < String >> entry: connection.getHeaderFields().entrySet()) {
resHeaders.put(entry.getKey(), entry.getValue().get(0));
}
This code can't work.The video tag in html can't play the video.
So I add some code.
byte[] bytes = new byte[30 * 1024 * 1024];
ByteArrayOutputStream byteBuffer = new ByteArrayOutputStream();
int len = 0;
while ((len = in.read(bytes)) != -1) {
byteBuffer.write(bytes, 0, len);
}
bytes = byteBuffer.toByteArray();
WebResourceResponse response = new WebResourceResponse("video/mp4", "UTF-8", new ByteArrayInputStream(bytes));
When I read inputstream into bytes and send bytes to WebResourceResponse, the video can be play.However it means my application will use lots of memory if the video is large.
So that, I want to know is there any way to play the video without saving inputstream into bytes.

OK.Fianlly,I found my way.
Actually, my goal is to add custom headers to resource request and now I found there is an easier way to do it.
For example, if I want to load a image,I will use img tag like this,<img src="the_url_of_image">,and I can't add any header to the request unless I interecept the request.
However,we can use blob now.We can request the resource by using something like ajax and use createObjectURL to create a url links to the resource.

Egnyte chunked api upload example in java

How to write java code for egnyte chunked upload and send to rest service of egnyte api.
https://developers.egnyte.com/docs/read/File_System_Management_API_Documentation#Chunked-Upload
long size = f.getTotalSpace();
int sizeOfFiles = 1024 * 1024;// 1MB
byte[] buffer = new byte[sizeOfFiles];
ResponseEntity<String> responseEntity = null;
String fileName = f.getName();
String url = DOWNLOAD_OR_UPLOAD + "-chunked" + egnyteSourcePath + f.getName();
HttpHeaders headers = buildEgnyteEntity();
HttpEntity entity = new HttpEntity<>(headers);
//try-with-resources to ensure closing stream
try (FileInputStream fis = new FileInputStream(f);
BufferedInputStream bis = new BufferedInputStream(fis)) {
int bytesAmount = 0;
while ((bytesAmount = bis.read(buffer)) > 0) {
//write each chunk of data into separate file with different number in name
String filePartName = String.format("%s.%03d", fileName, partCounter++);
File newFile = new File(f.getParent(), filePartName);
responseEntity = restTemplate.exchange(url, HttpMethod.POST, entity, String.class);
}
}
return responseEntity;

I think there's a couple of things missing in your code.
First thing is that you don't specify required headers. You should provide X-Egnyte-Chunk-Num with int value with number of your chunk, starting from 1. In X-Egnyte-Chunk-Sha512-Checksum header you should provide SHA512 checksum.
Second thing is that first request will give you an UploadId in response header in X-Egnyte-Upload-Id. You need to specify that as a header in your second and following requests.
Third thing is that I don't see you use your bytesAmount in the request. I'm not sure you're providing the data.
I'm not a Java guy, more of a C# one, but I've written a post how to upload and download big files with Egnyte API on my blog: http://www.michalbialecki.com/2018/02/11/sending-receiving-big-files-using-egnyte-api-nuget-package/. This can give you an idea how sending loop can be structured.
Hope that helps.

Receive image file through rest Api

How to receive an image file through Rest APIs. There is an option of MULTIPART_FORM_DATA which looks like it will send files in parts as in more than one request.
I want to receive images very fast on server. around 2 images per second.

Simply read image in a File and use Response class to build the response.
Response.ok(new File("myimage.jpg"), "image/jpeg").build();
There are other variations of the same.
Read the image using following.
URL url = new URL("http://localhost:8080/myimage/1");
URLConnection connection = url.openConnection();
input = connection.getInputStream();
byte[] buffer = new byte[1024];
int n = - 1;
OutputStream fos = new FileOutputStream("Output.jpg" );
while ( (n = input.read(buffer)) != -1)
{
fos.write(buffer, 0, n);
}
fos.close();
You can use Apache HTTP client to make it prettier.

Crawl web page encoding issue - negative value in byte

I use following code to crawl a web page.
CloseableHttpClient httpclient = HttpClients.createDefault();
HttpGet httpget = new HttpGet(url);
CloseableHttpResponse response = httpclient.execute(httpget);
HttpEntity entity = response.getEntity();
System.out.println(entity.getContentType());
//output: Content-Type: text/html; charset=ISO-8859-1
I found that the character "’" has the byte value -110, which cannot be mapped to a valid character in either iso-8859-1 or utf-8.
I try to manually open the page and copy the character and save as a text file, then I saw the byte value is actually 39. I think the OS did the conversion when the character gone through the clipboard
What I want is just to save the web page as original to local disk.
I made a simple code to save the content to disk. I directly read bytes and write bytes. When I open the saved file with Hex Editor, I can see the value of the byte is 146 (-110).
InputStream in = entity.getContent();
FileOutputStream fos = new FileOutputStream(new File("D:/test.html"));
byte[] buffer = new byte[1024];
int len = 0;
while((len = in.read(buffer)) > 0) {
fos.write(buffer, 0, len);
buffer = new byte[1024];
}
in.close();
fos.close();
So now the issue become how to reconstruct the character from the byte 146(-110). I will keep trying and update if I got anything.

Maybe you could give some code how you save the page to the disk? And did you check the value for ’?
It looks like the character ’ is 3 bytes long unless my pasting or your copying failed. Check this out:
public static void main(String[] args) {
char c = '’';
System.out.println("character: " + c);
System.out.println("int: " + (int)c);
String s = new String("’");
// Java uses UTF-16 encoding, other encodings will give different values
byte[] bytes = s.getBytes();
System.out.println("bytes: " + Arrays.toString(bytes));
}
Edit: I've found the following suggested approach to charset handling, give it a try:
ContentType contentType = ContentType.getOrDefault(entity);
Charset charset = contentType.getCharset();
Reader reader = new InputStreamReader(entity.getContent(), charset);
Source: https://hc.apache.org/httpcomponents-client-ga/tutorial/html/fundamentals.html

A byte in Java is a signed type with a value of -128 to 127. The most significant bit is used to indicate the sign. For example, 0111 1111 == 127, and 1000 0000 == -128.
I looked up your character (’) in an ANSI table and found that it has a value of 146 (which is of course greater than 127). The binary representation is 1001 0010, and so interpreting this as a signed value will yield -110.
To reproduce what you are seeing:
String s = new String("’"); // ’ is ansi character 146
byte[] bytes = s.getBytes();
System.out.println( (int)bytes[0] ); // prints -110
To convert the byte value to an unsigned representation:
char c = (char)(bytes[0] & 0xFF);
System.out.println( (int)c ); // prints 146

Get size of uncompressed gzip file while size of compressed file is available from server

I am using GZIPInputStream to download PDF file. I want to show the download progress of the file on a UI button. But, I am not getting the actual size of the file, what I am getting is compressed size due to which I am unable to show the correct download progress. This download progress is exceeding 100 as the actual file size is greater than the compressed size of file.
Header content of file from server: Following info I receive from server, from which I am using content-length which is giving compressed file size.
1.Connection
2.Content-Encoding
3.Content-length
4.Content-Type
5.Keep-Alive
6.Server
7.Date
Here is my code. Is there any way to get original size of file?
long fileLength = httpResponse.getEntity().getContentLength();//
GZIPInputStream input = new GZIPInputStream(new BufferedInputStream(
httpResponse.getEntity().getContent()));
FileOutputStream output = new FileOutputStream(destinationFilePath);
byte data[] = new byte[1024];
long total = 0;
float percentage = 0;
int count;
currentDownloadingPercentage=0;
while ((count = input.read(data)) != -1) {
total += count;
output.write(data, 0, count);
// publishing the progress....
percentage = (float)total/(float)fileLength;
percentage *= 100;
if ((int)percentage > (int)currentDownloadingPercentage) {
currentDownloadingPercentage = percentage;
Bundle resultData = new Bundle();
resultData.putBoolean(DOWNLOAD_FAILED, false);
resultData.putInt(DOWNLOAD_PROGRESS ,(int)percentage);
receiver.send(processID, resultData);
resultData = null;
}
}

You're looking at it the wrong way. You should be counting the compressed bytes that you read and calculating the progress based on those. Instead, you're counting the decompressed bytes and comparing it with the compressed file size. In answer to your question, there's no (reliable) way to determine the size of a gzipped file without decompressing it.
Update: Here's one way you could count the uncompressed bytes coming in. Wrap the raw input stream with a TeeInputStream before wrapping it with the GZIPInputStream. Make the TeeInputStream branch to a CountingOutputStream. Then you'll always have a current count of the compressed bytes that have been downloaded via getByteCount()

This issue discuss result seems not way to avoid HttpURLConnection.getInputStream() automatically returned GZIPInputStream, once you let HttpURLConnection accept gzip compression, you won't calculate download progress accurately, the only one we can do just disable gzip as acceptable encoding :
HttpURLConnection.setRequestProperty("Accept-Encoding", "identity");
Another choice is use AndroidHttpClient, I had tested about this, even we present accept gzip encoding like this :
HttpUriRequest.addHeader("Accept-Encoding", "gzip");
the InputStream instance that return by HttpResponse.getEntity().getContent() will be EofSensorInputStream, an original InputStream is what we wanted, isn't GZIPInputStream, that make us possible to wrap it to GZIPInputStream by myself, we can use TeeInputStream and CountingOutputStream to finish calculating download progress.
HttpResponse response = ...;
HttpEntity entity = response.getEntity();
long fileSize = entity.getContentLength();
InputStream ins = entity.getContent(); // instance of EofSensorInputStream
CountingOutputStream coStrem = new CountingOutputStream(new ByteArrayOutputStream(100));
GZIPInputStream inStrem = new GZIPInputStream(new TeeInputStream(ins, coStrem, true));
byte[] buffer = new byte[6 * 1024]; // 6K buffer
int offset;
while ((offset = inStrem.read(buffer)) != -1) {
tmpFileRaf.write(buffer, 0, offset);
postDownloadProgress(fileSize, coStrem.getByteCount());
}
I think that's all we can do with this problem, I tried pick up android libcore source with my project so we can customize HttpURLConnectionImpl then suppress it return GZIPInputStream, but many errors makes trouble, I discard this effort.
In this post, Jesse Wilson suggested we the best choice client of Android is HttpURLConnection, so I'm looking for how to solve this problem always, I hope I can get a way soon.