We Keep Coding

"IllegalsStateException closed" in HttpUrlConnection while download file

I have random IllegalstateException crash in Fabric. But can't reproduce it.
Users download file from server. For download I use regular HttpUrlConnection and read bytes from InputStream in while. Call this in foreground service with separate thread. On some iteration of reading this crash happen. I close InputStream only in finally, so it can't be closed before while finish.
Android 6, 7, 8, 9. Interesting that 100% Samsung, maybe Samsung has some specific behavior? Also interesting that this happen always near finish of download. For example from 2738688 bytes was downloaded 2716156 and only 22532‬ left. All crashes near the end of download.
public static Response downloadFile(String url, String tmpFile) throws IOException {
InputStream is = null;
OutputStream os = null;
Response result;
try {
URL urlFile = new URL(url);
HttpURLConnection connection = (HttpURLConnection) urlFile.openConnection();
connection.connect();
long size = connection.getContentLength();
int code = connection.getResponseCode();
Map<String, List<String>> responseHeaders = connection.getHeaderFields();
result = new Response(code, size, responseHeaders);
if (code != HttpURLConnection.HTTP_OK) {
return result;
}
is = connection.getInputStream();
os = new FileOutputStream(tmpFile);
final byte[] buf = new byte[1024];
int read;
//This read random crash
while ((read = is.read(buf)) != -1) {
os.write(buf, 0, read);
}
os.flush();
} finally {
if (is != null) {
is.close();
}
if (os != null) {
os.close();
}
}
return result;
}

Java gzip pdf from url to file - result gives minor character mismatch

I'm trying to download a gzip pdf from an url, unpacking it and writing it to a file. It almost works, but currently some characters in the pdf made from my code mismatches the real pdf. I checked this by opening both of the pdf's in notepad.
I provide some short text samples from the two pdfs.
From my code:
’8 /qªMiUe°Ä[H`ðKíulýªäqvA®v8;xÒhÖßÚ²ý!Æ¢ØK$áýçpF[¸t1#y$93
From the real pdf:
ƒ8 /qªMiUe°Ä[H`ðKíulªäqvA®—v8;ŸÒhÖßÚ²!ˆ¢ØK$áçpF[¸t1#y$‘‹3
Here is my code:
public void readPDFfromURL(String urlStr) throws IOException {
URL myURL = new URL(urlStr);
HttpURLConnection urlCon = (HttpURLConnection) myURL.openConnection();
urlCon.setRequestProperty("Accept-Encoding", "gzip");
urlCon.setRequestProperty("Content-Type", "application/pdf");
urlCon.setRequestMethod("GET");
urlCon.setDoInput(true);
urlCon.connect();
Reader reader;
if ("gzip".equals(urlCon.getContentEncoding())) {
reader = new InputStreamReader(new GZIPInputStream(urlCon.getInputStream()));
}
else {
reader = new InputStreamReader(urlCon.getInputStream());
}
FileOutputStream fos = new FileOutputStream("document.pdf");
int data = reader.read();
while(data != -1) {
char c = (char) data;
fos.write(c);
data = reader.read();
}
fos.close();
reader.close();
}
I can open the pdf, and it has the correct amount of pages, but the pages are all blank.
My initial thought is that it might got something to do with character codes to do, like some setting in my java project, intellij etc.
Alternatively, I don't actually need to put it in a file. I just need to download it so I can upload it to another place. However, the pdf should of course be working in either case. I'm really just putting it in an actual file to check if it works.
Thank you for your help!

Here is my new implementation, which solves my question:
public void readPDFfromURL(String urlStr) throws IOException {
URL myURL = new URL(urlStr);
HttpURLConnection urlCon = (HttpURLConnection) myURL.openConnection();
urlCon.setRequestProperty("Accept-Encoding", "gzip");
urlCon.setRequestProperty("Content-Type", "application/pdf");
urlCon.setRequestMethod("GET");
urlCon.setDoInput(true);
urlCon.connect();
GZIPInputStream reader = new GZIPInputStream(urlCon.getInputStream());
FileOutputStream fos = new FileOutputStream("document.pdf");
byte[] buffer = new byte[1024];
int len;
while((len = reader.read(buffer)) != -1){
fos.write(buffer, 0, len);
}
fos.close();
reader.close();
}

Android - Size in chars of an http response

I am not an pro developing android. I wanted to download a JSON object from my server, but only code I could find was this:
private String downloadUrl(String myurl) throws IOException {
InputStream is = null;
// Only display the first 500 characters of the retrieved
// web page content.
int len = 500;
try {
URL url = new URL(myurl);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setReadTimeout(10000 /* milliseconds */);
conn.setConnectTimeout(15000 /* milliseconds */);
conn.setRequestMethod("GET");
conn.setDoInput(true);
// Starts the query
conn.connect();
int response = conn.getResponseCode();
Log.d("ServerConnection", "The response is: " + response);
is = conn.getInputStream();;
//is.
// Convert the InputStream into a string
String contentAsString = readIt(is, len);
return contentAsString;
// Makes sure that the InputStream is closed after the app is
// finished using it.
} catch (MalformedURLException e) {
//
return "error";
} catch (IOException e) {
//
return "error";
} finally {
if (is != null) {
is.close();
}
}
}
And it works fine, I cant understand. But it has a int len = 500, and my returned json is cropped to 500 chars. I tried changing to a great number, but it puts spaces at the end. How can I know the size in chars of the String contained by the InputSteam?
Thanks

You can check the Content-Length header value of your response:
Map<String, List<String>> headers = connection.getHeaderFields();
for (Entry<String, List<String>> header : headers.entrySet()) {
if(header.getKey().equals("Content-Legth")){
len=Integer.parseInt(header.getValue());
}
}
or you can your response in a buffered reader like this:
InputStream is = connection.getInputStream();
InputStreamReader reader = new InputStreamReader(is);
StringBuilder builder = new StringBuilder();
int c = 0;
while ((c = reader.read()) != -1) {
builder.append((char) c);
}

Yout can use Apache Commons IO IOUtils.toString to convert InputStream to String or use Gson to read object from input stream directly:
return gson.fromJson(new InputStreamReader(inputStream), YourType.class);

HTTP GET request in java returns meaningless data from App Engine

I'm requesting a json file from an App Engine URL
http://1-1-26a.wordbuzzweb.appspot.com/json/level-images.json
The file encoding is UTF-8 without a BOM. If I look at this file on my local disk it's size is 12414 bytes. If I get the file in Chrome is reads it perfectly well. If I then save it it's 12414 bytes. However, if I try and download the file with a GET request in java I only get 780 bytes returned and the returned data would appear to be meaningless.
I've tried several different types of get request, both of the methods below I have used elsewhere perfectly effectively. The response code on the GET requests is 200. Interestingly, if I do a POST with no content instead of a GET, then I get the valid response.
If I download the file from this URL on Google Drive instead, then the GET methods below work perfectly.
edit This code is now working, however, this is a recurring issue that comes and goes. If anyone has any ideas what might be causing it then please say so!
This doesn't work
public static String doGetSync(String urlToRead) throws IOException {
URL url = new URL(urlToRead);
InputStream is = url.openStream();
ByteArrayOutputStream buffer = new ByteArrayOutputStream();
int nRead;
byte[] data = new byte[BUFFER_SIZE];
while ((nRead = is.read(data, 0, data.length)) != -1) {
buffer.write(data, 0, nRead);
}
buffer.flush();
byte[] bytes = buffer.toByteArray();
return new String(bytes, "UTF-8");
}
Neither does this
public static String doGetSync2(String urlToRead) throws IOException {
final String charset = "UTF-8";
// Create the connection
HttpURLConnection connection = (HttpURLConnection) new URL(urlToRead).openConnection();
// Check the error stream first, if this is null then there have been no issues with the request
InputStream inputStream = connection.getErrorStream();
if (inputStream == null)
inputStream = connection.getInputStream();
// Read everything from our stream
BufferedReader responseReader = new BufferedReader(new InputStreamReader(inputStream, charset));
String inputLine;
StringBuilder response = new StringBuilder();
while ((inputLine = responseReader.readLine()) != null) {
response.append(inputLine);
}
responseReader.close();
return response.toString();
}
This code works
public static String doPostSync(final String url, final String content) throws IOException {
final String charset = "UTF-8";
// Create the connection
HttpURLConnection connection = (HttpURLConnection) new URL(url).openConnection();
// setDoOutput(true) implicitly set's the request type to POST
connection.setDoOutput(true);
connection.setRequestProperty("Accept-Charset", charset);
connection.setRequestProperty("Content-type", "application/json");
// Write to the connection
OutputStream output = connection.getOutputStream();
output.write(content.getBytes(charset));
output.close();
// Check the error stream first, if this is null then there have been no issues with the request
InputStream inputStream = connection.getErrorStream();
if (inputStream == null)
inputStream = connection.getInputStream();
// Read everything from our stream
BufferedReader responseReader = new BufferedReader(new InputStreamReader(inputStream, charset));
String inputLine;
StringBuilder response = new StringBuilder();
while ((inputLine = responseReader.readLine()) != null) {
response.append(inputLine);
}
responseReader.close();
return response.toString();
}

Download zip returns SocketException: Connection reset

There is a billion of topics about Connection, I've tried so many ways to download this file and always fail. When I disable cookies on my web browser I can't download it, for that reason I believe my problem is with cookies.
The function of my program is extract the zip, parse the html inside, with Jsoup, insert the content on mysql database and load it on JApplet. Everything is working except the auto-download part, which I have to do manual download in my web browser.
I'm using this class for the cookie, which returns error on
read.CookieManager.storeCookies(CookieManager.java:89)
which corresponds to this line from Cookie class
for (int i=1; (headerName = conn.getHeaderFieldKey(i)) != null; i++) {
and this one from download class
cm.storeCookies(urlConnection);
the download method
public static void main(String args[]) throws Exception {
downloadFromUrl("http://www1.caixa.gov.br/loterias/_arquivos/loterias/D_mgsasc.zip", "Mozilla", "C:/", "D_mgsasc.zip", true);
}
public static void downloadFromUrl(String srcAddress, String userAgent, String destDir, String destFileName, boolean overwrite) throws Exception
{
InputStream is = null;
FileOutputStream fos = null;
try {
CookieManager cm = new CookieManager();
URL url = new URL(srcAddress);
URLConnection urlConnection = url.openConnection();
urlConnection.setRequestProperty("User-Agent", userAgent);
urlConnection.setDoInput(true);
urlConnection.setDoOutput(true);
urlConnection.setConnectTimeout(30000);
urlConnection.setReadTimeout(30000);
urlConnection.setUseCaches(true);
urlConnection.connect();
cm.storeCookies(urlConnection);
cm.setCookies(url.openConnection());
is = urlConnection.getInputStream();
fos = new FileOutputStream(destFile);
byte[] buffer = new byte[1024];
int len, totBytes = 0;
while((len = is.read(buffer)) > 0)
{
totBytes += len;
fos.write(buffer, 0, len);
}
fos.flush();
fos.close();
}
}
*
updated, removed unnecessary code
*
which returns the following error
java.net.SocketException: Connection reset at
zip.DownloadFile.downloadFromUrl(DownloadFile.java:71)
related to this line in code
is = urlConnection.getInputStream();
When I remove cookies set code, the same error of Connection reset persists.