In server side code, I have set buffer size and content length as File.length() and then Opened File using FileInputStream.
Later fetching output stream using HttpResponse.getOutputStream() and dumping bytes of data that is read using FileInputStream
I am using Apache Tomcat 7.0.52, Java 7
On Client
File Downloader.java
URL url = new URL("myFileURL");
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
con.setDoInput(true);
con.setConnectTimeout(10000);
con.setReadTimeout(10000);
con.setRequestMethod("GET");
con.setUseCaches(false);
con.setRequestProperty("User-Agent", "Mozilla/5.0");
con.connect();
FileOutputStream fos = new FileOutputStream("filename");
if(con.getResponseCode()==200){
InputStream is = con.getInputStream();
int readVal;
while((readVal=is.read())!=-1) fos.write(readVal);
}
fos.flush()
fos.close();
So above code failed to download large file.
On client using Java 7
Can You try this
FileOutputStream outputStream = new FileOutputStream(fileName);
int bytesRead;
byte[] buffer = new byte[1024];
while ((bytesRead = inputStream.read(buffer)) != -1) {
outputStream.write(buffer, 0, bytesRead);
}
Quoting from https://stackoverflow.com/a/45453874/4121845
Because you only want to write data that you actually read. Consider the case where the input consists of N buffers plus one byte. Without the len parameter you would write (N+1)*1024 bytes instead of N*1024+1 bytes. Consider also the case of reading from a socket, or indeed the general case of reading: the actual contract of InputStream.read() is that it transfers at least one byte, not that it fills the buffer. Often it can't, for one reason or another.
fos.flush();
} finally {
fos.close();
con.close();
}
Related
I have been searching the web for this particular problem. Maybe i'm doing something wrong or i'm missing something here...
So i'm trying to convert a File Stream ( an Excel file ) -> mimetype ( application/octet-stream or application/vnd.ms-excel ) doesn´t matter...to a Base64 encoded string.
The reason i'm doing this is because i want to provide the File in a REST API inside a JSON object for later decoding in the browser the base64 string and download the file.
When I receivethe InputStream and save to the disk everything works fine...
Even when i use POSTMAN to get the FILE if I save the file it opens in Excel with all the right data.
THE CODE -> Used this simple example to download a file from a URL
URL url = new URL(fileURL);
HttpURLConnection httpConn = (HttpURLConnection) url.openConnection();
//etc...i get response code OK(200) get file name etc
// opens input stream from the HTTP connection
InputStream inputStream = httpConn.getInputStream();
String saveFilePath1 = "C:\\test1.xlsx";
String saveFilePath2 = "C:\\test2.xlsx";
FileOutputStream outputStream = new FileOutputStream(saveFilePath1);
int bytesRead = -1;
byte[] buffer = new byte[BUFFER_SIZE];
while ((bytesRead = inputStream.read(buffer)) != -1) {
outputStream.write(buffer, 0, bytesRead);
}
//FOR TESTING PURPOSES AT THIS POINT I HAVE SAVED THE STREAM INTO
//**test1.xlsx** SUCCESSFULLY and opens into excel and everything
//is fine.
//THE PROBLEM RESIDES HERE IN THIS NEXT PIECE OF CODE
//import org.apache.commons.codec.binary.Base64;
//I try to encode the string to Base64
String encodedBytesBase64 = Base64.encodeBase64String(buffer);
//WHEN I DO THE DECODE AND WRITE THE BYTES into test2.xlsx this file doesn´t work...
FileOutputStream fos = new FileOutputStream(saveFilePath2);
byte[] bytes = Base64.decodeBase64(encodedBytesBase64);
fos.write(bytes);
//Close streams from saved file test2
fos.close();
//Close streams from saved file test1
outputStream.close();
inputStream.close();
I even took the string to check if it is a valid Base64 String, which it is accordind to this site -> Base64 Validator
But when i try to decode the string in the same website it tells me there's a different encoding:
Is it possible this is the problem ?
I think you can ignore those warnings. Rather, the issue is here:
int bytesRead = -1;
byte[] buffer = new byte[BUFFER_SIZE];
while ((bytesRead = inputStream.read(buffer)) != -1) {
outputStream.write(buffer, 0, bytesRead);
}
:
String encodedBytesBase64 = Base64.encodeBase64String(buffer);
As you can see in the first part, you are reusing buffer to read the input stream and write to the output stream. If this loops around more than once, buffer will be overwritten with the next chunk of data from the input stream. So, when you are encoding buffer, you are only using the last chunk of the file.
The next problem is that when you are encoding, you are encoding the full buffer array, ignoring the bytesRead.
One option might be to read the inputStream and write it to a ByteArrayOutputStream, and then encode that.
int bytesRead = -1;
byte[] buffer = new byte[BUFFER_SIZE];
ByteArrayOutputStream array = new ByteArrayOutputStream();
while ((bytesRead = inputStream.read(buffer)) != -1) {
array.write(buffer, 0, bytesRead);
}
String encoded = Base64.encodeBase64String(array.toByteArray());
I wrote a code like this.
URL obj = new URL(url);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
con.setRequestMethod("GET");
con.setRequestProperty("Cache-Control", "no-cache");
int responseCode = con.getResponseCode();
InputStream inputStream = con.getInputStream();
FileOutputStream outputStream = new FileOutputStream("C:\\aaaa\\ww.csv");
int bytesRead = -1;
byte[] buffer = new byte[4096];
while ((bytesRead = inputStream.read(buffer)) != -1) {
outputStream.write(buffer, 0, bytesRead);
}
outputStream.close();
inputStream.close();
The code works well.I send request and ı can download the csv file into my computer.But ı want to know that if the csv file have turkish characters(ş,ğ,ı,ç) can ı download the csv with that characters.
or what can ı do for that characters to see them in csv file.
The server sends you a stream of bytes and you just save the byte stream on your disk, so you don't need to worry about characters.
In java input or output stream , there always has a byte array size of 1024.
Just like below:
URL url = new URL(src);
URLConnection connection = url.openConnection();
InputStream is = connection.getInputStream();
OutputStream os = new FileOutputStream("D:\\images"+"\\"+getName(src)+getExtension(src));
byte[] byteArray = new byte[1024];
int len = 0;
while((len = is.read(byteArray))!= -1){
os.write(byteArray, 0, len);
}
is.close();
os.close();
Why initialize this array to 1024?
That is called buffering and ,each time you overwrite the contents of the buffer each time you go through the loop.
Simply reading file in chunks, instead of allocating memory for the file content at a time.
Reason behind this to do is you will become a clear victim of OutOfMemoryException if file is too large.
And coming to the specific question, That is need not to be 1024, even you can do with 500. But a minimum 1024 is a common usage.
In the following code. I'm reading a file into a small buffer ( len=CHUNK_SIZE) And I simply want to write this buffer to outputstream. But even though i am flushing after every chunk I get an heap overflow exception. Well if I want to stream small files everything is fine. But shouldn't flush also delete all data in the stream?
URL url = new URL(built);
HttpURLConnection con = (HttpURLConnection)url.openConnection();
con.setDoOutput(true);
con.setDoInput(true);
con.setUseCaches(false);
//con.setChunkedStreamingMode(CHUNK_SIZE);
con.setRequestProperty("Content-Type", "multipart/form-data; boundary="
+ Boundary);
FileInputStream is = new FileInputStream(m_FileList.get(i));
DataOutputStream os = new DataOutputStream(con.getOutputStream());
// .....
while((read = is.read(temp, 0, CHUNK_SIZE)) != -1) {
bytesTotal += read;
os.write(temp, 0, read); // heap overflow here if the file is to big
os.flush();
}
DataOutputStream doesn't buffer at all, but HttpURLConnection's output stream buffers everything by default, so it can set the Content-Length header. Use chunked transfer mode to prevent that.
You don't actually need the DataOutputStream at all here: just write to the connection's output stream.
Don't flush() inside the loop either.
I've seen examples with text files but is saving an audio file directly to a server done the same way with URLConnection?
Yes, the same. Although make sure you use a binary output stream to write the content to disk.
Something like:
URLConnection conn = new URL("http://www.gravatar.com/avatar/fd9e8761fad999a1bf1e095fc8f53ffe?s=32&d=identicon&r=PG")
.openConnection();
InputStream is = conn.getInputStream();
FileOutputStream outstream = new FileOutputStream("/tmp/myfile");
byte[] buffer = new byte[4096];
int len;
while ((len = is.read(buffer)) > 0) {
outstream.write(buffer, 0, len);
}
outstream.close();
is.close();
The example uses your gravatar, but same difference.