In the following code. I'm reading a file into a small buffer ( len=CHUNK_SIZE) And I simply want to write this buffer to outputstream. But even though i am flushing after every chunk I get an heap overflow exception. Well if I want to stream small files everything is fine. But shouldn't flush also delete all data in the stream?
URL url = new URL(built);
HttpURLConnection con = (HttpURLConnection)url.openConnection();
con.setDoOutput(true);
con.setDoInput(true);
con.setUseCaches(false);
//con.setChunkedStreamingMode(CHUNK_SIZE);
con.setRequestProperty("Content-Type", "multipart/form-data; boundary="
+ Boundary);
FileInputStream is = new FileInputStream(m_FileList.get(i));
DataOutputStream os = new DataOutputStream(con.getOutputStream());
// .....
while((read = is.read(temp, 0, CHUNK_SIZE)) != -1) {
bytesTotal += read;
os.write(temp, 0, read); // heap overflow here if the file is to big
os.flush();
}
DataOutputStream doesn't buffer at all, but HttpURLConnection's output stream buffers everything by default, so it can set the Content-Length header. Use chunked transfer mode to prevent that.
You don't actually need the DataOutputStream at all here: just write to the connection's output stream.
Don't flush() inside the loop either.
Related
In server side code, I have set buffer size and content length as File.length() and then Opened File using FileInputStream.
Later fetching output stream using HttpResponse.getOutputStream() and dumping bytes of data that is read using FileInputStream
I am using Apache Tomcat 7.0.52, Java 7
On Client
File Downloader.java
URL url = new URL("myFileURL");
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
con.setDoInput(true);
con.setConnectTimeout(10000);
con.setReadTimeout(10000);
con.setRequestMethod("GET");
con.setUseCaches(false);
con.setRequestProperty("User-Agent", "Mozilla/5.0");
con.connect();
FileOutputStream fos = new FileOutputStream("filename");
if(con.getResponseCode()==200){
InputStream is = con.getInputStream();
int readVal;
while((readVal=is.read())!=-1) fos.write(readVal);
}
fos.flush()
fos.close();
So above code failed to download large file.
On client using Java 7
Can You try this
FileOutputStream outputStream = new FileOutputStream(fileName);
int bytesRead;
byte[] buffer = new byte[1024];
while ((bytesRead = inputStream.read(buffer)) != -1) {
outputStream.write(buffer, 0, bytesRead);
}
Quoting from https://stackoverflow.com/a/45453874/4121845
Because you only want to write data that you actually read. Consider the case where the input consists of N buffers plus one byte. Without the len parameter you would write (N+1)*1024 bytes instead of N*1024+1 bytes. Consider also the case of reading from a socket, or indeed the general case of reading: the actual contract of InputStream.read() is that it transfers at least one byte, not that it fills the buffer. Often it can't, for one reason or another.
fos.flush();
} finally {
fos.close();
con.close();
}
Why the code below do some GET request instead of POST resquest. I receive also CONTENT_TYPE: "" :(
HashMap<String, String> PostDataMap = new HashMap<>();
PostDataMap.put("method","any");
String PostDataString = HTTPEncodeParamNameValues(PostDataMap);
URL url = new URL(ApiServerURL);
HttpURLConnection httpURLConnection = (HttpURLConnection) url.openConnection();
httpURLConnection.setDoInput(true);
httpURLConnection.setDoOutput(true);
httpURLConnection.setRequestMethod("POST");
httpURLConnection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
httpURLConnection.setUseCaches(false);
DataOutputStream dataOutputStream = new DataOutputStream(httpURLConnection.getOutputStream());
dataOutputStream.writeBytes(PostDataString);
dataOutputStream.flush();
dataOutputStream.close();
InputStream inputStream = url.openConnection().getInputStream();
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
byte[] buffer = new byte[1024];
int length;
while ((length = inputStream.read(buffer)) != -1) {
byteArrayOutputStream.write(buffer, 0, length);
}
when you pass some user input to server that time used post and only getting data no user input that time used get.
when used get that time all data call in url. and it limited size. and also not secure.
when you used post that not go data into url. and is unlimited data to be send .and also secure.
I don't get a good understanding about HttpUrlConnection.setChunkedStreamingMode, what is the effect of this mode?
I have following sample code:
HttpURLConnection conn = getHttpURLConnection(_url);
conn.setChunkedStreamingMode(4096); //4k
conn.setConnectTimeout(3000);
conn.setDoInput(true);
conn.setDoOutput(true);
conn.setRequestMethod("POST");
OutputStream out = conn.getOutputStream();
byte[] buffer = new byte[1024 * 10];//10k
FileInputStream in= new FileInputStream(file); //Write the content of the file to the server
int len;
while ((len = in.read(buffer)) != -1) {
out.write(buffer, 0, len);
}
out.flush();
in.close();
Say, The file size is 101k, and I set the chunk size to be 4096.
The HttpUrlConnection will send 4096 bytes to the server every write? and 1k for the last time?
Note that I have used a 10k buffer to write to the outputstream, Does it matter that the chunk size and buffer size are not the same?
If I disable the ChunkedStreamMode in my code, what's the effect compared to the code that I have set 4096?
The HttpUrlConnection will send 4096 bytes to the server every write? and 1k for the last time?
Yes.
Note that I have used a 10k buffer to write to the outputstream, Does it matter that the chunk size and buffer size are not the same?
No.
If I disable the ChunkedStreamMode in my code, what's the effect compared to the code that I have set 4096?
The effect is that the entire output is buffered until you close, so that the Content-length header can be set and sent first, which adds a lot of latency and memory. Not recommended in the case of large files.
I want to know when and why the "java.net.SocketTimeoutException readtime out" is raised.
URL obj = new URL(url);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
con.setConnectTimeout(5000);
con.setReadTimeout(421);
int responseCode = con.getResponseCode();
InputStream is = con.getInputStream();
System.out.println("Inputstream done");
FileOutputStream fos = fos = new FileOutputStream("D:\\tryfile\\file1.csv");
byte[] buffer = new byte[4096]; //declare 4KB buffer
int len;
while ((len = is.read(buffer)) > 0) {
fos.write(buffer, 0, len);
}
fos.close();
is.close();
Here is the question.I set read timeout value 421 and I take "java.net.SocketTimeoutException readtime out" exception at that line 55.
while ((len = is.read(buffer)) > 0) {
So I take the inputstream succesfuly but when I start reading/writing it I take this exception.And I check that the 732 Kb of the file is transfered until the exception.
So I really confused about it.Please explain readtimeout method exactly.
It is raised when no data arrives between the inception of the read() call and the expiration of the specified timeout starting at that point. In other words, no data arrived within the timeout period.
So I take the inputstream succesfully
Not surprising. It takes zero time and does nothing on the network, as you have already called getResponseCode() (and done nothing with it).
but when I start reading/writing it I take this exception.
So the data was slow arriving.
And I check that the 732 Kb of the file is transfered until the exception.
So the end of the data was slow arriving.
NB 421ms is far too short for a read timeout. It should be tens of seconds.
I wrote a code like this.
URL obj = new URL(url);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
con.setRequestMethod("GET");
con.setRequestProperty("Cache-Control", "no-cache");
int responseCode = con.getResponseCode();
InputStream inputStream = con.getInputStream();
FileOutputStream outputStream = new FileOutputStream("C:\\aaaa\\ww.csv");
int bytesRead = -1;
byte[] buffer = new byte[4096];
while ((bytesRead = inputStream.read(buffer)) != -1) {
outputStream.write(buffer, 0, bytesRead);
}
outputStream.close();
inputStream.close();
The code works well.I send request and ı can download the csv file into my computer.But ı want to know that if the csv file have turkish characters(ş,ğ,ı,ç) can ı download the csv with that characters.
or what can ı do for that characters to see them in csv file.
The server sends you a stream of bytes and you just save the byte stream on your disk, so you don't need to worry about characters.