I'm currently trying to read in a image file from the server but either getting a incomplete data or
Exception in thread "main"
java.lang.NegativeArraySizeException.
Has this something to do with the buffer size? I have tried to use static size instead of contentlength. Please kindly advise.
URL myURL = new URL(url);
HttpURLConnection connection = (HttpURLConnection)myURL.openConnection();
connection.setRequestMethod("GET");
status = connection.getResponseCode();
if (status == 200)
{
int size = connection.getContentLength() + 1024;
byte[] bytes = new byte[size];
InputStream input = new ByteArrayInputStream(bytes);
FileOutputStream out = new FileOutputStream(file);
input = connection.getInputStream();
int data = input.read(bytes);
while(data != -1){
out.write(bytes);
data = input.read(bytes);
}
out.close();
input.close();
Let's examine the code:
int size = connection.getContentLength() + 1024;
byte[] bytes = new byte[size];
why do you add 1024 bytes to the size? What's the point? The buffer size should be something large enough to avoid too many reads, but small enough to avoid consuming too much memory. Set it at 4096, for example.
InputStream input = new ByteArrayInputStream(bytes);
FileOutputStream out = new FileOutputStream(file);
input = connection.getInputStream();
Why do you create a ByteArrayInputStream, and then forget about it completely? You don't need a ByteArrayInputStream, since you don't read from a byte array, but from the connection's input stream.
int data = input.read(bytes);
This reads bytes from the input. The max number of bytes read is the length of the byte array. The actual number of bytes read is returned and stored in data.
while (data != -1) {
out.write(bytes);
data = input.read(bytes);
}
So you have read data bytes, but you don't write only the first data bytes of the array. You write the whole array of bytes. That is wrong. Suppose your array if of size 4096 and data is 400, instead of writing the 400 bytes that have been read, you write the 400 bytes + the remaining 3696 bytes of the array, which could be 0, or could have values coming from a previous read. It should be
out.write(bytes, 0, data);
Finally:
out.close();
input.close();
If any exception occurs before, those two streams will never be closed. Do that a few times, and your whold OS won't have file descriptos available anymore. Use the try-with-resources statement to be sure your streams are closed, no matter what happens.
This code can help you
input = connection.getInputStream();
byte[] buffer = new byte[4096];
int n = - 1;
OutputStream output = new FileOutputStream( file );
while ( (n = input.read(buffer)) != -1)
{
if (n > 0)
{
output.write(buffer, 0, n);
}
}
output.close();
Related
I have a C# server that is sending a file (in bytes) to my JAVA client. Currently I am using DataInputStream to read data from socket with a buffer size of 1024 bytes and then storing that data in a file.
Can anyone tell me how I would know how much data I have received in my socket so that I would adjust my bufferSize accordingly?
Is there any other way except DataInputStream to read incoming byte data in data from a socket?
Code:
DataInputStream In = new DataInputStream(socket.getInputStream());
FileOutputStream fos = new FileOutputStream(getFilePath(fileName));
int bytesRead = 0;
int bufferSize = 1024;
byte[] data = new byte[bufferSize];
// fileSize is the int Variable having total Bytes of file that is being sent by server
while (bytesRead < fileSize)
{
bytesRead += dIn.read(data, 0, bufferSize);
fos.write(data);
}
Hint: I know the size of the incoming file.
After a long search, I found one solution. Not perfectly fine, but somehow, it did the job. I used dataInputStream.available() to check how many bytes are available and then only read that number of available bytes.
Code:
DataInputStream dIn = new DataInputStream(socket.getInputStream());
FileOutputStream fos = new FileOutputStream(getFilePath(fileName));
int availableBytes = dIn.available();
while (availableBytes> 0) {
Log.d(TAG, "data is available");
size = dIn.read(data, 0, availableBytes);
fos.write(data, 0, size);
availableBytes = dIn.available();
}
Why i cant read POST-request with 150k chars?
I can only read ~15k chars all time
InputStream is = socket.getInputStream();
ByteArrayOutputStream baos = new ByteArrayOutputStream();
while (is.available() > 0 && (length = is.read(buffer)) != -1) {
baos.write(buffer, 0, length);
}
System.out.println(baos.toString(StandardCharsets.UTF_8.name()));
UPD: if we ignored is.available(), code freezes in the while:
InputStream is = socket.getInputStream();
ByteArrayOutputStream baos = new ByteArrayOutputStream();
while ((length = is.read(buffer)) != -1) {
baos.write(buffer, 0, length);
}
System.out.println(baos.toString(StandardCharsets.UTF_8.name()));
There are no exceptions.
Docs for avaiable() says:
available()
Returns an estimate of the number of bytes that can be read (or skipped over) from this input stream without blocking
So I'm going to guess that you internally have 15k buffers and you're only reading up to the end of your own buffer, not to the end of the stream. You should frankly be ignoring availabe() in this case and just call read( byte[] ) until it returns -1.
Your updated code example looks almost exactly like the code I use to read streams. I think the problem must be on the sender's side. Either the sender is not closing the stream properly, or there's some network issue that doesn't allow enough packets through.
For reference, here's the code I use to read an entire stream. (Lightly tested.)
public static ByteArrayOutputStream readFully( InputStream ins )
throws IOException
{
ByteArrayOutputStream bos = new ByteArrayOutputStream();
byte[] bytes = new byte[ 1024 ];
for( int length; ( length = ins.read( bytes ) ) != -1; )
bos.write( bytes, 0, length );
return bos;
}
When try to convert InputStream into byte array to know size of the file being uploaded. I am able to get size, but InputStream.read() becomes -1. How to check file size before writing it to server?
My current code gives me size , but InputStream reaches the end.
private static byte[] readFully(InputStream input) throws IOException
{
byte[] buffer = new byte[8192];
int bytesRead;
byte []bytes=null;
ByteArrayOutputStream output = new ByteArrayOutputStream();
while ((bytesRead = input.read(buffer)) != -1)
{
System.out.println("Buffer is "+input.read(buffer));
output.write(buffer, 0, bytesRead);
}
bytes=output.toByteArray();
output.close();
return bytes;
}
If you are implementing a web-server in Java, please take a look at the following link:
http://www.prasannatech.net/2008/11/http-web-server-java-post-file-upload.html
You must try to read the incomming information until you found its boundary.
You can't use read() method now, because the InputStream may not be ready to be read yet.
In java input or output stream , there always has a byte array size of 1024.
Just like below:
URL url = new URL(src);
URLConnection connection = url.openConnection();
InputStream is = connection.getInputStream();
OutputStream os = new FileOutputStream("D:\\images"+"\\"+getName(src)+getExtension(src));
byte[] byteArray = new byte[1024];
int len = 0;
while((len = is.read(byteArray))!= -1){
os.write(byteArray, 0, len);
}
is.close();
os.close();
Why initialize this array to 1024?
That is called buffering and ,each time you overwrite the contents of the buffer each time you go through the loop.
Simply reading file in chunks, instead of allocating memory for the file content at a time.
Reason behind this to do is you will become a clear victim of OutOfMemoryException if file is too large.
And coming to the specific question, That is need not to be 1024, even you can do with 500. But a minimum 1024 is a common usage.
When I am trying to download a large file which is of 260MB from server, I get this error: java.lang.OutOfMemoryError: Java heap space. I am sure my heap size is less than 252MB. Is there any way I can download large files without increasing heap size?
How I can download large files without getting this issue? My code is given below:
String path= "C:/temp.zip";
response.addHeader("Content-Disposition", "attachment; filename=\"test.zip\"");
byte[] buf = new byte[1024];
try {
File file = new File(path);
long length = file.length();
BufferedInputStream in = new BufferedInputStream(new FileInputStream(file));
ServletOutputStream out = response.getOutputStream();
while ((in != null) && ((length = in.read(buf)) != -1)) {
out.write(buf, 0, (int) length);
}
in.close();
out.close();
There are 2 places where I can see you could potentially be building up memory usage:
In the buffer reading your input file.
In the buffer writing to your output stream (HTTPOutputStream?)
For #1 I would suggest reading directly from the file via FileInputStream without the BufferedInputStream. Try this first and see if it resolves your issue. ie:
FileInputStream in = new FileInputStream(file);
instead of:
BufferedInputStream in = new BufferedInputStream(new FileInputStream(file));
If #1 does not resolve the issue, you could try periodically flushing the output stream after so much data is written (decrease chunk size if necessary):
ie:
try
{
FileInputStream fileInputStream = new FileInputStream(file);
byte[] buf=new byte[8192];
int bytesread = 0, bytesBuffered = 0;
while( (bytesread = fileInputStream.read( buf )) > -1 ) {
out.write( buf, 0, bytesread );
bytesBuffered += bytesread;
if (bytesBuffered > 1024 * 1024) { //flush after 1MB
bytesBuffered = 0;
out.flush();
}
}
}
finally {
if (out != null) {
out.flush();
}
}
Unfortunately you have not mentioned what type out is. If you have memory issues I guess it is ByteArrayOutpoutStream. So, replace it by FileOutputStream and write the byte you are downloading directly to file.
BTW, do not use read() method that reads byte-by-byte. Use read(byte[] arr) instead. This is much faster.
First you can remove the (in != null) from your while statement, it's unnecessary. Second, try removing the BufferedInputStream and just do:
FileInputStream in = new FileInputStream(file);
There's nothing wrong (in regard to memory usage) with the code you're show. Either the servlet container is configured to buffer the entire response (look at the web.xml configuration), or the memory is being leaked elsewhere.