I am trying to fetch object from bucket using openPrefetchingReadChannel GCSInputChannel. As Google developer tutorial says:
GcsInputChannel readChannel = gcsService.openPrefetchingReadChannel(fileName, 0, 1024 * 1024);
Prefetching provides is a major performance advantage for most applications, because it
allows processing part of the file while more data is being downloaded in the background
in parallel.The third parameter is the size of the prefetch buffer, set in the example
to 1 megabyte.
Well this is not happening for me. Please have a look at my snippet:
GcsInputChannel readChannel = gcsService.openPrefetchingReadChannel(fileName, 0, 1024);
copy(Channels.newInputStream(readChannel), resp.getOutputStream());
private void copy(InputStream input, OutputStream output) throws IOException {
try {
byte[] buffer = new byte[BUFFER_SIZE];
int bytesRead = input.read(buffer);
while (bytesRead != -1) {
output.write(buffer, 0, bytesRead);
bytesRead = input.read(buffer);
}
} finally {
input.close();
output.close();
}
}
Ref: https://code.google.com/p/appengine-gcs-client/source/browse/trunk/java/example/src/com/google/appengine/demos/GcsExampleServlet.java
Above code should deliver 1KB of data from uploaded object but it is returning the whole data of object i.e. 8.4KB. Please look at the screenshot:
I am not sure what is happening. Need your help guys
The third argument for openPrefetchingReadChannel is not the max size to read (or limit).
Is the the internal buffer size for prefetching. In your case you may want to track how much
you read and keep writing until reached the desired limit
Related
I want to split a file into multiple chunks (in this case, trying lengths of 300) and base64 encode it, since loading the entire file to memory gives a negative array exception when base64 encoding it. I tried using the following code:
int offset = 0;
bis = new BufferedInputStream(new FileInputStream(f));
while(offset + 300 <= f.length()){
byte[] temp = new byte[300];
bis.skip(offset);
bis.read(temp, 0, 300);
offset += 300;
System.out.println(Base64.encode(temp));
}
if(offset < f.length()){
byte[] temp = new byte[(int) f.length() - offset];
bis.skip(offset);
bis.read(temp, 0, temp.length);
System.out.println(Base64.encode(temp));
}
At first it appears to be working, however, at one point it switches to just printing out "AAAAAAAAA" and fills up the entire console with it, and the new file is corrupted when decoded. What could be causing this error?
skip() "Skips over and discards n bytes of data from the input stream", and read() returns "the number of bytes read".
So, you read some bytes, skip some bytes, read some more, skip, .... eventually reaching EOF at which point read() returns -1, but you ignore that and use the content of temp which contains all 0's, that are then encoded to all A's.
Your code should be:
try (InputStream in = new BufferedInputStream(new FileInputStream(f))) {
int len;
byte[] temp = new byte[300];
while ((len = in.read(temp)) > 0)
System.out.println(Base64.encode(temp, 0, len));
}
This code reuses the single buffer allocated before the loop, so it will also cause much less garbage collection than your code.
If Base64.encode doesn't have a 3 parameter version, do this:
try (InputStream in = new BufferedInputStream(new FileInputStream(f))) {
int len;
byte[] temp = new byte[300];
while ((len = in.read(temp)) > 0) {
byte[] data;
if (len == temp.length)
data = temp;
else {
data = new byte[len];
System.arraycopy(temp, 0, data, 0, len);
}
System.out.println(Base64.encode(data));
}
}
Be sure to use a buffer size that is a multiple of 3 for encoding and a multiple of 4 for decoding when using chunks of data.
300 fulfills both, so that is already OK. Just as an info for those trying different buffer sizes.
Keep in mind, that reading from a stream into a buffer can in some cicumstances result in a buffer not being fully filled, even though the end of the stream was not yet reached. Might be possible when reading from an internet stream and a timeout occures.
You can heal that, but taking that into account would lead to much more complex coding, that would not be educational anymore.
I want to read a file into a byte array. So, I am reading it using:
int len1 = (int)(new File(filename).length());
FileInputStream fis1 = new FileInputStream(filename);
byte buf1[] = new byte[len1];
fis1.read(buf1);
However, it is realy very slow. Can anyone inform me a very fast approach (possibly best one) to read a file into byte array. I can use java library also if needed.
Edit: Is there any benchmark which one is faster (including library approach).
It is not very slow, at least there is not way to make it faster. BUT it is wrong. If file is big enough the method read() will not return all bytes from fist call. This method returns number of bytes it managed to read as return value.
The right way is to call this method in loop:
public static void copy(InputStream input,
OutputStream output,
int bufferSize)
throws IOException {
byte[] buf = new byte[bufferSize];
int bytesRead = input.read(buf);
while (bytesRead != -1) {
output.write(buf, 0, bytesRead);
bytesRead = input.read(buf);
}
output.flush();
}
call this as following:
ByteArrayOutputStream baos = new ByteArrayOutputStream();
copy(new FileInputStream(myfile), baos);
byte[] bytes = baos.toByteArray();
Something like this is implemented in a lot of packages, e.g. FileUtils.readFileToByteArray() mentioned by #Andrey Borisov (+1)
EDIT
I think that reason for slowness in your case is the fact that you create so huge array. Are you sure you really need it? Try to re-think your design. I believe that you do not have to read this file into array and can process data incrementally.
apache commons-io FileUtils.readFileToByteArray
I am transferring file from client to server. I dont know the amount of time it will take to transfer. But my UI will simple remain the same without any intimation to user. I need to keep a progress bar in such a way it should be progress till file is uploaded. How can i acheive this.
I am abit aware of this scenario in .net. but how can we do it in java?
trashgod's answer is correct for actions that are truly 'indeterminate'. Why do you think that your file transfer fits into this category? Haven't you ever downloaded a file on the internet with some sort of progress bar associated with it? Can you imagine not having that?
See the example below that was provided among the answers to How do I use JProgressBar to display file copy progress?
public OutputStream loadFile(URL remoteFile, JProgressBar progress) throws IOException
{
URLConnection connection = remoteFile.openConnection(); //connect to remote file
InputStream inputStream = connection.getInputStream(); //get stream to read file
int length = connection.getContentLength(); //find out how long the file is, any good webserver should provide this info
int current = 0;
progress.setMaximum(length); //we're going to get this many bytes
progress.setValue(0); //we've gotten 0 bytes so far
ByteArrayOutputStream out = new ByteArrayOutputStream(); //create our output steam to build the file here
byte[] buffer = new byte[1024];
int bytesRead = 0;
while((bytesRead = inputStream.read(buffer)) != -1) //keep filling the buffer until we get to the end of the file
{
out.write(buffer, current, bytesRead); //write the buffer to the file offset = current, length = bytesRead
current += bytesRead; //we've progressed a little so update current
progress.setValue(current); //tell progress how far we are
}
inputStream.close(); //close our stream
return out;
}
As shown in How to Use Progress Bars, you can specify indeterminate mode until you either have enough data to gauge progress or the download concludes. The exact implementation depends on how the transfer takes place. Ideally, the sender provides the length first, but it may also be possible to calculate the rate dynamically as data accumulates.
I'm writing an Android application which copies files from the assets to one file on the device's drive (no permission problems, bytes get from the assets to the drive). The file that I need to copy is larger than 1 MB, so I split it up into multiple files, and I copy them with something like:
try {
out = new FileOutputStream(destination);
for (InputStream file : files /* InputStreams from assets */) {
copyFile(file);
file.close();
}
out.close();
System.out.println(bytesCopied); // shows 8716288
System.out.println(new File(destination).length()); // shows 8749056
} catch (IOException e) {
Log.e("ERROR", "Cannot copy file.");
return;
}
Then, the copyFile() method:
private void copyFile(InputStream file) throws IOException {
byte[] buffer = new byte[16384];
int length;
while ((length = file.read(buffer)) > 0) {
out.write(buffer);
bytesCopied += length;
out.flush();
}
}
The correct number of total bytes that the destination file should contain is 8716288 (that's what I get when I look at the original files and if I count the written bytes in the Android application), but new File(destination).length() shows 8749056.
What am I doing wrong?
The file size becomes too large because you are not writing length bytes for each write, you are actually writing the whole buffer each time, which is buffer.length() bytes long.
You should use the write(byte[] b, int off, int len) overload instead, to specify how many bytes in the buffer you want to be written on each iteration.
Didn't you mean to write
out.write(buffer, 0, length);
instead of
out.write(buffer);
Otherwise you would always write the complete buffer, even if less bytes were read. This may then lead to a larger file (filled with some garbage between your original data).
I have some working code in python that I need to convert to Java.
I have read quite a few threads on this forum but could not find an answer. I am reading in a JPG image and converting it into a byte array. I then write this buffer it to a different file. When I compare the written files from both Java and python code, the bytes at the end do not match. Please let me know if you have a suggestion. I need to use the byte array to pack the image into a message that needs to be sent over to a remote server.
Java code (Running on Android)
Reading the file:
File queryImg = new File(ImagePath);
int imageLen = (int)queryImg.length();
byte [] imgData = new byte[imageLen];
FileInputStream fis = new FileInputStream(queryImg);
fis.read(imgData);
Writing the file:
FileOutputStream f = new FileOutputStream(new File("/sdcard/output.raw"));
f.write(imgData);
f.flush();
f.close();
Thanks!
InputStream.read is not guaranteed to read any particular number of bytes and may read less than you asked it to. It returns the actual number read so you can have a loop that keeps track of progress:
public void pump(InputStream in, OutputStream out, int size) {
byte[] buffer = new byte[4096]; // Or whatever constant you feel like using
int done = 0;
while (done < size) {
int read = in.read(buffer);
if (read == -1) {
throw new IOException("Something went horribly wrong");
}
out.write(buffer, 0, read);
done += read;
}
// Maybe put cleanup code in here if you like, e.g. in.close, out.flush, out.close
}
I believe Apache Commons IO has classes for doing this kind of stuff so you don't need to write it yourself.
Your file length might be more than int can hold and than you end up having wrong array length, hence not reading entire file into the buffer.