I have very large file about 2GB and i want to encrypt it by cutting it on chunks, if I read it at once its too big and my JVM throws OutOfMemory Error. I am using AES/CBC/PKCS5padding. Can you recommend me a way to do it? I read about CipherStreams but I am not sure if they would work. Thanks in advance!
I am using this strategy right now, but its not working for bigger files.
try (FileInputStream input = new FileInputStream(in);
FileOutputStream output = new FileOutputStream(out)) {
byte[] encoded = cipher.doFinal(ByteStreams.toByteArray(input));
output.write(encoded);
}
Create a CipherOutputStream wrapping your FileOuputStream.
Use Files.copy() to copy your file to this CipherOutputStream.
Done.
Read the javadoc for details.
Related
I am trying to read the mails from the MS Exchange by using camel and getting the attachments as DataHandler. A 10MB file takes around 3hrs to write into the location.
File outputFile = new File(someDirectory, someFileName);
DataHandler attachment_data = destination1Attachments.get("someFileName.txt");
try (FileOutputStream fos = new FileOutputStream(outputFile)) {
attachment_data.writeTo(fos);
}
I have also noticed that sometimes a 6 to 7Mb file takes around 2 to 3 minutes and when another mail comes just after that it takes more time than expected.
Because of GC ?
Trying to find the exact root cause or any other method to write the data to the file.
Update 1 :
Tried using BufferedOutputStream around FileOutputSteam as mentioned by #user207421 in the comment. No much change could find (just 1sec or little more).
This could be due to the default implementation of write mechanism.
attachment_data.writeTo(fos);
If the DataHandler.getDataSource()!=null then this theory will work
In this method implementation 8 bytes are getting read at a time and writing into the stream. The number of read and writes are more and this might be causing the issue.
Try reading the on your own from DataHandler.getInputStream and write to file by increasing the read content from the input stream.
One must assume that the object is loaded in memory or writeTo very inefficient. Hence specify the DataFlavor and inspect attachment_data.getTransferDataFlavors().
DataFlavor flavor = new DataFlavor(InputStream.class, "application/octetstream");
try (InputStream in = (InputStream) attachment_data.getTransferData(flavor)) {
Some fiddling needed.
I was trying to read from file and then write to other file. I use code bellow to do so.
byte[] bytes = Files.readAllBytes(file1);
Writer Writer = new BufferedWriter(new OutputStreamWriter(new FileOutputStream(file2), "UTF-8"));
for(int i=0;i<bytes.length;i++)
Writer.write(bytes[i]);
Writer.close();
But when I change file1 to picture.png and file2 to picture2.png, this method doesn't work and I can't open picture2.png using image viewer.
What have I done wrong?
Writers are for writing text, possibly in different formats (ie utf-8 / 16, etc). For writing raw bytes, don't use writers. Just use (File)OutputStreams.
It is truly as simple as
byte[] bytes = ...;
FileOutputStream fos = ...;
fos.write(bytes);
The other answers explain why what you have potentially fails.
I'm curious why you're already using one Java NIO method, but not others? The library already has methods to do this for you.
byte[] bytes = Files.readAllBytes(file1);
Files.write(file2, bytes, StandardOpenOption.CREATE_NEW); // or relevant OpenOptions
or
FileOutputStream out = new FileOutputStream(file2); // or buffered
Files.copy(file1, out);
out.close();
or
Files.copy(file1, file2, options);
The problem is that Writer.write() doesn't take a byte. It takes a char, which is variable size, and often bigger than one byte.
But once you've got the whole thing read in as a byte[], you can just use Files.write() to send the whole array to a file in much the same way that you read it in:
Files.write(filename, bytes);
This is the more modern NIO idiom, rather than using an OutputStream.
It's worth reading the tutorial.
I'm trying to write code in Java that will encrypt file. I had used example from this site:
http://www.avajava.com/tutorials/lessons/how-do-i-encrypt-and-decrypt-files-using-des.html
Everything works fine but I need code that will overwrite original file with encrypted one. I'd changed only this:
FileInputStream fis = new FileInputStream("original.txt");
FileOutputStream fos = new FileOutputStream("original.txt");
encrypt(key, fis, fos);
FileInputStream fis2 = new FileInputStream("original.txt");
FileOutputStream fos2 = new FileOutputStream("original.txt");
Encryption works, but after decryption decrypted file is empty.
Can someone explain me what's the problem and how to solve it?
Thanks !
You shouldn't read and overwrite the same file simultaneously with FileInputStream and FileOutputStream. Often, you'll get lucky, but the behavior is going to vary based on the underlying system, and that's not good. Instead, write to a temporary file, then move the temporary file to the location of the original file.
I'm trying to output a byte array to a file. The String that I create displays correctly when I call System.out.println(ouput_stream). Hover, it does not output correctly when I use a FileOutputStream. Here's what I've got so far. Any suggestions?
FileOutputStream fos = new FileOutputStream("outputFile.txt");
OutputStreamWriter out = new OutputStreamWriter(fos, "UTF-8");
String received_string = new String(rPacket.getData(), 0, rPacket.getLength(), "UTF-8");
System.out.println(received_string);
out.write(received_string, 0, received_string.length());
The console displays the information the information when I call the System.out.println(received_string. However, it doesn't output the file correctly. I asked a similar question earlier, but now am struggling on the output. Thanks for any help.
Have a look at Apache Commons FileUtils.writeByteArrayToFile
Should do what you need
Your code looks a bit weird (do you want bytes or Strings), but should be okay.
What did you find in the file? Did you close the stream/writer before looking at it? If not, it is likely still being buffered somewhere.
Try to avoid the conversion to String, though, if you really want to just pipe that packet to a file.
Don't use a String or a Writer at all. Just copy the bytes directly from the packet to the file:
fos.write(rPacket.getData(), rPacket.getOffset(), rPacket.getLength());
Then you can't possibly corrupt the data.
Try using BufferedWriter ..
BufferedWriter writer = new BufferedWriter(new FileWriter("outfile"));
writer.write(received_string);
I need to send image from android app to java app. Basically, I need a byte array from the image to send to rf module which transmits.Another rf module receives and sends the byte array to java app which must make the image .
Android code:
FileInputStream fis = new FileInputStream(myFile);
byte[] b=new byte[(int)myFile.length()];
fis.read(b);server.send(b);
Java code:
FileOutputStream fwrite = new FileOutputStream(new File("my_xml"),true);
fwrite.write(bb);//bb is a byte from rf using input stream as soon as a byte comes it is read to file. This is necessary for some other reasons
fwrite.flush();
fwrite.close();
After getting full file:
FileInputStream fir=new FileInputStream("my_xml");
final BufferedImage bufferedImage = ImageIO.read(fir);
ImageIO.write(bufferedImage, "bmp", new File("image.bmp"));
fir.close();
I am getting error javax.imageio.IIOException: Bogus Huffman table definition
The rf is working fine because text file is being sent perfectly.Please help.Even without ImageIo code is not giving image even after changing extension to jpeg
The error means that the image file cant be read because the format is wrong.That is some bytes are missing or wrong or out of proper position and therefore file cant be decoded. My rf transfer does not have protocols like tcp/ip therefore some bytes are lost due to error in communication channel and hence the error.
You don't need to use ImageIO just to copy a file. Just read and write the bytes.
Your code has other problems:
You are assuming that read(byte[]) fills the buffer. It doesn't. Check the Javadoc.
You are also assuming that the file length fits into an int. If it does, fine. If it doesn't, you are hosed.
You appear to be opening and closing the FileOutputStream on every byte received. This could not be more inefficient. Open it once, write everything, close it.
flush() before close() is redundant.
You are storing the image in a file called 'my_xml'. This is only going to cause confusion, if it hasn't already.
You don't even need the file. Just load the image directly from the input stream.