I am using AES to encrypt files. The problem first came when i tried to encrypt a large file. So i did some reading online and figured that i need to use a buffer and only encrypt bytes of data at a time.
I divided my plaintext into chunks of 8192 bytes of data and then applied the encryption operation on each of these chunks but I am still getting the out of memory error.
public static File encrypt(File f, byte[] key) throws Exception
{
System.out.println("Starting Encryption");
byte[] plainText = fileToByte(f);
SecretKeySpec secretKey = new SecretKeySpec(key, ALGORITHM);
Cipher cipher = Cipher.getInstance(ALGORITHM);
cipher.init(Cipher.ENCRYPT_MODE, secretKey);
System.out.println(plainText.length);
List<byte[]> bufferedFile = divideArray(plainText, 8192);
System.out.println(bufferedFile.size());
List<byte[]> resultByteList = new ArrayList<>();
for(int i = 0; i < bufferedFile.size(); i++)
{
resultByteList.add(cipher.doFinal(bufferedFile.get(i)));
}
ByteArrayOutputStream baos = new ByteArrayOutputStream();
for(byte[] b : resultByteList)
baos.write(b);
byte[] cipherText = baos.toByteArray();
File temp = byteToFile(cipherText, "D:\\temp");
return temp;
}
The fileToByte() takes a file as input and returns a byte array
the divideArray() takes a byte array as input and divides it into an arraylist consisting of smaller byte arrays.
public static List<byte[]> divideArray(byte[] source, int chunkSize) {
List<byte[]> result = new ArrayList<byte[]>();
int start = 0;
while (start < source.length) {
int end = Math.min(source.length, start + chunkSize);
result.add(Arrays.copyOfRange(source, start, end));
start += chunkSize;
}
return result;
}
Here is the error I get
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3236)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
at java.io.OutputStream.write(OutputStream.java:75)
at MajorProjectTest.encrypt(MajorProjectTest.java:61)
at MajorProjectTest.main(MajorProjectTest.java:30)
I am not getting this error if I use a file of a smaller size, but then again, the sole purpose of using buffers was to eliminate the out of memory problem.
Thanks in advance. Any help is appreciated.
One problem is holding arrays and copies of arrays in memory.
Read and write in blocks.
Then doFinal should not be repeated. Use update instead. Many examples just use a single doFinal which is misleading.
So:
public static File encrypt(File f, byte[] key) throws Exception
{
System.out.println("Starting Encryption");
SecretKeySpec secretKey = new SecretKeySpec(key, ALGORITHM);
Cipher cipher = Cipher.getInstance(ALGORITHM);
cipher.init(Cipher.ENCRYPT_MODE, secretKey);
System.out.println(plainText.length);
Path outPath = Paths.get("D:/Temp");
byte[] plainBuf = new byte[8192];
try (InputStream in = Files.newInputStream(f.toPath());
OutputStream out = Files.newOutputStream(outPath)) {
int nread;
while ((nread = in.read(plainBuf)) > 0) {
byte[] enc = cipher.update(plainBuf, 0, nread);
out.write(enc);
}
byte[] enc = cipher.doFinal();
out.write(enc);
}
return outPath.toFile();
}
Explanation
Encryption of some byte blocks goes as:
Cipher.init
Cipher.update block[0]
Cipher.update block[1]
Cipher.update block[2]
...
Cipher.doFinal(block[n-1])
Or instead of the last doFinal:
Cipher.update(block[n-1])
Cipher.doFinal()
Every update or doFinal yielding a portion of the encrypted data.
doFinal also "flushes" final encryption data.
If one has only a single block of bytes, it suffices to call
byte[] encryptedBlock = cipher.doFinal(plainBlock);
Then no calls to cipher.update are needed.
For the rest I used the try-with-resources syntax which automatically closes input and output streams, even should a return happen, or an exception have been thrown.
Instead of File the newer Path is a bit more versatile, and in combination with Paths.get("...") and the very nice utility class Files can provide powerful code: like Files.readAllBytes(path) and much more.
Look at these four variables: plainText, bufferedFile, resultByteList, cipherText. All of them contain your entire file in a slightly different format, which means that each of them is 1.2GB big. Two of them are Lists which means they are likely to be even bigger, because you didn't set the initial size of ArrayLists and they resize automatically when needed. So we are talking about over 5GB of memory needed.
Actually, you add chunks into ByteArrayOutputStream baos, which means that it must store it internally, before you call toByteArray() on it. So it's 5 copies of your data, meaning 6GB+. The ByteArrayOutputStream is internally using an array so it grows similarly to ArrayLists so it will use more memory than needed (see the stacktrace - it tried to resize).
All these variables are in the same scope, never are assigned null which means that they cannot be garbage collected.
You can increase the maximum heap limit (see Increase heap size in Java), but this will be a serious limitation on your program.
Your program throws out of memory error when writing to ByteArrayOutputStream. This is the 4th time you copy all your data, which means that 3.6GB is already allocated. From this I deduce that your heap is set to 4GB (which is a maximum you can set on 32 bit operating system).
What you should do is have a loop, read part of the file, encrypt it and write to another file. This will avoid loading entire file into memory. Lines like List<byte[]> bufferedFile = divideArray(plainText, 8192); or resultByteList.add(...) is something that you shouldn't have in your code - you end up storing entire file in memory. The only thing that you need to keep track of is a cursor (i.e. position which says what bytes you already processed), which is O(1) memory complexity. Then you only need as much memory as the chunk your are encoding - which is far smaller than entire file.
As you're iterating over the file, keep a counter to track the number of bytes:
int encryptedBytesSize = 0;
for(int i = 0; i < bufferedFile.size(); i++) {
resultByteList.add(cipher.doFinal(bufferedFile.get(i)));
encryptedBytesSize += resultByteList.get(resultByteList.size() - 1).length;
}
Then use the constructor which takes a size parameter to create the output buffer:
ByteArrayOutputStream baos = new ByteArrayOutputStream(encryptedBytesSize);
This will avoid the internal buffer from having to grow. Growth could be non-linear so as more bytes are added each iteration even more space is allocated the next time it grows.
But this still might not work, depending on the file size. Another approach would be to:
Read a little chunk of the unencrypted file
Encrypt the chunk
Write to the encrypte file
This avoids having all of the regular and encrypted files in memory at the same time.
Related
I'm trying to decrypt an encrypted data which is stored in a text file. I use the same key and IV to encrypt and decrypt, and transfer it via configuration file.
When I print CipherInputStream to the console I do get some content, but when I try to write it to a text file I don't get any content in it.
This is the piece of code refferes to my problem:
File encryptedData = new File("C:\\Users\\Victoria\\Desktop\\encryptedData.txt");
File decryptedData = new File("C:\\Users\\Victoria\\Desktop\\decryptedData.txt");
FileInputStream inputStream = new FileInputStream(encryptedData);
byte[] inputBytes = new byte[(int) decryptedData.length()];
inputStream.read(inputBytes);
Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");
cipher.init(Cipher.DECRYPT_MODE, newkey, newiv, SecureRandom.getInstance("SHA1PRNG"));
CipherInputStream cipherInputStream = new CipherInputStream(inputStream, cipher);
FileOutputStream outputStream = new FileOutputStream(decryptedData);
System.out.println("cipherInputStream: " + cipherInputStream);
// Writing the decrypted content to an output file
byte[] buff = new byte[1024 * 10];
int length;
while ((length = cipherInputStream.read(buff)) > 0) {
outputStream.write(buff, 0, length);
}
bufin.close();
outputStream.close();
cipherInputStream.close();
Any solutions?
Thank you!
Encrypted data which is stored in a text file
This is already a contradiction in terms. Encrypted data is binary, not text, and should not be stored in files with the .txt extension.
byte[] inputBytes = new byte[(int) decryptedData.length()];
This line of code is meaningless. You don't yet know how long the decrypted data will be. The decrypted file may not even exist, in which case this will produce a zero length array; or it may be different from what is about to be produced, in which case it is the wrong length.
inputStream.read(inputBytes);
Remove this line and the one before it.
It reads into an array which is at best sized to the size of the decrypted data, which is the wrong size for encrypted data, and at worst is just the wrong size, or even zero length, as shown above.
It reads the input probably until that wrongly sized buffer is full, and you then (a) completely ignore the data read and (b) attempt to read the same stream further, which will fail in the decryption loop, or at best produce incorrect output, as you may not be decrypting all the data.
When I print CipherInputStream to the console I do get some content
No you don't. You get a piece of data of the general form CipherInputStream#0011223344, which is just the result of calling CipherInputStream.toString(), which does not contain any 'content'.
I have written code in vb.net to encrypt a file from a memory stream. I also decrypt the file as well as copy the memory stream to a file to assure encryption/ decryption works. My vb solution works.
However my need is to decrypt using Java. When I decrypt my file, I always get an extra "?" character at the very beginning of the file, but other than that the resullts are perfect. Has anyone seen anything like this before? I must admit, my results are from using only one set of data, but I've encrypted it twice using new keys and vectors both times.
A few details. I'm using AES, PKCS7 padding in vb, and PKCS5 padding in Java. The file can be of arbitrary length. Any help is appreciated.
I am posting this from my phone, and don't have the code handy. I can add it tomorrow. I'm just hoping that this description rings a bell with someone.
Thanks,
SH
When I wrote to the MemoryStream in VB, I declared a StreamWriter like so:
Writer = New IO.StreamWriter(MS, System.Text.Encoding.UTF8)
Here's my VB.NET encryption function.
Public Shared Function WriteEncryptedFile(ms As MemoryStream, FileName As String) As List(Of Byte())
Try
Dim original() As Byte
Dim myAes As System.Security.Cryptography.Aes = Aes.Create()
myAes.KeySize = 128
myAes.Padding = PadMode
Dim keys As New List(Of Byte())
keys.Add(myAes.Key)
keys.Add(myAes.IV)
original = ms.ToArray
Dim encryptor As ICryptoTransform = myAes.CreateEncryptor(myAes.Key, myAes.IV)
Using FileEncrypt As New FileStream(FileName, FileMode.Create, FileAccess.Write)
Using csEncrypt As New CryptoStream(FileEncrypt, encryptor, CryptoStreamMode.Write)
csEncrypt.Write(original, 0, original.Length)
csEncrypt.FlushFinalBlock()
FileEncrypt.Flush()
FileEncrypt.Close()
csEncrypt.Close()
End Using
End Using
Return keys
Catch e As Exception
MsgBox("Error during encryption." & vbCrLf & e.Message)
End Try
Return Nothing
End Function
And here's the Java decryption:
public static void DecryptLIGGGHTSInputFile(String fileIn, String fileOut, String base64Key, String base64IV) throws Exception
{
// Get the keys from base64 text
byte[] key = Base64.decodeBase64(base64Key);
byte[] iv= Base64.decodeBase64(base64IV);
// Read fileIn into a byte[]
int len = (int)(new File(fileIn).length());
byte[] cipherText = new byte[len];
FileInputStream bs = new FileInputStream(fileIn);
bs.read(cipherText, 1, len-1);
System.out.println(cipherText.length);
System.out.println((double)cipherText.length/128);
bs.close();
// Create an Aes object
// with the specified key and IV.
Cipher cipher = null;
cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");
// Encrypt the message.
SecretKey secret = new SecretKeySpec(key, "AES");
/*
cipher.init(Cipher.ENCRYPT_MODE, secret, ivspec);
cipherText = cipher.doFinal("Hello, World!".getBytes("UTF-8"));
System.out.println(cipherText);
*/
cipher.init(Cipher.DECRYPT_MODE, secret , new IvParameterSpec(iv));
String plaintext = new String(cipher.doFinal(cipherText), "UTF-8");
System.out.println(plaintext.length());
FileWriter fw = new FileWriter(fileOut);
fw.write(plaintext);
fw.close();
}
It was a BOM problem. When I created the MemoryStream with VB, I initialized it in UTF-8 encoding. The very first character in my file boosted the size and position of the stream from 0 bytes to 4 bytes, when it should have only been one. The solution was to create an encoding based on UTF-8 without Byte Order Marks, like so:
Dim UTF8EncodingWOBOM As New System.Text.UTF8Encoding(False) 'indicates to omit BOM
Writer = New IO.StreamWriter(MS, UTF8EncodingWOBOM)
I read here that there are frequently issues with encoding incompatibilities between platforms due to the presence or lack of byte order mark, as it is neither recommended or required. It's not right to use one, it's not wrong to use one. You basically have to find a way to deal with them. A plethora of other articles and postings suggested different ways to do it. The gist was, either identify them and deal with them if they exist. Since I have control of both the writing and the reading, it makes about as much sense to do away with them entirely.
SH
Preface: This is a homework assignment, and I am almost done with it -- it's just this tiny piece that is preventing me from finishing. With this information, please do not write any code for me, but possibly note what I might be doing wrong.
Okay, here is the simple idea.
Use RSA to encrypt/decrypt a file with ECB Mode. This means if there was a block size of 4, and the string was 'testdata', 'test' would be encrypted with the key, written to file, and then 'data' would be encrypted with the key and written to the file.
My implementation is using 128 as the block size, but I'm having a strange error.
Here is my code to encrypt a block of 128 and append to a file:
ArrayList<byte[]> bytes = new ArrayList<byte[]>();
String file = read_file(input_file);
int index = 0;
while (index<file.length()) {
byte[] block = file.substring(index, Math.min(index+128,file.length())).getBytes();
cipher = new BigInteger(block).modPow(public_exponent, public_modulus).toByteArray();
bytes.add(cipher);
append_bytes(output_file, cipher);
index+=128;
}
Encryption works perfectly. Here's why I think that encryption is not the issue:
Decrypting the data that is being written to the file works
Adding all encrypted data to a list contains the same data as reading the file
If decrypting from the list that I mentioned above, decryption works flawlessly.
It's the strangest issue, though.
This produces the right output:
for(int i = 0; i < bytes.size(); i++) {
decrypted = new BigInteger(bytes.get(i)).modPow(d, modulus).toByteArray();
System.out.print(new String(decrypted));
}
But that is useless, because what's the point of being able to decrypt only after encrypting.
This does not work every time, but it does work occassionaly:
index = 0;
file = new String(read_bytes(output_file));
while(index < file.length()) {
byte[] block = file.substring(index, Math.min(index+128,file.length())).getBytes();
decrypted = new BigInteger(block).modPow(d, modulus).toByteArray();
System.out.println(new String(decrypted));
index+= 128;
}
I am reading the file the same way that it was wrote to; in blocks of 128. But it does not read it properly, and because of that, decryption fails!
Any idea why this might be happening?
You are reading the cipher-text (which is binary data) into a String, then possible running into some charset convertion that messes up everything.
The decryption should read raw bytes. If you need each blocks in a different array, you may use Arrays.copyOfRange(original,from,to).
Another approach would be base64-encoding the ciphertext before writing it to the file, then base-64 decoding before decryption.
I think i'm looking for some sort of basic file encryption but don't know where to start.
I'm looking for someone to tell me where to start looking or, even better, offer some code.
I've written a game that currently saves data to a general text file. This of course could be changed by anyone who wished to do so.
What i need is to create a file that can store integers and strings that is difficult if not impossible to be edited outside of the game.
In my searching i came across .dat files but they seemed more complicated that what i'm looking for.
All help is appreciated, Alex.
You can write your data to a ByteBuffer and then you can distort your data by a simple algorithm. For example, assume that the data you want to save is a String array, you can do this:
String[] data; // the data you want to save
int byteLength = 0;
byte[][] bytes = new byte[data.length][];
// Calculate the length of the content.
for(int i=0; i<data.length; i++) {
bytes[i] = data[i].getBytes();
byteLength += bytes[i].length;
byteLength += 4; // this is for an integer, which is for the length of the String
}
// Transfer the content to a ByteBuffer object
ByteBuffer buffer = ByteBuffer.allocate(byteLength);
for(int i=0; i<bytes.length; i++) {
// Put the length of the current byte array
buffer.putInt(bytes[i].length);
for(int j=0; j<bytes[i].length; j++) {
// Reverse the byte so that it can't be understood
buffer.put((byte)(~bytes[i][j]));
}
}
After writing all of your content to the ByteBuffer object, you can take the resulting byte array and write it down to a file.
FileOutputStream fos = new FileOutputStream("YourFileName.anyExtension");
fos.write(buffer.array());
fos.close();
While reading the file back, you should first read an integer, which is the length of the data you should read as byte array, then you should read this byte array.
FileInputStream fis = new FileInputStream("YourFileName.anyExtension");
DataInputStream dis = new DataInputStream(fis);
ArrayList<String> list = new ArrayList<String>();
byte[] bytes;
while(dis.available()) {
int length = dis.readInt();
bytes = new byte[length];
for(int i=0; i<length; i++) {
// Those bytes were reversed, right?
bytes[i] = (byte)(~dis.readByte());
}
// Convert byte array to String
String str = new String(bytes);
list.add(str);
}
Now you have an ArrayList of your String data.
Of course this is not the best, the safest, and the fastest algorithm. You can always find or create faster. But I think this is a good example of doing those kind of things.
If you are using Java you can just try and create a class that implements Serializable This way you can just create an object with all your meta info stored inside, serialize it, and when you wanna load it just deserialize it again.
Its not very safe though since you only need to know have the class it was made with, to deserialize it. But it is something to begin with.
Look into digital signatures, specifically HMACs. Those are pretty much exactly what you need, and the Java Crypto framework should make things fairly straightforward. Here's a potentially relevant SO entry: How to generate an HMAC in Java equivalent to a Python example?
You could pass your file writing stream thru a CipherOutputStream
Generate a random string, or number or anything. get its byte array, produce a key, and use it to encrypt your file.
byte password[] = (WHAT YOUR WANT. STRING, NUMBER, etc.).getBytes();
DESKeySpec desKeySpec;
try {
desKeySpec = new DESKeySpec(password);
SecretKeyFactory keyFactory = SecretKeyFactory.getInstance("DES");
SecretKey key = keyFactory.generateSecret(desKeySpec);
Cipher desCipher = Cipher.getInstance("DES/ECB/PKCS5Padding");
desCipher.init(Cipher.ENCRYPT_MODE, key);
// Create stream
FileOutputStream fos = new FileOutputStream("Your file here");
BufferedOutputStream bos = new BufferedOutputStream(fos);
CipherOutputStream cos = new CipherOutputStream(bos, desCipher);
}
Now you can write to the file using cos
Reading the file is done the same way using the SecretKey object
SecretKey key = loadKey(); // Deserialize your SecretKey object
try {
Cipher cipher = Cipher.getInstance("DES/ECB/PKCS5Padding");
cipher.init(Cipher.DECRYPT_MODE, key);
FileInputStream fis = new FileInputStream("Your file here");
BufferedInputStream bis = new BufferedInputStream(fis);
CipherInputStream cis = new CipherInputStream(bis, cipher);
now you can read using cis
The downside is you need to keep the SecretKey object (Serialize it or something) it wouldn't be a problem for any low level hacker to get the data (since the key is stored on the device) but it wouldn't allow just changing your data using a text editor.
I am trying to decrypt a file in Java which was encrypted in C# using Rijndael/CBC/PKCS7. I keep getting the following exception:
javax.crypto.BadPaddingException: pad block corrupted
at org.bouncycastle.jce.provider.JCEBlockCipher.engineDoFinal(Unknown Source)
at javax.crypto.Cipher.doFinal(DashoA13*..)
at AESFileDecrypter.decrypt(AESFileDecrypter.java:57)
when the doFinal(inpbytes) method is called by the web server for the first byte[]. I am guessing this is a problem with the key or IV. I have the encrypted files on my file system for testing. Is there anything that anyone can see glaringly wrong with my code below?
***keyStr is base64 encoded
public AESFileDecrypter(String keyStr){
try {
Security.addProvider(new BouncyCastleProvider());
convertIvParameter();
key = new sun.misc.BASE64Decoder().decodeBuffer(keyStr);
//use the passed in Base64 decoded key to create a key object
decryptKey = new SecretKeySpec(key, "AES");
//specify the encryption algorithm
decryptCipher = Cipher.getInstance("AES/CBC/PKCS7Padding", "BC");
//make a parameter object for the initialization vector(IV)
IvParameterSpec ivs = new IvParameterSpec(_defaultIv);
//initialize the decrypter to the correct mode, key used and IV
decryptCipher.init(Cipher.DECRYPT_MODE, decryptKey, ivs);
}
catch (Exception e) {
e.printStackTrace();
}
}
public void convertIvParameter() {
int[] iv = new int[] {11, 190, 165, 33, 68, 88, 11, 200, 245, 35, 68, 23, 60, 24, 223, 67};
_defaultIv = new byte[16];
for(int x = 0; x < _defaultIv.length; x++) {
_defaultIv[x] = (byte)iv[x];
}
}
public void decryptUpdate(byte[] inpBytes) throws Exception {
//decrypt the byte passed in from the web server
decryptCipher.update(inpBytes);
}
public byte[] decryptFinal() throws Exception {
//decrypt the byte passed in from the web server
return decryptCipher.doFinal();
}
//sends bytes to the client for diaply
private void sendBytes(FileInputStream fis, OutputStream os)throws Exception {
//set the buffer size to send 4k segments of data
aesFileDecrypter = new AESFileDecrypter(<Insert Key string here>);
byte[] buffer = new byte[4096];
int bytes = 0, totalBytes = fis.available();
//while there is still data to be sent keep looping and write the data
//to the output stream as the buffer is filled
try {
while ((bytes = fis.read(buffer)) != -1) {
aesFileDecrypter.decryptUpdate(buffer);
//os.write(buffer, 0, bytes);
}
os.write(aesFileDecrypter.decryptFinal(), 0, totalBytes);
}
catch(Exception e) {
e.printStackTrace();
}
}
Firstly, just to be clear, from comments below, you shouldn't call doFinal() on every block, because doFinal() expects any padding at the end, which obviouslly won't be there in intermediate blocks. Either (a) call update() on intermediate data, then doFinal() at the end, or (b) just arrange to have all your data in one buffer or byte array, and call doFinal() once on the whole job lot.
It's not clear from the code you posted that that's actually what you're doing, but it should be mentioned just in case.
Failing that, then as a first step to debugging, I'd suggest whichever of these two is easier for you:
Decrypting in ECB mode with no padding and seeing what you get. Look at the first block of data this brings back. If you can XOR this with your IV bytes and get the expected decrypted data, you know your key is OK.
Dumping out the actual key bytes from C# before base 64 encoding and Java after decoding and checking they are the same.
As I recall, C# has unsigned bytes (whereas Java signed) so there are a few places where there's room for things subtly going wrong with byte signedness.
I have encountered this problem before.
When I wrote some code to do encryption and decryption like this:
Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");
cipher.init(Cipher.ENCRYPT_MODE, new SecretKeySpec(sec, "AES"),new IvParameterSpec(new byte[cipher.getBlockSize()]));
byte[] encode = cipher.doFinal(data);
cipher.init(Cipher.DECRYPT_MODE, new SecretKeySpec(sec, "AES"), new IvParameterSpec(new byte[cipher.getBlockSize()]));
byte[] decode = cipher.doFinal(encode);
I forgot the first IvParameterSpec(new byte[cipher.getBlockSize()]) when encrypting data, then I got an exception "pad block corrupted", so maybe you should check you encryption code.
As far as I know AES is based on Rijndael, but the specification is not exactly the same. I would suggest to check the key and block size you are using to cipher in C# and the sizes being use in Java. (.Net differences between Rijndael and AES).
The doFinal() was the undoing of the code above, and I ended up just using cipher streams instead of the update/doFinal approach. This way I could use the FileInputStream and my cipher as parameters for the CipherInputStream, and then pass the output to the web browser through an OutputStream. Breaking the update and doFinal out into their own method calls made the task much more difficult and both methods were deleted from the decrypter class (leaving a single while loop that read in chunks of data and output it to the browser). The Bouncy Castle Provider was also not needed in this case and PKCS5Padding was enough, which was given by the SunJCE.