Preface: This is a homework assignment, and I am almost done with it -- it's just this tiny piece that is preventing me from finishing. With this information, please do not write any code for me, but possibly note what I might be doing wrong.
Okay, here is the simple idea.
Use RSA to encrypt/decrypt a file with ECB Mode. This means if there was a block size of 4, and the string was 'testdata', 'test' would be encrypted with the key, written to file, and then 'data' would be encrypted with the key and written to the file.
My implementation is using 128 as the block size, but I'm having a strange error.
Here is my code to encrypt a block of 128 and append to a file:
ArrayList<byte[]> bytes = new ArrayList<byte[]>();
String file = read_file(input_file);
int index = 0;
while (index<file.length()) {
byte[] block = file.substring(index, Math.min(index+128,file.length())).getBytes();
cipher = new BigInteger(block).modPow(public_exponent, public_modulus).toByteArray();
bytes.add(cipher);
append_bytes(output_file, cipher);
index+=128;
}
Encryption works perfectly. Here's why I think that encryption is not the issue:
Decrypting the data that is being written to the file works
Adding all encrypted data to a list contains the same data as reading the file
If decrypting from the list that I mentioned above, decryption works flawlessly.
It's the strangest issue, though.
This produces the right output:
for(int i = 0; i < bytes.size(); i++) {
decrypted = new BigInteger(bytes.get(i)).modPow(d, modulus).toByteArray();
System.out.print(new String(decrypted));
}
But that is useless, because what's the point of being able to decrypt only after encrypting.
This does not work every time, but it does work occassionaly:
index = 0;
file = new String(read_bytes(output_file));
while(index < file.length()) {
byte[] block = file.substring(index, Math.min(index+128,file.length())).getBytes();
decrypted = new BigInteger(block).modPow(d, modulus).toByteArray();
System.out.println(new String(decrypted));
index+= 128;
}
I am reading the file the same way that it was wrote to; in blocks of 128. But it does not read it properly, and because of that, decryption fails!
Any idea why this might be happening?
You are reading the cipher-text (which is binary data) into a String, then possible running into some charset convertion that messes up everything.
The decryption should read raw bytes. If you need each blocks in a different array, you may use Arrays.copyOfRange(original,from,to).
Another approach would be base64-encoding the ciphertext before writing it to the file, then base-64 decoding before decryption.
Related
I'm writing a program to encrypt and decrypt data.
for encrypting,
I created a symmetric key using keyGenerator.
I transferred the key to the cipher, and created a string version of the key:
String keyString = Base64.getEncoder().encodeToString(symmetricKey.getEncoded());
in order to store it in a configuration file (so I can retrieve the key in the decrypt function).
Now, in the decrypt function I need to get that string back to key format, so I can send it as a parameter to the cipher in dercypt mode.
I convert it back to key this way:
byte[] keyBytes = key.getBytes(Charset.forName("UTF-8"));
Key newkey = new SecretKeySpec(keyBytes,0,keyBytes.length, "AES");
And I transffer it to the cipher and write the output (the decrypted data) using CipherInputStream:
Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");
cipher.init(Cipher.DECRYPT_MODE, newkey, newiv, SecureRandom.getInstance("SHA1PRNG"));
CipherInputStream cipherInputStream = new CipherInputStream(
new ByteArrayInputStream(encryptedBytes), cipher);
ArrayList<Byte> decryptedVal = new ArrayList<>();
int nextByte;
while ((nextByte = cipherInputStream.read()) != -1) {
decryptedVal.add((byte) nextByte);
}
byte[] bytes = new byte[decryptedVal.size()];
for (int i = 0; i < bytes.length; i++) {
bytes[i] = decryptedVal.get(i);
}
String decryptedData = new String(bytes);
cipherInputStream.close();
System.out.println("decryptedData: " + decryptedData);
I get this error:
Exception in thread "main" java.io.IOException: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
So I suspect that there might be a problem with the way I treat the key.
Any suggestions? help would be appreciated!
I think you have not sent IV to decryption function. For decryption in CBC mode, you must provide an IV which is used in encryption process.
Update:
IV will affect only first block in CBC decryption mode. So my answer may affect the unpadding if your data is less than 1 block. It will just change the decrypted plaintext of the first block otherwise.
Of course you get this error: first you apply base 64 encoding:
String keyString = Base64.getEncoder().encodeToString(symmetricKey.getEncoded());
and then you use character-encoding to turn it back into bytes:
byte[] keyBytes = key.getBytes(Charset.forName("UTF-8"));
which just keeps be base64 encoding, probably expanding the key size from 16 bytes to 24 bytes which corresponds with a 192 bit key instead of a 128 bit key. Or 24 bytes key to a 32 bytes key of course - both seem to work.
To solve this you need to use Base64.getDecoder() and decode the key.
Currently you get a key with a different size and value. That means that each block of plaintext, including the last one containing the padding, will decrypt to random plaintext. As random plaintext is unlikely to contain valid padding, you will be greeted with a BadPaddingException.
Reminder:
encoding, e.g. base 64 or hex: encoding bytes to a text string
character-encoding, e.g. UTF-8 or ASCII: encoding a text string into bytes
They are not opposites, that would be decoding and character-decoding respectively.
Remarks:
yes, listen to Ashfin; you need to use a random IV during encryption and then use it during decryption, for instance by prefixing it to the ciphertext (unencrypted);
don't use ArrayList<Byte>; that stores a reference to each separate byte (!) - use ByteArrayOutputStream or any other OutputStream instead;
you can better use a byte buffer and use that to read / write to the streams (note that the read function may not fill the buffer, even if at the start or in the middle of the stream) - reading a single byte at the time is not performant;
lookup try-with-resources for Java;
using a KeyStore may be better than storing in a config file;
GCM mode (AES/GCM/NoPadding) also authenticates data and should be preferred over CBC mode.
I am using AES to encrypt files. The problem first came when i tried to encrypt a large file. So i did some reading online and figured that i need to use a buffer and only encrypt bytes of data at a time.
I divided my plaintext into chunks of 8192 bytes of data and then applied the encryption operation on each of these chunks but I am still getting the out of memory error.
public static File encrypt(File f, byte[] key) throws Exception
{
System.out.println("Starting Encryption");
byte[] plainText = fileToByte(f);
SecretKeySpec secretKey = new SecretKeySpec(key, ALGORITHM);
Cipher cipher = Cipher.getInstance(ALGORITHM);
cipher.init(Cipher.ENCRYPT_MODE, secretKey);
System.out.println(plainText.length);
List<byte[]> bufferedFile = divideArray(plainText, 8192);
System.out.println(bufferedFile.size());
List<byte[]> resultByteList = new ArrayList<>();
for(int i = 0; i < bufferedFile.size(); i++)
{
resultByteList.add(cipher.doFinal(bufferedFile.get(i)));
}
ByteArrayOutputStream baos = new ByteArrayOutputStream();
for(byte[] b : resultByteList)
baos.write(b);
byte[] cipherText = baos.toByteArray();
File temp = byteToFile(cipherText, "D:\\temp");
return temp;
}
The fileToByte() takes a file as input and returns a byte array
the divideArray() takes a byte array as input and divides it into an arraylist consisting of smaller byte arrays.
public static List<byte[]> divideArray(byte[] source, int chunkSize) {
List<byte[]> result = new ArrayList<byte[]>();
int start = 0;
while (start < source.length) {
int end = Math.min(source.length, start + chunkSize);
result.add(Arrays.copyOfRange(source, start, end));
start += chunkSize;
}
return result;
}
Here is the error I get
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3236)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
at java.io.OutputStream.write(OutputStream.java:75)
at MajorProjectTest.encrypt(MajorProjectTest.java:61)
at MajorProjectTest.main(MajorProjectTest.java:30)
I am not getting this error if I use a file of a smaller size, but then again, the sole purpose of using buffers was to eliminate the out of memory problem.
Thanks in advance. Any help is appreciated.
One problem is holding arrays and copies of arrays in memory.
Read and write in blocks.
Then doFinal should not be repeated. Use update instead. Many examples just use a single doFinal which is misleading.
So:
public static File encrypt(File f, byte[] key) throws Exception
{
System.out.println("Starting Encryption");
SecretKeySpec secretKey = new SecretKeySpec(key, ALGORITHM);
Cipher cipher = Cipher.getInstance(ALGORITHM);
cipher.init(Cipher.ENCRYPT_MODE, secretKey);
System.out.println(plainText.length);
Path outPath = Paths.get("D:/Temp");
byte[] plainBuf = new byte[8192];
try (InputStream in = Files.newInputStream(f.toPath());
OutputStream out = Files.newOutputStream(outPath)) {
int nread;
while ((nread = in.read(plainBuf)) > 0) {
byte[] enc = cipher.update(plainBuf, 0, nread);
out.write(enc);
}
byte[] enc = cipher.doFinal();
out.write(enc);
}
return outPath.toFile();
}
Explanation
Encryption of some byte blocks goes as:
Cipher.init
Cipher.update block[0]
Cipher.update block[1]
Cipher.update block[2]
...
Cipher.doFinal(block[n-1])
Or instead of the last doFinal:
Cipher.update(block[n-1])
Cipher.doFinal()
Every update or doFinal yielding a portion of the encrypted data.
doFinal also "flushes" final encryption data.
If one has only a single block of bytes, it suffices to call
byte[] encryptedBlock = cipher.doFinal(plainBlock);
Then no calls to cipher.update are needed.
For the rest I used the try-with-resources syntax which automatically closes input and output streams, even should a return happen, or an exception have been thrown.
Instead of File the newer Path is a bit more versatile, and in combination with Paths.get("...") and the very nice utility class Files can provide powerful code: like Files.readAllBytes(path) and much more.
Look at these four variables: plainText, bufferedFile, resultByteList, cipherText. All of them contain your entire file in a slightly different format, which means that each of them is 1.2GB big. Two of them are Lists which means they are likely to be even bigger, because you didn't set the initial size of ArrayLists and they resize automatically when needed. So we are talking about over 5GB of memory needed.
Actually, you add chunks into ByteArrayOutputStream baos, which means that it must store it internally, before you call toByteArray() on it. So it's 5 copies of your data, meaning 6GB+. The ByteArrayOutputStream is internally using an array so it grows similarly to ArrayLists so it will use more memory than needed (see the stacktrace - it tried to resize).
All these variables are in the same scope, never are assigned null which means that they cannot be garbage collected.
You can increase the maximum heap limit (see Increase heap size in Java), but this will be a serious limitation on your program.
Your program throws out of memory error when writing to ByteArrayOutputStream. This is the 4th time you copy all your data, which means that 3.6GB is already allocated. From this I deduce that your heap is set to 4GB (which is a maximum you can set on 32 bit operating system).
What you should do is have a loop, read part of the file, encrypt it and write to another file. This will avoid loading entire file into memory. Lines like List<byte[]> bufferedFile = divideArray(plainText, 8192); or resultByteList.add(...) is something that you shouldn't have in your code - you end up storing entire file in memory. The only thing that you need to keep track of is a cursor (i.e. position which says what bytes you already processed), which is O(1) memory complexity. Then you only need as much memory as the chunk your are encoding - which is far smaller than entire file.
As you're iterating over the file, keep a counter to track the number of bytes:
int encryptedBytesSize = 0;
for(int i = 0; i < bufferedFile.size(); i++) {
resultByteList.add(cipher.doFinal(bufferedFile.get(i)));
encryptedBytesSize += resultByteList.get(resultByteList.size() - 1).length;
}
Then use the constructor which takes a size parameter to create the output buffer:
ByteArrayOutputStream baos = new ByteArrayOutputStream(encryptedBytesSize);
This will avoid the internal buffer from having to grow. Growth could be non-linear so as more bytes are added each iteration even more space is allocated the next time it grows.
But this still might not work, depending on the file size. Another approach would be to:
Read a little chunk of the unencrypted file
Encrypt the chunk
Write to the encrypte file
This avoids having all of the regular and encrypted files in memory at the same time.
I have written code in vb.net to encrypt a file from a memory stream. I also decrypt the file as well as copy the memory stream to a file to assure encryption/ decryption works. My vb solution works.
However my need is to decrypt using Java. When I decrypt my file, I always get an extra "?" character at the very beginning of the file, but other than that the resullts are perfect. Has anyone seen anything like this before? I must admit, my results are from using only one set of data, but I've encrypted it twice using new keys and vectors both times.
A few details. I'm using AES, PKCS7 padding in vb, and PKCS5 padding in Java. The file can be of arbitrary length. Any help is appreciated.
I am posting this from my phone, and don't have the code handy. I can add it tomorrow. I'm just hoping that this description rings a bell with someone.
Thanks,
SH
When I wrote to the MemoryStream in VB, I declared a StreamWriter like so:
Writer = New IO.StreamWriter(MS, System.Text.Encoding.UTF8)
Here's my VB.NET encryption function.
Public Shared Function WriteEncryptedFile(ms As MemoryStream, FileName As String) As List(Of Byte())
Try
Dim original() As Byte
Dim myAes As System.Security.Cryptography.Aes = Aes.Create()
myAes.KeySize = 128
myAes.Padding = PadMode
Dim keys As New List(Of Byte())
keys.Add(myAes.Key)
keys.Add(myAes.IV)
original = ms.ToArray
Dim encryptor As ICryptoTransform = myAes.CreateEncryptor(myAes.Key, myAes.IV)
Using FileEncrypt As New FileStream(FileName, FileMode.Create, FileAccess.Write)
Using csEncrypt As New CryptoStream(FileEncrypt, encryptor, CryptoStreamMode.Write)
csEncrypt.Write(original, 0, original.Length)
csEncrypt.FlushFinalBlock()
FileEncrypt.Flush()
FileEncrypt.Close()
csEncrypt.Close()
End Using
End Using
Return keys
Catch e As Exception
MsgBox("Error during encryption." & vbCrLf & e.Message)
End Try
Return Nothing
End Function
And here's the Java decryption:
public static void DecryptLIGGGHTSInputFile(String fileIn, String fileOut, String base64Key, String base64IV) throws Exception
{
// Get the keys from base64 text
byte[] key = Base64.decodeBase64(base64Key);
byte[] iv= Base64.decodeBase64(base64IV);
// Read fileIn into a byte[]
int len = (int)(new File(fileIn).length());
byte[] cipherText = new byte[len];
FileInputStream bs = new FileInputStream(fileIn);
bs.read(cipherText, 1, len-1);
System.out.println(cipherText.length);
System.out.println((double)cipherText.length/128);
bs.close();
// Create an Aes object
// with the specified key and IV.
Cipher cipher = null;
cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");
// Encrypt the message.
SecretKey secret = new SecretKeySpec(key, "AES");
/*
cipher.init(Cipher.ENCRYPT_MODE, secret, ivspec);
cipherText = cipher.doFinal("Hello, World!".getBytes("UTF-8"));
System.out.println(cipherText);
*/
cipher.init(Cipher.DECRYPT_MODE, secret , new IvParameterSpec(iv));
String plaintext = new String(cipher.doFinal(cipherText), "UTF-8");
System.out.println(plaintext.length());
FileWriter fw = new FileWriter(fileOut);
fw.write(plaintext);
fw.close();
}
It was a BOM problem. When I created the MemoryStream with VB, I initialized it in UTF-8 encoding. The very first character in my file boosted the size and position of the stream from 0 bytes to 4 bytes, when it should have only been one. The solution was to create an encoding based on UTF-8 without Byte Order Marks, like so:
Dim UTF8EncodingWOBOM As New System.Text.UTF8Encoding(False) 'indicates to omit BOM
Writer = New IO.StreamWriter(MS, UTF8EncodingWOBOM)
I read here that there are frequently issues with encoding incompatibilities between platforms due to the presence or lack of byte order mark, as it is neither recommended or required. It's not right to use one, it's not wrong to use one. You basically have to find a way to deal with them. A plethora of other articles and postings suggested different ways to do it. The gist was, either identify them and deal with them if they exist. Since I have control of both the writing and the reading, it makes about as much sense to do away with them entirely.
SH
I'm trying to encryt/decrypt using RSAEngine library at bouncy castle, with a 2048 bit length key. I'm able to create the keys, store in different files and get from the files, but when I decrypt an image it makes something that I don't know that the file decrypted is not shown correctly.Files are created correctly,and I think the problem is at processBlock method while encrypting and/or decrypting.The code is the following to encrypt:
InputStream clearTextFile;
FileOutputStream textFileProcessed=new FileOutputStream(fileName);
//getKey is a method I implemented and works correctly
RSAKeyParameters key=getKey(keyFileName);
RSAEngine rsaEngine=new RSAEngine();
rsaEngine.init(true,key);
clearTextFile=new FileInputStream(nameClearTextFile);
byte[] bytesReaded;
int nBytesReaded;
int inputBlockSize=rsaEngine.getInputBlockSize();
do
{
bytesReaded = new byte[inputBlockSize];
nBytesReaded=clearTextFile.read(bytesReaded);
if(nBytesReaded>-1)
{ //This is for the last block if it's not 256 byte length
if(nBytesReaded<inputBlockSize)
{
byte[] temp=new byte[nBytesReaded];
for(int i=0;i<nBytesReaded;i++)
{
temp[i]=bytesReaded[i];
}
byte[] encryptedText=rsaEngine.processBlock(temp,0,nBytesReaded);
textFileProcessed.write(encryptedText);
}
else
{
byte[] encryptedText=rsaEngine.processBlock(bytesReaded,0,inputBlockSize);
textFileProcessed.write(encryptedText);
}
}
}while(nBytesReaded>-1);
textFileProcessed.flush();
textFileProcessed.close();
textFileProcessed.close();
And to decrypt:
InputStream encryptedTextFile=new FileInputStream(nameOfFile);
OutputStream decryptedTextFile=new FileOutputStream(nameOfFile);
RSAKeyParameters key=getKey(nameKeyFile);
RSAEngine rsaEngine=new RSAEngine();
rsaEngine.init(false,key);
byte[] bytesReaded;
int nBytesReaded;
int inputBlockSize=rsaEngine.getInputBlockSize();
do
{
bytesLeidos = new byte[inputBlockSize];
nBytesReaded=encryptedTextFile.read(bytesReaded);
if(nBytesReaded>-1)
{
byte[] decryptedText=rsaEngine.processBlock(bytesReaded,0,inputBlockSize);
decryptedTextFile.write(decryptedText);
}
}while(nBytesReaded>-1);
decryptedTextFile.flush();
decryptedTextFile.close();
encryptedTextFile.close();
Thanks in advance
RSAEngine does not add padding, you will lose any leading zeros in your data blocks as a result. You need to use one of the encoding modes available as well.
I'd recommend using a symmetric key algorithm as well and just using RSA to encrypt the symmetric key. It will be a lot faster, and depending on your data, safer as well.
Regards,
David
I think you need to change this line:
if(nBytesReaded>1)
to this
if(nBytesReaded>-1)
And change this in the decypt part, maybe:
rsaEngine.init(false,clave);
to this
rsaEngine.init(false,key);
But there may be more. You aren't encrypting the whole input if the last block isn't full size.
I am trying to decrypt a file in Java which was encrypted in C# using Rijndael/CBC/PKCS7. I keep getting the following exception:
javax.crypto.BadPaddingException: pad block corrupted
at org.bouncycastle.jce.provider.JCEBlockCipher.engineDoFinal(Unknown Source)
at javax.crypto.Cipher.doFinal(DashoA13*..)
at AESFileDecrypter.decrypt(AESFileDecrypter.java:57)
when the doFinal(inpbytes) method is called by the web server for the first byte[]. I am guessing this is a problem with the key or IV. I have the encrypted files on my file system for testing. Is there anything that anyone can see glaringly wrong with my code below?
***keyStr is base64 encoded
public AESFileDecrypter(String keyStr){
try {
Security.addProvider(new BouncyCastleProvider());
convertIvParameter();
key = new sun.misc.BASE64Decoder().decodeBuffer(keyStr);
//use the passed in Base64 decoded key to create a key object
decryptKey = new SecretKeySpec(key, "AES");
//specify the encryption algorithm
decryptCipher = Cipher.getInstance("AES/CBC/PKCS7Padding", "BC");
//make a parameter object for the initialization vector(IV)
IvParameterSpec ivs = new IvParameterSpec(_defaultIv);
//initialize the decrypter to the correct mode, key used and IV
decryptCipher.init(Cipher.DECRYPT_MODE, decryptKey, ivs);
}
catch (Exception e) {
e.printStackTrace();
}
}
public void convertIvParameter() {
int[] iv = new int[] {11, 190, 165, 33, 68, 88, 11, 200, 245, 35, 68, 23, 60, 24, 223, 67};
_defaultIv = new byte[16];
for(int x = 0; x < _defaultIv.length; x++) {
_defaultIv[x] = (byte)iv[x];
}
}
public void decryptUpdate(byte[] inpBytes) throws Exception {
//decrypt the byte passed in from the web server
decryptCipher.update(inpBytes);
}
public byte[] decryptFinal() throws Exception {
//decrypt the byte passed in from the web server
return decryptCipher.doFinal();
}
//sends bytes to the client for diaply
private void sendBytes(FileInputStream fis, OutputStream os)throws Exception {
//set the buffer size to send 4k segments of data
aesFileDecrypter = new AESFileDecrypter(<Insert Key string here>);
byte[] buffer = new byte[4096];
int bytes = 0, totalBytes = fis.available();
//while there is still data to be sent keep looping and write the data
//to the output stream as the buffer is filled
try {
while ((bytes = fis.read(buffer)) != -1) {
aesFileDecrypter.decryptUpdate(buffer);
//os.write(buffer, 0, bytes);
}
os.write(aesFileDecrypter.decryptFinal(), 0, totalBytes);
}
catch(Exception e) {
e.printStackTrace();
}
}
Firstly, just to be clear, from comments below, you shouldn't call doFinal() on every block, because doFinal() expects any padding at the end, which obviouslly won't be there in intermediate blocks. Either (a) call update() on intermediate data, then doFinal() at the end, or (b) just arrange to have all your data in one buffer or byte array, and call doFinal() once on the whole job lot.
It's not clear from the code you posted that that's actually what you're doing, but it should be mentioned just in case.
Failing that, then as a first step to debugging, I'd suggest whichever of these two is easier for you:
Decrypting in ECB mode with no padding and seeing what you get. Look at the first block of data this brings back. If you can XOR this with your IV bytes and get the expected decrypted data, you know your key is OK.
Dumping out the actual key bytes from C# before base 64 encoding and Java after decoding and checking they are the same.
As I recall, C# has unsigned bytes (whereas Java signed) so there are a few places where there's room for things subtly going wrong with byte signedness.
I have encountered this problem before.
When I wrote some code to do encryption and decryption like this:
Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");
cipher.init(Cipher.ENCRYPT_MODE, new SecretKeySpec(sec, "AES"),new IvParameterSpec(new byte[cipher.getBlockSize()]));
byte[] encode = cipher.doFinal(data);
cipher.init(Cipher.DECRYPT_MODE, new SecretKeySpec(sec, "AES"), new IvParameterSpec(new byte[cipher.getBlockSize()]));
byte[] decode = cipher.doFinal(encode);
I forgot the first IvParameterSpec(new byte[cipher.getBlockSize()]) when encrypting data, then I got an exception "pad block corrupted", so maybe you should check you encryption code.
As far as I know AES is based on Rijndael, but the specification is not exactly the same. I would suggest to check the key and block size you are using to cipher in C# and the sizes being use in Java. (.Net differences between Rijndael and AES).
The doFinal() was the undoing of the code above, and I ended up just using cipher streams instead of the update/doFinal approach. This way I could use the FileInputStream and my cipher as parameters for the CipherInputStream, and then pass the output to the web browser through an OutputStream. Breaking the update and doFinal out into their own method calls made the task much more difficult and both methods were deleted from the decrypter class (leaving a single while loop that read in chunks of data and output it to the browser). The Bouncy Castle Provider was also not needed in this case and PKCS5Padding was enough, which was given by the SunJCE.