I have a class that compresses and decompresses a byte array;
public class Compressor
{
public static byte[] compress(final byte[] input) throws IOException
{
try (ByteArrayOutputStream bout = new ByteArrayOutputStream();
GZIPOutputStream gzipper = new GZIPOutputStream(bout))
{
gzipper.write(input, 0, input.length);
gzipper.close();
return bout.toByteArray();
}
}
public static byte[] decompress(final byte[] input) throws IOException
{
try (ByteArrayInputStream bin = new ByteArrayInputStream(input);
GZIPInputStream gzipper = new GZIPInputStream(bin))
{
// Not sure where to go here
}
}
}
How do I decompress the input and return a byte array?
Note: I don't want to do any conversion to strings because of character encoding issues.
your missing code will be something like
byte[] buffer = new byte[1024];
ByteArrayOutputStream out = new ByteArrayOutputStream();
int len;
while ((len = gzipper.read(buffer)) > 0) {
out.write(buffer, 0, len);
}
gzipper.close();
out.close();
return out.toByteArray();
Related
As per my requirement I want to create checksum value using SHA-256, from InputStream,
As below:
private InputStream createZipInput(List<ResponsePack> aList, byte[] manifestData)
{
final int bufferSize = 2048;
byte buffer[] = new byte[bufferSize];
ByteArrayOutputStream byteStream = new ByteArrayOutputStream();
ZipOutputStream zipFileToSend = new ZipOutputStream(byteStream);
LOG.trace("Compressing the file {}");
try
{
for (ResponsePack info : aList)
{
ByteArrayOutputStream byteStreamCheckSum = new ByteArrayOutputStream();
ZipOutputStream zipFileToSendCheckSum = new ZipOutputStream(byteStreamCheckSum);
zipFileToSend.putNextEntry(new ZipEntry(info.getFileName()));
zipFileToSendCheckSum.putNextEntry(new ZipEntry(info.getFileName()));
InputStream in = info.getFileContentStream();
int length;
while ((length = in.read(buffer)) >= 0)
{
zipFileToSend.write(buffer, 0, length);
zipFileToSendCheckSum.write(buffer, 0, length);
}
zipFileToSend.closeEntry();
zipFileToSendCheckSum.closeEntry();
String checksum = validChecksum(byteStreamCheckSum.toByteArray());
LOG.error("Checksum {}", checksum);
zipFileToSendCheckSum.flush();
zipFileToSendCheckSum.close();
}
zipFileToSend.close();
}
catch (IOException e)
{
return e;
}
return new ByteArrayInputStream(byteStream.toByteArray());
}
private static String validChecksum(byte[] dataCopy)
{
printLOG("Byte Array Size {}", dataCopy.length);
try (ZipInputStream zipInputStream = new ZipInputStream(new ByteArrayInputStream(dataCopy)))
{
ZipEntry zipEntry;
MessageDigest digest = DigestUtils.getSha256Digest();
DWriter writer = new DWriter(digest);
while ((zipEntry = zipInputStream.getNextEntry()) != null)
{
org.apache.commons.io.output.ByteArrayOutputStream dest = StreamUtils.extractFileAsByteArrayStream(zipInputStream);
LOG.error("CheckSum Entity creating");
if(zipEntry != null)
{
printLOG("CheckSum Entity file Name {}", zipEntry.getName());
}
LOG.error("Byte array size {}", dest.toByteArray().length);
writer.write(dest.toByteArray());
dest.flush();
dest.close();
}
if (writer.getChecksum() != null)
{
return writer.getChecksum();
}
else
{
return "";
}
}
catch (Exception e)
{
printLOG("Exception encountered while creating checksum: {}", e.getMessage());
return "";
}
}
static class DWriter
{
private final MessageDigest myDigest;
DWriter(MessageDigest digest)
{
myDigest = digest;
}
public void write(byte[] data)
{
myDigest.update(data);
}
public String getChecksum()
{
return new String(Hex.encodeHex(myDigest.digest()));
}
}
But the problem is when I checked the log, found byte array contains value but still checksum always creating for empty string, as below
Byte Array Size 3948
CheckSum Entity creating
CheckSum Entity file Name 20200911104812526.json
Byte array size 20854
Checksum e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Help me where I am doing wrong, due to which I am getting checksum for an empty string
I'm not sure what's wrong with the code but it seems overly complicated: you're writing the input into a zipped stream and the dezip it in memory to read it again.
You don't need to do that: storing the input in a (non-zipped) byte array should be enough.
I think you need to make sure that in.read() works as intended (and that there's actually some data to read).
You get the checksum for a null input and your zip entry is also empty, so it looks like the input was empty. Add some logs or use a debugger to investigate what's happening.
private InputStream createZipInput(List<ResponsePack> aList, byte[] manifestData) {
final int bufferSize = 2048;
byte buffer[] = new byte[bufferSize];
ByteArrayOutputStream byteStream = new ByteArrayOutputStream();
ZipOutputStream zipFileToSend = new ZipOutputStream(byteStream);
LOG.trace("Compressing the file {}");
try {
for (ResponsePack info : aList) {
ByteArrayOutputStream byteStreamCheckSum = new ByteArrayOutputStream();
zipFileToSend.putNextEntry(new ZipEntry(info.getFileName()));
InputStream in = info.getFileContentStream();
int length;
while ((length = in.read(buffer)) != -1) {
zipFileToSend.write(buffer, 0, length);
byteStreamCheckSum.write(buffer, 0, length);
}
zipFileToSend.closeEntry();
MessageDigest digest = DigestUtils.getSha256Digest();
digest.update(byteStreamCheckSum.toByteArray());
String checksum = new String(Hex.encodeHex(digest.digest()));
LOG.error("Checksum {}", checksum);
}
zipFileToSend.close();
} catch (IOException e) {
throw e;
}
return new ByteArrayInputStream(byteStream.toByteArray());
// XML needs to encrypt and deflate and encode using BASE64?I've used this code for this but I am not getting the exact output.
public String encryptDeflateAndBase64(String decryptedString) throws IOException {
byte[] b=decryptedString.getBytes("UTF-8");
byte[] c = new byte[550];
Deflater compresser=new Deflater();
compresser.setInput(b);
compresser.finish();
System.out.println(compresser.deflate(c));
compresser.end();
/*ByteArrayOutputStream bos = new ByteArrayOutputStream(decryptedString.length());
byte[] buf = new byte[4096];
while (!compresser.finished()) {
int count = compresser.deflate(buf);
bos.write(buf, 0, count);
}
bos.close();
byte[] compressedData = bos.toByteArray();
*/ byte[] output = Base64.encodeBase64(c);
return new String(output);
}
//Remove: System.out.println(compresser.deflate(c));
int count = compresser.deflate(c);
ByteBuffer bb = ByteBuffer.wrap(c, 0, count);
byte[] output = Base64.getEncoder().encode(bb);
Where I used the java.util.Base64. To not copy bytes, I used a ByteBuffer.
I want to use Deflater and Inflater (NOT DeflaterOutputStream and InflaterInputStream) to compress files. The problem is that deflater stops working after mentioned buffer size in this case is 1024. I am using the following code:
public class CompressionUtils {
static String deflateInput = "pic.jpg";
static String deflateOutput = "picDeflate.raw";
static String inflateOutput = "picInflate.jpg";
public static void compress() throws IOException {
Deflater deflater = new Deflater();
byte[] data = new byte[1024];
FileInputStream in = new FileInputStream(new File(deflateInput));
FileOutputStream out = new FileOutputStream(new File(deflateOutput));
long readBytes = 0;
while ((readBytes = in.read(data, 0, 1024)) != -1) {
deflater.setInput(data);
deflater.finish();
byte[] buffer = new byte[1024];
while (!deflater.finished()) {
int count = deflater.deflate(buffer); // returns the generated code... index
out.write(buffer, 0, count);
}
}
}
public static void decompress() throws IOException, DataFormatException {
Inflater inflater = new Inflater();
byte[] data = new byte[1024];
FileInputStream in = new FileInputStream(new File(deflateOutput));
FileOutputStream out = new FileOutputStream(new File(inflateOutput));
long readBytesCount = 0;
long readCompressedBytesCount = 0;
long readBytes = 0;
while ((readBytes = in.read(data, 0, 1024)) != -1) {
readBytesCount = readBytesCount + readBytes;
inflater.setInput(data);
byte[] buffer = new byte[1024];
while (!inflater.finished()) {
int count = inflater.inflate(buffer);
System.out.println("Remaining: " + inflater.getRemaining());
out.write(buffer, 0, count);
}
}
System.out.println("readBytesCount: " + readBytesCount);
}
public static void main(String[] args) {
System.out.println("Operation started");
try {
compress();
} catch (Exception e) {
e.printStackTrace();
}
System.out.println("Operation ended");
}
}
And this is output (in windows) of dir:
01-04-2018 16:52 220,173 pic.jpg
28-04-2018 20:50 943 picDeflate.raw
28-04-2018 20:28 1,024 picInflate.jpg
Why does the compress code stops after reading 1024 bytes?
finish() is only for when you're finished. It is the last thing called after you have provided all of the input data to the object.
I have a task that
read a zip file from local into binary message
transfer binary message through EMS as String (done by java API)
receive transferred binary message as String (done by java API)
decompress the binary message and then print it out
The problem I am facing is DataFormatException while decompress the message.
I have no idea which part went wrong.
I use this to read file into binary message:
static String readFile_Stream(String fileName) throws IOException {
File file = new File(fileName);
byte[] fileData = new byte[(int) file.length()];
FileInputStream in = new FileInputStream(file);
in.read(fileData);
String content = "";
System.out.print("Sent message: ");
for(byte b : fileData)
{
System.out.print(getBits(b));
content += getBits(b);
}
in.close();
return content;
}
static String getBits(byte b)
{
String result = "";
for(int i = 0; i < 8; i++)
result = ((b & (1 << i)) == 0 ? "0" : "1") + result;
return result;
}
I use this to decompress message:
private static byte[] toByteArray(String input)
{
byte[] byteArray = new byte[input.length()/8];
for (int i=0;i<input.length()/8;i++)
{
String read_data = input.substring(i*8, i*8+8);
short a = Short.parseShort(read_data, 2);
byteArray[i] = (byte) a;
}
return byteArray;
}
public static byte[] unzipByteArray(byte[] file) throws IOException {
byte[] byReturn = null;
Inflater oInflate = new Inflater(false);
oInflate.setInput(file);
ByteArrayOutputStream oZipStream = new ByteArrayOutputStream();
try {
while (! oInflate.finished() ){
byte[] byRead = new byte[4 * 1024];
int iBytesRead = oInflate.inflate(byRead);
if (iBytesRead == byRead.length){
oZipStream.write(byRead);
}
else {
oZipStream.write(byRead, 0, iBytesRead);
}
}
byReturn = oZipStream.toByteArray();
}
catch (DataFormatException ex){
throw new IOException("Attempting to unzip file that is not zipped.");
}
finally {
oZipStream.close();
}
return byReturn;
}
The message I got is
java.io.IOException: Attempting to unzip file that is not zipped.
at com.sourcefreak.example.test.TibcoEMSQueueReceiver.unzipByteArray(TibcoEMSQueueReceiver.java:144)
at com.sourcefreak.example.test.TibcoEMSQueueReceiver.main(TibcoEMSQueueReceiver.java:54)
After check, the binary message does not corrupted after transmission.
Please help to figure out the problem.
Have you tried using InflaterInputStream? Based on my experience, using Inflater directly is rather tricky. You can use this to get started:
public static byte[] unzipByteArray(byte[] file) throws IOException {
InflaterInputStream iis = new InflaterInputStream(new ByteArrayInputStream(file));
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte[] buffer = new byte[512];
int length = 0;
while ((length = iis.read(buffer, 0, buffer.length) != 0) {
baos.write(buffer, 0, length);
}
iis.close();
baos.close();
return baos.toByteArray();
}
I finally figure out the problem.
The problem is the original file is a .zip file, so I should use zipInputStream to unzip the file before further processing.
public static byte[] unzipByteArray(byte[] file) throws IOException {
// create a buffer to improve copy performance later.
byte[] buffer = new byte[2048];
byte[] content ;
// open the zip file stream
InputStream theFile = new ByteArrayInputStream(file);
ZipInputStream stream = new ZipInputStream(theFile);
ByteArrayOutputStream output = new ByteArrayOutputStream();
try
{
ZipEntry entry;
while((entry = stream.getNextEntry())!=null)
{
//String s = String.format("Entry: %s len %d added %TD", entry.getName(), entry.getSize(), new Date(entry.getTime()));
//System.out.println(s);
// Once we get the entry from the stream, the stream is
// positioned read to read the raw data, and we keep
// reading until read returns 0 or less.
//String outpath = outdir + "/" + entry.getName();
try
{
//output = new FileOutputStream(outpath);
int len = 0;
while ((len = stream.read(buffer)) > 0)
{
output.write(buffer, 0, len);
}
}
finally
{
// we must always close the output file
if(output!=null) output.close();
}
}
}
finally
{
// we must always close the zip file.
stream.close();
}
content = output.toByteArray();
return content;
}
This code work for zip file containing single file inside.
I know that there's a way of converting a file to byte array in chunks, here's a sample code:
InputStream inputStream = new FileInputStream(videoFile);
ByteArrayOutputStream bos = new ByteArrayOutputStream();
byte[] b = new byte[1024];
int bytesRead =0;
while ((bytesRead = inputStream.read(b)) != -1)
{
bos.write(b, 0, bytesRead);
}
I'm looking for the opposite: a way of converting a byte array into a file in chunks. I didn't find any example of doing it in chunks.
You just have to use either the write(byte[]) or write(byte[],int,int) methods from the FileOutputStream class.
byte[] to file:
FileOutputStream fop = null; File file;
try {
file = new File(filePath);
fop = new FileOutputStream(file, true);
fop.write(chunk);
fop.flush();
fop.close();
System.out.println("Done");
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
if (fop != null) {
fop.close();
}
} catch (IOException e) {
e.printStackTrace();
}
}
Try this for file to byte[]:
InputStream is = new FileInputStream(file);
int length = (int) file.length();
int take = 262144;//size of your chunk
byte[] bytes = new byte[take];
int offset=0;
int a = 0;
do {
a = is.read(bytes, 0, take);
offset += a;
//And you can add here each chunk created in to a list, etc, etc.
//encode to base 64 this is extra :)
String str = Base64.encodeToString(bytes, Base64.DEFAULT);
} while (offset < length);=
is.close();
is=null;
Consider generalizing the problem.
This method copies data in chunks:
public static <T extends OutputStream> T copy(InputStream in, T out)
throws IOException {
byte[] buffer = new byte[1024];
for (int r = in.read(buffer); r != -1; r = in.read(buffer)) {
out.write(buffer, 0, r);
}
return out;
}
This can then be used in both reading to and from byte arrays:
try (InputStream in = new FileInputStream("original.txt");
OutputStream out = new FileOutputStream("copy.txt")) {
byte[] contents = copy(in, new ByteArrayOutputStream()).toByteArray();
copy(new ByteArrayInputStream(contents), out);
}