Incorrect decompression of Java Gzip in .NET - java

I'm trying to repeat in .NET the algorithm that was originally written in Java and I'm having troubles with the GZIP decompression.
At the bottom of the post I inserted the hex string that is converted to byte array in both .NET and Java. The resulting byte array is then decompressed in Java with the following method:
public static Object readObjectFromByte(byte[] bytes)
{
ObjectInputStream oos = null;
try {
ByteArrayInputStream baos = new ByteArrayInputStream(bytes);
zis = new GZIPInputStream(baos);
oos = new ObjectInputStream(zis);
return oos.readObject();
} catch (Throwable t) { GZIPInputStream zis;
return null;
} finally {
try {
if (oos != null) {
oos.close();
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
After decompression the resulting byte array has a length of 3952 which is probably correct. At the same time I tried different .NET classes/libs to decompress, but it always gives a byte array of 3979 bytes which is probably incorrect.
I tried:
GZipStream
DotNetZip
SevenZipLib
SharpZipLib
I read a lot of articles about GZIP issues in .NET trying to fix this. I use .NET 4.5, and for example my last decompression version is this:
Ionic.Zlib.GZipStream.UncompressBuffer(compressedBytes)
It's weird but even if I try:
Ionic.Zlib.GZipStream.CompressBuffer(Ionic.Zlib.GZipStream.UncompressBuffer(compressedBytes)).SequenceEquals(compressedBytes)
It gives me FALSE.
The hex string:
EDIT:
Java Code:
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;
import java.util.zip.GZIPInputStream;
import java.util.zip.GZIPOutputStream;
public class JavaFiddle
{
public static void main(String[] args)
{
String hex = "PLEASE_UPDATE"; //update this from the hex constant at the end of the post
byte[] compressedBytes = hexStringToByteArray(hex);
byte[] decompressedBytes = (byte[])readObjectFromByte(compressedBytes);
System.out.println(decompressedBytes.length); //THIS GIVES 3952
}
public static byte[] hexStringToByteArray(String s) {
int len = s.length();
byte[] data = new byte[len / 2];
for (int i = 0; i < len; i += 2) {
data[i / 2] = (byte) ((Character.digit(s.charAt(i), 16) << 4)
+ Character.digit(s.charAt(i+1), 16));
}
return data;
}
public static Object readObjectFromByte(byte[] bytes)
{
ObjectInputStream oos = null;
try {
ByteArrayInputStream baos = new ByteArrayInputStream(bytes);
GZIPInputStream zis = new GZIPInputStream(baos);
oos = new ObjectInputStream(zis);
return oos.readObject();
} catch (Throwable t) { GZIPInputStream zis;
return null;
} finally {
try {
if (oos != null) {
oos.close();
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
.NET Code
private byte[] StringToByteArray(string hex)
{
int NumberChars = hex.Length;
byte[] bytes = new byte[NumberChars / 2];
for (int i = 0; i < NumberChars; i += 2)
bytes[i / 2] = Convert.ToByte(hex.Substring(i, 2), 16);
return bytes;
}
...
var hex = "PLEASE_UPDATE"; //update this from the hex constant at the end of the post
var compressedBytes = StringToByteArray(hex);
var decompressedBytes = Ionic.Zlib.GZipStream.UncompressBuffer(compressedBytes);
//decompressedBytes.Length is 3979, Note that this is using one of the external libraries, the same result is for built-in GZipStream in .NET
1F8B0800000000000000ADCEF923130A0000E0596C8A1CB90AA1D0E428916C4AE6889C61071E1D2A77B42C67D1731BF2DE6C662C5744C6DE7BC8DC4791DCB79E35315EC2C62437E1FDBE9FF7FD055FC53240282418007437AB5893DB060923594000201C030088639882D19C31C860FB7838A60DBBF039C3B41821FF606044C89D24EB4C1D6A627FBF8E978D3616796C2ACDD165ABB32B81A0C493A35F163889E74665A9341C7C36F06FA0F991201BEAAB6A47842C43A50DA6B8C000AD794840CF03E29A527D0C921FAE6D607C71AEF7C01DB34037A7F8CDD8CB1F61DB9C15A87DBBDD1F933FDF58312B054D7B5AF72FE75A0AA101714EEA6980BF405A15A1F13A0979FB5BBE1636A367CB4C943FB7F87A3245573BB6857DA6AA0B412CCB553B39E06973EB2A3864E98915E46C16E4EE06E9DA63DB642915AF6E86811FD2C8F3A1F325C82677EA701D1E3574885696DEB3B8A0C29AE1FDD41DA26C55C6381506A3CF77DD15BD37783F545F76E8DCF9FB6A8B0776E2427FBFF431B73748CE96786EA569E307B5DC8990A68A52838D9DC1BA19291F283BF2C7EE43D3E4516D77A07FDE04934BB7C254057BDFC3AD153A0A711E7EFDDDED31AED367FAAEA6F494A55E029BF6AECA9433CA7D45B52EC24BCF9074315BFE5F5494F4F0DDDE8D73F56397A0667992950E62AD4020A4102DBE7C4D6B85E457F184DC5AA51D2D536313E1E16636928BD6500207571B83C8830DB9E7CF2C92AE4DDB12B8DF3C6AAF83B09CAACB78AA78CDB0DB393D71A599F185B413BFDB907FC99598D2E56ADE701EE72DB71C9650E820DB452F7DD4904E4679FFD0CB6FEEDF7527AF21BE5A5404B8BAE4E9951AE5C88BF9B5D5FFC6447989445C7A2DB80B3E4A2DBCB51978BA1729F97BDCAD174D345417B96941BF2CC5C10539C952B2526770BA4637A5820DCF150E16AED3159AD9724A8DFCFEA8BADA2A08BD1A0C9B05B019CE61E01EDECF7F028B466E4B53A5F1FAE56CB137EF269E1946896D1A3DDCDB756364A96D8E47F3FB33E9B70BBED37FA8719F2D2EA89BBFB0CAFB71CC56947F7907439E9FE97C61D2A326F830592509674860BA7CC7E713EFD69D2724DED740A2959846CFE801B313BB1DAB45594D70E888F2A3063C67085BFDB345D64D116372F5BE42BB08ABBC6D65D90E1E50C849EDD9A6F977751C4DB5A454C66B997D6A0CA148827E03AEE30F1697392124A8C44C6A914C3C69C21447F9C5660665C5D0D3F758D57086D0938608E9BECE9BA95D359C41E255B0524C55EDC97816A76E32B29BF55BAE0EC1F71DBF3FF548DC2C181755D85ABCAFEDF23C248EF7D3C1580AB88D28CDCF9935F1DA797145A7F0B18603C1DA39EA8E969A2279F20F077E7F62BD5C1F7C095BF1B21D36561A254836F37EBC0AEB8C8EAE8EC725044EE47F107DE413DE804EED41473E4D3BEC93E9234255F9FDD1FD91F00B32BF13B2629C4373888813E5FD08F66AB7295D281E180ACE99A77952DE6DCD675B20E021A71E91F78B899A1FC2F8FD913C0E4D980B1D6CF13F7A76E823BDF75FDE8FC513C3C491F51305F1A483B81A11884254CB2DB396F864B08DDEDD005A90EF457E7F8EDFB64FF7C7443C6E2135183619ED69F27E1464BF862FB14A66509C379818517F6B2423D0D1BE794E4D9C7130D9F54AB19DDF9FC825CF8B3B5EBD8090811D372C0315CEFBB926701AC2B84CB4820D25DD2D71F2F6630FB3FA88A3EAAF9E44C709EFACC05FF2FBD342E844AB8A22F433B7C2161FEDE70BF07EF29A6B6355DAF2D33E7A2679C8A38511939EEE73DABDE98B021A9F0E21A5CFFEE6F7A7C44CE2358158F5B15F837CE96B75622FEFE702653D87222B9319D46D321EEDA628713D1B355A30E0D07A655F12A55B6397CFEFCFA3DB3B76DB80347FDCA613FD1F3F6022EF276A1666DDA3BE518CB634F67B185BD2016DB2DC20E56C1C9B58CD411710E802FCFED0E3E1E182C37A9684A007EF0DEBBBCB783F37A0D39E2B6FAB147A3BCD75B1C24FA5939BDD9501D5E0365A60E58BE429C821BF3F774DC4603D43CE355412CC3DBF47639DF7B33CFEF4138133255770F2659E44F2FC9EE285FCDCEC414B473CCB31BA39BA798CDF1FF28A3EBD7549341C8E809E8AABF836C7FB59DB4025ED1DC5BF1874EDF4511ECCC397050C8C3F404FB04BF52DF4F35411EBFCFEFCB70C2AF6EE8AFC32A539F624F0E30D47DE8F9138B0C9B24F03F5CD53E05CA77D63CAA8800CA567E4D60D8D4ECBA9F9FE3C75FADED32267AB07ED635AD14CB6A87B1F8D5B5BD43F7C04E14DB30935A6B6CA780F2FBD10CD2EAE8DD13D6C9FA89334A8ADEAC40D63C54B6E881F9F382D3056BF7DACFFC6914A6DF60F64E7C5FE8EB222FD71632318CDC5E3CF8EC48602D238E95FC06C9BDBBD3D85535DED236D1FFA4EB1F3939B334A222BBF7FE034EB1599A7E20DFF89F92E9B3EE0D45FA7103A9B8BE6F727EC6D338C464AAA1B6B8AF426BA2153793FF7C43EEE7A43E6038296B83E130BAA9D467DB7A9B8E29E18B8F4E4493DB3E1BE4CDC1ABA3074A8D69058420FB2F56BCCD3CAB0806FA9C1E825D6F18644CD9137AB9195DD63A8BD3BA9F5C9BF0AEC9F5B6E605B2C02CAA93FB10A5F03ADD7370A981F3A99F0F3CC2DA93C298292E9859C01BBB088FE6CD979FC5A02127305D63C72B319EDD48A0EE6BE431C17B3F8E4FFDAC1F5D9A9F4EB7443EE2D3CAD6A3508A918C48DBB7F89081B7087C94A60F9FD297839305DA999BE163F234E3ED7D8D8CDFBA16AE5327F3E9D5925DF533EF5FEA2F26D3784CFB578877E15799DB73DAEC8D767F9FD81597235251D2DC413C2E1F58887AF75783F8248EC923F57F93D81A88D9E3E1E045997FD3C40B1BF83CEFC9CC2CCC079197928232333623BE67DA6A70B4818095DEEC23ABEC73E4DC9E1A74E04AEC1D420EB533CF3C06C242E1612F85A560A2861A0E54C218AC1834FCCDBD2F61FBC3B32983F25B29F5967A7219C74A10C769002C1869E9DC926A99C9D1AC92CC8BC3A5D151D405693BA0DBE292D94F40B918AFCEE6851DFC215B3738975F1EFAA32923037D96656C2D2B94B015CD1DA5F43D0A8FFA063099BA0BFB246724C8E61150FB2126EDE245963F4454EBCFF5105BDA267D598E2D1E9985136AF4348D8FCF96F355B27AE6E6F5378415D8A8D1CAD3EC3426969BF5790DBA0C01EF9B7E86095D9313F74309E01C7BA00B9B0E85DA89900D6F89DEA5BCA9273C017B56139F9CF13E7DB74724240262CBDFCEC05FB5DDAA64CB98257C9AAF7AA48AAD36577B3D6F11CD0A074D5B15860B2791E8E8B87AB2A5B2419F61561E6ECF56746E2185EBFFF0F96F821B38B0F0000
Thanks,

Now we've got more of the Java code, we can see the problem: you've got an extra layer of serialization around your real data. That has nothing to do with compression really.
Here's an example to show what I mean:
import java.io.*;
public class Test {
public static void main(String[] args) throws Exception {
try (ByteArrayOutputStream output = new ByteArrayOutputStream()) {
try (ObjectOutputStream oos = new ObjectOutputStream(output)) {
oos.writeObject(new byte[5]);
}
byte[] data = output.toByteArray();
System.out.println(data.length);
}
}
}
That's writing a byte array that's 10 bytes long - but the result is 32 bytes long, because of the extra "wrapper" information. Note that the extra 27 bytes is the same as the discrepancy you've seen.
Fundamentally, it's odd to wrap a byte array in this way, and if you can possibly change the original code, that would be for the best. If you absolutely can't do that, it may be safe to just ignore the first 27 bytes of the resulting data.

Related

Why does C# and Java application calculate different MD5 values for the same file?

I have a C# project to verify a file's MD5. I use System.Security.Cryptography.MD5 to calculate the MD5 in C#.
But it is different from the MD5 in Java.
EDIT: I have found the c# code is correct one. Thanks to Andy. May I know how to correct the Java code?
C# code:
public static String ComputeMD5(String fileName)
{
String hashMD5 = String.Empty;
if (System.IO.File.Exists(fileName))
{
using (System.IO.FileStream fs = new System.IO.FileStream(fileName, System.IO.FileMode.Open, System.IO.FileAccess.Read))
{
System.Security.Cryptography.MD5 calculator = System.Security.Cryptography.MD5.Create();
Byte[] buffer = calculator.ComputeHash(fs);
calculator.Clear();
StringBuilder stringBuilder = new StringBuilder();
for (int i = 0; i < buffer.Length; i++){
stringBuilder.Append(buffer[i].ToString("x2"));
}
hashMD5 = stringBuilder.ToString();
}
}
return hashMD5;
}
Java Code:
public static String ComputeMD5(File file) {
if (!file.isFile()) {
return null;
}
MessageDigest digest = null;
FileInputStream in = null;
byte buffer[] = new byte[1024];
int len;
try {
digest = MessageDigest.getInstance("MD5");
in = new FileInputStream(file);
while ((len = in.read(buffer, 0, 1024)) != -1) {
digest.update(buffer, 0, len);
}
in.close();
} catch (Exception e) {
e.printStackTrace();
return null;
}
return bytesToHexString(digest.digest());
}
Your bytesToHexString function is wrong. After using the function from here, in this complete example, I get the same result as the Linux md5sum command and`onlinemd5.com as Andy suggested. The best way to handle this is to use a library, such as Apache Commons, that has a function to convert from bytes to a hex string. That way, you offload the work to get it right to a reputable library.
import java.io.File;
import java.io.FileInputStream;
import java.security.MessageDigest;
public class Md5{
public static String computeMd5(File file) {
if (!file.isFile()) {
return null;
}
MessageDigest digest = null;
FileInputStream in = null;
byte buffer[] = new byte[1024];
int len;
try {
digest = MessageDigest.getInstance("MD5");
in = new FileInputStream(file);
while ((len = in.read(buffer, 0, 1024)) != -1) {
digest.update(buffer, 0, len);
}
in.close();
} catch (Exception e) {
e.printStackTrace();
return null;
}
return byteArrayToHex(digest.digest());
}
public static String byteArrayToHex(byte[] a) {
StringBuilder sb = new StringBuilder(a.length * 2);
for(byte b: a)
sb.append(String.format("%02x", b));
return sb.toString();
}
public static void main(String[] args){
System.out.println(computeMd5(new File("./text.txt")));
}
}

How to read a file using ByteArrayInputStream in java? [duplicate]

How do I convert a java.io.File to a byte[]?
From JDK 7 you can use Files.readAllBytes(Path).
Example:
import java.io.File;
import java.nio.file.Files;
File file;
// ...(file is initialised)...
byte[] fileContent = Files.readAllBytes(file.toPath());
It depends on what best means for you. Productivity wise, don't reinvent the wheel and use Apache Commons. Which is here FileUtils.readFileToByteArray(File input).
Since JDK 7 - one liner:
byte[] array = Files.readAllBytes(Paths.get("/path/to/file"));
No external dependencies needed.
import java.io.RandomAccessFile;
RandomAccessFile f = new RandomAccessFile(fileName, "r");
byte[] b = new byte[(int)f.length()];
f.readFully(b);
Documentation for Java 8: http://docs.oracle.com/javase/8/docs/api/java/io/RandomAccessFile.html
Basically you have to read it in memory. Open the file, allocate the array, and read the contents from the file into the array.
The simplest way is something similar to this:
public byte[] read(File file) throws IOException, FileTooBigException {
if (file.length() > MAX_FILE_SIZE) {
throw new FileTooBigException(file);
}
ByteArrayOutputStream ous = null;
InputStream ios = null;
try {
byte[] buffer = new byte[4096];
ous = new ByteArrayOutputStream();
ios = new FileInputStream(file);
int read = 0;
while ((read = ios.read(buffer)) != -1) {
ous.write(buffer, 0, read);
}
}finally {
try {
if (ous != null)
ous.close();
} catch (IOException e) {
}
try {
if (ios != null)
ios.close();
} catch (IOException e) {
}
}
return ous.toByteArray();
}
This has some unnecessary copying of the file content (actually the data is copied three times: from file to buffer, from buffer to ByteArrayOutputStream, from ByteArrayOutputStream to the actual resulting array).
You also need to make sure you read in memory only files up to a certain size (this is usually application dependent) :-).
You also need to treat the IOException outside the function.
Another way is this:
public byte[] read(File file) throws IOException, FileTooBigException {
if (file.length() > MAX_FILE_SIZE) {
throw new FileTooBigException(file);
}
byte[] buffer = new byte[(int) file.length()];
InputStream ios = null;
try {
ios = new FileInputStream(file);
if (ios.read(buffer) == -1) {
throw new IOException(
"EOF reached while trying to read the whole file");
}
} finally {
try {
if (ios != null)
ios.close();
} catch (IOException e) {
}
}
return buffer;
}
This has no unnecessary copying.
FileTooBigException is a custom application exception.
The MAX_FILE_SIZE constant is an application parameters.
For big files you should probably think a stream processing algorithm or use memory mapping (see java.nio).
As someone said, Apache Commons File Utils might have what you are looking for
public static byte[] readFileToByteArray(File file) throws IOException
Example use (Program.java):
import org.apache.commons.io.FileUtils;
public class Program {
public static void main(String[] args) throws IOException {
File file = new File(args[0]); // assume args[0] is the path to file
byte[] data = FileUtils.readFileToByteArray(file);
...
}
}
If you don't have Java 8, and agree with me that including a massive library to avoid writing a few lines of code is a bad idea:
public static byte[] readBytes(InputStream inputStream) throws IOException {
byte[] b = new byte[1024];
ByteArrayOutputStream os = new ByteArrayOutputStream();
int c;
while ((c = inputStream.read(b)) != -1) {
os.write(b, 0, c);
}
return os.toByteArray();
}
Caller is responsible for closing the stream.
// Returns the contents of the file in a byte array.
public static byte[] getBytesFromFile(File file) throws IOException {
// Get the size of the file
long length = file.length();
// You cannot create an array using a long type.
// It needs to be an int type.
// Before converting to an int type, check
// to ensure that file is not larger than Integer.MAX_VALUE.
if (length > Integer.MAX_VALUE) {
// File is too large
throw new IOException("File is too large!");
}
// Create the byte array to hold the data
byte[] bytes = new byte[(int)length];
// Read in the bytes
int offset = 0;
int numRead = 0;
InputStream is = new FileInputStream(file);
try {
while (offset < bytes.length
&& (numRead=is.read(bytes, offset, bytes.length-offset)) >= 0) {
offset += numRead;
}
} finally {
is.close();
}
// Ensure all the bytes have been read in
if (offset < bytes.length) {
throw new IOException("Could not completely read file "+file.getName());
}
return bytes;
}
You can use the NIO api as well to do it. I could do this with this code as long as the total file size (in bytes) would fit in an int.
File f = new File("c:\\wscp.script");
FileInputStream fin = null;
FileChannel ch = null;
try {
fin = new FileInputStream(f);
ch = fin.getChannel();
int size = (int) ch.size();
MappedByteBuffer buf = ch.map(MapMode.READ_ONLY, 0, size);
byte[] bytes = new byte[size];
buf.get(bytes);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} finally {
try {
if (fin != null) {
fin.close();
}
if (ch != null) {
ch.close();
}
} catch (IOException e) {
e.printStackTrace();
}
}
I think its very fast since its using MappedByteBuffer.
Simple way to do it:
File fff = new File("/path/to/file");
FileInputStream fileInputStream = new FileInputStream(fff);
// int byteLength = fff.length();
// In android the result of file.length() is long
long byteLength = fff.length(); // byte count of the file-content
byte[] filecontent = new byte[(int) byteLength];
fileInputStream.read(filecontent, 0, (int) byteLength);
Simplest Way for reading bytes from file
import java.io.*;
class ReadBytesFromFile {
public static void main(String args[]) throws Exception {
// getBytes from anyWhere
// I'm getting byte array from File
File file = null;
FileInputStream fileStream = new FileInputStream(file = new File("ByteArrayInputStreamClass.java"));
// Instantiate array
byte[] arr = new byte[(int) file.length()];
// read All bytes of File stream
fileStream.read(arr, 0, arr.length);
for (int X : arr) {
System.out.print((char) X);
}
}
}
Guava has Files.toByteArray() to offer you. It has several advantages:
It covers the corner case where files report a length of 0 but still have content
It's highly optimized, you get a OutOfMemoryException if trying to read in a big file before even trying to load the file. (Through clever use of file.length())
You don't have to reinvent the wheel.
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Path;
File file = getYourFile();
Path path = file.toPath();
byte[] data = Files.readAllBytes(path);
Using the same approach as the community wiki answer, but cleaner and compiling out of the box (preferred approach if you don't want to import Apache Commons libs, e.g. on Android):
public static byte[] getFileBytes(File file) throws IOException {
ByteArrayOutputStream ous = null;
InputStream ios = null;
try {
byte[] buffer = new byte[4096];
ous = new ByteArrayOutputStream();
ios = new FileInputStream(file);
int read = 0;
while ((read = ios.read(buffer)) != -1)
ous.write(buffer, 0, read);
} finally {
try {
if (ous != null)
ous.close();
} catch (IOException e) {
// swallow, since not that important
}
try {
if (ios != null)
ios.close();
} catch (IOException e) {
// swallow, since not that important
}
}
return ous.toByteArray();
}
This is one of the simplest way
String pathFile = "/path/to/file";
byte[] bytes = Files.readAllBytes(Paths.get(pathFile ));
I belive this is the easiest way:
org.apache.commons.io.FileUtils.readFileToByteArray(file);
ReadFully Reads b.length bytes from this file into the byte array, starting at the current file pointer. This method reads repeatedly from the file until the requested number of bytes are read. This method blocks until the requested number of bytes are read, the end of the stream is detected, or an exception is thrown.
RandomAccessFile f = new RandomAccessFile(fileName, "r");
byte[] b = new byte[(int)f.length()];
f.readFully(b);
If you want to read bytes into a pre-allocated byte buffer, this answer may help.
Your first guess would probably be to use InputStream read(byte[]). However, this method has a flaw that makes it unreasonably hard to use: there is no guarantee that the array will actually be completely filled, even if no EOF is encountered.
Instead, take a look at DataInputStream readFully(byte[]). This is a wrapper for input streams, and does not have the above mentioned issue. Additionally, this method throws when EOF is encountered. Much nicer.
Not only does the following way convert a java.io.File to a byte[], I also found it to be the fastest way to read in a file, when testing many different Java file reading methods against each other:
java.nio.file.Files.readAllBytes()
import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
public class ReadFile_Files_ReadAllBytes {
public static void main(String [] pArgs) throws IOException {
String fileName = "c:\\temp\\sample-10KB.txt";
File file = new File(fileName);
byte [] fileBytes = Files.readAllBytes(file.toPath());
char singleChar;
for(byte b : fileBytes) {
singleChar = (char) b;
System.out.print(singleChar);
}
}
}
//The file that you wanna convert into byte[]
File file=new File("/storage/0CE2-EA3D/DCIM/Camera/VID_20190822_205931.mp4");
FileInputStream fileInputStream=new FileInputStream(file);
byte[] data=new byte[(int) file.length()];
BufferedInputStream bufferedInputStream=new BufferedInputStream(fileInputStream);
bufferedInputStream.read(data,0,data.length);
//Now the bytes of the file are contain in the "byte[] data"
Let me add another solution without using third-party libraries. It re-uses an exception handling pattern that was proposed by Scott (link). And I moved the ugly part into a separate message (I would hide in some FileUtils class ;) )
public void someMethod() {
final byte[] buffer = read(new File("test.txt"));
}
private byte[] read(final File file) {
if (file.isDirectory())
throw new RuntimeException("Unsupported operation, file "
+ file.getAbsolutePath() + " is a directory");
if (file.length() > Integer.MAX_VALUE)
throw new RuntimeException("Unsupported operation, file "
+ file.getAbsolutePath() + " is too big");
Throwable pending = null;
FileInputStream in = null;
final byte buffer[] = new byte[(int) file.length()];
try {
in = new FileInputStream(file);
in.read(buffer);
} catch (Exception e) {
pending = new RuntimeException("Exception occured on reading file "
+ file.getAbsolutePath(), e);
} finally {
if (in != null) {
try {
in.close();
} catch (Exception e) {
if (pending == null) {
pending = new RuntimeException(
"Exception occured on closing file"
+ file.getAbsolutePath(), e);
}
}
}
if (pending != null) {
throw new RuntimeException(pending);
}
}
return buffer;
}
public static byte[] readBytes(InputStream inputStream) throws IOException {
byte[] buffer = new byte[32 * 1024];
int bufferSize = 0;
for (;;) {
int read = inputStream.read(buffer, bufferSize, buffer.length - bufferSize);
if (read == -1) {
return Arrays.copyOf(buffer, bufferSize);
}
bufferSize += read;
if (bufferSize == buffer.length) {
buffer = Arrays.copyOf(buffer, bufferSize * 2);
}
}
}
Another Way for reading bytes from file
Reader reader = null;
try {
reader = new FileReader(file);
char buf[] = new char[8192];
int len;
StringBuilder s = new StringBuilder();
while ((len = reader.read(buf)) >= 0) {
s.append(buf, 0, len);
byte[] byteArray = s.toString().getBytes();
}
} catch(FileNotFoundException ex) {
} catch(IOException e) {
}
finally {
if (reader != null) {
reader.close();
}
}
Try this :
import sun.misc.IOUtils;
import java.io.IOException;
try {
String path="";
InputStream inputStream=new FileInputStream(path);
byte[] data=IOUtils.readFully(inputStream,-1,false);
}
catch (IOException e) {
System.out.println(e);
}
Can be done as simple as this (Kotlin version)
val byteArray = File(path).inputStream().readBytes()
EDIT:
I've read docs of readBytes method. It says:
Reads this stream completely into a byte array.
Note: It is the caller's responsibility to close this stream.
So to be able to close the stream, while keeping everything clean, use the following code:
val byteArray = File(path).inputStream().use { it.readBytes() }
Thanks to #user2768856 for pointing this out.
try this if you have target version less than 26 API
private static byte[] readFileToBytes(String filePath) {
File file = new File(filePath);
byte[] bytes = new byte[(int) file.length()];
// funny, if can use Java 7, please uses Files.readAllBytes(path)
try(FileInputStream fis = new FileInputStream(file)){
fis.read(bytes);
return bytes;
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
return null;
}
In JDK8
Stream<String> lines = Files.lines(path);
String data = lines.collect(Collectors.joining("\n"));
lines.close();

How to convert binary text into useable file

So I use the following methods
(File is converted to Byte Array through 'convertFileToByteArray()', then written to .txt file by 'convertByteArrayToBitTextFile()'
to convert any kind of file into a Binary Text file (and by that I mean only 1's and 0's in human readable form.)
public static byte[] convertFileToByteArray(String path) throws IOException
{
File file = new File(path);
byte[] fileData;
fileData = new byte[(int)file.length()];
FileInputStream in = new FileInputStream(file);
in.read(fileData);
in.close();
return fileData;
}
public static boolean convertByteArrayToBitTextFile(String path, byte[] bytes)
{
String content = convertByteArrayToBitString(bytes);
try
{
PrintWriter out = new PrintWriter(path);
out.println(content);
out.close();
return true;
}
catch (FileNotFoundException e)
{
return false;
}
}
public static String convertByteArrayToBitString(byte[] bytes)
{
String content = "";
for (int i = 0; i < bytes.length; i++)
{
content += String.format("%8s", Integer.toBinaryString(bytes[i] & 0xFF)).replace(' ', '0');
}
return content;
}
Edit: Additional Code:
public static byte[] convertFileToByteArray(String path) throws IOException
{
File file = new File(path);
byte[] fileData;
fileData = new byte[(int)file.length()];
FileInputStream in = new FileInputStream(file);
in.read(fileData);
in.close();
return fileData;
}
public static boolean convertByteArrayToBitTextFile(String path, byte[] bytes)
{
try
{
PrintWriter out = new PrintWriter(path);
for (int i = 0; i < bytes.length; i++)
{
out.print(String.format("%8s", Integer.toBinaryString(bytes[i] & 0xFF)).replace(' ', '0'));
}
out.close();
return true;
}
catch (FileNotFoundException e)
{
return false;
}
}
public static boolean convertByteArrayToByteTextFile(String path, byte[] bytes)
{
try
{
PrintWriter out = new PrintWriter(path);
for(int i = 0; i < bytes.length; i++)
{
out.print(bytes[i]);
}
out.close();
return true;
}
catch (FileNotFoundException e)
{
return false;
}
}
public static boolean convertByteArrayToRegularFile(String path, byte[] bytes)
{
try
{
PrintWriter out = new PrintWriter(path);
for(int i = 0; i < bytes.length; i++)
{
out.write(bytes[i]);
}
out.close();
return true;
}
catch (FileNotFoundException e)
{
return false;
}
}
public static boolean convertBitFileToByteTextFile(String path)
{
try
{
byte[] b = convertFileToByteArray(path);
convertByteArrayToByteTextFile(path, b);
return true;
}
catch (IOException e)
{
return false;
}
}
I do this to try methods of compression on a very fundamental level, so please let's not discuss why use human-readable form.
Now this works quite well so far, however I got two problems.
1)
It takes foreeeever (>20 Minutes for 230KB into binary text). Is this just a by-product of the relatively complicated conversion or are there other methods to do this faster?
2) and main problem:
I have no idea how to convert the files back to what they used to be. Renaming from .txt to .exe does not work (not too surprising as the resulting file is two times larger than the original)
Is this still possible or did I lose Information about what the file is supposed to represent by converting it to a human-readable text file?
If so, do you know any alternative that prevents this?
Any help is appreciated.
The thing that'll cost you most time is the construction of an ever increasing String. A better approach would be to write the data as soon as you have it.
The other problem is very easy. You know that every sequence of eight characters ('0' or '1') was made from a byte. Hence, you know the values of each character in an 8-character block:
01001010
^----- 0*1
^------ 1*2
^------- 0*4
^-------- 1*8
^--------- 0*16
^---------- 0*32
^----------- 1*64
^------------ 0*128
-----
64+8+2 = 74
You only need to add the values where an '1' is present.
You can do it in Java like this, without even knowing the individual bit values:
String sbyte = "01001010";
int bytevalue = 0;
for (i=0; i<8; i++) {
bytevalue *= 2; // shifts the bit pattern to the left 1 position
if (sbyte.charAt(i) == '1') bytevalue += 1;
}
Use StringBuilder to avoid generating enormous numbers of unused String instances.
Better yet, write directly to the PrintWriter instead of building it in-memory at all.
Loop through every 8-character subsequence and call Byte.parseByte(text, 2) to parse it back to a byte.

Java: Extracting bytes from audio file

What is the difference between the following two implementations in extracting the bytes of data from an audio file ?
The file is a .wav file and i want to extract only the data, without headers or any other thing.
Implementation 1:
public byte[] extractAudioFromFile(String filePath) {
try {
// Get an input stream on the byte array
// containing the data
File file = new File(filePath);
final AudioInputStream audioInputStream = AudioSystem
.getAudioInputStream(file);
byte[] buffer = new byte[4096];
int counter;
while ((counter = audioInputStream.read(buffer, 0, buffer.length)) != -1) {
if (counter > 0) {
byteOut.write(buffer, 0, counter);
}
}
audioInputStream.close();
byteOut.close();
} catch (Exception e) {
System.out.println(e);
System.exit(0);
}// end catch
return ((ByteArrayOutputStream) byteOut).toByteArray();
}
Implementation 2:
public byte[] readAudioFileData(String filePath) throws IOException,
UnsupportedAudioFileException {
final AudioInputStream audioInputStream = AudioSystem
.getAudioInputStream(new File(filePath));
AudioSystem.write(audioInputStream, AudioFileFormat.Type.WAVE, byteOut);
audioInputStream.close();
byteOut.close();
return ((ByteArrayOutputStream) byteOut).toByteArray();
}
Every implementation returns a different size of bytes.
The first one return byte[] with length less than second implementation.
I trying to extract the bytes of data to visualize the Spectrogram of the file.
Any explanation appreciated.
Thanks,
Samer
The 2nd impl is writing the full WAVE 'file format'. Is 2nd buffer 44 bytes larger than the first?
[edit: curious enough to actually try it - the above is correct]
package so_6933920;
import java.io.ByteArrayOutputStream;
import java.io.File;
import javax.sound.sampled.AudioFileFormat;
import javax.sound.sampled.AudioInputStream;
import javax.sound.sampled.AudioSystem;
public class AudioFiles {
public static void main(String[] args) {
String file = "clarinet.wav";
AudioFiles afiles = new AudioFiles();
byte[] data1 = afiles.readAudioFileData(file);
byte[] data2 = afiles.readWAVAudioFileData(file);
System.out.format("data len: %d\n", data1.length);
System.out.format("data len: %d\n", data2.length);
System.out.format("diff len: %d\n", data2.length - data1.length);
}
public byte[] readAudioFileData(final String filePath) {
byte[] data = null;
try {
final ByteArrayOutputStream baout = new ByteArrayOutputStream();
final File file = new File(filePath);
final AudioInputStream audioInputStream = AudioSystem
.getAudioInputStream(file);
byte[] buffer = new byte[4096];
int c;
while ((c = audioInputStream.read(buffer, 0, buffer.length)) != -1) {
baout.write(buffer, 0, c);
}
audioInputStream.close();
baout.close();
data = baout.toByteArray();
} catch (Exception e) {
e.printStackTrace();
}
return data;
}
public byte[] readWAVAudioFileData(final String filePath){
byte[] data = null;
try {
final ByteArrayOutputStream baout = new ByteArrayOutputStream();
final AudioInputStream audioInputStream = AudioSystem.getAudioInputStream(new File(filePath));
AudioSystem.write(audioInputStream, AudioFileFormat.Type.WAVE, baout);
audioInputStream.close();
baout.close();
data = baout.toByteArray();
} catch (Exception e) {
e.printStackTrace();
}
return data;
}
}
I tried this with this sample WAV file.
Results:
data len: 489708
data len: 489752
diff len: 44
Note: I took some liberties with your snippet to clean it up.
That System.exit(0) is a definite no-no.
if(counter > 0) isn't really necessary since counter must be greater than 0 if return value of the read method is not -1.

How to make read a PDF file as byte array?

After compile and run it shows "no pdf printer available", How to solve this?
I have created a file in c:\print.pdf (using PHP TCPDF). And i am
trying to read that file in byte array, so that i can silently print
it without showing any popup of print etc.
I cant make it working, can anyone please show guide how to read a
file in Byte array? To do the following:
import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.ObjectInputStream;
import java.util.logging.Level;
import java.util.logging.Logger;
import javax.print.Doc;
import javax.print.DocFlavor;
import javax.print.DocPrintJob;
import javax.print.PrintException;
import javax.print.PrintService;
import javax.print.PrintServiceLookup;
import javax.print.SimpleDoc;
public class print
{
private static Object pdfBytes;
// Byte array reader
public static byte[] getBytesFromFile(File file) throws IOException {
InputStream is = new FileInputStream(file);
long length = file.length();
if (length > Integer.MAX_VALUE) {}
byte[] bytes = new byte[(int)length];
int offset = 0;
int numRead = 0;
while (offset < bytes.length
&& (numRead=is.read(bytes, offset, bytes.length-offset)) >= 0) {
offset += numRead;
}
if (offset < bytes.length) {
throw new IOException("Could not completely read file "+file.getName());
}
is.close();
return bytes;
}
// Convert Byte array to Object
public static Object toObject(byte[] bytes)
{
Object obj = null;
try {
ByteArrayInputStream bis = new ByteArrayInputStream(bytes);
ObjectInputStream ois = new ObjectInputStream (bis);
obj = ois.readObject();
} catch (IOException ex) {
} catch (ClassNotFoundException ex) {
}
return obj;
}
private static File fl = new File("c:\\print.pdf");
public static void main(String argc[])
{
DocFlavor flavor = DocFlavor.BYTE_ARRAY.PDF;
PrintService[] services =
PrintServiceLookup.lookupPrintServices(flavor,
null);
//Object pdfBytes = null;
try {
byte[] abc = getBytesFromFile(fl);
pdfBytes =toObject(abc);
} catch (IOException ex) {
Logger.getLogger(print.class.getName()).log(Level.SEVERE, null, ex);
}
if (services.length>0)
{
DocPrintJob printJob = services[0].createPrintJob();
Doc document = new SimpleDoc(pdfBytes,flavor,null);
try {
printJob.print(document, null);
} catch (PrintException ex) {
Logger.getLogger(print.class.getName()).log(Level.SEVERE, null, ex);
}
} else {
System.out.println("no pdf printer available");
}
}
}
I tried this and it solves my silent printing: https://gist.github.com/1094612
Here's an example on how to read a file into a byte[]:
// Returns the contents of the file in a byte array.
public static byte[] getBytesFromFile(File file) throws IOException {
InputStream is = new FileInputStream(file);
// Get the size of the file
long length = file.length();
// You cannot create an array using a long type.
// It needs to be an int type.
// Before converting to an int type, check
// to ensure that file is not larger than Integer.MAX_VALUE.
if (length > Integer.MAX_VALUE) {
// File is too large
}
// Create the byte array to hold the data
byte[] bytes = new byte[(int)length];
// Read in the bytes
int offset = 0;
int numRead = 0;
while (offset < bytes.length
&& (numRead=is.read(bytes, offset, bytes.length-offset)) >= 0) {
offset += numRead;
}
// Ensure all the bytes have been read in
if (offset < bytes.length) {
throw new IOException("Could not completely read file "+file.getName());
}
// Close the input stream and return bytes
is.close();
return bytes;
}
import java.io.*;
import java.util.*;
import com.lowagie.text.*;
import com.lowagie.text.pdf.*;
public class ReadPDFFile {
public static void main(String[] args) throws IOException {
try {
Document document = new Document();
document.open();
PdfReader reader = new PdfReader("file.pdf");
PdfDictionary dictionary = reader.getPageN(1);
PRIndirectReference reference = (PRIndirectReference) dictionary
.get(PdfName.CONTENTS);
PRStream stream = (PRStream) PdfReader.getPdfObject(reference);
byte[] bytes = PdfReader.getStreamBytes(stream);
PRTokeniser tokenizer = new PRTokeniser(bytes);
StringBuffer buffer = new StringBuffer();
while (tokenizer.nextToken()) {
if (tokenizer.getTokenType() == PRTokeniser.TK_STRING) {
buffer.append(tokenizer.getStringValue());
}
}
String test = buffer.toString();
System.out.println(test);
} catch (Exception e) {
}
}
}
After compile and run it shows "no pdf printer available"
From my reading of the documentation here and here, the problem is you haven't configured print service provider that understands how to print documents with that DocFlavour.
One solution is to find a JAR file that implements the PrinterService SPI for PDF documents and add it to your classpath. A Google search will show examples. (I can't recommend any particular SP because I've never had to use one. You'll need to do some investigation / testing to find one that works for you.)

Categories