We Keep Coding

How to write array of bytes to a binary file in Java

How to write an array of bytes b[i] to a binary file in Java.
I need to write those bytes it into a "binary file" to be able to read it later using hex editor (AXE).
Some readers might be confused by "binary file", by binary file I don't mean a file filled by zeros and ones, I mean machine-readable form, something like this :
binary files in text editor
The hex editor suppose to read this data, hex editor
From what I understand I need to byte stream that data into a file
Is there a command I could use for this purpose.
Any code would be appreciated.

Just write the byte[] to a FileOutputStream pointing to the file:
private static void writeBytesToFile(byte[] b, String f) {
try (FileOutputStream out = new FileOutputStream(f)){
out.write(b);
}
catch (IOException e) {
e.printStackTrace();
}
}

How to unpack a binary file in java?

May somebody help me to know how can I do in java what I do in ruby with the code below.
The ruby code below uses unpack('H*')[0] to stores the complete binary file content in variable "var" in ASCII format.
IO.foreach(ARGV[0]){ |l|
var = l.unpack('H*')[0]
} if File.exists?(ARGV[0])
Update:
Hi Aru. I've tested the way you say in the form below
byte[] bytes = Files.readAllBytes(testFile.toPath());
str = new String(bytes,StandardCharsets.UTF_8);
System.out.println(str);
But when I print the content of variable "str", the printout shows only little squares, like is not decoding the content. I'd like to store in "str" the content of binary file in ASCII format.
Update #2:
Hello Aru, I'm trying to store in array of bytes all the binary file's content but I don't know how to do it. It worked
with "FileUtils.readFileToByteArray(myFile);" but this is an external library, is there a built in option to do it?
File myFile = new File("./Binaryfile");
byte[] binary = FileUtils.readFileToByteArray(myFile); //I have issues here to store in array of bytes all binary content
String hexString = DatatypeConverter.printHexBinary(binary);
System.out.println(hexString);
Update #3:
Hello ursa and Aru, Thanks for your help. I've tried both of your solutions and works so fine, but seeing Files.readAllBytes() documentation
it says that is not intended to handle big files and the binary file I want to analyse is more than 2GB :(. I see an option with your solutions, read
chunk by chunk. The chunks inside the binary are separated by the sequence FF65, so is there a way to tweak your codes to only process one chunk at a
time based on the chunk separator? If not, maybe with some external library.
Update #4:
Hello, I'm trying to modify your code since I'd like to read variable size chunks based of
value of "Var".
How can I set an offset to read the next chunk in your code?
I mean,
- in first iteration read the first 1024,
- In this step Var=500
- in 2d iteration read the next 1024 bytes, beginning from 1024 - Var = 1024-500 = 524
- In this step Var=712
- in 3rd iteration read the next 1024 bytes, beginning from 1548 - Var = 1548-712 = 836
- and so on
is there a method something like read(number of bytes, offset)?

You can use commons-codec Hex class + commons-io FileUtils class:
byte[] binary = FileUtils.readFileToByteArray(new File("/Users/user/file.bin");
String hexEncoded = Hex.encodeHex(binary);
But if you just want to read content of TEXT file you can use:
String content = FileUtils.readFileToString(new File("/Users/user/file.txt", "ISO-8859-1");
With JRE 7 you can use standard classes:
public static void main(String[] args) throws Exception {
Path path = Paths.get("path/to/file");
byte[] data = Files.readAllBytes(path);
char[] hexArray = "0123456789ABCDEF".toCharArray();
char[] hexChars = new char[data.length * 2];
for ( int j = 0; j < data.length; j++ ) {
int v = data[j] & 0xFF;
hexChars[j * 2] = hexArray[v >>> 4];
hexChars[j * 2 + 1] = hexArray[v & 0x0F];
}
System.out.println(new String(hexChars));
}

This should do what you want:
try {
File inputFile = new File("someFile");
byte inputBytes[] = Files.readAllBytes(inputFile.toPath());
String hexCode = DatatypeConverter.printHexBinary(inputBytes);
System.out.println(hexCode);
} catch (IOException e) {
System.err.println("Couldn't read file: " + e);
}
If you don't want to read the entire file at once, you can do so as well. You'll need an InputStream of some sort.
File inputFile = new File("C:\\Windows\\explorer.exe");
try (InputStream input = new FileInputStream(inputFile)) {
byte inputBytes[] = new byte[1024];
int readBytes;
// Read until all bytes were read
while ((readBytes = input.read(inputBytes)) != -1) {
System.out.printf("%4d bytes were read.\n", readBytes);
System.out.println(DatatypeConverter.printHexBinary(inputBytes));
}
} catch (FileNotFoundException ex) {
System.err.println("Couldn't read file: " + ex);
} catch (IOException ex) {
System.err.println("Error while reading file: " + ex);
}

Zlib compression is too big in size

I am completely new to java, I have decided to learn it by doing a small project in it. I need to compress some string using zlib and write it to a file. However, file turn out to be too big in size. Here is code example:
String input = "yasar\0yasar"; // test input. Input will have null character in it.
byte[] compressed = new byte[100]; // hold compressed content
Deflater compresser = new Deflater();
compresser.setInput(input.getBytes());
compresser.finish();
compresser.deflate(compressed);
File test_file = new File(System.getProperty("user.dir"), "test_file");
try {
if (!test_file.exists()) {
test_file.createNewFile();
}
try (FileOutputStream fos = new FileOutputStream(test_file)) {
fos.write(compressed);
}
} catch (IOException e) {
e.printStackTrace();
}
This write a 1 kilobytes file, while the file should be at most 11 bytes (because the content is 11 bytes here.). I think problem is in the way I initialize the byte array compressed as 100 bytes, but I don't know how big the compreesed data will be in advance. What am I doing wrong here? How can I fix it?

If you don't want to write the whole array and instead write just the part of it that was filled by Deflater use OutputStream#write(byte[] array, int offset, int lenght)
Roughly like
String input = "yasar\0yasar"; // test input. Input will have null character in it.
byte[] compressed = new byte[100]; // hold compressed content
Deflater compresser = new Deflater();
compresser.setInput(input.getBytes());
compresser.finish();
int length = compresser.deflate(compressed);
File test_file = new File(System.getProperty("user.dir"), "test_file");
try {
if (!test_file.exists()) {
test_file.createNewFile();
}
try (FileOutputStream fos = new FileOutputStream(test_file)) {
fos.write(compressed, 0, length); // starting at 0th byte - lenght(-1)
}
} catch (IOException e) {
e.printStackTrace();
}
You will probably still see 1kB or so in Windows because what you see there seems to be either rounded (you wrote 100 bytes before) or it refers to the size on the filesystem which is at least 1 block large (should be 4kb IIRC). Rightclick the file and check the size in the properties, that should show the actual size.
If you don't know the size in advance, don't use Deflater, use a DeflaterOutputStream that writes data of any length compressed.
try (OutputStream out = new DeflaterOutputStream(new FileOutputStream(test_file))) {
out.write("hello!".getBytes());
}
Above example will use the default values for deflating but you can pass a configured Deflater in the constructor of DeflaterOutputStream to change the behavior.

you write to file all 100 bytes of compressed array, but you have to write only really compressed bytes returned by deflater.
int compressedsize = compresser.deflate(compressed);
fos.write(compressed, 0, compressedsize);

Where to start in building an app/system that compresses and decompresses a file

I'm currently developing an application in Java/Android that allows the user to compress and decompress files. At first, I started to study the file size such as:
1Byte = 8Bits
1KB = 1024Byte
1MB = 1024KB
1GB = 1024MB
1TB = 1024GB
1PB = 1024TB
1EB = 1024PB
1ZB = 1024EB
1YB = 1024ZB
After I studied this, I studied and read some articles on the net and found out there are 2 types of file compression (Correct me if I'm wrong): Lossless and Lossy. Lossless compression means that a file is compressed into a smaller bit without losing any single file while lossy compression means that important files were being removed while compressing the file.
I also read that compression(run-length coding method) is like this:
AAABBCCDFFFFEEEEH
to this:
3A2B2CD4F4EH
which gives me an idea on how compressing/decompressing works on file.
I also searched the net that there is an API for compressing file on java(also applicable on android) which is
java.util.zip
I also tried some codes on compressing and decompressing file from various helpful websites/forum/etc (including stackoverflow.com) which gives me an experience to this study.
I also read about algorithms used in data compression which are
Huffman encoding algorithm - assigns a code to characters in a file based on how frequently those characters occur
run-length encoding - generates a two-part value for repeated characters: the first part specifies the number of times the character is repeated, and the second part identifies the character
Lempel-Ziv algorithm - converts variable-length strings into fixed-length codes that consume less space than the original strings.
Now, I need to know how to code an algo in compressing and decompressing file by using java.util.zip(I also don't know how to use it. tutorials on net is not working for me :/). What algo does winzip, winrar, compressed folder(windows), and androzip(android app) is using? Will someone please teach me step by step(treat me as an unschooled person) on how java.util.zip works and the different algorithms. sorry for the long post folks. Thanks for the future help and posts(if there will be)!

public static final byte[] unzip(byte[] in) throws IOException {
// decompress using GZIPInputStream
ByteArrayOutputStream outStream =
new ByteArrayOutputStream(EXPECTED_COMPRESSION_RATIO * in.length);
GZIPInputStream inStream =
new GZIPInputStream ( new ByteArrayInputStream(in) );
byte[] buf = new byte[BUF_SIZE];
while (true) {
int size = inStream.read(buf);
if (size <= 0)
break;
outStream.write(buf, 0, size);
}
outStream.close();
return outStream.toByteArray();
}
public static final byte[] zip(byte[] in) {
try {
// compress using GZIPOutputStream
ByteArrayOutputStream byteOut=
new ByteArrayOutputStream(in.length / EXPECTED_COMPRESSION_RATIO);
GZIPOutputStream outStream= new GZIPOutputStream(byteOut);
try {
outStream.write(in);
} catch (Exception e) {
LOG.error("", e);
}
try {
outStream.close();
} catch (IOException e) {
LOG.error("", e);
}
return byteOut.toByteArray();
} catch (IOException e) {
LOG.error("", e);
return null;
}
}

Counted bytes and total bytes differ

I'm writing an Android application which copies files from the assets to one file on the device's drive (no permission problems, bytes get from the assets to the drive). The file that I need to copy is larger than 1 MB, so I split it up into multiple files, and I copy them with something like:
try {
out = new FileOutputStream(destination);
for (InputStream file : files /* InputStreams from assets */) {
copyFile(file);
file.close();
}
out.close();
System.out.println(bytesCopied); // shows 8716288
System.out.println(new File(destination).length()); // shows 8749056
} catch (IOException e) {
Log.e("ERROR", "Cannot copy file.");
return;
}
Then, the copyFile() method:
private void copyFile(InputStream file) throws IOException {
byte[] buffer = new byte[16384];
int length;
while ((length = file.read(buffer)) > 0) {
out.write(buffer);
bytesCopied += length;
out.flush();
}
}
The correct number of total bytes that the destination file should contain is 8716288 (that's what I get when I look at the original files and if I count the written bytes in the Android application), but new File(destination).length() shows 8749056.
What am I doing wrong?

The file size becomes too large because you are not writing length bytes for each write, you are actually writing the whole buffer each time, which is buffer.length() bytes long.
You should use the write(byte[] b, int off, int len) overload instead, to specify how many bytes in the buffer you want to be written on each iteration.

Didn't you mean to write
out.write(buffer, 0, length);
instead of
out.write(buffer);
Otherwise you would always write the complete buffer, even if less bytes were read. This may then lead to a larger file (filled with some garbage between your original data).