We Keep Coding

java.io.UTFDataFormatException while reading file entry name

Im trying to "pack" several files (previously inside a jar archive) in another single non-jar file by using DataInputStream / DataOutputStream.
The idea was:
First int = number of entries
First UTF is the first entry name
Second Int is entry byte array length (entry size)
Then repeat for every entry.
The code:
public static void main(String[] args) throws Throwable {
test();
System.out.println("========================================================================================");
final DataInputStream dataInputStream = new DataInputStream(new FileInputStream(new File("C:\\Users\\Admin\\Desktop\\randomJarOut")));
for (int int1 = dataInputStream.readInt(), i = 0; i < int1; ++i) {
final String utf = dataInputStream.readUTF();
System.out.println("Entry name: " + utf);
final byte[] array = new byte[dataInputStream.readInt()];
for (int j = 0; j < array.length; ++j) {
array[j] = dataInputStream.readByte();
}
System.out.println("Entry bytes length: " + array.length);
}
}
Unpacking original & packing to new one:
private static void test() throws Throwable {
JarInputStream stream = new JarInputStream(new FileInputStream(new File("C:\\Users\\Admin\\Desktop\\randomJar.jar")));
JarInputStream stream1 = new JarInputStream(new FileInputStream(new File("C:\\Users\\Admin\\Desktop\\randomJar.jar")));
final byte[] buffer = new byte[2048];
final DataOutputStream outputStream = new DataOutputStream(new FileOutputStream(new File("C:\\Users\\Admin\\Desktop\\randomJarOut")));
int entryCount = 0;
for (ZipEntry entry; (entry = stream.getNextJarEntry()) != null; ) {
entryCount++;
}
outputStream.writeInt(entryCount);
for (JarEntry entry; (entry = stream1.getNextJarEntry()) != null; ) {
int entryRealSize = stream1.read(buffer);
if (!(entryRealSize == -1)) {
System.out.println("Writing: " + entry.getName() + " Length: " + entryRealSize);
outputStream.writeUTF(entry.getName());
outputStream.writeInt(entryRealSize);
for (int len = stream1.read(buffer); len != -1; len = stream1.read(buffer)) {
outputStream.write(buffer, 0, len);
}
}
}
outputStream.flush();
outputStream.close();
}
Apparently im able to unpack the first entry without any problems, the second one and others:
Entry name: META-INF/services/org.jd.gui.spi.ContainerFactory
Entry bytes length: 434
Exception in thread "main" java.io.UTFDataFormatException: malformed input around byte 279
at java.io.DataInputStream.readUTF(DataInputStream.java:656)
at java.io.DataInputStream.readUTF(DataInputStream.java:564)
at it.princekin.esercizio.Bootstrap.main(Bootstrap.java:29)
Disconnected from the target VM, address: '127.0.0.1:54384', transport: 'socket'
Process finished with exit code 1
Does anyone knows how to fix this? Why is this working for the first entry but not the others?

My take on this is that the jar file (which in fact is a zip file) has a Central Directory which is only read with the ZipFile (or JarFile) class.
The Central Directory contains some data about the entries such as the size.
I think the ZipInputStream will not read the Central Directory and thus the ZipEntry will not contain the size (returning -1 as it is unknown) whereas reading ZipEntry from ZipFile class will.
So if you first read the size of each entry using a ZipFile and store that in a map, you can easily get it when reading the data with the ZipInputStream.
This page includes some good examples as well.
So my version of your code would be:
import java.io.*;
import java.util.HashMap;
import java.util.Map;
import java.util.zip.ZipEntry;
import java.util.zip.ZipFile;
import java.util.zip.ZipInputStream;
public class JarRepacker {
public static void main(String[] args) throws Throwable {
JarRepacker repacker = new JarRepacker();
repacker.repackJarToMyFileFormat("commons-cli-1.3.1.jar", "randomJarOut.bin");
repacker.readMyFileFormat("randomJarOut.bin");
}
private void repackJarToMyFileFormat(String inputJar, String outputFile) throws Throwable {
int entryCount;
Map<String, Integer> sizeMap = new HashMap<>();
try (ZipFile zipFile = new ZipFile(inputJar)) {
entryCount = zipFile.size();
zipFile.entries().asIterator().forEachRemaining(e -> sizeMap.put(e.getName(), (int) e.getSize()));
}
try (final DataOutputStream outputStream = new DataOutputStream(new FileOutputStream(outputFile))) {
outputStream.writeInt(entryCount);
try (ZipInputStream stream = new ZipInputStream(new BufferedInputStream(new FileInputStream(inputJar)))) {
ZipEntry entry;
final byte[] buffer = new byte[2048];
while ((entry = stream.getNextEntry()) != null) {
final String name = entry.getName();
outputStream.writeUTF(name);
final Integer size = sizeMap.get(name);
outputStream.writeInt(size);
//System.out.println("Writing: " + name + " Size: " + size);
int len;
while ((len = stream.read(buffer)) > 0) {
outputStream.write(buffer, 0, len);
}
}
}
outputStream.flush();
}
}
private void readMyFileFormat(String fileToRead) throws IOException {
try (DataInputStream dataInputStream
= new DataInputStream(new BufferedInputStream(new FileInputStream(fileToRead)))) {
int entries = dataInputStream.readInt();
System.out.println("Entries in file: " + entries);
for (int i = 1; i <= entries; i++) {
final String name = dataInputStream.readUTF();
final int size = dataInputStream.readInt();
System.out.printf("[%3d] Reading: %s of size: %d%n", i, name, size);
final byte[] array = new byte[size];
for (int j = 0; j < array.length; ++j) {
array[j] = dataInputStream.readByte();
}
// Still need to do something with this array...
}
}
}
}

The problem, probably, lies in that you are mixing not reciprocal read/write methods:
The writer method writes with outputStream.writeInt(entryCount) and the main method reads with dataInputStream.readInt(). That is OK.
The writer method writes with outputStream.writeUTF(entry.getName()) and the main method reads with dataInputStream.readUTF(). That is OK.
The writer method writes with outputStream.writeInt(entryRealSize) and the main method reads with dataInputStream.readInt(). That is OK.
The writer method writes with outputStream.write(buffer, 0, len) and the main method reads with dataInputStream.readByte() several times. WRONG.
If you write an array of bytes with write(buffer, offset, len), you must read it with read(buffer, offset, len), because write(buffer, offset, len) writes exactly len physical bytes onto the output stream, while writeByte (the counterpart of readByte) writes a lot of metadata overhead about the object type, and then its state variables.
Bugs in the writer method
There is also a mayor bug in the writer method: It invokes up to three times stream1.read(buffer), but it just uses once the buffer contents. The result is that the real size of file is actually written onto the output stream metadata, but it is followed by just a small part of the data.
If you need to know the input file size before writing it in the output stream, you have two choices:
Either chose a large enough buffer size (like 204800) which will allow you to read the whole file in just one read and write it in just one write.
Or either separate read from write algorithms: First a method to read the whole file and store it in memory (a byte[], for example), and then another method to write the byte[] onto the output stream.
Full fixed solution
I've fixed your program, with specific, decoupled methods for each task. The process consists in parsing the input file to a memory model, write it to an intermediate file according to your custom definition, and then read it back.
public static void main(String[] args)
throws Throwable
{
File inputJarFile=new File(args[0]);
File intermediateFile=new File(args[1]);
List<FileData> fileDataEntries=parse(inputJarFile);
write(fileDataEntries, intermediateFile);
read(intermediateFile);
}
public static List<FileData> parse(File inputJarFile)
throws IOException
{
List<FileData> list=new ArrayList<>();
try (JarInputStream stream=new JarInputStream(new FileInputStream(inputJarFile)))
{
for (ZipEntry entry; (entry=stream.getNextJarEntry()) != null;)
{
byte[] data=readAllBytes(stream);
if (data.length > 0)
{
list.add(new FileData(entry.getName(), data));
}
stream.closeEntry();
}
}
return list;
}
public static void write(List<FileData> fileDataEntries, File output)
throws Throwable
{
try (DataOutputStream outputStream=new DataOutputStream(new FileOutputStream(output)))
{
int entryCount=fileDataEntries.size();
outputStream.writeInt(entryCount);
for (FileData fileData : fileDataEntries)
{
int entryRealSize=fileData.getData().length;
{
System.out.println("Writing: " + fileData.getName() + " Length: " + entryRealSize);
outputStream.writeUTF(fileData.getName());
outputStream.writeInt(entryRealSize);
outputStream.write(fileData.getData());
}
}
outputStream.flush();
}
}
public static void read(File intermediateFile)
throws IOException
{
try (DataInputStream dataInputStream=new DataInputStream(new FileInputStream(intermediateFile)))
{
for (int entryCount=dataInputStream.readInt(), i=0; i < entryCount; i++)
{
String utf=dataInputStream.readUTF();
int entrySize=dataInputStream.readInt();
System.out.println("Entry name: " + utf + " size: " + entrySize);
byte[] data=readFixedLengthBuffer(dataInputStream, entrySize);
System.out.println("Entry bytes length: " + data.length);
}
}
}
private static byte[] readAllBytes(InputStream input)
throws IOException
{
byte[] buffer=new byte[4096];
byte[] total=new byte[0];
int len;
do
{
len=input.read(buffer);
if (len > 0)
{
byte[] total0=total;
total=new byte[total0.length + len];
System.arraycopy(total0, 0, total, 0, total0.length);
System.arraycopy(buffer, 0, total, total0.length, len);
}
}
while (len >= 0);
return total;
}
private static byte[] readFixedLengthBuffer(InputStream input, int size)
throws IOException
{
byte[] buffer=new byte[size];
int pos=0;
int len;
do
{
len=input.read(buffer, pos, size - pos);
if (len > 0)
{
pos+=len;
}
}
while (pos < size);
return buffer;
}
private static class FileData
{
private final String name;
private final byte[] data;
public FileData(String name, byte[] data)
{
super();
this.name=name;
this.data=data;
}
public String getName()
{
return this.name;
}
public byte[] getData()
{
return this.data;
}
}

Extracting SFX 7-Zip

I want to extract two specific files from a .zip file. I tried the following library:
ZipFile zipFile = new ZipFile("myZip.zip");
Result:
Exception in thread "main" java.util.zip.ZipException: error in opening zip file
I also tried:
public void extract(String targetFileName) throws IOException
{
OutputStream outputStream = new FileOutputStream("targetFile.foo");
FileInputStream fileInputStream = new FileInputStream("myZip.zip");
ZipInputStream zipInputStream = new ZipInputStream(new BufferedInputStream(fileInputStream));
ZipEntry zipEntry;
while ((zipEntry = zipInputStream.getNextEntry()) != null)
{
if (zipEntry.getName().equals("targetFile.foo"))
{
byte[] buffer = new byte[8192];
int length;
while ((length = zipInputStream.read(buffer)) != -1)
{
outputStream.write(buffer, 0, length);
}
outputStream.close();
break;
}
}
}
Result:
No exception, but an empty targetFile.foo file.
Note that the .zip file is of type SFX 7-zip and initially had the .exe extensions so that may be the reason for the failure.

As in Comments, Extracting SFX 7-Zip file is basically not supported with your library. But you can do with commons compress and xz Libary together with a quick "hack":
import org.apache.commons.compress.archivers.sevenz.SevenZArchiveEntry;
import org.apache.commons.compress.archivers.sevenz.SevenZFile;
import org.apache.commons.io.FileUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
...
protected File un7zSFXFile(File file, String password)
{
SevenZFile sevenZFile = null;
File tempFile = new File("/tmp/" + file.getName() + ".temp");
try
{
FileInputStream in = new FileInputStream(file);
/**
* Yes this is Voodoo Code:
* first 205824 Bytes get skipped as these is are basically the 7z-sfx-runnable.dll
* common-compress does fail if this information is not cut away
* ATTENTION: the amount of bytes may vary depending of the 7z Version used!
*/
in.skip(205824);
// EndOfVoodoCode
tempFile.getParentFile().mkdirs();
tempFile.createNewFile();
FileOutputStream temp = new FileOutputStream(tempFile);
byte[] buffer = new byte[1024];
int length;
while((length = in.read(buffer)) > 0)
{
temp.write(buffer, 0, length);
}
temp.close();
in.close();
LOGGER.info("prepared exefile for un7zing");
if (password!=null) {
sevenZFile = new SevenZFile(tempFile, password.toCharArray());
} else {
sevenZFile = new SevenZFile(tempFile);
}
SevenZArchiveEntry entry;
boolean first = true;// accept only files with
while((entry = sevenZFile.getNextEntry()))
{
if(entry.isDirectory())
{
continue;
}
File curfile = new File(file.getParentFile(), entry.getName());
File parent = curfile.getParentFile();
if(!parent.exists())
{
parent.mkdirs();
}
FileOutputStream out = new FileOutputStream(curfile);
byte[] content = new byte[(int) entry.getSize()];
sevenZFile.read(content, 0, content.length);
out.write(content);
out.close();
}
}
catch(Exception e)
{
throw e;
}
finally
{
try
{
tempFile.delete();
sevenZFile.close();
}
catch(Exception e)
{
LOGGER.trace("error on cloasing Stream: " + sevenZFile.getDefaultName(), e);
}
}
}
Please acknowledge that this simple solution does only unpack in to the same directory as the as sfx-file is placed!

java zip to binary format and then decompress

I have a task that
read a zip file from local into binary message
transfer binary message through EMS as String (done by java API)
receive transferred binary message as String (done by java API)
decompress the binary message and then print it out
The problem I am facing is DataFormatException while decompress the message.
I have no idea which part went wrong.
I use this to read file into binary message:
static String readFile_Stream(String fileName) throws IOException {
File file = new File(fileName);
byte[] fileData = new byte[(int) file.length()];
FileInputStream in = new FileInputStream(file);
in.read(fileData);
String content = "";
System.out.print("Sent message: ");
for(byte b : fileData)
{
System.out.print(getBits(b));
content += getBits(b);
}
in.close();
return content;
}
static String getBits(byte b)
{
String result = "";
for(int i = 0; i < 8; i++)
result = ((b & (1 << i)) == 0 ? "0" : "1") + result;
return result;
}
I use this to decompress message:
private static byte[] toByteArray(String input)
{
byte[] byteArray = new byte[input.length()/8];
for (int i=0;i<input.length()/8;i++)
{
String read_data = input.substring(i*8, i*8+8);
short a = Short.parseShort(read_data, 2);
byteArray[i] = (byte) a;
}
return byteArray;
}
public static byte[] unzipByteArray(byte[] file) throws IOException {
byte[] byReturn = null;
Inflater oInflate = new Inflater(false);
oInflate.setInput(file);
ByteArrayOutputStream oZipStream = new ByteArrayOutputStream();
try {
while (! oInflate.finished() ){
byte[] byRead = new byte[4 * 1024];
int iBytesRead = oInflate.inflate(byRead);
if (iBytesRead == byRead.length){
oZipStream.write(byRead);
}
else {
oZipStream.write(byRead, 0, iBytesRead);
}
}
byReturn = oZipStream.toByteArray();
}
catch (DataFormatException ex){
throw new IOException("Attempting to unzip file that is not zipped.");
}
finally {
oZipStream.close();
}
return byReturn;
}
The message I got is
java.io.IOException: Attempting to unzip file that is not zipped.
at com.sourcefreak.example.test.TibcoEMSQueueReceiver.unzipByteArray(TibcoEMSQueueReceiver.java:144)
at com.sourcefreak.example.test.TibcoEMSQueueReceiver.main(TibcoEMSQueueReceiver.java:54)
After check, the binary message does not corrupted after transmission.
Please help to figure out the problem.

Have you tried using InflaterInputStream? Based on my experience, using Inflater directly is rather tricky. You can use this to get started:
public static byte[] unzipByteArray(byte[] file) throws IOException {
InflaterInputStream iis = new InflaterInputStream(new ByteArrayInputStream(file));
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte[] buffer = new byte[512];
int length = 0;
while ((length = iis.read(buffer, 0, buffer.length) != 0) {
baos.write(buffer, 0, length);
}
iis.close();
baos.close();
return baos.toByteArray();
}

I finally figure out the problem.
The problem is the original file is a .zip file, so I should use zipInputStream to unzip the file before further processing.
public static byte[] unzipByteArray(byte[] file) throws IOException {
// create a buffer to improve copy performance later.
byte[] buffer = new byte[2048];
byte[] content ;
// open the zip file stream
InputStream theFile = new ByteArrayInputStream(file);
ZipInputStream stream = new ZipInputStream(theFile);
ByteArrayOutputStream output = new ByteArrayOutputStream();
try
{
ZipEntry entry;
while((entry = stream.getNextEntry())!=null)
{
//String s = String.format("Entry: %s len %d added %TD", entry.getName(), entry.getSize(), new Date(entry.getTime()));
//System.out.println(s);
// Once we get the entry from the stream, the stream is
// positioned read to read the raw data, and we keep
// reading until read returns 0 or less.
//String outpath = outdir + "/" + entry.getName();
try
{
//output = new FileOutputStream(outpath);
int len = 0;
while ((len = stream.read(buffer)) > 0)
{
output.write(buffer, 0, len);
}
}
finally
{
// we must always close the output file
if(output!=null) output.close();
}
}
}
finally
{
// we must always close the zip file.
stream.close();
}
content = output.toByteArray();
return content;
}
This code work for zip file containing single file inside.

Compressing a folder into a ZipFile

I have this Java method to upload a file. I am trying to cater for users trying to upload a folder by compressing that folder into a zip file and upload it instead. For some reason in my case file.isDirectory() and file.isFile() are not working correctly.. even though the filename does not contain any extension, file.isFile() is returning true and isDirectory() returns false. Also directory.list() is also acting weird by returning null.
What can be the problem? Am I doing something wrong?
public File uploadFile(FileItem item, String filename, int ticket_id) throws IOException
{
FileOutputStream out = null;
InputStream fileContent = null;
File file = null;
try
{
//fullpath returns C://MyDocuments//zerafbe//Documents//apache-tomcat-7.0.29//webapps//attachments//t50\test
StringBuffer fullPath = new StringBuffer();
fullPath.append(Attachment.attachments_path);
fullPath.append("t");
fullPath.append(Integer.toString(ticket_id));
fullPath.append(File.separator);
fullPath.append(filename);
System.out.println("filename " + filename);
file = new File(fullPath.toString());
if (!file.exists())
{
// if directory does not exist, create it
file.getParentFile().mkdirs();
}
if (file.isFile())
{
// if file is not a folder
out = new FileOutputStream(file);
fileContent = item.getInputStream();
int read = 0;
final byte[] bytes = new byte[1024];
// read all the file and write it to created file
while ((read = fileContent.read(bytes)) != -1)
{
out.write(bytes, 0, read);
}
}
else if (file.isDirectory())
{
ZipFile appZip = new ZipFile(fullPath.toString());
appZip.generateFileList(file);
appZip.zipIt(filename + ".zip");
}
}
catch (FileNotFoundException e)
{
LogFile.logError("[FileUpload.uploadFile()] " + e.getMessage());
}
catch (IOException e1)
{
LogFile.logError("[FileUpload.uploadFile()] " + e1.getMessage());
}
finally
{
if (out != null)
{
out.close();
}
if (fileContent != null)
{
fileContent.close();
}
}
return file;
}
This is the ZipFile class I am using
public class ZipFile
{
List<String> fileList = null;
String source_folder = "";
public ZipFile(String source_folder)
{
fileList = new ArrayList<String>();
this.source_folder = source_folder;
}
public void zipIt(String zipFile)
{
byte[] buffer = new byte[1024];
String source = "";
try
{
try
{
source = source_folder.substring(source_folder.lastIndexOf("\\") + 1, source_folder.length());
}
catch(Exception e)
{
source = source_folder;
}
FileOutputStream fos = new FileOutputStream(zipFile);
ZipOutputStream zos = new ZipOutputStream(fos);
for (String file : this.fileList)
{
ZipEntry ze = new ZipEntry(source + File.separator + file);
zos.putNextEntry(ze);
FileInputStream in = new FileInputStream(source_folder + File.separator + file);
int len;
while ((len = in.read(buffer)) > 0)
{
zos.write(buffer, 0, len);
}
in.close();
}
zos.closeEntry();
//remember close it
zos.close();
}
catch(IOException ex)
{
ex.printStackTrace();
}
}
public void generateFileList(File node)
{
// add file only
if(node.isFile())
{
fileList.add(generateZipEntry(node.toString()));
}
if(node.isDirectory())
{
String[] subNode = node.list();
if (subNode != null) {
for(String filename : subNode)
{
generateFileList(new File (node, filename));
}
}
}
}
private String generateZipEntry(String path)
{
return path.substring(source_folder.length() + 1, path.length());
}
}
file.list() is being done in the generateFileList method in ZipFile class. I know this is returning null since I tried detecting whether the file is a folder or a file by using filename.indexOf(".") instead of isDirectory() and isFile() since they were not working. But I wish I had an explanation for this.
Thanks for your help!

if (!file.exists()) {
// if directory does not exist, create it
file.mkdirs();
}
will create directory and test file.isDirectory() will return true

It could be a problem with the path?
C://MyDocuments//zerafbe//Documents//apache-tomcat-7.0.29//webapps//attachments//t50\test
You are mixing backslash with slash...

I tested your code block
ZipFile appZip = new ZipFile(file.toString());
appZip.generateFileList(file);
appZip.zipIt(filename + ".zip");
with a local folder and it's working perfectly. I think you are passing a invalid path. This may be the cause isFile or isDirectory methods are acting strangely. Try to add a validation statement at the starting of generateFileList method using File API:
if(!node.exists) {
// return some flag to signify error OR throw a suitable Exception
}
This should work.

public String compressData(String srcDir) {
String zipFile = srcDir+".zip";
try {
// create byte buffer
byte[] buffer = new byte[1024];
FileOutputStream fos = new FileOutputStream(zipFile);
ZipOutputStream zos = new ZipOutputStream(fos);
File dir = new File(srcDir);
File[] files = dir.listFiles();
for (int i = 0; i < files.length; i++) {
System.out.println("Adding file: " + files[i].getName());
FileInputStream fis = new FileInputStream(files[i]);
// begin writing a new ZIP entry, positions the stream to the start of the entry data
zos.putNextEntry(new ZipEntry(files[i].getName()));
int length;
while ((length = fis.read(buffer)) > 0) {
zos.write(buffer, 0, length);
}
zos.closeEntry();
// close the InputStream
fis.close();
}
// close the ZipOutputStream
zos.close();
}
catch (IOException ioe) {
System.out.println("Error creating zip file" + ioe);
}
return zipFile;
}

A better way to convert a directory of files into bytes

I have been messing with this for some time and it's getting better and better, but it's still a little slow for me. Can anyone help speed this up / make the design better, please?
Also, the files must only be numbers and the file must end with the file extension ".dat"
I never added the checks because I didn't feel is was necessary.
public void preloadModels() {
try {
File directory = new File(signlink.findcachedir() + "raw", File.separator);
File[] modelFiles = directory.listFiles();
for (int modelIndex = modelFiles.length - 1;; modelIndex--) {
String modelFileName = modelFiles[modelIndex].getName();
byte[] buffer = getBytesFromInputStream(new FileInputStream(new File(directory, modelFileName)));
Model.method460(buffer, Integer.parseInt(modelFileName.replace(".dat", "")));
}
} catch (Throwable e) {
return;
}
}
public static final byte[] getBytesFromInputStream(InputStream inputStream) throws IOException {
byte[] buffer = new byte[32 * 1024];
int bufferSize = 0;
for (;;) {
int read = inputStream.read(buffer, bufferSize, buffer.length - bufferSize);
if (read == -1) {
return Arrays.copyOf(buffer, bufferSize);
}
bufferSize += read;
if (bufferSize == buffer.length) {
buffer = Arrays.copyOf(buffer, bufferSize * 2);
}
}
}

I would do the following.
public void preloadModels() throws IOException {
File directory = new File(signlink.findcachedir() + "raw");
for (File file : directory.listFiles()) {
if (!file.getName().endsWith(".dat")) continue;
byte[] buffer = getBytesFromFile(file);
Model.method460(buffer, Integer.parseInt(file.getName().replace(".dat", "")));
}
}
public static byte[] getBytesFromFile(File file) throws IOException {
byte[] buffer = new byte[(int) file.length()];
try (DataInputStream dis = new DataInputStream(new FileInputStream(file))) {
dis.readFully(buffer);
return buffer;
}
}
If this is still too slow, most likely the limitation is the speed of hard drive.

How about using Apache Commons IOUtils class.
IOUtils.toByteArray(InputStream input)

I think the easiest way is to add all directories content to archive. Have a look at java.util.zip. It has some bugs with file names before 7th version. There is also Apache Commons implementation.