We Keep Coding

java.io.UTFDataFormatException while reading file entry name

Im trying to "pack" several files (previously inside a jar archive) in another single non-jar file by using DataInputStream / DataOutputStream.
The idea was:
First int = number of entries
First UTF is the first entry name
Second Int is entry byte array length (entry size)
Then repeat for every entry.
The code:
public static void main(String[] args) throws Throwable {
test();
System.out.println("========================================================================================");
final DataInputStream dataInputStream = new DataInputStream(new FileInputStream(new File("C:\\Users\\Admin\\Desktop\\randomJarOut")));
for (int int1 = dataInputStream.readInt(), i = 0; i < int1; ++i) {
final String utf = dataInputStream.readUTF();
System.out.println("Entry name: " + utf);
final byte[] array = new byte[dataInputStream.readInt()];
for (int j = 0; j < array.length; ++j) {
array[j] = dataInputStream.readByte();
}
System.out.println("Entry bytes length: " + array.length);
}
}
Unpacking original & packing to new one:
private static void test() throws Throwable {
JarInputStream stream = new JarInputStream(new FileInputStream(new File("C:\\Users\\Admin\\Desktop\\randomJar.jar")));
JarInputStream stream1 = new JarInputStream(new FileInputStream(new File("C:\\Users\\Admin\\Desktop\\randomJar.jar")));
final byte[] buffer = new byte[2048];
final DataOutputStream outputStream = new DataOutputStream(new FileOutputStream(new File("C:\\Users\\Admin\\Desktop\\randomJarOut")));
int entryCount = 0;
for (ZipEntry entry; (entry = stream.getNextJarEntry()) != null; ) {
entryCount++;
}
outputStream.writeInt(entryCount);
for (JarEntry entry; (entry = stream1.getNextJarEntry()) != null; ) {
int entryRealSize = stream1.read(buffer);
if (!(entryRealSize == -1)) {
System.out.println("Writing: " + entry.getName() + " Length: " + entryRealSize);
outputStream.writeUTF(entry.getName());
outputStream.writeInt(entryRealSize);
for (int len = stream1.read(buffer); len != -1; len = stream1.read(buffer)) {
outputStream.write(buffer, 0, len);
}
}
}
outputStream.flush();
outputStream.close();
}
Apparently im able to unpack the first entry without any problems, the second one and others:
Entry name: META-INF/services/org.jd.gui.spi.ContainerFactory
Entry bytes length: 434
Exception in thread "main" java.io.UTFDataFormatException: malformed input around byte 279
at java.io.DataInputStream.readUTF(DataInputStream.java:656)
at java.io.DataInputStream.readUTF(DataInputStream.java:564)
at it.princekin.esercizio.Bootstrap.main(Bootstrap.java:29)
Disconnected from the target VM, address: '127.0.0.1:54384', transport: 'socket'
Process finished with exit code 1
Does anyone knows how to fix this? Why is this working for the first entry but not the others?

My take on this is that the jar file (which in fact is a zip file) has a Central Directory which is only read with the ZipFile (or JarFile) class.
The Central Directory contains some data about the entries such as the size.
I think the ZipInputStream will not read the Central Directory and thus the ZipEntry will not contain the size (returning -1 as it is unknown) whereas reading ZipEntry from ZipFile class will.
So if you first read the size of each entry using a ZipFile and store that in a map, you can easily get it when reading the data with the ZipInputStream.
This page includes some good examples as well.
So my version of your code would be:
import java.io.*;
import java.util.HashMap;
import java.util.Map;
import java.util.zip.ZipEntry;
import java.util.zip.ZipFile;
import java.util.zip.ZipInputStream;
public class JarRepacker {
public static void main(String[] args) throws Throwable {
JarRepacker repacker = new JarRepacker();
repacker.repackJarToMyFileFormat("commons-cli-1.3.1.jar", "randomJarOut.bin");
repacker.readMyFileFormat("randomJarOut.bin");
}
private void repackJarToMyFileFormat(String inputJar, String outputFile) throws Throwable {
int entryCount;
Map<String, Integer> sizeMap = new HashMap<>();
try (ZipFile zipFile = new ZipFile(inputJar)) {
entryCount = zipFile.size();
zipFile.entries().asIterator().forEachRemaining(e -> sizeMap.put(e.getName(), (int) e.getSize()));
}
try (final DataOutputStream outputStream = new DataOutputStream(new FileOutputStream(outputFile))) {
outputStream.writeInt(entryCount);
try (ZipInputStream stream = new ZipInputStream(new BufferedInputStream(new FileInputStream(inputJar)))) {
ZipEntry entry;
final byte[] buffer = new byte[2048];
while ((entry = stream.getNextEntry()) != null) {
final String name = entry.getName();
outputStream.writeUTF(name);
final Integer size = sizeMap.get(name);
outputStream.writeInt(size);
//System.out.println("Writing: " + name + " Size: " + size);
int len;
while ((len = stream.read(buffer)) > 0) {
outputStream.write(buffer, 0, len);
}
}
}
outputStream.flush();
}
}
private void readMyFileFormat(String fileToRead) throws IOException {
try (DataInputStream dataInputStream
= new DataInputStream(new BufferedInputStream(new FileInputStream(fileToRead)))) {
int entries = dataInputStream.readInt();
System.out.println("Entries in file: " + entries);
for (int i = 1; i <= entries; i++) {
final String name = dataInputStream.readUTF();
final int size = dataInputStream.readInt();
System.out.printf("[%3d] Reading: %s of size: %d%n", i, name, size);
final byte[] array = new byte[size];
for (int j = 0; j < array.length; ++j) {
array[j] = dataInputStream.readByte();
}
// Still need to do something with this array...
}
}
}
}

The problem, probably, lies in that you are mixing not reciprocal read/write methods:
The writer method writes with outputStream.writeInt(entryCount) and the main method reads with dataInputStream.readInt(). That is OK.
The writer method writes with outputStream.writeUTF(entry.getName()) and the main method reads with dataInputStream.readUTF(). That is OK.
The writer method writes with outputStream.writeInt(entryRealSize) and the main method reads with dataInputStream.readInt(). That is OK.
The writer method writes with outputStream.write(buffer, 0, len) and the main method reads with dataInputStream.readByte() several times. WRONG.
If you write an array of bytes with write(buffer, offset, len), you must read it with read(buffer, offset, len), because write(buffer, offset, len) writes exactly len physical bytes onto the output stream, while writeByte (the counterpart of readByte) writes a lot of metadata overhead about the object type, and then its state variables.
Bugs in the writer method
There is also a mayor bug in the writer method: It invokes up to three times stream1.read(buffer), but it just uses once the buffer contents. The result is that the real size of file is actually written onto the output stream metadata, but it is followed by just a small part of the data.
If you need to know the input file size before writing it in the output stream, you have two choices:
Either chose a large enough buffer size (like 204800) which will allow you to read the whole file in just one read and write it in just one write.
Or either separate read from write algorithms: First a method to read the whole file and store it in memory (a byte[], for example), and then another method to write the byte[] onto the output stream.
Full fixed solution
I've fixed your program, with specific, decoupled methods for each task. The process consists in parsing the input file to a memory model, write it to an intermediate file according to your custom definition, and then read it back.
public static void main(String[] args)
throws Throwable
{
File inputJarFile=new File(args[0]);
File intermediateFile=new File(args[1]);
List<FileData> fileDataEntries=parse(inputJarFile);
write(fileDataEntries, intermediateFile);
read(intermediateFile);
}
public static List<FileData> parse(File inputJarFile)
throws IOException
{
List<FileData> list=new ArrayList<>();
try (JarInputStream stream=new JarInputStream(new FileInputStream(inputJarFile)))
{
for (ZipEntry entry; (entry=stream.getNextJarEntry()) != null;)
{
byte[] data=readAllBytes(stream);
if (data.length > 0)
{
list.add(new FileData(entry.getName(), data));
}
stream.closeEntry();
}
}
return list;
}
public static void write(List<FileData> fileDataEntries, File output)
throws Throwable
{
try (DataOutputStream outputStream=new DataOutputStream(new FileOutputStream(output)))
{
int entryCount=fileDataEntries.size();
outputStream.writeInt(entryCount);
for (FileData fileData : fileDataEntries)
{
int entryRealSize=fileData.getData().length;
{
System.out.println("Writing: " + fileData.getName() + " Length: " + entryRealSize);
outputStream.writeUTF(fileData.getName());
outputStream.writeInt(entryRealSize);
outputStream.write(fileData.getData());
}
}
outputStream.flush();
}
}
public static void read(File intermediateFile)
throws IOException
{
try (DataInputStream dataInputStream=new DataInputStream(new FileInputStream(intermediateFile)))
{
for (int entryCount=dataInputStream.readInt(), i=0; i < entryCount; i++)
{
String utf=dataInputStream.readUTF();
int entrySize=dataInputStream.readInt();
System.out.println("Entry name: " + utf + " size: " + entrySize);
byte[] data=readFixedLengthBuffer(dataInputStream, entrySize);
System.out.println("Entry bytes length: " + data.length);
}
}
}
private static byte[] readAllBytes(InputStream input)
throws IOException
{
byte[] buffer=new byte[4096];
byte[] total=new byte[0];
int len;
do
{
len=input.read(buffer);
if (len > 0)
{
byte[] total0=total;
total=new byte[total0.length + len];
System.arraycopy(total0, 0, total, 0, total0.length);
System.arraycopy(buffer, 0, total, total0.length, len);
}
}
while (len >= 0);
return total;
}
private static byte[] readFixedLengthBuffer(InputStream input, int size)
throws IOException
{
byte[] buffer=new byte[size];
int pos=0;
int len;
do
{
len=input.read(buffer, pos, size - pos);
if (len > 0)
{
pos+=len;
}
}
while (pos < size);
return buffer;
}
private static class FileData
{
private final String name;
private final byte[] data;
public FileData(String name, byte[] data)
{
super();
this.name=name;
this.data=data;
}
public String getName()
{
return this.name;
}
public byte[] getData()
{
return this.data;
}
}

java: Length of data changes after write in stream

I have saved a binary data in FileOutputStream but when I check the length of the data before and after I found that it changes from 72 to 106.
This is my method:
inputStream = new FileInputStream(certificate_file);
/*Certificate file is a Path of a binary file */
pubkey = readFromStream(inputStream, 0, 71);
System.out.println("length of pubkey: "+pubkey.length());
/* This return : length of pubkey: 72 */
writeToStream(path + "pubkey.bin", pubkey);
inputStream = new FileInputStream(path + "pubkey.bin");
pubkey = readFromStream(inputStream);
System.out.println("length of pubkey: "+pubkey.length());
/* This return : length of pubkey: 106 */
writeToStream method to write data into outputstream:
public void writeToStream(String path, String data)
throws FileNotFoundException {
OutputStream os = new FileOutputStream(path);
PrintStream printStream = new PrintStream(os);
printStream.print(data);
}
readFromStream method to read data from stream:
public static String readFromStream(InputStream inputStream, int begin, int end) throws Exception {
int i = 0;
int data = inputStream.read();
String out = "";
while (data != -1) {
if (i >= begin && i <= end) {
out += (char) data;
}
data = inputStream.read();
i++;
}
return out;
}
public static String readFromStream(InputStream inputStream) throws Exception {
int i = 0;
int data = inputStream.read();
String out = "";
while (data != -1) {
out += (char) data;
data = inputStream.read();
i++;
}
return out;
}
Why I have this problem?

I have solved the problem, I transformed the data from String to bytes[] and I changed the read in readFromStream to readAllBytes.

How to effeciently read chunk of bytes of a given range from a large encrypted file in java?

I have a large encrypted file(10GB+) in server. I need to transfer the decrypted file to the client in small chunks. When a client make a request for a chunk of bytes (say 18 to 45) I have to random access the file, read the specific bytes, decrypt it and transfer it to the client using ServletResponseStream.
But since the file is encrypted I have to read the file as blocks of 16 bytes in order to decrypt correctly.
So if client requests to get from byte 18 to 45, in the server I have to read the file in multiples of 16 bytes block. So I have to random access the file from byte 16 to 48. Then decrypt it. After decryption I have to skip 2 bytes from the first and 3 bytes from the last to return the appropriate chunk of data client requested.
Here is what I am trying to do
Adjust start and end for encrypted files
long start = 15; // input from client
long end = 45; // input from client
long skipStart = 0; // need to skip for encrypted file
long skipEnd = 0;
// encrypted files, it must be access in blocks of 16 bytes
if(fileisEncrypted){
skipStart = start % 16; // skip 2 byte at start
skipEnd = 16 - end % 16; // skip 3 byte at end
start = start - skipStart; // start becomes 16
end = end + skipEnd; // end becomes 48
}
Access the encrypted file data from start to end
try(final FileChannel channel = FileChannel.open(services.getPhysicalFile(datafile).toPath())){
MappedByteBuffer mappedByteBuffer = channel.map(FileChannel.MapMode.READ_ONLY, start, end-start);
// *** No idea how to convert MappedByteBuffer into input stream ***
// InputStream is = (How do I get inputstream for byte 16 to 48 here?)
// the medhod I used earlier to decrypt the all file atonce, now somehow I need the inputstream of specific range
is = new FileEncryptionUtil().getCipherInputStream(is,
EncodeUtil.decodeSeedValue(encryptionKeyRef), AESCipher.DECRYPT_MODE);
// transfering decrypted input stream to servlet response
OutputStream outputStream = response.getOutputStream();
// *** now for chunk transfer, here I also need to
// skip 2 bytes at the start and 3 bytes from the end.
// How to do it? ***/
org.apache.commons.io.IOUtils.copy(is, outputStream)
}
I am missing few steps in the code given above. I know I could try to read byte by byte and the ignore 2byte from first and 3 byte from last. But I am not sure if it will be efficient enough. Moreover, the client could request a large chunk say from byte 18 to 2048 which would require to read and decrypt almost two gigabytes of data. I am afraid creating a large byte array will consume too much memory.
How can I efficiently do it without putting too much pressure on server processing or memory?
Any ideas?

As you haven't specified which cipher mode you're using, I'll assume that you're using AES in CTR mode, as it's designed to read random chunks of big files without having to decrypt them completely.
With AES-CTR, you can stream the file through the decryption code and send the blocks back to the client as soon as they are available. So you only need a few arrays the size of the AES block in memory, all the rest is read from the disk. You would need to add special logic to skip some byes on the first and last block (but you don't need to load the whole thing in memory).
There's an example of how to do this in another SO question (this only performs the seek): Seeking in AES-CTR-encrypted input . After that you can skip the first few bytes, read until the last block and adjust that to the number of bytes your client requested.

After researching for awhile. This is how I solved it.
First I created a ByteBufferInputStream class. To read from MappedByteBuffer
public class ByteBufferInputStream extends InputStream {
private ByteBuffer byteBuffer;
public ByteBufferInputStream () {
}
/** Creates a stream with a new non-direct buffer of the specified size. The position and limit of the buffer is zero. */
public ByteBufferInputStream (int bufferSize) {
this(ByteBuffer.allocate(bufferSize));
byteBuffer.flip();
}
/** Creates an uninitialized stream that cannot be used until {#link #setByteBuffer(ByteBuffer)} is called. */
public ByteBufferInputStream (ByteBuffer byteBuffer) {
this.byteBuffer = byteBuffer;
}
public ByteBuffer getByteBuffer () {
return byteBuffer;
}
public void setByteBuffer (ByteBuffer byteBuffer) {
this.byteBuffer = byteBuffer;
}
public int read () throws IOException {
if (!byteBuffer.hasRemaining()) return -1;
return byteBuffer.get();
}
public int read (byte[] bytes, int offset, int length) throws IOException {
int count = Math.min(byteBuffer.remaining(), length);
if (count == 0) return -1;
byteBuffer.get(bytes, offset, count);
return count;
}
public int available () throws IOException {
return byteBuffer.remaining();
}
}
Then created BlockInputStream class by extending InputStream which will allow to skip the extra bytes and read internal input stream in multiples of 16 bytes block.
public class BlockInputStream extends InputStream {
private final BufferedInputStream inputStream;
private final long totalLength;
private final long skip;
private long read = 0;
private byte[] buff = new byte[16];
private ByteArrayInputStream blockInputStream;
public BlockInputStream(InputStream inputStream, long skip, long length) throws IOException {
this.inputStream = new BufferedInputStream(inputStream);
this.skip = skip;
this.totalLength = length + skip;
if(skip > 0) {
byte[] b = new byte[(int)skip];
read(b);
b = null;
}
}
private int readBlock() throws IOException {
int count = inputStream.read(buff);
blockInputStream = new ByteArrayInputStream(buff);
return count;
}
#Override
public int read () throws IOException {
byte[] b = new byte[1];
read(b);
return (int)b[1];
}
#Override
public int read(byte[] b) throws IOException {
return read(b, 0, b.length);
}
#Override
public int read (byte[] bytes, int offset, int length) throws IOException {
long remaining = totalLength - read;
if(remaining < 1){
return -1;
}
int bytesToRead = (int)Math.min(length, remaining);
int n = 0;
while(bytesToRead > 0){
if(read % 16 == 0 && bytesToRead % 16 == 0){
int count = inputStream.read(bytes, offset, bytesToRead);
read += count;
offset += count;
bytesToRead -= count;
n += count;
} else {
if(blockInputStream != null && blockInputStream.available() > 0) {
int len = Math.min(bytesToRead, blockInputStream.available());
int count = blockInputStream.read(bytes, offset, len);
read += count;
offset += count;
bytesToRead -= count;
n += count;
} else {
readBlock();
}
}
}
return n;
}
#Override
public int available () throws IOException {
long remaining = totalLength - read;
if(remaining < 1){
return -1;
}
return inputStream.available();
}
#Override
public long skip(long n) throws IOException {
return inputStream.skip(n);
}
#Override
public void close() throws IOException {
inputStream.close();
}
#Override
public synchronized void mark(int readlimit) {
inputStream.mark(readlimit);
}
#Override
public synchronized void reset() throws IOException {
inputStream.reset();
}
#Override
public boolean markSupported() {
return inputStream.markSupported();
}
}
This is my final working implementation using this two classes
private RangeData getRangeData(RangeInfo r) throws IOException, GeneralSecurityException, CryptoException {
// used for encrypted files
long blockStart = r.getStart();
long blockEnd = r.getEnd();
long blockLength = blockEnd - blockStart + 1;
// encrypted files, it must be access in blocks of 16 bytes
if(datafile.isEncrypted()){
blockStart -= blockStart % 16;
blockEnd = blockEnd | 15; // nearest multiple of 16 for length n = ((n−1)|15)+1
blockLength = blockEnd - blockStart + 1;
}
try ( final FileChannel channel = FileChannel.open(services.getPhysicalFile(datafile).toPath()) )
{
MappedByteBuffer mappedByteBuffer = channel.map(FileChannel.MapMode.READ_ONLY, blockStart, blockLength);
InputStream inputStream = new ByteBufferInputStream(mappedByteBuffer);
if(datafile.isEncrypted()) {
String encryptionKeyRef = (String) settingsManager.getSetting(AppSetting.DEFAULT_ENCRYPTION_KEY);
inputStream = new FileEncryptionUtil().getCipherInputStream(inputStream,
EncodeUtil.decodeSeedValue(encryptionKeyRef), AESCipher.DECRYPT_MODE);
long skipStart = r.getStart() - blockStart;
inputStream = new BlockInputStream(inputStream, skipStart, r.getLength()); // this will trim the data to n bytes at last
}
return new RangeData(r, inputStream);
}
}

Byte array not fully consumed

I am writing an FLV parser in Java and have come up against an issue. The program successfully parses and groups together tags into packets and correctly identifies and assigns a byte array for each tag's body based upon the BodyLength flag in the header. However in my test files it successfully completes this but stops before the last 4 bytes.
The byte sequence left out in the first file is :
00 00 14 C3
And in the second:
00 00 01 46
Clearly it is an issue with the final 4 bytes of both files however I cannot spot the error in my logic. I suspect it might be:
while (in.available() != 0)
However I also doubt this is the case as the program is successfully entering the loop for the final tag however it is just stopping 4 bytes short. Any help is greatly appreciated. (I know proper exception handling is as yet not taking place)
Parser.java
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.lang.reflect.Array;
import java.net.URI;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.InputMismatchException;
/**
*
* #author A
*
* Parser class for FLV files
*/
public class Parser {
private static final int HEAD_SIZE = 9;
private static final int TAG_HEAD_SIZE = 15;
private static final byte[] FLVHEAD = { 0x46, 0x4C, 0x56 };
private static final byte AUDIO = 0x08;
private static final byte VIDEO = 0x09;
private static final byte DATA = 0x12;
private static final int TYPE_INDEX = 4;
private File file;
private FileInputStream in;
private ArrayList<Packet> packets;
private byte[] header = new byte[HEAD_SIZE];
Parser() throws FileNotFoundException {
throw new FileNotFoundException();
}
Parser(URI uri) {
file = new File(uri);
init();
}
Parser(File file) {
this.file = file;
init();
}
private void init() {
packets = new ArrayList<Packet>();
}
public void parse() {
boolean test = false;
try {
test = parseHeader();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
if (test) {
System.out.println("Header Verified");
// Add header packet to beginning of list & then null packet
Packet p = new Packet(PTYPE.P_HEAD);
p.setSize(header.length);
p.setByteArr(header);
packets.add(p);
p = null;
try {
parseTags();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
} else {
try {
in.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
// throw FileNotFoundException because incorrect file
}
}
private boolean parseHeader() throws FileNotFoundException, IOException {
if (file == null)
throw new FileNotFoundException();
in = new FileInputStream(file);
in.read(header, 0, 9);
return Arrays.equals(FLVHEAD, Arrays.copyOf(header, FLVHEAD.length));
}
private void parseTags() throws IOException {
if (file == null)
throw new FileNotFoundException();
byte[] tagHeader = new byte[TAG_HEAD_SIZE];
Arrays.fill(tagHeader, (byte) 0x00);
byte[] body;
byte[] buf;
PTYPE pt;
int OFFSET = 0;
while (in.available() != 0) {
// Read first 5 - bytes, previous tag size + tag type
in.read(tagHeader, 0, 5);
if (tagHeader[TYPE_INDEX] == AUDIO) {
pt = PTYPE.P_AUD;
} else if (tagHeader[TYPE_INDEX] == VIDEO) {
pt = PTYPE.P_VID;
} else if (tagHeader[TYPE_INDEX] == DATA) {
pt = PTYPE.P_DAT;
} else {
// Header should've been dealt with - if previous data types not
// found then throw exception
System.out.println("Unexpected header format: ");
System.out.print(String.format("%02x\n", tagHeader[TYPE_INDEX]));
System.out.println("Last Tag");
packets.get(packets.size()-1).diag();
System.out.println("Number of tags found: " + packets.size());
throw new InputMismatchException();
}
OFFSET = TYPE_INDEX;
// Read body size - 3 bytes
in.read(tagHeader, OFFSET + 1, 3);
// Body size buffer array - padding for 1 0x00 bytes
buf = new byte[4];
Arrays.fill(buf, (byte) 0x00);
// Fill size bytes
buf[1] = tagHeader[++OFFSET];
buf[2] = tagHeader[++OFFSET];
buf[3] = tagHeader[++OFFSET];
// Calculate body size
int bSize = ByteBuffer.wrap(buf).order(ByteOrder.BIG_ENDIAN)
.getInt();
// Initialise Array
body = new byte[bSize];
// Timestamp
in.read(tagHeader, ++OFFSET, 3);
Arrays.fill(buf, (byte) 0x00);
// Fill size bytes
buf[1] = tagHeader[OFFSET++];
buf[2] = tagHeader[OFFSET++];
buf[3] = tagHeader[OFFSET++];
int milliseconds = ByteBuffer.wrap(buf).order(ByteOrder.BIG_ENDIAN)
.getInt();
// Read padding
in.read(tagHeader, OFFSET, 4);
// Read body
in.read(body, 0, bSize);
// Diagnostics
//printBytes(body);
Packet p = new Packet(pt);
p.setSize(tagHeader.length + body.length);
p.setByteArr(concat(tagHeader, body));
p.setMilli(milliseconds);
packets.add(p);
p = null;
// Zero out for next iteration
body = null;
Arrays.fill(buf, (byte)0x00);
Arrays.fill(tagHeader, (byte)0x00);
milliseconds = 0;
bSize = 0;
OFFSET = 0;
}
in.close();
}
private byte[] concat(byte[] tagHeader, byte[] body) {
int aLen = tagHeader.length;
int bLen = body.length;
byte[] C = (byte[]) Array.newInstance(tagHeader.getClass()
.getComponentType(), aLen + bLen);
System.arraycopy(tagHeader, 0, C, 0, aLen);
System.arraycopy(body, 0, C, aLen, bLen);
return C;
}
private void printBytes(byte[] b) {
System.out.println("\n--------------------");
for (int i = 0; i < b.length; i++) {
System.out.print(String.format("%02x ", b[i]));
if (((i % 8) == 0 ) && i != 0)
System.out.println();
}
}
}
Packet.java
public class Packet {
private PTYPE type = null;
byte[] buf;
int milliseconds;
Packet(PTYPE t) {
this.setType(t);
}
public void setSize(int s) {
buf = new byte[s];
}
public PTYPE getType() {
return type;
}
public void setType(PTYPE type) {
if (this.type == null)
this.type = type;
}
public void setByteArr(byte[] b) {
this.buf = b;
}
public void setMilli(int milliseconds) {
this.milliseconds = milliseconds;
}
public void diag(){
System.out.println("|-- Tag Type: " + type);
System.out.println("|-- Milliseconds: " + milliseconds);
System.out.println("|-- Size: " + buf.length);
System.out.println("|-- Bytes: ");
for(int i = 0; i < buf.length; i++){
System.out.print(String.format("%02x ", buf[i]));
if (((i % 8) == 0 ) && i != 0)
System.out.println();
}
System.out.println();
}
}
jFLV.java
import java.net.URISyntaxException;
public class jFLV {
/**
* #param args
*/
public static void main(String[] args) {
// TODO Auto-generated method stub
Parser p = null;
try {
p = new Parser(jFLV.class.getResource("sample.flv").toURI());
} catch (URISyntaxException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
p.parse();
}
}
PTYPE.java
public enum PTYPE {
P_HEAD,P_VID,P_AUD,P_DAT
};

Both your use of available() and your call to read are broken. Admittedly I would have somewhat expected this to be okay for a FileInputStream (until you reach the end of the stream, at which point ignoring the return value for read could still be disastrous) but I personally assume that streams can always return partial data.
available() only tells you whether there's any data available right now. It's very rarely useful - just ignore it. If you want to read until the end of the stream, you should usually keep calling read until it returns -1. It's slightly tricky to combine that with "I'm trying to read the next block", admittedly. (It would be nice if InputStream had a peek() method, but it doesn't. You can wrap it in a BufferedInputStream and use mark/reset to test that at the start of each loop... ugly, but it should work.)
Next, you're ignoring the result of InputStream.read (in multiple places). You should always use the result of this, rather than assuming it has read the amount of data you've asked for. You might want a couple of helper methods, e.g.
static byte[] readExactly(InputStream input, int size) throws IOException {
byte[] data = new byte[size];
readExactly(input, data);
return data;
}
static void readExactly(InputStream input, byte[] data) throws IOException {
int index = 0;
while (index < data.length) {
int bytesRead = input.read(data, index, data.length - index);
if (bytesRead < 0) {
throw new EOFException("Expected more data");
}
}
}

You should use one of the read methods instead of available, as available() "Returns an estimate of the number of bytes that can be read (or skipped over) from this input stream without blocking by the next invocation of a method for this input stream."
It is not designed to check how long you can read.

Reading files bits and saving them

i have file reader which read entire file and write it's bits.
I have this class which help reading:
import java.io.*;
public class FileReader extends ByteArrayInputStream{
private int bitsRead;
private int bitPosition;
private int currentByte;
private int myMark;
private final static int NUM_BITS_IN_BYTE = 8;
private final static int END_POSITION = -1;
private boolean readingStarted;
/**
* Create a BitInputStream for a File on disk.
*/
public FileReader( byte[] buf ) throws IOException {
super( buf );
myMark = 0;
bitsRead = 0;
bitPosition = NUM_BITS_IN_BYTE-1;
currentByte = 0;
readingStarted = false;
}
/**
* Read a binary "1" or "0" from the File.
*/
public int readBit() throws IOException {
int theBit = -1;
if( bitPosition == END_POSITION || !readingStarted ) {
currentByte = super.read();
bitPosition = NUM_BITS_IN_BYTE-1;
readingStarted = true;
}
theBit = (0x01 << bitPosition) & currentByte;
bitPosition--;
if( theBit > 0 ) {
theBit = 1;
}
return( theBit );
}
/**
* Return the next byte in the File as lowest 8 bits of int.
*/
public int read() {
currentByte = super.read();
bitPosition = END_POSITION;
readingStarted = true;
return( currentByte );
}
/**
*
*/
public void mark( int readAheadLimit ) {
super.mark(readAheadLimit);
myMark = bitPosition;
}
/**
* Add needed functionality to super's reset() method. Reset to
* the last valid position marked in the input stream.
*/
public void reset() {
super.pos = super.mark-1;
currentByte = super.read();
bitPosition = myMark;
}
/**
* Returns the number of bits still available to be read.
*/
public int availableBits() throws IOException {
return( ((super.available() * 8) + (bitPosition + 1)) );
}
}
In class where i call this, i do:
FileInputStream inputStream = new FileInputStream(file);
byte[] fileBits = new byte[inputStream.available()];
inputStream.read(fileBits, 0, inputStream.available());
inputStream.close();
FileReader bitIn = new FileReader(fileBits);
and this work correctly.
However i have problems with big files above 100 mb because byte[] have the end.
So i want to read bigger files. Maybe some could suggest how i can improve this code ?
Thanks.

If scaling to large file sizes is important, you'd be better off not reading the entire file into memory. The downside is that handling the IOException in more locations can be a little messy. Also, it doesn't look like your application needs something that implements the InputStream API, it just needs the readBit() method. So, you can safely encapsulate, rather than extend, the InputStream.
class FileReader {
private final InputStream src;
private final byte[] bits = new byte[8192];
private int len;
private int pos;
FileReader(InputStream src) {
this.src = src;
}
int readBit() throws IOException {
int idx = pos / 8;
if (idx >= len) {
int n = src.read(bits);
if (n < 0)
return -1;
len = n;
pos = 0;
idx = 0;
}
return ((bits[idx] & (1 << (pos++ % 8))) == 0) ? 0 : 1;
}
}
Usage would look similar.
FileInputStream src = new FileInputStream(file);
try {
FileReader bitIn = new FileReader(src);
...
} finally {
src.close();
}
If you really do want to read in the entire file, and you are working with an actual file, you can query the length of the file first.
File file = new File(path);
if (file.length() > Integer.MAX_VALUE)
throw new IllegalArgumentException("File is too large: " + file.length());
int len = (int) file.length();
FileInputStream inputStream = new FileInputStream(file);
try {
byte[] fileBits = new byte[len];
for (int pos = 0; pos < len; ) {
int n = inputStream.read(fileBits, pos, len - pos);
if (n < 0)
throw new EOFException();
pos += n;
}
/* Use bits. */
...
} finally {
inputStream.close();
}

org.apache.commons.io.IOUtils.copy(InputStream in, OutputStream out)