public static void main(String[] args) throws Exception
{
// Server sends 3 numbers to the client
ByteArrayOutputStream bos = new ByteArrayOutputStream();
bos.write(1000);
bos.write(2000);
bos.write(3000);
// Client receive the bytes
final byte[] bytes = bos.toByteArray();
ByteArrayInputStream bis = new ByteArrayInputStream(bytes);
System.out.println(bis.read());
System.out.println(bis.read());
System.out.println(bis.read());
}
The code above is breaking because bis.read() returns an int in the range 0 to 255
How can I receive those numbers properly? Should I use a delimiter and keep reading the stream until I find it? If so, what if I'm sending multiple files, I think if the delimiter as a single byte it could matched somewhere in the file and also break.
Use decorators for your streams!
All you have to do is to wrap your Output- and InputStream by java.io.ObjectOutputStream / and java.io.ObjectInputStream. These classes support writing and reading ints (a 4-byte value) with a single method call to writeInt/readInt.
ByteArrayOutputStream bos = new ByteArrayOutputStream();
ObjectOutputStream os = new ObjectOutputStream(bos);
os.writeInt(1000);
os.writeInt(2000);
os.writeInt(3000);
os.close();
// Client receive the bytes
final byte[] bytes = bos.toByteArray();
ObjectInputStream is = new ObjectInputStream(new ByteArrayInputStream(bytes));
System.out.println(is.readInt());
System.out.println(is.readInt());
System.out.println(is.readInt());
Don't forget to close the streams. Use try/finally or try-with-resources.
Byte stream is a stream of bytes. So if you're reading stream and want to differentiate between different parts of the stream then you should "create" some sort of protocol.
Here are some ideas that can be relevant:
Use delimiter as you've stated by yourself, If you're concerned about the length - do not one byte length, but something more unique - something that you're sure you won't see in the parts themselves.
At the beginning of the part allocate N bytes (2-4 or maybe more, depending on data) and write the size of the part that will follow.
So that when you create the stream (writer), before actually streaming the "part" - calculate its size and encode it. This is a protocol between reader and writer.
When you read - read the size (=N bytes for example), and then read N bytes. Now you know that the part is ended, and the next part (again, size + content) will follow
Can you try ByteBuffer class?
ByteStream is just a stream of bytes. It doesn't understand integer which actually needs more than one byte. If you print bytes.length it will return you 3. Surely you need more bytes than that. Allocate 4 bytes before you write an integer and then write to it. Check out this class above. Hope that helps!
How to write an array of bytes b[i] to a binary file in Java.
I need to write those bytes it into a "binary file" to be able to read it later using hex editor (AXE).
Some readers might be confused by "binary file", by binary file I don't mean a file filled by zeros and ones, I mean machine-readable form, something like this :
binary files in text editor
The hex editor suppose to read this data, hex editor
From what I understand I need to byte stream that data into a file
Is there a command I could use for this purpose.
Any code would be appreciated.
Just write the byte[] to a FileOutputStream pointing to the file:
private static void writeBytesToFile(byte[] b, String f) {
try (FileOutputStream out = new FileOutputStream(f)){
out.write(b);
}
catch (IOException e) {
e.printStackTrace();
}
}
I need to save a pdf document, generated by aspose.pdf for java library to memory (without using temporary file)
I was looking at the documentation and didn't find the save method with the appropriate signature. (I was looking for some kind of outputstream, or at least byte array).
Is it possible? If it is, how can I manage that?
Thanks
Aspose.Pdf for Java supports saving output to both file and stream. Please check following code snippet, It will help you to accomplish the task.
byte[] input = getBytesFromFile(new File("C:/data/HelloWorld.pdf"));
ByteArrayOutputStream output = new ByteArrayOutputStream();
com.aspose.pdf.Document pdfDocument = new com.aspose.pdf.Document(new ByteArrayInputStream(input));
pdfDocument.save(output);
//If you want to read the result into a Document object again, in Java you need to get the
//data bytes and wrap into an input stream.
InputStream inputStream=new ByteArrayInputStream(output.toByteArray());
I am Tilal Ahmad, developer evangelist at Aspose.
I did similar thing.
Here is method to write data to byte:
public byte[] toBytes() {
//create byte array output stream object
ByteArrayOutputStream byteOutStream = new ByteArrayOutputStream();
//create new data output stream object
DataOutputStream outStream = new DataOutputStream(byteOutStream);
try {//write data to bytes stream
if (data != null) {
outStream.write(data);//write data
}//return array of bytes
return byteOutStream.toByteArray();
}
Then you do something like
yourFileName.toBytes;
My program needs to do calculations against the entire bytes of a file and it breaks whenever the file gets above a certain size.
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
I know I can allocate the amount of memory to my program using command line switches, but I'm wondering if there is a more effective way of handling this in my program?
I'm basically trying to figure out a way to read the file in chunks and pass those chunks to another method and essentially rebuild the file in that method.
This is the problem method. I need these bytes to be used in another method.
This method converts the stream to a byte array:
private byte[] inputStreamToByteArray(InputStream inputStream) {
BufferedInputStream bis = null;
ByteArrayOutputStream baos = null;
try {
bis = new BufferedInputStream(inputStream);
baos = new ByteArrayOutputStream(bis);
byte[] buffer = new byte[1024];
int nRead;
while((nRead = bis.read(buffer)) != -1) {
baos.write(buffer, 0, nRead);
}
} catch(IOException ioe) {
ioe.printStackTrace();
}
return baos.toByteArray();
}
This method checks the file type:
private final boolean isMyFileType(byte[] bytes) {
// do stuff
return theBoolean;
}
The reason it is breaking makes sense to me - the byte array ends up being gigantic if I have a gigantic file AND I'm passing around a gigantic byte array.
My goal, I want to read the bytes from a file, determine what type of file it is using another method I wrote, run compression/decompression method against those bytes after determining the file type.
I have most of my goal completed, I just don't know how to handle file streams and large byte arrays effectively.
You are already using a BufferedInputStream. Use the "mark" method to place a mark in the steam. Make sure the "readlimit" argument to "mark" is large enough for you to detect the file type. Read the first X bytes from the stream (but not more than readlimit) and try to figure out the content. Then call reset() to set the stream back to the beginning and continue withw whatever you want to do with the stream.
Using Base64 from Apache commons
public byte[] encode(File file) throws FileNotFoundException, IOException {
byte[] encoded;
try (FileInputStream fin = new FileInputStream(file)) {
byte fileContent[] = new byte[(int) file.length()];
fin.read(fileContent);
encoded = Base64.encodeBase64(fileContent);
}
return encoded;
}
Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: Java heap space
at org.apache.commons.codec.binary.BaseNCodec.encode(BaseNCodec.java:342)
at org.apache.commons.codec.binary.Base64.encodeBase64(Base64.java:657)
at org.apache.commons.codec.binary.Base64.encodeBase64(Base64.java:622)
at org.apache.commons.codec.binary.Base64.encodeBase64(Base64.java:604)
I'm making small app for mobile device.
You cannot just load the whole file into memory, like here:
byte fileContent[] = new byte[(int) file.length()];
fin.read(fileContent);
Instead load the file chunk by chunk and encode it in parts. Base64 is a simple encoding, it is enough to load 3 bytes and encode them at a time (this will produce 4 bytes after encoding). For performance reasons consider loading multiples of 3 bytes, e.g. 3000 bytes - should be just fine. Also consider buffering input file.
An example:
byte fileContent[] = new byte[3000];
try (FileInputStream fin = new FileInputStream(file)) {
while(fin.read(fileContent) >= 0) {
Base64.encodeBase64(fileContent);
}
}
Note that you cannot simply append results of Base64.encodeBase64() to encoded bbyte array. Actually, it is not loading the file but encoding it to Base64 causing the out-of-memory problem. This is understandable because Base64 version is bigger (and you already have a file occupying a lot of memory).
Consider changing your method to:
public void encode(File file, OutputStream base64OutputStream)
and sending Base64-encoded data directly to the base64OutputStream rather than returning it.
UPDATE: Thanks to #StephenC I developed much easier version:
public void encode(File file, OutputStream base64OutputStream) {
InputStream is = new FileInputStream(file);
OutputStream out = new Base64OutputStream(base64OutputStream)
IOUtils.copy(is, out);
is.close();
out.close();
}
It uses Base64OutputStream that translates input to Base64 on-the-fly and IOUtils class from Apache Commons IO.
Note: you must close the FileInputStream and Base64OutputStream explicitly to print = if required but buffering is handled by IOUtils.copy().
Either the file is too big, or your heap is too small, or you've got a memory leak.
If this only happens with really big files, put something into your code to check the file size and reject files that are unreasonably big.
If this happens with small files, increase your heap size by using the -Xmx command line option when you launch the JVM. (If this is in a web container or some other framework, check the documentation on how to do it.)
If the file recurs, especially with small files, the chances are that you've got a memory leak.
The other point that should be made is that your current approach entails holding two complete copies of the file in memory. You should be able to reduce the memory usage, though you'll typically need a stream-based Base64 encoder to do this. (It depends on which flavor of the base64 encoding you are using ...)
This page describes a stream-based Base64 encoder / decoder library, and includes lnks to some alternatives.
Well, do not do it for the whole file at once.
Base64 works on 3 bytes at a time, so you can read your file in batches of "multiple of 3" bytes, encode them and repeat until you finish the file:
// the base64 encoding - acceptable estimation of encoded size
StringBuilder sb = new StringBuilder(file.length() / 3 * 4);
FileInputStream fin = null;
try {
fin = new FileInputStream("some.file");
// Max size of buffer
int bSize = 3 * 512;
// Buffer
byte[] buf = new byte[bSize];
// Actual size of buffer
int len = 0;
while((len = fin.read(buf)) != -1) {
byte[] encoded = Base64.encodeBase64(buf);
// Although you might want to write the encoded bytes to another
// stream, otherwise you'll run into the same problem again.
sb.append(new String(buf, 0, len));
}
} catch(IOException e) {
if(null != fin) {
fin.close();
}
}
String base64EncodedFile = sb.toString();
You are not reading the whole file, just the first few kb. The read method returns how many bytes were actually read. You should call read in a loop until it returns -1 to be sure that you have read everything.
The file is too big for both it and its base64 encoding to fit in memory. Either
process the file in smaller pieces or
increase the memory available to the JVM with the -Xmx switch, e.g.
java -Xmx1024M YourProgram
This is best code to upload image of more size
bitmap=Bitmap.createScaledBitmap(bitmap, 100, 100, true);
ByteArrayOutputStream stream = new ByteArrayOutputStream();
bitmap.compress(Bitmap.CompressFormat.PNG, 100, stream); //compress to which format you want.
byte [] byte_arr = stream.toByteArray();
String image_str = Base64.encodeBytes(byte_arr);
Well, looks like your file is too large to keep the multiple copies necessary for an in-memory Base64 encoding in the available heap memory at the same time. Given that this is for a mobile device, it's probably not possible to increase the heap, so you have two options:
make the file smaller (much smaller)
Do it in a stram-based way so that you're reading from an InputStream one small part of the file at a time, encode it and write it to an OutputStream, without ever keeping the enitre file in memory.
In Manifest in applcation tag write following
android:largeHeap="true"
It worked for me
Java 8 added Base64 methods, so Apache Commons is no longer needed to encode large files.
public static void encodeFileToBase64(String inputFile, String outputFile) {
try (OutputStream out = Base64.getEncoder().wrap(new FileOutputStream(outputFile))) {
Files.copy(Paths.get(inputFile), out);
} catch (IOException e) {
throw new UncheckedIOException(e);
}
}