We Keep Coding

Java OutOfMemoryError while merge large file parts from chunked files

I have a problem when the user upload large files (> 1 GB) (I'm using flow.js library), it creates hundred of thousand small chunked files (e.g 100KB each) inside temporary directory but failed to merge into single file, due to MemoryOutOfException. This is not happened when the file is under 1 GB. I know it sound tedious and you probably suggest me to increase the XmX in my container-but I want to have another angle besides that.
Here is my code
private void mergeFile(String identifier, int totalFile, String outputFile) throws AppException{
File[] fileDatas = new File[totalFile]; //we know the size of file here and create specific amount of the array
byte fileContents[] = null;
int totalFileSize = 0;
int filePartUploadSize = 0;
int tempFileSize = 0;
//I'm creating array of file and append the length
for (int i = 0; i < totalFile; i++) {
fileDatas[i] = new File(identifier + "." + (i + 1)); //indentifier is the name of the file
totalFileSize += fileDatas[i].length();
}
try {
fileContents = new byte[totalFileSize];
InputStream inStream;
for (int j = 0; j < totalFile; j++) {
inStream = new BufferedInputStream(new FileInputStream(fileDatas[j]));
filePartUploadSize = (int) fileDatas[j].length();
inStream.read(fileContents, tempFileSize, filePartUploadSize);
tempFileSize += filePartUploadSize;
inStream.close();
}
} catch (FileNotFoundException ex) {
throw new AppException(AppExceptionCode.FILE_NOT_FOUND);
} catch (IOException ex) {
throw new AppException(AppExceptionCode.ERROR_ON_MERGE_FILE);
} finally {
write(fileContents, outputFile);
for (int l = 0; l < totalFile; l++) {
fileDatas[l].delete();
}
}
}
Please show the "inefficient" of this method, once again... only large files that cannot be merge using this method, smaller one ( < 1 GB) no problem at all....
I appreciate if you do not suggest me to increase the heap memory instead show me the fundamental error of this method... thanks...
Thanks

It's unnecessary to allocate the entire file size in memory by declaring a byte array of the entire size. Building the concatenated file in memory in general is totally unnecessary.
Just open up an outputstream for your target file, and then for each file that you are combining to make it, just read each one as an input stream and write the bytes to outputstream, closing each one as you finish. Then when you're done with them all, close the output file. Total memory use will be a few thousand bytes for the buffer.
Also, don't do I/O operations in finally block (except closing and stuff).
Here is a rough example you can play with.
ArrayList<File> files = new ArrayList<>();// put your files here
File output = new File("yourfilename");
BufferedOutputStream boss = null;
try
{
boss = new BufferedOutputStream(new FileOutputStream(output));
for (File file : files)
{
BufferedInputStream bis = null;
try
{
bis = new BufferedInputStream(new FileInputStream(file));
boolean done = false;
while (!done)
{
int data = bis.read();
boss.write(data);
done = data < 0;
}
}
catch (Exception e)
{
//do error handling stuff, log it maybe?
}
finally
{
try
{
bis.close();//do this in a try catch just in case
}
catch (Exception e)
{
//handle this
}
}
}
} catch (Exception e)
{
//handle this
}
finally
{
try
{
boss.close();
}
catch (Exception e) {
//handle this
}
}

... show me the fundamental error of this method
The implementation flaw is that you are creating a byte array (fileContents) whose size is the total file size. If the total file size is too big, that will cause an OOME. Inevitably.
Solution - don't do that! Instead "stream" the file by reading from the "chunk" files and writing to the final file using a modest sized buffer.
There are other problems with your code too. For instance, it could leak file descriptors because you are not ensure that inStream is closed under all circumstances. Read up on the "try-with-resources" construct.

Why does my program stop running and does not return error?

I put the declaration in the while loop, and the program would not running and also does not return any error. I suspect the while loop become an infinite loop.
try
{
while (true)
{
inputStream = new ObjectInputStream (new FileInputStream (fileName));
Ship copyObject = (Ship) inputStream.readObject();
String nameCompany = copyObject.getCompanyName();
if (compName.equalsIgnoreCase(nameCompany)){
listShipName += (copyObject.getShipName() + ", ");
numberOfShip ++;
}
}
}
catch (EOFException e)
{
}
catch (Exception e)
{
e.printStackTrace();
}
But if I put the declaration of input stream out of the while loop, the program runs successfully. Can someone explain why this happens?
try
{
inputStream = new ObjectInputStream (new FileInputStream (fileName));
}
catch (Exception e)
{
e.printStackTrace();
}
try
{
while (true)
{
Ship copyObject = (Ship) inputStream.readObject();
String nameCompany = copyObject.getCompanyName();
if (compName.equalsIgnoreCase(nameCompany)){
listShipName += (copyObject.getShipName() + ", ");
numberOfShip ++;
}
}
}
catch (EOFException e)
{
}
catch (Exception e)
{
e.printStackTrace();
}

You're reopening your file on every iteration through the loop, which means you are only ever reading the first object from the file. But you're reading the same object over and over again.
As well as opening your file only once, you really should try to detect the end of file without throwing an exception. As a matter of style, exceptions should be thrown when things go wrong, not as a matter of course.

Now I realize that in each iteration, I reopen the input stream, so the loop would not reach to the end of the file, and it becomes infinite.

EOFException - how to handle?

I'm a beginner java programmer following the java tutorials.
I am using a simple Java Program from the Java tutorials's Data Streams Page, and at runtime, it keeps on showing EOFException. I was wondering if this was normal, as the reader has to come to the end of the file eventually.
import java.io.*;
public class DataStreams {
static final String dataFile = "F://Java//DataStreams//invoicedata.txt";
static final double[] prices = { 19.99, 9.99, 15.99, 3.99, 4.99 };
static final int[] units = { 12, 8, 13, 29, 50 };
static final String[] descs = {
"Java T-shirt",
"Java Mug",
"Duke Juggling Dolls",
"Java Pin",
"Java Key Chain"
};
public static void main(String args[]) {
try {
DataOutputStream out = new DataOutputStream(new BufferedOutputStream(new FileOutputStream(dataFile)));
for (int i = 0; i < prices.length; i ++) {
out.writeDouble(prices[i]);
out.writeInt(units[i]);
out.writeUTF(descs[i]);
}
out.close();
} catch(IOException e){
e.printStackTrace(); // used to be System.err.println();
}
double price;
int unit;
String desc;
double total = 0.0;
try {
DataInputStream in = new DataInputStream(new BufferedInputStream(new FileInputStream(dataFile)));
while (true) {
price = in.readDouble();
unit = in.readInt();
desc = in.readUTF();
System.out.format("You ordered %d" + " units of %s at $%.2f%n",
unit, desc, price);
total += unit * price;
}
} catch(IOException e) {
e.printStackTrace();
}
System.out.format("Your total is %f.%n" , total);
}
}
It compiles fine, but the output is:
You ordered 12 units of Java T-shirt at $19.99
You ordered 8 units of Java Mug at $9.99
You ordered 13 units of Duke Juggling Dolls at $15.99
You ordered 29 units of Java Pin at $3.99
You ordered 50 units of Java Key Chain at $4.99
java.io.EOFException
at java.io.DataInputStream.readFully(Unknown Source)
at java.io.DataInputStream.readLong(Unknown Source)
at java.io.DataInputStream.readDouble(Unknown Source)
at DataStreams.main(DataStreams.java:39)
Your total is 892.880000.
From the Java tutorials's Data Streams Page, it says:
Notice that DataStreams detects an end-of-file condition by catching EOFException, instead of testing for an invalid return value. All implementations of DataInput methods use EOFException instead of return values.
So, does this mean that catching EOFException is normal, so just catching it and not handling it is fine, meaning that the end of file is reached?
If it means I should handle it, please advise me on how to do it.
EDIT
From the suggestions, I've fixed it by using in.available() > 0 for the while loop condition.
Or, I could do nothing to handle the exception, because it's fine.

While reading from the file, your are not terminating your loop. So its read all the values and correctly throws EOFException on the next iteration of the read at line below:
price = in.readDouble();
If you read the documentation, it says:
Throws:
EOFException - if this input stream reaches the end before reading eight bytes.
IOException - the stream has been closed and the contained input stream does not support reading after close, or another I/O error occurs.
Put a proper termination condition in your while loop to resolve the issue e.g. below:
while(in.available() > 0) <--- if there are still bytes to read

The best way to handle this would be to terminate your infinite loop with a proper condition.
But since you asked for the exception handling:
Try to use two catches. Your EOFException is expected, so there seems to be no problem when it occures. Any other exception should be handled.
...
} catch (EOFException e) {
// ... this is fine
} catch(IOException e) {
// handle exception which is not expected
e.printStackTrace();
}

You can use while(in.available() != 0) instead of while(true).

Alternatively, you could write out the number of elements first (as a header) using:
out.writeInt(prices.length);
When you read the file, you first read the header (element count):
int elementCount = in.readInt();
for (int i = 0; i < elementCount; i++) {
// read elements
}

You may come across code that reads from an InputStream and uses the snippet
while(in.available()>0) to check for the end of the stream, rather than checking for an
EOFException (end of the file).
The problem with this technique, and the Javadoc does echo this, is that it only tells you the number of blocks that can be read without blocking the next caller. In other words, it can return 0 even if there are more bytes to be read. Therefore, the InputStream available() method should never be used to check for the end of the stream.
You must use while (true) and
catch(EOFException e) {
//This isn't problem
} catch (Other e) {
//This is problem
}

You catch IOException which also catches EOFException, because it is inherited. If you look at the example from the tutorial they underlined that you should catch EOFException - and this is what they do. To solve you problem catch EOFException before IOException:
try
{
//...
}
catch(EOFException e) {
//eof - no error in this case
}
catch(IOException e) {
//something went wrong
e.printStackTrace();
}
Beside that I don't like data flow control using exceptions - it is not the intended use of exceptions and thus (in my opinion) really bad style.

Put your code inside the try catch block:
i.e :
try{
if(in.available()!=0){
// ------
}
}catch(EOFException eof){
//
}catch(Exception e){
//
}
}

EOFException being a child of IOException
I prefer it like below ==>
try {
.
.
.
} catch (IOException e) {
if (!(e instanceof EOFException)) {
throw new RuntimeException(e);
}
}

input.read() func. lock in the while loop

After connecting to server, I run some commands on server and then trying to take the server knowledge to console with;
while(i!=-1){
String c="";
String line = "";
try {
while ((i = input.read()) != 10 && i!=-1) {
bx[0] = (byte) i;
c = new String(bx);
line = line + c ;
System.out.print(c);
}
} catch (IOException e2) {
e2.printStackTrace();
}
File outfile = new File("calltrak.txt");
boolean append = true;
try
{
if (!outfile.exists())
{
append = false;
}
FileWriter fout1 = new FileWriter("calltrak.txt",append);
PrintWriter fileout = new PrintWriter(fout1,true);
fileout.println(line);
fileout.flush();
fileout.close();
} catch (IOException e1) {
e1.printStackTrace();
}
disp.append(line);
}
But the problem is when the program read all lines from server windows, in server it waits to new input and my prog still tring to read the line and so it locked... How can I solve this problem... (Note:Using a timer isn't a way to solve because the lines which the program read can be 100 line or 100000 and sometimes server can work slow) (In the code "disp" is Jpanel name)

I solved this problem with using paralel thread. With starting Inputstream read method I also started another thread and put inside of it a timer.If read method wait more than 5 seconds, other thread sen -1 to first loop and so loop terminated.

There are several problems performance wise with your code, but to answer your question you should let the server send a EndOfText 0x3 or EndOfTransmission 0x4 at the end see AsciiTable this way you can terminate then.

Android - Socket time out

I have some strange socket behavior going on. I've set an timeout of 5 seconds using setSoTimeout. This should be plenty of time in my situation. According to online java documentation a SocketTimeoutException should be thrown if it times out. It also says that the socket is still valid. So I want to catch it and then continue. However instead of the inner catch, the outer catch IOException is catching the expception and when I output to the log the details it says it was a SocketTimeoutException. Another perplexing thing is I change the timeout from 5 seconds to say, 15 seconds and log the amount of time it take for every read, the times are always in the milli-second range, never even close to a second. Any ideas are GREATLY appreciated.
ReadThread code snippet
#Override
public void run()
{
try
{
while (true)
{
byte[] sizeBuffer = new byte[BYTES_FOR_MESSAGE_SIZE];
int bytesRead = this.inputStream.read(sizeBuffer);
int length = 0;
for (int i = 0; i < BYTES_FOR_MESSAGE_SIZE; i++)
{
int bitsToShift = 8 * i;
int current = ((sizeBuffer[i] & 0xff) << bitsToShift);
length = length | current;
}
byte[] messageBuffer = new byte[length];
this.socket.setSoTimeout(5000); //5 second timeout
try
{
this.inputStream.read(messageBuffer);
}
catch(java.net.SocketTimeoutException ste)
{
Log.e(this.toString(), "---- SocketTimeoutException caught ----");
Log.e(this.toString(), ste.toString());
}
}
}
catch (IOException ioe)
{
Log.e(this.toString(), "IOException caught in ReadThread");
Log.e(this.toString(), ioe.toString());
ioe.printStackTrace();
}
catch (Exception e)
{
Log.e(this.toString(), "Exception caught in ReadThread");
Log.e(this.toString(), e.toString());
e.printStackTrace();
}
this.interfaceSocket.socketClosed();
}// end run

I agree with Brian. You are probably getting the timeout on the first read, not the second. The timeout once set remains in effect until you change it again.
Your second read call where you read the 'message' seems to assume (a) that it will read the entire message and (b) that it will timeout if the entire message doesn't arrive within 5s. It doesn't work like that. It will timeout if nothing arrives within 5s, or else it will read whatever has arrived, up to message.length. But it could only be one byte.
You should use DataInputStream.readFully() to read the entire message, and you need to completely reconsider your timeout strategy.

The exception is probably caught in the first try catch because of the earlier call to this.inputStream.read(). You have two of these calls: one in the outer try, one in the inner try.
Have you validated if data is being read? If data is being read then you should expect the read operation to return after a few milliseconds. If data is not being read, then the read operation should block there for the time you specify. Maybe this has to do with the order by which you setSoTimeout (perhaps doing it earlier will help).
Good luck,
B-Rad