I have a synchronize method, in whoich i am using datainputstream.readfully() .Its throwing me the "EOF exception". Why the reallyfully method throws EOF when it is still inside the synchronize method? below is the code for reference
private static synchronized String getTransactionId() {
try {
String txnId_fname = SiteConfiguration.getInstance().getProperty("TRANSACTION.INFO_FILE", //
LaneProcessor.DEFAULT_TRANSACTION_ID_FILE_NAME);
File tmpFile = new File(txnId_fname);
if (!tmpFile.exists()) {
tmpFile.createNewFile();
}
else {
long sz = tmpFile.length();
if ( 12 == sz ) {
// read the transaction id from the file, the ID must be 12 bytes long to be valid.
DataInputStream dis = new DataInputStream(new FileInputStream(tmpFile));
byte[] datainBytes = new byte[dis.available()];
dis.readFully(datainBytes);
transactionIdLog = new String(datainBytes, 0, datainBytes.length);
if ( Stringer.isNumeric(transactionIdLog))
{
transactionId = Long.valueOf(transactionIdLog);
}
dis.close();
//log.debug("transaction id from the existing file"+transactionId);
}
}
transactionId = ConvertUtils.incrementLong(transactionId);
transactionIdLog = Long.toString(transactionId);
transactionIdLog = Stringer.zpad(transactionIdLog, 12);
_out = new FileOutputStream(tmpFile);
_out.write(transactionIdLog.getBytes());
_out.flush();
_out.close();
}
catch (Exception e) {
log.error("Error in transaction id generation" + e.getMessage(), e);
}
return transactionIdLog;
}
The contract for available is that it returns an estimate of the number of bytes available; if you try to read that many bytes, the program won't block but it may read fewer bytes than available says. If available's result is too high, then readFully could get an EOF exception. Unfortunately, I tried looking at the source of FileInputStream.available to see how it worked, but it's native, so I can't tell whether it could return a "too large" value. All I can say is, based on the javadoc, I don't think your code is guaranteed to work.
To see whether this really is the problem, I'd recommend having the program output datainBytes.length after the array is created, and then check that against the actual file size.
Will the synchronize method throw EOF Exception?
Literally No. Any exception in the method will be caught and logged. So it won't propagate an EOFException. What is more, there is no throw new EOFException(...).
But could your method catch EOFException and log it? I think the answer is Yes!
The readFully method will throws EOFException if it cannot fill the buffer, and you have set the buffer size to the number of bytes that available() says are readable. But consider this scenario:
Your application executes to the point where available() returns.
Your application is paused (e.g. by the OS scheduler).
Some other application truncates the file.
Your application is resumed, and calls readFully ... only to discover that there are ZERO bytes to be read.
EOFException ...
This illustrates the point that the result of isAvailable() is only a hint. You can't entirely rely on it.
But, I don't think it is technically possible to code that method in such a way that an EOFException cannot occur. You certainly can't do it without some kind of file locking ... to prevent other applications truncating the file while your application is reading it.
Related
Here is my sample snippet for reading and writing by output stream and I am getting out of memory exception.
public static void readFileContent(InputStream in, OutputStream out) throws IOException {
byte[] buf = new byte[500000];
int nread;
int navailable;
int total = 0;
synchronized (in) {
try {
while((nread = in.read(buf, 0, buf.length)) >= 0) {
out.write(buf, 0, nread);
total += nread;
}
}
finally {
if (in != null) {
try {
in.close();
} catch (Exception e) {
e.printStackTrace();
}
}
}
}
out.flush();
buf = null;
}
What are the possible scenarios with the above snippet to get "out of memory exception" ?
Is it necessary to close the output stream here? And does stream, flush is enough or do we need to close the stream always? If so why?
How could I avoid Out of memory exception in general?
Please clarify me.
What are the possible scenarios with the above snippet to get "out of memory exception" ?
There are various root causes for out of memory exceptions. Refer to oracle documentation page for more details.
java.lang.OutOfMemoryError: Java heap space:
Cause: The detail message Java heap space indicates object could not be allocated in the Java heap.
java.lang.OutOfMemoryError: GC Overhead limit exceeded:
Cause: The detail message "GC overhead limit exceeded" indicates that the garbage collector is running all the time and Java program is making very slow progress
java.lang.OutOfMemoryError: Requested array size exceeds VM limit:
Cause: The detail message "Requested array size exceeds VM limit" indicates that the application (or APIs used by that application) attempted to allocate an array that is larger than the heap size.
java.lang.OutOfMemoryError: Metaspace:
Cause: Java class metadata (the virtual machines internal presentation of Java class) is allocated in native memory (referred to here as metaspace)
java.lang.OutOfMemoryError: request size bytes for reason. Out of swap space?:
Cause: The detail message "request size bytes for reason. Out of swap space?" appears to be an OutOfMemoryError exception. However, the Java HotSpot VM code reports this apparent exception when an allocation from the native heap failed and the native heap might be close to exhaustion
Is it necessary to close the output stream here? And does stream, flush is enough or do we need to close the stream always? If so why?
since you are using raw InputStream and OutputStream in your method, we don' t know which type of actual Stream is getting passed to this method and hence explicitly close these Streams is good idea.
How could I avoid Out of memory exception in general?
This question is already answered in response to your first question.
Refer to this SE question on handling large files for IO operations :
Java OutOfMemoryError in reading a large text file
Change the buf to new byte[1*1024]
Read using just buf no need to specify length e.g. pos = in.read(buf)
The rest of the code looks good. No need to increase the memory.
Also, any points of synchronised inputStream?
I think it's obvious that the problem is that you allocate 500000 bytes at once, and they may not be available in the heap at runtime.
Explanation:
I would not suggest it, but you could increment the heap size of your program.
The default heap size for a java program is determined at runtime, but it can also be parameterized.
Recommendation:
As far as I can see by the provided snippet, it's not absolutely necessary to read 500000 bytes at once. So, you can initialize your byte array with a smaller number that would result in having more reading loops. But if it's not a problem for your program... I guess.
Conclusion:
Try by setting the initial byte array size to 5000, or even 1000.
EDIT:
An extra point to take into consideration is that in the above code snippet you only flush once at the end. The bytes you are writting to the OutputStream are kept in memory, and their size may cause an OutOfMemoryException also.
In order to overcome this, you should flush more often. It will affect your performance if you flush too often, but you can always experiment with a condition in your loop e.g.
...
if (total % 5000 == 0) {
out.flush();
}
...
EDIT 2:
As the InputStream and OutputStream objects are passed to the given method as parameters, so, in my opinion this method is not responsible for closing them. The method that initializes the Streams is also responsible for close them gracefully. Flush is enough for this method. But consider doing it in smaller chunks.
EDIT 3:
To summarize the suggested tweaks:
public static void readFileContent(InputStream in, OutputStream out) throws IOException {
byte[] buf = new byte[1000];
// wrap your OutputStream in a BufferedOutputStream
BufferedOutputStream bos = new BufferedOutputStream(out, 5000);
int nread;
int navailable;
int total = 0;
synchronized (in) {
try {
while((nread = in.read(buf, 0, buf.length)) >= 0) {
// use the BufferedOutputStream to write data
// you don't need to flush regularly as it is handled automatically every time the buffer is full
bos.write(buf, 0, nread);
total += nread;
}
}
finally {
if (in != null) {
try {
in.close();
} catch (Exception e) {
e.printStackTrace();
}
}
}
}
// flush the last contents of the BufferedOutputStream
bos.flush();
buf = null;
}
Please note also that BufferedOutputStream will automatically call flush() when you will close it gracefully.
EDIT 4:
Example calling the above method:
public static void main(String[] args) {
String filename = "test.txt";
String newFilename = "newtest.txt";
File file = new File(filename);
File newFile = new File(newFilename);
try (InputStream fis = new FileInputStream(file);
OutputStream fout = new FileOutputStream(newFile)) {
readFileContent(fis, fout);
}
catch(IOException ioe) {
System.out.println(ioe.getMessage());
}
}
In Java there isn't any brute way of freeing memory. Even calling the built-in Garbage Collector (System.gC()) might not solve the problem, as the GC only frees objects that are not referenced anymore. You need to take care of the code that you are writing so that it can employ the resources in its best way it can. Certainly, there are cases that you are left out of options, especially when you are using big or giant data structures regardless of any code optimization you can think of (in your case, you are creating an array with half a million records of bytes).
As a partial solution, you can increase your Heap Size Memory so that Java can allocate more memory.
Is there a way to ask a DataInputStream, if it has content to read? .readByte() will just hang it, waiting for a byte to be read :( Or do I always have to send a Dummy-Byte, to make sure it always sees something?
dis.available();
Returns:
an estimate of the number of bytes that can be read (or skipped over) from this input stream without blocking.
Is this what you looking for?
also check answers here. You might get even more informations. "available" of DataInputStream from Socket
Look at
public int available() throws IOException
according to docs it "Returns an estimate of the number of bytes that can be read"
so you should call dis.available()
When reading past the end of the file, an EOFException is thrown. So you can tell there's no more data to read. For examle:
DataInputStream inputStream = new DataInputStream(new FileInputStream(file));
int data = 0;
try {
while (true) {
data += inputStream.readInt();
}
} catch (EOFException e) {
System.out.println("All data were read");
System.out.println(data);
}
I am trying to read text file whilst running the program from a jar archive.
I come accros that I need to use InputStream to read file. The snippet of code:
buffer = new BufferedInputStream(this.getClass().getResourceAsStream((getClass().getClassLoader().getResource("English_names.txt").getPath())));
System.out.println(buffer.read()+" yeas");
At this line System.out.println(buffer.read()+" yeas"); program stops and nothing happens since then. Once you output the contents of buffer object it is not null.
What might be the problem?
From InputStream#read():
This method blocks until input data is available, the end of the stream is detected, or an exception is thrown.
So basically, the stream appears to be waiting on content. I'm guessing it's how you've constructed the stream, you can simplify your construction to:
InputStream resourceStream = getClass().getResourceAsStream("/English_names.txt");
InputStream buffer = new BufferedInputStream(resourceStream);
I'd also check to make sure that resourceStream is not-null.
You should not worry about InputStream being null when passed into BufferedInputStream constructor since it, the constructor handles null parameters just fine. When supplied with null it will just return null without throwing any exception. Also since InputStream implements AutoClosable the try-with-resources block will take care of closing your streams properly.
try (
final InputStream is = getClass().getResourceAsStream("/English_names.txt");
final BufferedInputStream bis = new BufferedInputStream(is);
) {
if (null == bis)
throw new IOException("requsted resource was not found");
// Do your reading.
// Do note that if you are using InputStream.read() you may want to call it in a loop until it returns -1
} catch (IOException ex) {
// Either resource is not found or other I/O error occurred
}
Following is a part of the code snippet that I will be using for my project.
public String fetchFromStream()
{
try
{
int charVal;
StringBuffer sb = new StringBuffer();
while((charVal = inputStream.read()) > 0) {
sb.append((char)charVal);
}
return sb.toString();
} catch (Exception e)
{
m_log.error("readUntil(..) : " + e.getMessage());
return null;
} finally {
System.out.println("<<<<<<<<<<<<<<<<<<<<<< Called >>>>>>>>>>>>>>>>>>>>>>>>>>>");
}
}
Initially the while loop start working pretty fine. But after the probable last character is read from the stream I was expecting to get -1 return value. But this is where my problem starts. The code gets hanged, even the finally block is not executed.
I was debugging this code in Eclipse to see what is actually happening during the run-time. I set a pointer (debug) inside the while loop and was constantly monitoring the StringBuffer getting populated with char values one by one. But suddenly while checking the condition inside the while loop, the debugging control is getting lost and this is the point where the code goes to hangup state !! No exception is thrown as well !!
What is happening here ?
Edit::
This is how I'm getting my InputStream. Basically I'm using Apache Commons Net for Telnet.
private TelnetClient getTelnetSession(String hostname, int port)
{
TelnetClient tc = new TelnetClient();
try
{
tc.connect(hostname, port != 0 ? port : 23);
//These are instance variables
inputStream = tc.getInputStream();
outputStream = new PrintStream(tc.getOutputStream());
//More codes...
return tc;
} catch (SocketException se)
{
m_log.error("getTelnetSession(..) : " + se.getMessage());
return null;
} catch (IOException ioe)
{
m_log.error("getTelnetSession(..) : " + ioe.getMessage());
return null;
} catch (Exception e)
{
m_log.error("getTelnetSession(..) : " + e.getMessage());
return null;
}
}
Look at the JavaDocs:
Reads the next byte of data from the input stream. The value byte is returned as an int in the range 0 to 255. If no byte is available because the end of the stream has been reached, the value -1 is returned. This method blocks until input data is available, the end of the stream is detected, or an exception is thrown.
In simple turns: if your stream ended (e.g. end of file), read() returns -1 immediately. However if the stream is still open but JVM is waiting for data (slow disk, socket connection), read() will block (not really hung).
Where are you getting the stream from? Check out the available() - but please do not call it in a loop exhausting CPU.
Finally: casting int/byte to char will only work for ASCII characters, consider using Reader on top of InputStream.
read the docs
read() will wait until there is more data on the InputStream if the InputStream is not closed.
I suspect you are doing this with sockets? This is the most common area where this comes up.
"Reads the next byte of data from the input stream. The value byte is returned as an int in the range 0 to 255. If no byte is available because the end of the stream has been reached, the value -1 is returned. This method blocks until input data is available, the end of the stream is detected, or an exception is thrown"
I have the same issue with the Apache Commons on Android ...
the read() command on the inputstream hangs forever for some reason. And no, it is not just blocking "until data is available" ...
my debugging information shows that there are several 100 chars available() ... yet it just randomly blocks at some read. However, whenever I send something to the telnet server the block is suddenly released and it will continue reading for several chars until it suddenly stops/blocks again at some arbitrary point!
I believe there is some bug within the Apache Commons library! This is really annoying because there isn't a lot that can be done ... no timeout for the read command or anything else ...
EDIT: I was able to get around it ... by setting the TelNetClient.setReaderThread(false) ... obviously there is a bug within the Library that exists as long as a thread handles the input data ... when dispabled it works just fine for me!
I just wanted to see if there was a better way I should be handling this. My understanding of streams is that as long as you close a stream, any streams encapsulated within it will be closed which is why I only close TarArchiveOutputStream in finally. If I get FileNotFound on the rawDir or archiveDir I want to log it, otherwise anything else I want to throw.
public static void createTarGzOfDirectory(File rawDir, File archiveFile) throws IOException {
FileOutputStream fOut = null;
BufferedOutputStream bOut = null;
GzipCompressorOutputStream gzOut = null;
TarArchiveOutputStream tOut = null;
try {
fOut = new FileOutputStream(archiveFile);
bOut = new BufferedOutputStream(fOut);
gzOut = new GzipCompressorOutputStream(bOut);
tOut = new TarArchiveOutputStream(gzOut);
addFileToTarGz(tOut, rawDir, "");
} catch (FileNotFoundException e) {
log.error("File not found: " + e);
} finally {
if(tOut != null) {
tOut.finish();
tOut.close();
}
}
Any other considerations or thoughts on improving things?
My understanding of streams is that as long as you close a stream, any streams encapsulated within it will be closed ...
That is correct.
However, your code is (effectively) assuming that if tOut is null, then none of the other streams in the chain have been created. That's a somewhat dodgy assumption. Consider this sequence:
The FileOutputStream is created and is assigned to fOut.
The BufferedOutputStream is created and is assigned to bOut.
The GzipCompressorOutputStream constructor throws an exception or error. (Maybe the heap is full ...).
The catch is skipped ... wrong exception.
The finally checks tOut, finds it is null, and does nothing.
Net result: we've leaked the file descriptor / channel held by the FileOUtputStream.
The key to getting this example (absolutely) right is to understand which of those stream objects holds the critical resources, and ensuring that THAT stream gets closed. The other streams that don't hold resources don't have to be closed.
} finally {
if (fOut != null) {
fOut.close();
}
}
The other point is that you need to move the tOut.finish() call into the try block after the addFileToTarGz call.
If the addFileToTarGz call throws an exception, or if you don't get that far, the finish call is a waste of time.
The finish call will attempt to write the index to the archive, and THAT could throw an IOException. If this happens in the finally block, then any following code in the finally block to close the stream chain won't get executed ... and a file descriptor will be leaked.
Although it would look ugly and is,maybe, unlikely to be the case, you should close them all in cascade. Yes, if you close the TarArchiveOutputStream, it is supposed to close the underlyning streams. But, depending on the implementation, it may not always be the case. Moreover, and probably mainly, if one of the intermediate constructors throw an exception, tOut will be null, but the other ones may not be. Meaning that your streams are opened but your did not close any.
You could chain all your constructors together like so:
tOut = new TarArchiveOutputStream(new GzipCompressorOutputStream(new BufferedOutputStream(new FileOutputStream(archiveFile))));
And save yourself 6 lines of initialization and 3 local variables for debugging. Not everyone likes chaining things that way - I personally find it more readable but the rest of your team may prefer it your way.
As far as closing the stream, it looks correct to me.