htsjdk intermittent indexed fasta read errror - java

I have an issue with intermittently getting an exception from htsjdk.samtools.reference.IndexedFastaSequenceFile:
htsjdk.samtools.SAMException: Sequence dictionary and index contain different numbers of contigs
or
htsjdk.samtools.SAMException: Unable to get ...
The problem is that I'm invoking the same code over and over and only getting this error intermittently.
I propose that the error message is in fact misleading, particularly in the case of the sequence dictionary response.

I eventually tracked this down to a thread interrupt exception.
The program was using multiple threads each with it's own instance of the fasta reader. I redesigned the software so that I do not call interrupt on the thread and close by a different mechanism but this took quite a while to track down.
Perhaps obvious in retrospect.
Suggestion, htsjdk.samtools should try to catch interrupt exceptions so that it's easier to track down what the problem is.

Related

Is catching Throwable in schedulers a good practice?

I rather say that fail fast and not catching Throwable is a good practice. But in case unhandled exception in request processor like StackOverflowError, the task may be stopped. And that doesn't always sound good. I'd rather catch StackOverflowError over and over, but some tasks might be processed. What is good practice here?
There is no "good practice" or "best practice"1 about catching Error and hence Throwable.
On the one hand, a JVM can recover successfully from some kinds of Error in some circumstances. For example, if you are running "task" on a single thread that doesn't interact with other threads (directly or indirectly), then that thread can safely recover from a StackOverflowError and possibly2 a OutOfMemoryError.
On the other hand, many Error subclasses indicate that the application or JVM is in a state where recovery is not possible, or not practical:
A class loading or initialization Error means that certain classes will be an unusable state. An application that depends the class won't be able to proceed.
If an application that uses notify / wait or higher level synchronization constructs gets (say) an OutOfMemoryError error on one thread, it is liable to leave other threads waiting for notifications, etc that may never arrive.
My recommendation would be to heed the implied advice in the javadoc for Error and not to catch and attempt recover from Error or its subclasses. If you attempt to recover, allow for the possibility that the application may "wedge".
1 - Actually, there is not "best practice" at all; see https://www.satisfice.com/blog/archives/5164
2 - This depends on the root cause. If the root cause is a memory leak elsewhere in your codebase, then recovery is a bad idea. The OOME is likely to recur ... with increasing frequency.
This is from the javadoc of Error :
An Error is a subclass of Throwable that indicates serious problems that a reasonable application should not try to catch.
I usually follow this rule to avoid suppressing real unexpected errors that should be fixed.
You mentionned StackoverflowError, if you have this kind of error, there's probably a problem with your algorithm and your code should be optimized.
If you know that a specific error is likely to be thrown and you're okay with that, you can catch it.
However if you don't expect it, it's better to raise the alarm ASAP and deal with it.
Good practice is catch scheduled jobs, write log about exception, metrics and(optional) reschedule it.
Also, sometimes your thread will be stoped/destroyed (hello spring schedulers) cause you receive some sort of exception, that not critical to your job...

How do I prevent an exception from stopping my program?

I'm using a thread to run a half of an application, and the other half is running by itself. I did this so that the one part fetching the data (because I'm running an infinite loop) would not stop the other part from doing it's job. Now I was wondering: if an exception occurred on one part it would block the some of the other part because they are dependent on each other.
So my question is, how could I prevent an exception from stopping my whole program and make it continue do things?
Try-catch or throw an exception. If neither of those sound familiar, I would look them up.
an example of try-catch
try
{
//your code that may throw exception
}catch(TheException e){
e.printStackTrace(); // or however you handle exceptions
}
good resource for throwing exceptions: https://docs.oracle.com/javase/tutorial/essential/exceptions/throwing.html
Throwing an exception from one thread generally will not impact the runtime of the others. Threads are kind of off in their own little world in that regard.
That said, if two threads are sharing some state (whether directly through the heap or indirectly through a database or something like that) and one thread leaves the application's data in an unstable/corrupted state because it bombed out, that could obviously impact the behavior of the other threads, making them do further "bad stuff." You will want to prevent this whenever possible; best way to do it is to document your exceptions as best you can and to catch and handle them when thrown, and to back your logic with robust unit tests.
You can create a boolean variable maybe called running so that while your while loop is working fine, this variable will be true. Immediately an exception is caught, the variable becomes false. Your main task can then check the variable anytime it wants to use the shared data. If the variable is true, read the data; else, do something else like restarting the thread depending on the type of exception caught.
Remember, in order to make the variable false, you will need to include its update in your catch clause.
There are better and more complex methods out there like the use of ExecutorService.

Java: Call method on all caught exceptions [duplicate]

This question already has answers here:
Java: Global Exception Handler
(6 answers)
Closed 8 years ago.
I know I can set my own UncaughtExceptionHandler to execute code on uncaught exceptions, but is there a way to have some code execute on all caught exception without having to call that method in every single try-catch block?
My custom code sends me an email with the stack trace on uncaught exceptions. I also want that email for caught exception but I don't want to track down every single try-catch statement and add it.
I'm not looking for a global exception handler (I mentioned UncaughtExceptionHandler), I'm looking for a global CAUGHT exception handler.
You can do it by altering the Exception class to record every Exception. This can be done by changing the boot class path, runtime code injection or using the JVMTI.
I suspect it's a bad idea as the system will catch more exception than you might expect. You could get thousands of emails a second if something goes wrong.
I suggest you revisit your exceptions handling policy and strategy. When an exception happens it generally means something went wrong, i.e., something exceptional (as in unexpected) happened in the code (database connection lost, file not found, network unreachable, etc.) or a bug in the code causes an unexpected problem (NPE comes to mind...). In these scenarios the normal program flow can't continue.
It's usually considered best practice to fail fast in such cases, unless you are expecting some exception to be thrown and know how to recover, i.e., the exceptional condition wasn't that exceptional after all.
If you need to fail fast, you don't need to explicitly handle the exceptions as your thread or program will just die and you can just use the UncaughtExceptionHandler to log the stack trace, send emails or notifications etc.
Situation in which you can successfully recover and resume the normal program flow are far less common in my opinion and usually do not require developers/ops to be notified as it is part of the normal functioning of the system (otherwise you would not be able to handle the exception). So simply logging the specific message and useful data associated to this scenario is usually enough, e.g., "WARNING: Transient error x was recovered. State was y and z.".

Thread Handling after exception

used thread.sleep when exception occurs in my application
if any thing exception occured in any one of the thread it should wait for some time and have to try again
what i am facing the issue means for 100 request if exception occurs for 50 means that should be wait for some time and after try again if any
exception occurs means it should also wait till success
am feeling this going to affect the performance and system usage.
Please let me know your suggestions. Is it better to create new thread if any exception occurs?
Or Any other way?
Retrying on exception isn't a bad idea, but you should make sure that:
The exception is transient (i.e. you have reasons to believe it may
succeed on retry, like a network error)
You have ways to prevent retrying forever (like "max retry")
You don't overload the system by too frequent retries by e.g. using exponential back-off strategy (or even just a reasonably long wait time)
Exception to rule 2 is possible (sometimes you do want stuff to retry forever).
From the question, I understand that, you are talking about RunTimeExceptions.
1) Firstly I would like to suggest that, you should examine the code areas which may really throw the exceptions at run time i.e processing of files.
2) another thing is that you have to handle this scenarios so that exception cases are reduced. i.e first check file is available then only try to read it, check permissions before writing it etc. In this way frequency of retrying may be reduced.
3) If both above actions fails, means the situation which can never be handled such as services are down etc., you can retry the considering all necessary aspects like - MAX try, MAX time out foe which you are re-trying, leave the attempt after certain amount of try OR time etc.
Hope this helps.

When is it OK to catch a RuntimeException

On a recent project I recommended catching a RuntimeException within a test harness code and logging it. The code processes a series of inputs from a database, and I do not want the test to stop due to failure of any one input (Null values, Illegal arguments, etc.). Needless to say, my suggestion triggered a passionate discussion.
Is catching any kind of RuntimeException acceptable? If yes, what are other scenarios where it is OK to catch RuntimeExceptions?
You catch RuntimeException for the same reason that you catch any exception: You plan to do something with it. Perhaps you can correct whatever caused the exception. Perhaps you simply want to re-throw with a different exception type.
Catching and ignoring any exception, however, is extremely bad practice.
Unless you can correct a RuntimeException, you don't want to catch it...
...only true from a developers point of view....
you have to catch all exceptions before they reach up to the UI and make your user sad. This means on the "highest level" you want to catch anything that happend further down. Then you can let the user know there was a problem and at the same time take measures to inform the developers, like sending out alarm mails or whatever...
It is basically considered a data/programming error that could not be forseen, thus you want to improve future releases of the software while at the same time take the user by the hand and move on in a controlled manner...
RuntimeException is intended to be used for programmer errors. As such it should never be caught. There are a few cases where it should be:
you are calling code that comes from a 3rd party where you do not have control over when they throw exception. I would argue that you should do this on a case by case basis and wrap the usage of the 3rd party code within your own classes so you can pass back non-runtime exceptions.
your program cannot crash and leave a stack trace for the user to see. In this case it should go around main and around any threads and event handling code. The program should probably exit when such exception occurs as well.
In your specific case I would have to question why you are having RuntimeExceptions occur in the tests - you should be fixing them instead of working around them.
So you should guarantee that your code only throws RuntimeExceptions when you want to have the program exit. You should only catch RuntimeExceptions when you want to log it and exit. That is what is in line with the intent of RuntimeExceptions.
You can look at this discussion for some other reasons that people give... I personally haven't found a compelling reason in the answers though.
In my code 99% of my exceptions are derived from runtime_exception.
The reasons I catch exceptions are:
Catch Log and Fix problem.
Catch Log and Generate a more specific exception and throw
Catch Log and rethrow.
Catch Log and Kill operation (discard exception)
User/request initiated action fails.
An HTTP request handler for example. I would rather the requested operation die rather than bring the Service down. (Though preferably the handler has enough sense to return a 500 error code.)
Test case passed/failed with an exception.
All exceptions not in the main thread.
Allowing exceptions to escape a thread is usually badly documented but usually causes program termination (without stack unwinding).
Years ago, we wrote a control system framework and the Agent objects caught runtime exceptions, logged them if they could and continued.
Yes we caught Runtime exceptions including OutOfMemory in our framework code( and forced a GC, and it's surprising how well that kept even quite leaky code running.)
We had code that was doing very mathematical things involving the real world; and from time to time a Not-A-Number would get in due to tiny rounding errors and it coped okay with that too.
So in framework / "must not exit" code I think it can be justifiable. And when it works it's pretty cool.
The code was pretty solid, but it ran hardware, and hardware tends to give screwy answers sometimes.
It was designed to run without human intervention for months at a time.
It worked extremely well in our tests.
As part of the error recovery code, it could resort to rebooting the entire building using the UPS's ability to turn off in N minutes and turn on in M minutes.
Sometimes hardware faults need to power cycled :)
If I remember, the last resort after an unsuccessful power cycle was it sending an email to it's owners, saying
"I tried to fix myself and I can't; the problem is in subsystem XYZ", and included a link to raise a support call back to us.
Sadly the project got canned before it could become self aware :)>
Personally, I've always been told that you want to catch all RuntimeExceptions; however, you also want to do something about the exception, such as running a failsafe or possibly just informing the user that an error occurred.
The last Java project that I worked on had a similar approach, at the very least, we would log the exception so that if a user called complaining about a bug, we could find out exactly what happened and see where the error occurred.
Edit 1: As kdgregory said, catching and ignoring are two different things, generally, people are opposed to the latter :-)
We all know that checked exceptions and RuntimeExceptions are the two categories of exceptions. It is always suggested that we handle (either try-catch or throw) the checked exceptions because they are the programming conditions where unfortunately programmer can not to do anything on its own;
Like FileNotFoundException it is not the programmer who puts files on user's drive if program is actually trying to read the file 1.txt which is supposed to be there on f:\ of user with the statements:
File f11 = new File("f:\\1.txt");
FileInputStream fos = new FileInputStream(f11);
If the file is found it's all ok, but what happens in the other case if the file is not found is that, program crashes down with 0 error from the user. In this scenario programmer did not do anything wrong. This could be a checked exception which must be caught for the program to continue running.
Let me also explain the second scenario with which the concept of RuntimeException will be clear. Consider following code:
int a = {1,2,3,4,5};
System.out.println(a[9]);
This is poor coding which generates the ArrayIndexOutOfBoundsException. Which is an example of RuntimeException. So programmer should not actually handle the exception, let it crash the program, and later fix the logic.
You catch RuntimeException when you want to process it. Maybe you want to rethrow it as a different exception or log it to a file or database, or you want to turn on some exception flag in the return type, etc.
You catch RuntimeExceptions (in any language: unexpected exceptions/“all” exceptions) when your program is doing multiple subtasks and it makes sense to complete every one you can rather than stopping on the first unexpected situation. A test suite is a fine situation to do this — you want to know which of all the tests failed, not just the first test. The key characteristic is that each test is independent of all the others — it doesn't matter whether a previous test doesn't run because the order is not significant anyway.
Another common situation is a server; you don’t want to shut down just because one request was malformed in a way you didn't expect. (Unless it’s really, really important to minimize the chances of inconsistent state.)
In any of these situations, the appropriate thing to do is log/report the exception and continue with the remaining tasks.
One could vaguely generalize to any exception: it is “appropriate to catch” an exception if and only if there is something sensible to do after catching it: how your program should continue.
If a client can reasonably be expected to recover from an exception, make it a checked exception.
If a client cannot do anything to recover from the exception, make it an unchecked exception.
Here's the bottom line guideline.
From Java Docs. Please read this Unchecked Exceptions — The Controversy

Categories