Whenever our code throws an exception that says com.netflix.hystrix.exception.HystrixRuntimeException: MyClass timed-out and fallback failed., we always have to second guess ourselves as to whether or not the problem is that Hystrix is configured correctly. An easy way we could answer that question would be if the log said how long it took for the Hystrix thread to run before it errored. That way, if it says 1000ms, we know that Hystrix is not configured correctly (because that's the default) and if it says 5000ms, we know that it is configured the way we want.
you can do something like this:
command.getExecutionTimeInMilliseconds()
this method tells you how many seconds your run method took, to capture in a timeout you can wrap this in a try catch inside your command class
#Override
protected HttpResponse run() throws Exception {
try {
return executeSomeCall();
} catch (Exception e) {
System.out.println(getExecutionTimeInMilliseconds());
throw e;
}
}
or outside it on the invocation
try {
HttpResponse response = command.execute();
return response;
}
catch (HystrixRuntimeException e) {
command.getExecutionTimeInMilliseconds() // do something with the info
}
just one point, if you use Threads strategy its ok, but if you use Semaphor the timeout will only happen after you get a response. even that your timeout is 1000 ms and the request took 5000 ms. hystryx will wait 5000 ms to give you a timeout error and the getExecutionTimeInMilliseconds will return 1000 ms to you.
Related
Recently i am reading the source code in OKHTTP and found that in RetryAndFollowUpInterceptor.java has a while(true) recycle. But, the program dose not stacked in that place or occur 'Application not responding'.Then I attached the debugger and put a breakpoint in that place and found it is running in main thread.I dont know why the program runs normal at all.Who can help me?
Thanks to stephenCs answer, but you might not get my point.In my question I mean why does the program not stuck in while(true) recycle in main thread.
For example, if there is a while(true) recycle in activity`s onCreate() lifecycle, the app might not run correctly ,just stuck in that place and can not respond touch event, which means the application not responding(ANR).
How will the recycle exit?
The following is the source code:
#Override
public Response intercept(Chain chain) throws IOException {
...
//This is the start of the recycle!Just recycle forever
while (true) {
if (canceled) {
streamAllocation.release();
throw new IOException("Canceled");
}
Response response;
boolean releaseConnection = true;
try {
response = realChain.proceed(request, streamAllocation, null, null);
releaseConnection = false;
} catch (RouteException e) {
// The attempt to connect via a route failed. The request will not have been sent.
if (!recover(e.getLastConnectException(), streamAllocation, false, request)) {
throw e.getFirstConnectException();
}
releaseConnection = false;
continue;
} catch (IOException e) {
// An attempt to communicate with a server failed. The request may have been sent.
boolean requestSendStarted = !(e instanceof ConnectionShutdownException);
if (!recover(e, streamAllocation, requestSendStarted, request)) throw e;
releaseConnection = false;
continue;
} finally {
// We're throwing an unchecked exception. Release any resources.
if (releaseConnection) {
streamAllocation.streamFailed(null);
streamAllocation.release();
}
...
}
}
}
If you are talking about this:
if (!recover(e.getLastConnectException(), streamAllocation, false, request)) {
throw e.getFirstConnectException();
}
releaseConnection = false;
continue;
or the similar code for IOException, the recover(...) call tests to see if failed request is recoverable. (See the method for what that actually means. But one of the criteria is that there is an alternative route that has not been tried yet.) If the call returns true, then the intercept method will retry. If it returns false then the relevant exception is rethrown.
In general, the logic in class is complicated, but it needs to be. And it clearly does work. So maybe you just need to read / study more of the context to understand what is happening.
Note that using a debugger to trace this could be difficult because the behavior is liable to change due to timeouts happening differently and altering the normal flow of control. Consider that possibility ...
UPDATE
How will the recycle exit?
It is called a loop, not a recycle.
As I have explained above, the particular loop paths that you highlighted will terminate (by rethrowing an exception) if the recover(...) call returns false. Let us take a look at recover.
private boolean recover(IOException e, boolean routeException, Request userRequest) {
streamAllocation.streamFailed(e);
// The application layer has forbidden retries.
if (!client.retryOnConnectionFailure()) return false;
// We can't send the request body again.
if (!routeException && userRequest.body() instanceof UnrepeatableRequestBody) return false;
// This exception is fatal.
if (!isRecoverable(e, routeException)) return false;
// No more routes to attempt.
if (!streamAllocation.hasMoreRoutes()) return false;
// For failure recovery, use the same route selector with a new connection.
return true;
}
The first statement is doing some cleanup on the StreamAllocation object. It isn't relevant to this.
The rest of the method is testing various things:
The application layers forbids retries, it says don't retry.
If a request was sent and it is not a resendable request, it says don't retry.
If the exception indicates an error that won't be fixed by retrying, it says don't retry.
If the route selector has no more routes to try, it says don't retry.
Otherwise it says retry. Next time that proceed is called, it will try the next route from the route selector.
Note that eventually it will run out of alternative routes to try.
My code essentially does something like this:
public void run() {
try {
while(true) {
// Do a bunch of stuff with
// Kafka and Redis over the
// network.
}
}
catch (Exception outerE) {
System.out.println("I never print this.");
}
finally {
System.out.println("I always print this.");
}
}
Something is causing the thread to die, but it isn't falling out of the while loop, and it isn't triggering an Exception, but I do see the message from the finally.
What is the best way to identify what killed a thread?
It is running under a ThreadPoolExecutor, so I can't just add an uncaughtExceptionHandler to the thread.
The first idea that works gets a bounty as soon as I can assign one.
Edit: So the "finally" clause let me know that it was breaking without an Exception. A few hours after I posted this, I did indeed use a custom thread factory to add an uncaught handler to the threads and it turned out to be a deadly mix of a self-referential Avro record with the Confluent Kafka Schema Registry. It was causing infinite recursion and blowing out the stack.
At my shop, it is utterly impossible to get a "catch (Throwable t)" into production, as it is considered a bad idea, even though, in this case, it is exactly the idea that is needed. But still, utterly impossible or I would have tried that.
It might be that this Throwable is a type of Error and not a standard Exception
Try this:
public void run() {
try {
while(true) {
...
}
} catch (final Throwable t) {
System.out.println("I never print this.");
} finally {
System.out.println("I always print this.");
}
}
I have two sql database connections for which health checks are automatically added by dropwizard. But when the application loses connection to one of them, the /healthcheck endpoint takes indefinitely long to respond, where I would want it to timeout after a few seconds.
I've already set the maxWaitForConnection setting, and I've also experimented with the various checkConnectionOn.. settings, but nothing helped.
UPDATE: The health check correctly fails if the server actively refuses the connection, but it hangs indefinitely if it's not the case, for instance, a network issue.
Is it possible to have sql health checks timeout at a specified time value whatever the problem is?
If the JDBC timeout settings aren't working you could always just wrap the DB check in a Future and limit how long it can run:
protected Result check() throws Exception {
ExecutorService executor = Executors.newSingleThreadExecutor();
Future<Void> future = executor.submit(new DbConnectionChecker());
try {
future.get(SECONDS_THRESHOLD, TimeUnit.SECONDS);
} catch (TimeoutException e) {
return Result.unhealthy("DB timed out");
}
executor.shutdownNow();
return Result.healthy();
}
Where DbConnectionChecker is something like:
static class DbConnectionChecker implements Callable<Void> {
public Void call() throws Exception {
// Check your DB connection as you normally would
}
}
Would be nice to figure out why the JDBC connection settings aren't working as expected, though.
I have a Java web app A that sends Pojo's to anothet web app B. I want to create a mechanism that retries the sending of the Pojos from A to B if B is down. The way B's api is configured in A is via Spring with B's jars in A's classpath. I have made a stab at it but the retry is not working. Can anyone tell me if I'm going in the right direction?
What is wrong here is that my code first executes, then rightly throws an exception. The exception is caught in the catch block. The catch block re-calls callB() which rightly throws an exception again but then my retry code goes into non source-file exception handling code and stops.
EDIT: I cannot use jcabi-aspects because it is not supported in Gradle.
public Object getB(Object request){
if (bClient != null) {
int retryCount = 0;
try {
return callB(request);
} catch (Exception e) {
if (++retryCount > 3) {
return null;
}
return callB(request);
}
}
return null;
}
private Object callB(Object request){
return bClient.callServiceB(request);
}
As you want code to repeatedly do something until it is successful, consider writing the control logic as a loop. You will find this makes it easy for you to change the number of retries before the code should give up.
This is what I have:
#Test
public testSendMessageToStub() {
// under the hood sends message
// if exception occurrs
// it will be catched and message will be put on retry
object.sendMessage();
}
Is there any way to mark test as failed if exception has occurred but was handled in catch block in the sendMessage() method?
Thanks
EDIT: It seems like I was too fixated on these legacy tests and how they were used, that totally missed the fact of sendMessage returning a response with a status code (!!!). So now I just assert status codes, can expand these tests into more detailed scenarios and spin them on jenkins. I would like to avoid to answer how these tests were checked previously. The thought to check for status codes came to me after reading Plux's answer. Thanks!
Exactly what you are looking for is not possible with JUnit as far as I know.
If you really would want to test this, you could store some information about the exception in the catch-block where it is handled in the sendMessage() method.
A better option, in my opinion, could be to test the output or state of the object. If the state/output is exactly the same as when an exception doesn't occur, then whats the point of testing it? Do you have an overly broad catch-block?
EDIT: To AdityaTS, I dont have enough reputation to comment on a post, but my comment: you have not supplied all the code, so I can not say for sure, but my guess is that its the Logger.getLogger IN the catch-block that casts the ClassNotFoundException. (Either that or loadConnectionInfo()) see http://docs.oracle.com/javase/7/docs/api/java/lang/ClassNotFoundException.html
You cannot do this without modifying sendMessage method. If for example you catch the exception there but choose to ignore it and just return some value, code outside of the method doesn't know it. You can get around this by refactoring the code of object: move the code that handles the exception to a separate method, called e.g. handleException. Then, in your test you can create a subclass where handleException will execute the original handleException from superclass, but additionally set some flag which you will be able to read in your test and in this way tell that the exception was thrown. However, if you cannot modify the code for object's class, I'm afraid you're out of luck.
So you expect the exception to propagate out of the sendMessage() method, right?
This is another way to write a test that verifies an exception you expect will be thrown.
#Test (expected = MyExpectedException.class)
public testSendMessageToStub() {
// under the hood sends message
// if exception occurrs
// it will be catched and message will be put on retry
object.sendMessage();
}
And it's usually best to be as specific as possible (e.g. MyExpectedException.class over Exception.class)
The exception generated in the sendMessage() class will be available in the test method. Add a try catch block around the sendMessage() method like this
#Test
public testSendMessageToStub() {
try
{
object.sendMehssage();
}
catch(Excpetion e) //Use more specific exception type if you know
{
fail(e.getMessage());
}
}
I have tried this in my code. It worked for me. Let me know.
public DBConnectionInfo connectionInit()
{
loadConnectionInfo();
try
{
Class.forName(dbObject.getDriver());
} catch (Exception e)
{
Logger lgr = Logger.getLogger(PostgreLocationManager.class.getName());
lgr.log(Level.SEVERE, e.getMessage(), e);
}
try
{
dbObject.setConnection(DriverManager.getConnection(dbObject.getDatabaseURL(), dbObject.getUserName(),
dbObject.getPassword()));
} catch (Exception e)
{
Logger lgr = Logger.getLogger(PostgreLocationManager.class.getName());
lgr.log(Level.SEVERE, e.getMessage(), e);
}
return dbObject;
}
The test case for the above class.
#Test
public void testDriverFailure()
{
when(dbModelObject.getDriver()).thenReturn("driver");
when(dbModelObject.getDatabaseURL()).thenReturn("jdbc:postgresql://127.0.0.1:5432/testdb");
when(dbModelObject.getUserName()).thenReturn("postgres");
when(dbModelObject.getPassword()).thenReturn("postgres");
try
{
dbConnector.connectionInit();
} catch (Exception e)
{
assertTrue(e instanceof ClassNotFoundException);
}
verify(dbModelObject).getDriver();
}