We're on version 2.7 of Play Framework and we've been getting what seems like a random exception in the logs (see below), however we can not seem to trace it back to our code. Here is the stack trace:
[2020-07-15 14:02:36,294] - [ERROR] - from akka.actor.ActorSystemImpl at [akka.actor.ActorSystemImpl(application)]
Internal server error, sending 500 response
akka.http.impl.util.One2OneBidiFlow$OutputTruncationException: Inner flow was completed without producing result elements for 1 outstanding elements
at akka.http.impl.util.One2OneBidiFlow$OutputTruncationException$.apply(One2OneBidiFlow.scala:22)
at akka.http.impl.util.One2OneBidiFlow$OutputTruncationException$.apply(One2OneBidiFlow.scala:22)
at akka.http.impl.util.One2OneBidiFlow$One2OneBidi$$anon$1$$anon$4.onUpstreamFinish(One2OneBidiFlow.scala:97)
at akka.stream.impl.fusing.GraphInterpreter.processEvent(GraphInterpreter.scala:506)
at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:376)
at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:606)
at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:485)
at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:581)
at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:749)
at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:764)
at akka.actor.Actor$class.aroundReceive(Actor.scala:539)
at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:671)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:614)
at akka.actor.ActorCell.invoke(ActorCell.scala:583)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:268)
at akka.dispatch.Mailbox.run(Mailbox.scala:229)
at akka.dispatch.Mailbox.exec(Mailbox.scala:241)
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
As you can see there are no references to any non-play code. The play app runs normally with visible problems in functionality. Googling hasn't yielded much insight apart from similar issues due to malformed config files (which causes the same stack trace but under different conditions) and possibly Kamon related issues. We do use kamon version 1 . What could be causing this exception? Any help would be greatly appreciated
It looks like this is caused by this issue:
https://github.com/playframework/playframework/issues/9020
Meaning that your configuration is malformed (it couldn't be parsed). Hitting ctrl+c should at least show another error message with more information about where the error is.
Related
I am currently working on R3 Corda 4.4 and generating my network using the network Bootstrapper tool. I have a 3 node setup which is running on 3 different VMs with a load balancer attached to each of them. I am also using postgresql as my db for the nodes. All the steps till node-info creation run successfully, during node-info creation the 2nd party's node-info is generated after the last party node-info causing an error.
picocli.CommandLine$ExecutionException: Error while calling command (net.corda.bootstrapper.NetworkBootstrapperRunner#4e28bdd1): java.lang.IllegalThreadStateException: process hasn't exited
The error occurs as soon as the node-info for the 3rd party is generated and the 2nd party one is in progress. When I open the logs for each of the parties there is no Error and all node-info are getting generated and I can see them in their respective folders.
Detailed Error:
picocli.CommandLine$ExecutionException: Error while calling command (net.corda.bootstrapper.NetworkBootstrapperRunner#4e28bdd1): java.lang.IllegalThreadStateException: process hasn't exited
at picocli.CommandLine.execute(CommandLine.java:1180)
at picocli.CommandLine.access$800(CommandLine.java:141)
at picocli.CommandLine$RunLast.handle(CommandLine.java:1367)
at picocli.CommandLine$RunLast.handle(CommandLine.java:1335)
at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1243)
at picocli.CommandLine.parseWithHandlers(CommandLine.java:1526)
at net.corda.cliutils.CordaCliWrapperKt.start(CordaCliWrapper.kt:73)
at net.corda.bootstrapper.MainKt.main(Main.kt:19)
Caused by: java.lang.IllegalThreadStateException: process hasn't exited
at java.lang.UNIXProcess.exitValue(UNIXProcess.java:421)
at net.corda.nodeapi.internal.network.NetworkBootstrapper$Companion$generateNodeInfo$1.invoke(NetworkBootstrapper.kt:116)
at net.corda.nodeapi.internal.network.NetworkBootstrapper$Companion$generateNodeInfo$1.invoke(NetworkBootstrapper.kt:69)
at net.corda.nodeapi.internal.network.NetworkBootstrapper$Companion.printNodeInfoGenLogToConsole(NetworkBootstrapper.kt:128)
at net.corda.nodeapi.internal.network.NetworkBootstrapper$Companion.generateNodeInfo(NetworkBootstrapper.kt:116)
at net.corda.nodeapi.internal.network.NetworkBootstrapper$Companion.access$generateNodeInfo(NetworkBootstrapper.kt:69)
at net.corda.nodeapi.internal.network.NetworkBootstrapper$Companion$generateNodeInfos$1$1.invoke(NetworkBootstrapper.kt:95)
at net.corda.nodeapi.internal.network.NetworkBootstrapper$Companion$generateNodeInfos$1$1.invoke(NetworkBootstrapper.kt:69)
at net.corda.core.internal.concurrent.ValueOrException$DefaultImpls.capture(CordaFutureImpl.kt:141)
at net.corda.core.internal.concurrent.OpenFuture$DefaultImpls.capture(CordaFutureImpl.kt)
at net.corda.core.internal.concurrent.CordaFutureImpl.capture(CordaFutureImpl.kt:153)
at net.corda.core.internal.concurrent.CordaFutureImplKt$fork$$inlined$also$lambda$1.run(CordaFutureImpl.kt:22)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
This error message unfortunately doesn't give us a lot to work with but it would seem that there's a problem with a particular node config option.
Is this issue happening when running the deploy nodes task? (meaning compiling your cordapps) Or is this issue happening when you use the runnodes script?
can you share more info on what the node configs look like? that's probably going to be the most likely culprit as it's not clear what you've changed since the last working version.
Here's my recommendation:
See if you're able to isolate the issue and run the network without the problematic node and then share the config of the one that was giving the issue.
Here's a page on the configuration options for nodes : https://docs.corda.net/docs/corda-os/4.6/corda-configuration-fields.html
If you're able to find a specific node config option that's causing the breaking definitely update here or on slack so we can get that bug reported if it turns out to be that.
Why does DisconnectNonTransientException occur? It happened only once and the error could not be reproduced after that. What is the fix to avoid the error from occurring in future. More importantly how to fix an issue that is no longer reproducible?
edit: More updates on this question.
Using mybatis,DB2,tomcat.(trying to access remote DB)
This error occurs when code hits data source for first time after a long gap since the application in tomcat was last accessed. When refreshed, error disappears and the application works as expected.
Connection to data source is closed after every access
SqlSession created is not closed(Does this cause the trouble?)
Error says : "The database manager is not able to accept new requests, has terminated all requests in progress, or has terminated this particular request due to unexpected error conditions detected at the target system. ERRORCODE=-4499, SQLSTATE=58009"
Is there any default timeout for mybatis SqlSession?
More importantly how to fix an issue that is no longer reproducible?
In the stacktrace you can find where the exception is thrown. Then debugging the code you can find why it's thrown.
Many exceptions occurs because something wrong with the code or with the data. You should find what is wrong and fix it. If you can't fix it in the code where it occurred then you catch the exception from your code and then decide what to do with it.
If you can't reproduce the exception then you can trace the code that may throw the exception. Show or write the debugging code by changing the logging mode.
If you got the exception stacktrace then you should be aware what the code was when it thrown.
If you have some tests that might reproduce the exception then you can write additional code to intercept the exception. You should write such code trying to reproduce the exception on the test environment. Then you should know how to handle it.
I've been having some issues with NoHandlerException's in a multi server configuration. I've been trying to figure out exactly when I get this exception but
I can not find any good description on what it actually means that no handler was found.
The thing here is that everything actually seems to work fine, we are not receiving any error reports on this from our production system, and we are not able
to reproduce the error in our test systems. But we can clearly see a big amount of no handler found errors in our production logs.
So my question is, could this error be due to some bad load-balancing? Like that we send our users between
different servers and the server receiving server does not have an updated state for this user/session? Or should it be some configuration error on the Spring-application
that can not be affected by the load balancing?
When I have searched for other people with the same error they seem to get it all the time, but I get it only sporadically
The error we receive:
Uncaught service() exception root cause AppName: javax.servlet.ServletException: org.springframework.web.portlet.NoHandlerFoundException: No handler found for portlet request: mode 'view', phase 'ACTION_PHASE', parameters map['action' -> array<String>['myController.parameter']]
Try to check xml somewhere contains portlet. Normally every handler stage error cased by configuration.
I'm trying to copy messages from one IMAP store to another.
The line of code that does the actual copying is:
folder.appendMessages(new Message[] { m });
Most messages are copied with no problem, but some fail with:
javax.mail.MessagingException: IOException while appending messages;
nested exception is:
java.io.IOException: No content
The stacktrace points to IMAPFolder.java:1767
Some of the messages that were copied successfully were as large as 6MB, while some of the ones that failed were as small as 16KB in my test case, so it is not a size issue.
I'm not certain, but it's possible that the messages that fail has an attachment. The content type multipart/ALTERNATIVE was found in both in the group of messages that failed and the group that succeeded.
Opening the failed messages in a mail client like Thunderbird works properly, so the messages are not corrupted.
UPDATE:
mail.imap.fetchsize plays a role with this problem. When it is the default of 16,384b, running over a sample of 1000 messages the smallest failed message is 16,432b -- slightly over the fetchsize.
When I increased it to 65,536b -- the smallest failed message of that same sample is 65,787b -- slightly over the fetchsize again.
The message with size 16,432b that consistently failed with the default fetchsize, copied successfully with the higher fetchsize, so this is definitely an issue here.
Any idea what causes this problem?
How can I copy the messages with having this issue?
TIA!
p.s. I'm using Java 1.7u71 and Javamail 1.5.2
Based on our offline discussion, setting the mail.imap.partialfetch property to false causes it to work. When that happens, it's almost always because the server is broken in some way.
See also:
Javamail Notes under "A list of the known limitations, bugs, issues" sections 4 and 5.
Javamail FAQ, Fetch
The bug was fixed in Java Mail 1.6
https://javaee.github.io/javamail/
we experience the following error pattern:
sometimes we have GAE app request processing lasting long, this throws DeadlineExceededException as GAE has a limit for 1 min. This is a described by docs, ok.
apart from DeadlineExceededException we get A problem was encountered with the process that handled this request, causing it to exit. This is likely to cause a new process to be used for the next request to your application. If you see this message frequently, you may be throwing exceptions during the initialization of your application. (Error code 104)
subsequent requests coming to GAE app within next few milliseconds fail with the same Error code 104
Questions:
Why #2 is reported?
How can we avoid #3? Is it a bug in GAE? What is the mechanism of such failure?
Thanks for help.
As Bruyere kindly pointed, related threads killing in result of timeout exception is detailed here:
If concurrent requests are enabled through the "threadsafe" flag, every other running concurrent request is killed with error code 104: