trying to run gitblit, on tomcat 9, using JDK 11 occassionaly results in this stack trace:
gitblit | 07-May-2020 04:30:39.247 SEVERE [https-jsse-nio-8443-exec-10] org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun Error running socket processor
gitblit | java.lang.NullPointerException
gitblit | at java.base/sun.security.ssl.HKDF.extract(HKDF.java:93)
gitblit | at java.base/sun.security.ssl.HKDF.extract(HKDF.java:119)
gitblit | at java.base/sun.security.ssl.ServerHello.setUpPskKD(ServerHello.java:1167)
gitblit | at java.base/sun.security.ssl.ServerHello$T13ServerHelloProducer.produce(ServerHello.java:545)
gitblit | at java.base/sun.security.ssl.SSLHandshake.produce(SSLHandshake.java:436)
gitblit | at java.base/sun.security.ssl.ClientHello$T13ClientHelloConsumer.goServerHello(ClientHello.java:1234)
gitblit | at java.base/sun.security.ssl.ClientHello$T13ClientHelloConsumer.consume(ClientHello.java:1170)
gitblit | at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:852)
gitblit | at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:813)
gitblit | at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
gitblit | at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444)
gitblit | at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061)
gitblit | at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1048)
gitblit | at java.base/java.security.AccessController.doPrivileged(Native Method)
gitblit | at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:995)
gitblit | at org.apache.tomcat.util.net.SecureNioChannel.tasks(SecureNioChannel.java:443)
gitblit | at org.apache.tomcat.util.net.SecureNioChannel.handshakeUnwrap(SecureNioChannel.java:507)
gitblit | at org.apache.tomcat.util.net.SecureNioChannel.handshake(SecureNioChannel.java:238)
gitblit | at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1616)
gitblit | at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
gitblit | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
gitblit | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
gitblit | at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
gitblit | at java.base/java.lang.Thread.run(Thread.java:834)
When the clients are trying to pull files from the gitblit GUI.
Possibly of interest, until I updated to the 11.0.7 version of the JDF, I was seeing this error:
Open JDK 11 HTTP/2 Handshake ServerHello java.util.NoSuchElementException
Where the fixed the mis-use of the Optional here: https://bugs.openjdk.java.net/browse/JDK-8218889 but perhaps, didn't actually address the root problem?
Or any other thoughts as to what is triggering this error? I'm using a self-signed cert here, FYI. The client is Firefox, and the java release is
openjdk version "11.0.7" 2020-04-14
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.7+10)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.7+10, mixed mode)
Running inside an alpine linux docker system.
Chasing down an issue where gitblit has occassional 1 minute timeouts, and finding this in the log. Not sure if related, or not....
Looks like it has also been found in tomcat https://bz.apache.org/bugzilla/show_bug.cgi?id=64105, and reported here
https://bugs.openjdk.java.net/browse/JDK-8241248
Since I can't provide info on the openjdk bug tracker, I can tell you that the client that typically causes it for me is Firefox 75 on linux.
As pointed out by the bugtracker references that you provide, this is a bug related to session resumption.
While this answer does not address the bug itself, there is a possibility to ask the SSLEngine to disallow resumption for a particular connection. This comes at a performance penalty for future connections because the client is required to redo the handshake for new connections instead of leveraging the session resumption mechanism.
At any time after the handshake is established, you can call invalidate() on the SSLSession. As pointed in the doc:
Future connections will not be able to resume or join this session. However, any existing connection using this session can continue to use the session until the connection is closed.
Meaning that it has no effect on the current connection, but will prevent session resumption and thus avoid the JDK bug.
My snippet for the handshake loop:
case NOT_HANDSHAKING:
case FINISHED:
{
if( !sslEngine.getSession().isValid() || sslEngine.getSession().getId().length == 0 )
throw new SSLHandshakeException("Handshake failed");
// prevent bug with rejoin session
sslEngine.getSession().invalidate();
return;
}
Related
I'm trying to run the official "Hello, World" example with TLS, provided by the gRPC team (code on GitHub).
I've cloned the official repo and checked out tag v1.15.0.
I've run the installation script as follows (as shown in the documentation):
./gradlew installDist
I've edited the launch scripts for hello-world-server and hello-world-client to run the classes io.grpc.examples.helloworldtls.HelloWorldServerTls and io.grpc.examples.helloworldtls.HelloWorldClientTls respectively.
I've created the certificates needed for the TLS authentication to happen using the script provided as part of the documentation and stored them in a directory I named cert.
Finally, I've run the server as follows:
./build/install/examples/bin/hello-world-server localhost 50440 cert/server.crt cert/server.pem
The server starts correctly and outputs the following:
Oct 08, 2018 9:15:10 AM io.grpc.examples.helloworldtls.HelloWorldServerTls start
INFO: Server started, listening on 50440
Finally, I try to start the client on another shell with the following command:
./build/install/examples/bin/hello-world-client localhost 50440 cert/ca.crt
Unfortunately, the client fails with the following output:
Oct 08, 2018 9:25:22 AM io.grpc.examples.helloworldtls.HelloWorldClientTls greet
INFO: Will try to greet localhost ...
Oct 08, 2018 9:25:22 AM io.grpc.examples.helloworldtls.HelloWorldClientTls greet
WARNING: RPC failed: Status{code=UNKNOWN, description=channel closed, cause=java.nio.channels.ClosedChannelException
at io.grpc.netty.Utils.statusFromThrowable(Utils.java:169)
at io.grpc.netty.NettyClientTransport$5.operationComplete(NettyClientTransport.java:260)
at io.grpc.netty.NettyClientTransport$5.operationComplete(NettyClientTransport.java:254)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424)
at io.netty.util.concurrent.DefaultPromise.setFailure(DefaultPromise.java:112)
at io.netty.channel.DefaultChannelPromise.setFailure(DefaultChannelPromise.java:89)
at io.grpc.netty.ProtocolNegotiators$AbstractBufferingHandler.fail(ProtocolNegotiators.java:564)
at io.grpc.netty.ProtocolNegotiators$BufferUntilTlsNegotiatedHandler.userEventTriggered(ProtocolNegotiators.java:661)
at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)
at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)
at io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307)
at io.netty.handler.ssl.SslUtils.handleHandshakeFailure(SslUtils.java:318)
at io.netty.handler.ssl.SslHandler.setHandshakeFailure(SslHandler.java:1551)
at io.netty.handler.ssl.SslHandler.channelInactive(SslHandler.java:1023)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1429)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)
at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:947)
at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:822)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:464)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:844)
Caused by: java.nio.channels.ClosedChannelException
at io.netty.handler.ssl.SslHandler.channelInactive(...)(Unknown Source)
}
The error occurs with mutual authentication as well.
I'm not sure what's going wrong or how to find the root cause of this, do you have any pointers?
As it turns out, I was using incompatible versions of gRPC and BoringSSL.
The "Troubleshooting" section of this document contains a table with known compatible versions of the library.
The following is a table with known compatible versions as of October 2018:
grpc-netty version | netty-handler version | netty-tcnative-boringssl-static version
------------------ | --------------------- | ---------------------------------------
1.0.0-1.0.1 | 4.1.3.Final | 1.1.33.Fork19
1.0.2-1.0.3 | 4.1.6.Final | 1.1.33.Fork23
1.1.x-1.3.x | 4.1.8.Final | 1.1.33.Fork26
1.4.x | 4.1.11.Final | 2.0.1.Final
1.5.x | 4.1.12.Final | 2.0.5.Final
1.6.x | 4.1.14.Final | 2.0.5.Final
1.7.x-1.8.x | 4.1.16.Final | 2.0.6.Final
1.9.x-1.10.x | 4.1.17.Final | 2.0.7.Final
1.11.x-1.12.x | 4.1.22.Final | 2.0.7.Final
1.13.x | 4.1.25.Final | 2.0.8.Final
1.14.x- | 4.1.27.Final | 2.0.12.Final
I'm trying to get a very basic Fitnesse test to run; using Fitnesse / Slim; Java in Eclipse.
Test runs, or at least starts, but doesn't finish; get this error:
Unable to start test system 'slim': fitnesse.slim.SlimError: Error SLiM server died before a connection could be established. JDWP exit error AGENT_ERROR_TRANSPORT_INIT(197): No transports initialized [debugInit.c:750]
Fitnesse test is:
!define TEST_SYSTEM {slim}
!path: C:\Users\<my folder>\EclipseWorkspace\CalculatorProject
| import | packageCalculator |
| TestCalculator |
| operand | result? |
| 4 | 8 |
| 2 | 4 |
| 7 | 14 |
| 2 | 4 |
This is a similar issue: Fitnesse: SLiM server died before a connection could be established; suggests this might be a regression bug; but I tried earlier versions of fitnesse, which made no difference
And it's not this one either: https://github.com/unclebob/fitnesse/issues/726; there is no space in the classpath
Help!!!
My Fuse ESB application suddenly stopped, with no clues in fuseesb.log and the following weird log items in wrapper.log:
ERROR | wrapper | 2014/02/12 02:36:59 | JVM appears hung: Timed out waiting for signal from JVM.
ERROR | wrapper | 2014/02/12 02:36:59 | JVM did not exit on request, terminated
STATUS | wrapper | 2014/02/12 02:37:02 | JVM exited in response to signal SIGKILL (9).
ERROR | wrapper | 2014/02/12 02:37:02 | Unable to start a JVM
STATUS | wrapper | 2014/02/12 02:37:02 | <-- Wrapper Stopped
We are using Nagios to monitor the system.
System info: FuseESBEnterprise-7.1.0 / linux-gnu (x86_64).
For various reasons we need to sick with Fuse ESB (not possible to migrate to JBoss Fuse...)
This was a system issue - the vm hung.
You can catch this kind of condition in one of two ways:
monitor the java jvm process
monitor the wrapper.log file for errors
I try to run derby database as windows service using java service wrapper (JSW). I downloaded community edition 3.5.15.
there is wrapper config I use:
wrapper.java.command=java
#wrapper.java.mainclass=org.tanukisoftware.wrapper.WrapperSimpleApp
wrapper.java.mainclass=org.tanukisoftware.wrapper.WrapperStartStopApp
wrapper.java.classpath.1=../lib/derby.jar
wrapper.java.classpath.2=../lib/derbynet.jar
wrapper.java.classpath.3=../lib/derbytools.jar
wrapper.java.classpath.4=Wrapper.jar
wrapper.java.library.path.1=
wrapper.java.additional.1=-Dderby.system.home=c:/data/derby
#wrapper.app.parameter.1=org.apache.derby.drda.NetworkServerControl
#wrapper.app.parameter.2=start
wrapper.app.parameter.1=org.apache.derby.drda.NetworkServerControl
wrapper.app.parameter.2=3
wrapper.app.parameter.3=start
wrapper.app.parameter.4=-h
wrapper.app.parameter.5=127.0.0.1
wrapper.app.parameter.6=org.apache.derby.drda.NetworkServerControl
wrapper.app.parameter.7=true
wrapper.app.parameter.8=1
wrapper.app.parameter.9=shutdown
wrapper.console.format=PM
wrapper.console.loglevel=INFO
wrapper.logfile=log/wrapper.log
wrapper.logfile.format=LPTM
wrapper.logfile.loglevel=INFO
wrapper.logfile.maxsize=5m
wrapper.logfile.maxfiles=10
wrapper.syslog.loglevel=ERROR
wrapper.console.title=Derby DB Server
wrapper.ntservice.name=derby
wrapper.ntservice.displayname=Apache Derby Database
wrapper.ntservice.description=Apache Derby Relational Database Engine (Network Server)
wrapper.ntservice.starttype=AUTO_START
wrapper.ntservice.interactive=false
#wrapper.ntservice.account=.\derby
#wrapper.ntservice.password=derbypw
and when I try to start derby server via wrapper I get security violation exception:
c:\derby\bin>wrapper -c derby.conf
wrapper | --> Wrapper Started as Console
wrapper | Java Service Wrapper Community Edition 32-bit 3.5.15
wrapper | Copyright (C) 1999-2012 Tanuki Software, Ltd. All Rights Reserved.
wrapper | http://wrapper.tanukisoftware.com
wrapper |
wrapper | Launching a JVM...
jvm 1 | WrapperManager: Initializing...
jvm 1 | Sun Jul 22 22:11:11 BST 2012 : Security manager installed using the Basic server security policy.
jvm 1 | Sun Jul 22 22:11:11 BST 2012 : Apache Derby Network Server - 10.9.1.0 - (1344872) started and ready to accept connections on port 1527
jvm 1 | WrapperManager Error: Error in WrapperListener.start callback. java.security.AccessControlException: access denied (org.tanukisoftware.wrapper.security.WrapperPerm
ission signalStarting)
jvm 1 | WrapperManager Error: java.security.AccessControlException: access denied (org.tanukisoftware.wrapper.security.WrapperPermission signalStarting)
jvm 1 | WrapperManager Error: at java.security.AccessControlContext.checkPermission(Unknown Source)
jvm 1 | WrapperManager Error: at java.security.AccessController.checkPermission(Unknown Source)
jvm 1 | WrapperManager Error: at java.lang.SecurityManager.checkPermission(Unknown Source)
jvm 1 | WrapperManager Error: at org.tanukisoftware.wrapper.WrapperManager.signalStarting(WrapperManager.java:3268)
jvm 1 | WrapperManager Error: at org.tanukisoftware.wrapper.WrapperStartStopApp.start(WrapperStartStopApp.java:437)
jvm 1 | WrapperManager Error: at org.tanukisoftware.wrapper.WrapperManager$11.run(WrapperManager.java:3963)
jvm 1 | WrapperManager Error: Unable to remove the Wrappers shudownhook: {0}
jvm 1 | Exception in thread "WrapperListener_start_runner" java.security.AccessControlException: access denied (org.tanukisoftware.wrapper.security.WrapperPermission signal
Stopped)
jvm 1 | at java.security.AccessControlContext.checkPermission(Unknown Source)
jvm 1 | at java.security.AccessController.checkPermission(Unknown Source)
jvm 1 | at java.lang.SecurityManager.checkPermission(Unknown Source)
jvm 1 | at org.tanukisoftware.wrapper.WrapperManager.signalStopped(WrapperManager.java:3320)
jvm 1 | at org.tanukisoftware.wrapper.WrapperManager.shutdownJVM(WrapperManager.java:4058)
jvm 1 | at org.tanukisoftware.wrapper.WrapperManager.privilegedStopInner(WrapperManager.java:4363)
jvm 1 | at org.tanukisoftware.wrapper.WrapperManager.access$2900(WrapperManager.java:124)
jvm 1 | at org.tanukisoftware.wrapper.WrapperManager$11.run(WrapperManager.java:3983)
wrapper | CTRL-C trapped. Shutting down.
wrapper | Shutdown failed: Timed out waiting for signal from JVM.
wrapper | JVM did not exit on request, terminated
wrapper | <-- Wrapper Stopped
I could add permission grant to java.policy to solve this problem (I have not tried, but suppose that will work).
The thing I want to know is: what enforces security constraints (and throws security exception) in this case? I thought that all locally started java application are granted all permissions.
One more thing that confuses me: I have h2 database server (1.3.162 (2011-11-26)) running via JSW (but earlier version) there are no security grants for it. I searched for *.policy files containing wrapper word, and there was nothing. How in this case security is configured, and if it is configured at all?
I would be grateful if someone make these security issues more clear :P.
Putting Leif's comment as answer:
When you run with the Wrapper, the Wrapper classes are launching your application's main method. This is inserting classes from the wrapper.jar into the call stack. Java's security model works by limiting access to the permissions granted to the weakest method in the call stack. In order to make this work, you will need to give classes in the wrapper.jar permission to do what you want.
This is described on our site here:
http://wrapper.tanukisoftware.com/doc/english/security-model.html
We are using JBoss_4_0_4_GA with JDK 1.5.0 (no updates) on a Windows
The JBoss server is run within a Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org.
Since the JVM is so old I cannot even use the -XX:+HeapDumpOnOutOfMemoryError option on the JVM.
What are my option to find out the issue?
As usual the Out of Memory exception is happening on different parts of the application.
I do not have the liberty to upgrade the JVM right away.
The current VM settings
Java Additional Parameters
wrapper.java.additional.1=-Xms512m
wrapper.java.additional.2=-Xmx1024m
wrapper.java.additional.3=-Dsun.rmi.dgc.client.gcInterval=3600000
wrapper.java.additional.4=-Dsun.rmi.dgc.server.gcInterval=3600000
wrapper.java.additional.5=-Dorg.xml.sax.driver=org.apache.xerces.parsers.SAXParser
wrapper.java.additional.6=-Djava.endorsed.dirs=D:/jboss-4.0.4.GA/lib/endorsed
Snippets of the exception
INFO | jvm 1 | 2012/05/31 11:25:03 | 11:25:03,502 ERROR [SOAPFaultExceptionHelper] SOAP request exception INFO | jvm 1 | 2012/05/31 11:25:03 | java.rmi.RemoteException: java.lang.OutOfMemoryError: Java heap space; nested exception is: INFO | jvm 1 | 2012/05/31 11:25:03 | java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
INFO | jvm 1 | 2012/05/31 11:25:03 | Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space INFO | jvm 1 | 2012/05/31 11:25:03 | at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:205) INFO | jvm 1 | 2012/05/31 11:25:03 | at java.util.concurrent.FutureTask.get(FutureTask.java:80)
You can try to do memory dumps with jmap (in tags you mentioned java 5 so this should be possible). Do several dumps when the server still works (like every hour or so).
Then analyze them in Eclipse MAT. Search for objects, or collections of objects that grow bigger on every dump. This will most probably be your memory leak.
I see two options, each of them having their pros and cons:
Install a profiler agent and connect a profiler on your application. This has of course some serious performance impact which may be not acceptable in your production environment. But this would allow you to monitor the application and perform a memory dump when Heap space is almost full.
Reproduce your production environment elsewhere and also use a profiler. If your issues only appear with big loads, you may have to create load tests so that you reach the OOME.
Using a profiler is your best chance to find memory leaks.
Otherwise, you can always try to perform static code reviewing, using PMD and Findbugs (amongst other static analysis tools) which may spot some mistakes.