We have a Spring MVC web app (WAR) deploying to Tomcat (6.0.35) that launches a thread inside a separate JVM at deploy time (don't ask why - not my design) and then communicates with that thread via RMI over port 8888.
Despite being totally convoluded, this was working perfectly fine up until yesterday, and now the thread is failing at startup and despite our best efforts to add logging into the mix, we are hitting a wall. This is the only exception we are able to find in the logs:
Jun 12, 2012 3:11:36 AM com.ourapp.ImageController destroy
SEVERE: Shutdown Error: Lookup of RMI stub failed; nested exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused
Jun 12, 2012 3:11:37 AM org.apache.catalina.core.StandardContext listenerStop
SEVERE: Exception sending context destroyed event to listener instance of class org.springframework.web.context.ContextLoaderListener
java.lang.NoClassDefFoundError: org/springframework/web/context/ContextCleanupListener
at org.springframework.web.context.ContextLoaderListener.contextDestroyed(ContextLoaderListener.java:80)
at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:3973)
at org.apache.catalina.core.StandardContext.stop(StandardContext.java:4577)
at org.apache.catalina.startup.HostConfig.checkResources(HostConfig.java:1165)
at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1271)
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:296)
at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119)
at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1337)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1601)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1610)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1590)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.ClassNotFoundException: org.springframework.web.context.ContextCleanupListener
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1387)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1233)
... 12 more
The ImageController is the Spring MVC Controller that is responsible for kicking off this daemon/spawned RMI thread. Based on the verbage of this error, does anybody have any idea what might be causing this "connection refused" error?
Running a netstat -an | grep 8888 (this is a Linux machine) produces no output which means nothing is listening on that port. Thanks in advance for any ideas/suggestions that lead to a fix.
Edit: Here's another ConnectionException we're seeing:
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at java.net.Socket.connect(Socket.java:478)
at java.net.Socket.<init>(Socket.java:375)
at java.net.Socket.<init>(Socket.java:189)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595)
... 74 more
I'll move my comments to this answer.
Everything in your logs suggests that the problem is on your other machine where you connect to on port 8888. The netstat results prove that there's no process listening on that port - hence connection errors in your logs.
Next step I would take is check whether the application on the other machine (or in other JVM) is running at all (e.g. ps ax|grep java). And if it does - check its logs for clues why it failed to start the RMI service, or if it doesn't - start it.
Typically if an application can not open a socket, it's due to the fact that some other application is already using the port. It's not the case in your situation. It could be a case that two instances of the same application have been started: first application took 8888 port, second application couldn't take 8888 and failed, and then first application was shut down. After all you end up with nobody listening on the port.
does anybody have any idea what might be causing this "connection refused" error?
It's right there in the stack trace:
java.lang.NoClassDefFoundError: org/springframework/web/context/ContextCleanupListener
Your deployment is missing that class, i.e. the JAR file that it comes in.
Related
This question already has answers here:
Official reasons for "Software caused connection abort: socket write error"
(14 answers)
Closed 7 years ago.
We have a Tomcat server that is in one case, responding to a web service and in another attempting to call a web service. In both cases the caller and end point are running on the local box. Yet there is a connection being lost and we know that it is not a network appliance.
Here are part of the exceptions:
The response:
Caused by: ClientAbortException: java.net.SocketException: Software caused connection abort: socket write error
at org.apache.x.connector.OutputBuffer.doFlush(OutputBuffer.java:364)
at org.apache.x.connector.OutputBuffer.flush(OutputBuffer.java:326)
at org.apache.x.connector.Response.flushBuffer(Response.java:571)
And the call:
Caused by: java.net.SocketTimeoutException: SocketTimeoutException invoking http://localhost:7720/x/xService: Read timed out
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
The particulars:
Server version: Apache Tomcat/7.0.40
Server number: 7.0.40.0
OS Name: Windows Server 2008 R2
OS Version: 6.1
Architecture: x86
JVM Version: 1.7.0_25-b17
JVM Vendor: Oracle Corporation
Any ideas what might be causing this?
I should add that the server, once these errors occur, does not recover gracefully. It has to be restarted.
Well,since the whole log is not present the best guess I have is that either your port(7720) or url is incorrect or blocked locally (i.e being used by some other application) or the localhost entry in the hosts file is overridden and changed to something else instead of 127.0.0.1
I am unable to connect VisualVM to a remote JVM. I have started the remote JVM with the following parameters:
java -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=9000 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Djava.rmi.server.hostname=[server ip] -Dlog4j.configurationFile=file:///home/ubuntu/servicesLog4j2.xml -jar /home/ubuntu/Program.jar &
Note the -Dcom.sun.management.jmxremote.ssl=false
I then start VisualVM and add a new JMX connection to the server. I specifically check the option:
Do not require SLL connection
When the connection fails, I can see the following error in the VisualVM log. I am unsure why SSL is involved if I have disabled SSL with a flag on the remote VM and specified not to require SSL in VisualVM. But it would appear this error is preventing the connection. What am I missing?
java.io.EOFException: SSL peer shut down incorrectly
at sun.security.ssl.InputRecord.read(InputRecord.java:505)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:954)
Caused: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1343)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:728)
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:229)
Caused: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:304)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:342)
at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:118)
Caused: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake]
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:122)
at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:205)
at javax.naming.InitialContext.lookup(InitialContext.java:417)
at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1929)
at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1896)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287)
Caused: java.io.IOException: Failed to retrieve RMIServer stub
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369)
at com.sun.tools.visualvm.jmx.impl.JmxModelImpl$ProxyClient.tryConnect(JmxModelImpl.java:569)
[catch] at com.sun.tools.visualvm.jmx.impl.JmxModelImpl$ProxyClient.connect(JmxModelImpl.java:506)
at com.sun.tools.visualvm.jmx.impl.JmxModelImpl.connect(JmxModelImpl.java:234)
at com.sun.tools.visualvm.jmx.impl.JmxModelImpl.<init>(JmxModelImpl.java:223)
at com.sun.tools.visualvm.jmx.impl.JmxModelProvider.createModelFor(JmxModelProvider.java:60)
at com.sun.tools.visualvm.jmx.impl.JmxModelProvider.createModelFor(JmxModelProvider.java:41)
at com.sun.tools.visualvm.core.model.ModelFactory.getModel(ModelFactory.java:111)
at com.sun.tools.visualvm.tools.jmx.JmxModelFactory.getJmxModelFor(JmxModelFactory.java:69)
at com.sun.tools.visualvm.jmx.impl.JmxApplicationProvider.addJmxApplication(JmxApplicationProvider.java:295)
at com.sun.tools.visualvm.jmx.impl.JmxApplicationProvider.createJmxApplication(JmxApplicationProvider.java:200)
at com.sun.tools.visualvm.jmx.JmxApplicationsSupport.createJmxApplicationImpl(JmxApplicationsSupport.java:319)
at com.sun.tools.visualvm.jmx.JmxApplicationsSupport.createJmxApplicationInteractive(JmxApplicationsSupport.java:296)
at com.sun.tools.visualvm.jmx.impl.AddJMXConnectionAction$1.run(AddJMXConnectionAction.java:80)
at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:1423)
at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:2033)
How I managed to solve this after some headaches:
If you have a firewall at the remote machine, you might have issues with the second random port (where remote objects are exported). Try to disable the firewall and if it works then you might have to do something like this. Tomcat have a listener to solve it: link
Make sure that the value under -Djava.rmi.server.hostname resolves to the host from your client location (that was my case)
And finally jconsole (at the JDK bin folder, the same than VisualVM) provides very useful information. Execute it from a console with the -debug option. With the debug option, Jconsole will pop up stacktraces explaining the real reason why your client can't connect. When you try to connect to nonSSL servers JConsole will initially fail and then it will ask you to connect "insecure", next you'll see the real reason, in my case it was looking for the object export port at the loopback IP 127.0.0.1 (because I wasn't using the -Djava.rmi.server.hostname option).
Hope it helps!
Sep 4, 2013 9:59:40 AM org.apache.tomcat.util.net.JIoEndpoint$Acceptor run
SEVERE: Socket accept failed
java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:375)
at java.net.ServerSocket.implAccept(ServerSocket.java:470)
at java.net.ServerSocket.accept(ServerSocket.java:438)
at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:61)
at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:352)
at java.lang.Thread.run(Thread.java:679)
I have deploy my Jsp Application in Linux base tomcat server and log file coming its size approx 11 Gb i dont know how to Handle this What i have to do to fix for this please help am totally new in this Error its developer work some thing Else. please help.
You are leaking file handles or sockets. Check your closes.
I've been learning about RMI and I started with the Hello World program, which I got to work, and then began using other sample programs which all worked. (well there were errors, but I debugged them with the help of google/stackoverflow). and now I began writing my own project, which is basically a Traveling Salesman implementation that offloads the intense computation to a server.
Everything was working fine, when suddenly all my server RMI implementations broke. ie. when I run computeEngine from Eclipse as an RMI application (I use the RMI plugin), I get either the error:
java.rmi.ConnectException: Connection refused to host
or the error:
java.rmi.UnmarshalException: Error unmarshaling return header; nested exception is: java.io.EOFException
I have 4-5 different RMI applications that were all working and then all of a sudden the "ComputeEngine.java" file which I run on the server started returning one of these 2 errors for every single one of my applications.
Note: since I am testing these applications, I use "localhost" as my server hostname. In otherwords, the server and the client are the same machine.
Things I have tried:
I have checked my etc/hosts/ file and found that 127.0.0.1 is set as localhost. I also created a new machine entitled "virtualmachine" and used my public IP address. Both did not work.
I have double and triple checked my server.policy and client.policy files, and both seem to be fine.
I have tried running the RMI application from terminal (instead of using the RMI plugin from eclipse) and I get the same errors
I quit all java processes and reran everything making sure I started the server first (other threads suggested that these errors come from running the client first)
I've portscanned myself and port 1099 (the default rmi port) says "LISTEN" when I run my server (even though the server is returning an exception).
I've tried changing java.rmi.server.hostname, but nothing has seemed to help so far.
I'm not understanding how I'm not getting connection to host when I am running both the server and client on the same computer.
I preemptively apologize from being sucky at RMI and Java and missing some likely obvious solution.
Should I include any source code from what I have been trying to run?
EDIT: Here is the stack trace:
ComputeEngine exception:
java.rmi.UnmarshalException: Error unmarshaling return header; nested exception is:
java.io.EOFException
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:227)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:377)
at sun.rmi.registry.RegistryImpl_Stub.rebind(Unknown Source)
at engine.ComputeEngine.main(ComputeEngine.java:33)
Caused by: java.io.EOFException
at java.io.DataInputStream.readByte(DataInputStream.java:267)
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:213)
... 3 more
Thank You!
EDIT 2:
I noticed that if I restarted the RMIregistry when I started the application (as in, I selected the option in the Eclipse RMI plugin that restarted the rmiregistry) I would get a ClassNotFoundError for a class that is in my 'codebase.jar'.
Caused by: java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is:
java.lang.ClassNotFoundException: compute.Compute
Any ideas?
When I call, the service's method, I have Web Service Exception happening which I couldn't figure out for now.
The issue is this, I am running NetBeans 6.8 against IIS 7.0 with the Web Service written using WCF:
javax.xml.ws.WebServiceException: java.net.SocketException: Connection reset
at com.sun.xml.internal.ws.transport.http.client.HttpClientTransport.readResponseCodeAndMessage(HttpClientTransport.java:201)
at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.process(HttpTransportPipe.java:151)
at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.processRequest(HttpTransportPipe.java:83)
at com.sun.xml.internal.ws.transport.DeferredTransportPipe.processRequest(DeferredTransportPipe.java:105)
at com.sun.xml.internal.ws.api.pipe.Fiber.__doRun(Fiber.java:587)
at com.sun.xml.internal.ws.api.pipe.Fiber._doRun(Fiber.java:546)
at com.sun.xml.internal.ws.api.pipe.Fiber.doRun(Fiber.java:531)
at com.sun.xml.internal.ws.api.pipe.Fiber.runSync(Fiber.java:428)
at com.sun.xml.internal.ws.client.Stub.process(Stub.java:211)
at com.sun.xml.internal.ws.client.sei.SEIStub.doProcess(SEIStub.java:124)
at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:98)
at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:78)
at com.sun.xml.internal.ws.client.sei.SEIStub.invoke(SEIStub.java:107)
at $Proxy30.getTwo(Unknown Source)
at HighBeamWcfClient.Main.main(Main.java:28)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:168)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:687)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:632)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:652)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1072)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:373)
at com.sun.xml.internal.ws.transport.http.client.HttpClientTransport.readResponseCodeAndMessage(HttpClientTransport.java:198)
How to fix it?
"Connection reset" typically means that someone/something between you and the server is resetting the connection or somehow blocking it. Check that you can connect to the server on the port correctly without any issue, and that there are no firewalls in the way.
A very simple test that you can connect to the port in question is to simply telnet to it with telnet serverhostname portnum, for example telnet stackoverflow.com 80.