After Java OutOfMemoryError, Tomcat Won't Start - java

After running my App in Tomcat many times, I've maxed out the PermGen memory (a problem with my quartz servlet, I suspect) and Java threw a OutOfMemoryError. Normally, I just restart tomcat from time to time, but since this happened I'm unable to start/restart. Tomcat is running in our qa environment and is normally started/stoped/restarted via the sbin folder. I've tried to start tomcat using the alternative, catalina.sh run, but it says the port is being used! That led me to suspect that tomcat was already running, but when I try to stop it, I get this error:
SEVERE: Could not contact localhost:8005. Tomcat may not be running.
Jan 19, 2014 3:10:58 PM org.apache.catalina.startup.Catalina
stopServer SEVERE: Catalina.stop: java.net.ConnectException:
Connection refused at java.net.PlainSocketImpl.socketConnect(Native
Method) at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at
java.net.Socket.connect(Socket.java:579) ...
So, if it's not started or stopped, then what state is tomcat in?! How can I bring it back to a startable state?
Update
As per Boris's comment, I've used ps auxfww to list processes and the only one with tomcat as the user is the following:
tomcat 5111 8.0 29.0 2052336 556512 ? Sl Jan17 221:48
/usr/java/latest/bin/java
-Djava.util.logging.config.file=/var/lib/tomcat/conf/logging.properties
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -javaagent:/usr/share/tomcat/newrelic/newrelic.jar -Xmx512m -Djava.awt.headless=true -Djava.endorsed.dirs=/usr/share/tomcat/endorsed -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar
-Dcatalina.base=/var/lib/tomcat -Dcatalina.home=/usr/share/tomcat -Djava.io.tmpdir=/var/lib/tomcat/temp org.apache.catalina.startup.Bootstrap start
For some reason, even as root, I can't delete the process. I use
sudo kill 5111
but the process is still listed after and tomcat still won't restart...

Read about kill here
sudo kill -9 5111 will do the work. Tomcat is binded to 8005 port: until you kill the process with a SIGKILL signal it won't release the port (and you are not allowed to start another tomcat on this port).

Related

Tomcat not starting after modifying a xml file

I modified the web.xml file from a dynamic web project in Eclipse EE and now I can't start the Tomcat server.
Error shows:
Server Tomcat v8.0 Server at localhost failed to start.
and in the console:
SEVERE: Failed to destroy end point associated with ProtocolHandler ["ajp-nio-8009"]
How can i fix this ?
The service is already running on port 8009. You should kill it and try tomcat restart.
Try this :
Terminate the tomcat, restart it from command prompt and then first undeploy and then redeploy your web application.

Tomcat is not shutting down

I am unable to shutdown tomcat. When I do ./shutdown.sh, I don't see any error, but the process still seems to be running. When I kill the process (kill -9), the process gets killed. After this, if I do a startup, and shutdown again, I see the same problem. How do I gracefully shut it down? Why is this happening in the first place? Please help me.
devServer:bin dev$ ps -ef|grep tomcat
501 34933 1 0 11:24PM test 0:02.02 /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java -Djava.util.logging.config.file=/Users/dev/admin/runtime/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/Users/dev/admin/runtime/endorsed -classpath /Users/dev/admin/runtime/bin/bootstrap.jar:/Users/dev/admin/runtime/bin/tomcat-juli.jar -Dcatalina.base=/Users/dev/admin/runtime -Dcatalina.home=/Users/dev/admin/runtime -Djava.io.tmpdir=/Users/dev/admin/runtime/temp org.apache.catalina.startup.Bootstrap start
501 34982 33786 0 11:45PM test 0:00.00 grep tomcat
devServer:bin dev$ ./shutdown.sh
Using CATALINA_BASE: /Users/dev/admin/runtime
Using CATALINA_HOME: /Users/dev/admin/runtime
Using CATALINA_TMPDIR: /Users/dev/admin/runtime/temp
Using JRE_HOME: /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
Using CLASSPATH: /Users/dev/admin/runtime/bin/bootstrap.jar:/Users/dev/admin/runtime/bin/tomcat-juli.jar
devServer:bin dev$ ps -ef|grep tomcat
501 34933 1 0 11:24PM test 0:02.03 /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java -Djava.util.logging.config.file=/Users/dev/admin/runtime/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/Users/dev/admin/runtime/endorsed -classpath /Users/dev/admin/runtime/bin/bootstrap.jar:/Users/dev/admin/runtime/bin/tomcat-juli.jar -Dcatalina.base=/Users/dev/admin/runtime -Dcatalina.home=/Users/dev/admin/runtime -Djava.io.tmpdir=/Users/dev/admin/runtime/temp org.apache.catalina.startup.Bootstrap start
501 34992 33786 0 11:45PM test 0:00.00 grep tomcat
devServer:bin dev$
Also, I see my catalina.out is rather too short. There is no port number info getting written to it at all. This is all I see everytime I startup.
Feb 15, 2014 11:24:23 PM org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: .:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java
Here is my log file -
devServer:logs dev$ cat catalina.2014-02-15.log
Feb 15, 2014 11:24:23 PM org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: .:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java
Something seems to be terribly messed up. I can't understand what. Please advise.
I am running tomcat 7.0.50 on Mac OS.
Judging by the logs, it is evident that tomcat is not starting up at all as it should be. I can advise a couple of other tests as well like to
Verify whether or not the ports are getting bound by the tomcat service or not. You can do this by issueing the command netstat -tuplen on the terminal and check to see if port 8080 is getting bound by the tomcat instance or not.
Check to see if you can ping the instance. Try to ping the tomcat instance from a local machine. Then try to ping the public ip of the server to verify whether that server is responding correctly or not
Verify that you have, infact, the complete tomcat template as is used by your other servers. Verify the tomcat installtion folder and compare it with the other tomcat templates as used in rest of your servers. Make sure all folders are there, all configuration (default tomcat config) files are present.
(Just a hunch) Verify read/write/execute privileges as well
Remove your application, shutdown tomcat, empty the logs, and then restart tomcat and re-deploy your application and then view the logs to see if you get some error
Point 3 seems more likely that you have an incomplete tomcat instance, so I advise you to make a copy of the tomcat instance used in any other server and use it in this server to make sure that the server environment is infact how it should be

Jenkins headless linux slave connection error

I get java.net.SocketException: Connection reset when trying to connect a headless debian slave to my main Jenkins master. It's been running good for a couple of weeks without error and suddenly I can't get it to connect.
I can telnet into the master on port 6256.
Slave output:
#> java -jar ~/slave.jar -jnlpUrl https://test.tv/jenkins/computer/debian_slave_node_1/slave-agent.jnlp
Aug 12, 2013 10:10:46 AM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Hudson agent is running in headless mode.
Aug 12, 2013 10:10:46 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [https://test.tv/jenkins/, http://test.tv/jenkins/]
Aug 12, 2013 10:10:46 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to test.tv:6256
Aug 12, 2013 10:10:46 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Aug 12, 2013 10:10:46 AM hudson.remoting.jnlp.Main$CuiListener error
SEVERE: Connection reset
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:189)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at hudson.remoting.ClassicCommandTransport.create(ClassicCommandTransport.java:98)
at hudson.remoting.Channel.<init>(Channel.java:391)
at hudson.remoting.Channel.<init>(Channel.java:387)
at hudson.remoting.Channel.<init>(Channel.java:348)
at hudson.remoting.Channel.<init>(Channel.java:344)
at hudson.remoting.Channel.<init>(Channel.java:332)
at hudson.remoting.Engine.run(Engine.java:238)
Jenkins Master output:
java.io.IOException: Unexpected termination of the channel
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
Caused by: java.io.EOFException
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2596)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1316)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at hudson.remoting.Command.readFrom(Command.java:92)
at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:72)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
Hard to say what happens before knowing more. In particular:
does it fail at every connection request ? If so please check your auth.log on your server as it seems the failure happens after "status INFO: Handshaking" and before "Connected"
are your clock properly synchronized ?
what OSes are you using ? Which java version on your slaves ? check your PATH and JAVA_HOME
have you updated jenkins lately ? If so, have you updated the slave(s) ? Also have you tried reverting the update ? Which version are you using ? 1.520 took out support for JDK 1.5.
how do you run your java command ? From a terminal in an SSH session ? Is this scripted from the master ?
See also Jenkins slave jobs failing on "Unexpected termination of channel"
I've run into the same issue, which seemed to be worked-around by running jenkins-slave as root. Now, the thing periodically disconnects with that very same error, which I'm wondering is due to a LB keepalive. I'll update this answer as soon as I get more info.
In my case everything was working under root account. But under user account, node client failed to connect to server with message "failed Unexpected termination of the channel".
The problem was with access-or-write to working folder. As soon as I've reassigned owner throug "chown -R", node starts successfully.

Tomcat process suddenly crushes

Trying to diagnose some bizarre Tomcat 6 and/or JVM errors on a 64-bit linux machine:
OS VERSION : RED HAT ENTERPRISE LINUX SERVER RELEASE 6.3 (SANTIAGO)
SERVER MODEL : VMWARE VIRTUAL PLATFORM ()
I am running some application tests on virtual machines and facing a weird issue.
The problem occures on certain VMs where exactly 2 process are running: Tomcat Application server process (java process) and a monitor process, at one randomal point, after some random hours of execution the following message can be found on catalina.out (server log) of the VM that fails (the problem doesnt occurs always on same machine):
org.apache.coyote.http11.Http11Protocol pause
INFO: Pausing Coyote HTTP/1.1 on http-50100
INFO: Stopping service Catalina
org.apache.coyote.http11.Http11Protocol destroy
INFO: Stopping Coyote HTTP/1.1 on http-8080
After this moment the Tomcat process is shutted down and execution fails.
No picks can be seen on memory or CPU use.
No system messages can be seen under /var/logs.
At first stage we suspected that an external signal is killing our Tomcat process and we tried to use the -Xrs flag for JVM arguments of the process but issue still occurred.
The java flag (-xrs ) should prevent the Java runtime environment handling exception signals such SIGSEGV and SIGABRT not help
Before adding xrs flag we saw INFO: Pausing Coyote HTTP/1.1 on http-50100 error that happened because outcome signal
After adding `xrs flag the kettle/tomcat behaves the same way and crushes without any errors in log files.
Running starce command on relevant process gives us the following output:
Process 12917 attached - interrupt to quit
futex(0x7f6b9e8ab9d0, FUTEX_WAIT, 12918, NULLPANIC: attached pid 12917 exited with 143
We cannot run starce -f to include all threads because it causes our application to be stuck.
However, killing son process with kill command behaves similary - we see that parent JVM
exit with error code 143 and we see also the signal sent to the son process.
We are running also with java -XX:+HeapDumpOnOutOfMemoryError optind to create a heap dump for jvm crash due to out of memory error but no core dump or heap dumps are creted due to the crush.
Any ideas?
is it possible, that before the crash, there is some sort of memory depletion, so that the java process could be killed by the OOM.
Also, I found a link http://bugs.centos.org/view.php?id=4059 that describes a similar problem.

Remote debugging on Tomcat with Eclipse

I'm trying to initiate a remote debugging session on my PC with Eclipse & Tomcat.
I managed to run tomcat (not through eclipse) with the following params:
set JPDA_TRANSPORT=dt_socket
set JPDA_ADDRESS=5050
cataline jpda start
The server starts ok, this is the outcome:
Using CATALINA_BASE: "C:\Java\Tomcat"
Using CATALINA_HOME: "C:\Java\Tomcat"
Using CATALINA_TMPDIR: "C:\Java\Tomcat\temp"
Using JRE_HOME: "C:\Java\jdk1.6.0_27"
Using CLASSPATH: ...
Listening for transport dt_socket at address: 5050
...
...
INFO: Server startup in 12502 ms
Now, I opened eclipse, and loaded the relevant project.
I set a new debugging configuration using Remote Java Application with localhost and the right port. however when I run it, I get this error message:
Failed to connect to remote VM. Connection refused.
Connection refused: connect
Can't really understand where is the problem. this is all local inside the PC, so there shouldn't be any firewalls involved, can someone think of something ?
Found the solution. apparently the connection was made, but Eclipse & Tomcat don'tt show any notification or status regarding it.
Then, when you initiate the connection again, you get the error because you're already connected.
I think that a small notice, either from Eclipse, or from Tomcat would be nice.
Try adding the following debug options directly to the JVM startup by directly modifying the catalina startup script
-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5050
There could be problems when there are code mismatches between eclipse and tomcat. So when it matches the error will disappear
You can configure it in remote java application section in debug configuration.
For more details you can refer this link

Categories