Trying to diagnose some bizarre Tomcat 6 and/or JVM errors on a 64-bit linux machine:
OS VERSION : RED HAT ENTERPRISE LINUX SERVER RELEASE 6.3 (SANTIAGO)
SERVER MODEL : VMWARE VIRTUAL PLATFORM ()
I am running some application tests on virtual machines and facing a weird issue.
The problem occures on certain VMs where exactly 2 process are running: Tomcat Application server process (java process) and a monitor process, at one randomal point, after some random hours of execution the following message can be found on catalina.out (server log) of the VM that fails (the problem doesnt occurs always on same machine):
org.apache.coyote.http11.Http11Protocol pause
INFO: Pausing Coyote HTTP/1.1 on http-50100
INFO: Stopping service Catalina
org.apache.coyote.http11.Http11Protocol destroy
INFO: Stopping Coyote HTTP/1.1 on http-8080
After this moment the Tomcat process is shutted down and execution fails.
No picks can be seen on memory or CPU use.
No system messages can be seen under /var/logs.
At first stage we suspected that an external signal is killing our Tomcat process and we tried to use the -Xrs flag for JVM arguments of the process but issue still occurred.
The java flag (-xrs ) should prevent the Java runtime environment handling exception signals such SIGSEGV and SIGABRT not help
Before adding xrs flag we saw INFO: Pausing Coyote HTTP/1.1 on http-50100 error that happened because outcome signal
After adding `xrs flag the kettle/tomcat behaves the same way and crushes without any errors in log files.
Running starce command on relevant process gives us the following output:
Process 12917 attached - interrupt to quit
futex(0x7f6b9e8ab9d0, FUTEX_WAIT, 12918, NULLPANIC: attached pid 12917 exited with 143
We cannot run starce -f to include all threads because it causes our application to be stuck.
However, killing son process with kill command behaves similary - we see that parent JVM
exit with error code 143 and we see also the signal sent to the son process.
We are running also with java -XX:+HeapDumpOnOutOfMemoryError optind to create a heap dump for jvm crash due to out of memory error but no core dump or heap dumps are creted due to the crush.
Any ideas?
is it possible, that before the crash, there is some sort of memory depletion, so that the java process could be killed by the OOM.
Also, I found a link http://bugs.centos.org/view.php?id=4059 that describes a similar problem.
Related
I am trying to set up a Zookeeper cluster for Pulsar. I am following the instructions here, but I keep failing.
In my setup, I have two nodes, that should be part of the cluster. Since I need to deploy bookie to the same nodes, I executed
$ PULSAR_EXTRA_OPTS="-Dstats_server_port=8001" bin/pulsar-daemon start zookeeper
to start zookeeper. Afterwards, I am trying to init the cluster using this command:
bin/pulsar initialize-cluster-metadata \
--cluster pulsar-cluster-1 \
--zookeeper 10.100.100.77:2181 \
--configuration-store 10.100.100.77:2181 \
--web-service-url http://10.100.100.77:8080 \
--broker-service-url pulsar://10.100.100.77:6650 \
But I keep getting this error:
17:12:24.146 [main-SendThread(10.100.100.77:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: 10.100.100.77/10.100.100.77:2181: Verbindungsaufbau abgelehnt
17:12:25.251 [main-SendThread(10.100.100.77:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 10.100.100.77/10.100.100.77:2181. Will not attempt to authenticate using SASL (unknown error)
I read here that I need to have an odd number of nodes, so I added a virtual machine on one of the nodes. When I start Zookeeper on it, it doesn't print an error message, but but shows:
$ PULSAR_EXTRA_OPTS="-Dstats_server_port=8001" bin/pulsar-daemon start zookeeper
doing start zookeeper ...
starting zookeeper, logging to /home/host1/apache-pulsar-2.4.0/logs/pulsar-zookeeper-host1-VirtualBox.log
OpenJDK 64-Bit Server VM warning: Option AggressiveOpts was deprecated in version 11.0 and will likely be removed in a future release.
[AppClassLoader#27c170f0] info AspectJ Weaver Version 1.9.2 built on Wednesday Oct 24, 2018 at 15:43:33 GMT
[AppClassLoader#27c170f0] info register classloader jdk.internal.loader.ClassLoaders$AppClassLoader#27c170f0
[AppClassLoader#27c170f0] info using configuration file:/home/host1/apache-pulsar-2.4.0/lib/org.apache.pulsar-pulsar-zookeeper-utils-2.4.0.jar!/META-INF/aop.xml
[AppClassLoader#27c170f0] info using configuration file:/home/host1/apache-pulsar-2.4.0/lib/org.apache.pulsar-pulsar-zookeeper-2.4.0.jar!/META-INF/aop.xml
[AppClassLoader#27c170f0] info register aspect org.apache.pulsar.zookeeper.SerializeUtilsAspect
[AppClassLoader#27c170f0] info register aspect org.apache.pulsar.broker.zookeeper.aspectj.ClientCnxnAspect
However the Zookeeper service is not started, even if the setup is very similar to its host and I can't make up why.
Any Ideas how I could proceed from here? Thanks in advance!
The first error you posted seems to indicate that the connection to 10.100.100.77:2181 is refused "Verbindungsaufbau abgelehnt", and therefore the ZK server isn't listening on that server and port. You should first confirm that ZK is up and running and check the ZK log for any errors.
HTH
I found the soulution. The original error was indeed caused by having an odd number of nodes. The third (virtual) one wouldn't start, because of a mislocation of Zookepers data-directory. Now that the third server started, also the configuration passed successfully.
I have deployed my application in jboss-4.0.2RC1. It is working fine. But after some time unexpectedly jboss is getting shutdown.
Please find the below logs
2015-09-21 21:09:28,995 INFO [org.jboss.system.server.Server] JBoss SHUTDOWN: Undeploying all packages
2015-09-21 21:09:28,995 DEBUG [org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread] Notified to shutdown
2015-09-21 21:09:28,996 DEBUG [org.jboss.deployment.MainDeployer] Undeploying
I am not able to find the root cause?
Please suggest
What operating system is hosting your jboss? There are a number of reasons this could happen depending on your configuration. For instance, if you are running on Windows and running JBoss as a service, it will shutdown upon server logoff, unless you have added to your run.conf.bat something like:
set JAVA_OPTS=%JAVA_OPTS% -Xrs
I am unable to shutdown tomcat. When I do ./shutdown.sh, I don't see any error, but the process still seems to be running. When I kill the process (kill -9), the process gets killed. After this, if I do a startup, and shutdown again, I see the same problem. How do I gracefully shut it down? Why is this happening in the first place? Please help me.
devServer:bin dev$ ps -ef|grep tomcat
501 34933 1 0 11:24PM test 0:02.02 /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java -Djava.util.logging.config.file=/Users/dev/admin/runtime/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/Users/dev/admin/runtime/endorsed -classpath /Users/dev/admin/runtime/bin/bootstrap.jar:/Users/dev/admin/runtime/bin/tomcat-juli.jar -Dcatalina.base=/Users/dev/admin/runtime -Dcatalina.home=/Users/dev/admin/runtime -Djava.io.tmpdir=/Users/dev/admin/runtime/temp org.apache.catalina.startup.Bootstrap start
501 34982 33786 0 11:45PM test 0:00.00 grep tomcat
devServer:bin dev$ ./shutdown.sh
Using CATALINA_BASE: /Users/dev/admin/runtime
Using CATALINA_HOME: /Users/dev/admin/runtime
Using CATALINA_TMPDIR: /Users/dev/admin/runtime/temp
Using JRE_HOME: /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
Using CLASSPATH: /Users/dev/admin/runtime/bin/bootstrap.jar:/Users/dev/admin/runtime/bin/tomcat-juli.jar
devServer:bin dev$ ps -ef|grep tomcat
501 34933 1 0 11:24PM test 0:02.03 /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java -Djava.util.logging.config.file=/Users/dev/admin/runtime/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/Users/dev/admin/runtime/endorsed -classpath /Users/dev/admin/runtime/bin/bootstrap.jar:/Users/dev/admin/runtime/bin/tomcat-juli.jar -Dcatalina.base=/Users/dev/admin/runtime -Dcatalina.home=/Users/dev/admin/runtime -Djava.io.tmpdir=/Users/dev/admin/runtime/temp org.apache.catalina.startup.Bootstrap start
501 34992 33786 0 11:45PM test 0:00.00 grep tomcat
devServer:bin dev$
Also, I see my catalina.out is rather too short. There is no port number info getting written to it at all. This is all I see everytime I startup.
Feb 15, 2014 11:24:23 PM org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: .:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java
Here is my log file -
devServer:logs dev$ cat catalina.2014-02-15.log
Feb 15, 2014 11:24:23 PM org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: .:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java
Something seems to be terribly messed up. I can't understand what. Please advise.
I am running tomcat 7.0.50 on Mac OS.
Judging by the logs, it is evident that tomcat is not starting up at all as it should be. I can advise a couple of other tests as well like to
Verify whether or not the ports are getting bound by the tomcat service or not. You can do this by issueing the command netstat -tuplen on the terminal and check to see if port 8080 is getting bound by the tomcat instance or not.
Check to see if you can ping the instance. Try to ping the tomcat instance from a local machine. Then try to ping the public ip of the server to verify whether that server is responding correctly or not
Verify that you have, infact, the complete tomcat template as is used by your other servers. Verify the tomcat installtion folder and compare it with the other tomcat templates as used in rest of your servers. Make sure all folders are there, all configuration (default tomcat config) files are present.
(Just a hunch) Verify read/write/execute privileges as well
Remove your application, shutdown tomcat, empty the logs, and then restart tomcat and re-deploy your application and then view the logs to see if you get some error
Point 3 seems more likely that you have an incomplete tomcat instance, so I advise you to make a copy of the tomcat instance used in any other server and use it in this server to make sure that the server environment is infact how it should be
After running my App in Tomcat many times, I've maxed out the PermGen memory (a problem with my quartz servlet, I suspect) and Java threw a OutOfMemoryError. Normally, I just restart tomcat from time to time, but since this happened I'm unable to start/restart. Tomcat is running in our qa environment and is normally started/stoped/restarted via the sbin folder. I've tried to start tomcat using the alternative, catalina.sh run, but it says the port is being used! That led me to suspect that tomcat was already running, but when I try to stop it, I get this error:
SEVERE: Could not contact localhost:8005. Tomcat may not be running.
Jan 19, 2014 3:10:58 PM org.apache.catalina.startup.Catalina
stopServer SEVERE: Catalina.stop: java.net.ConnectException:
Connection refused at java.net.PlainSocketImpl.socketConnect(Native
Method) at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at
java.net.Socket.connect(Socket.java:579) ...
So, if it's not started or stopped, then what state is tomcat in?! How can I bring it back to a startable state?
Update
As per Boris's comment, I've used ps auxfww to list processes and the only one with tomcat as the user is the following:
tomcat 5111 8.0 29.0 2052336 556512 ? Sl Jan17 221:48
/usr/java/latest/bin/java
-Djava.util.logging.config.file=/var/lib/tomcat/conf/logging.properties
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -javaagent:/usr/share/tomcat/newrelic/newrelic.jar -Xmx512m -Djava.awt.headless=true -Djava.endorsed.dirs=/usr/share/tomcat/endorsed -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar
-Dcatalina.base=/var/lib/tomcat -Dcatalina.home=/usr/share/tomcat -Djava.io.tmpdir=/var/lib/tomcat/temp org.apache.catalina.startup.Bootstrap start
For some reason, even as root, I can't delete the process. I use
sudo kill 5111
but the process is still listed after and tomcat still won't restart...
Read about kill here
sudo kill -9 5111 will do the work. Tomcat is binded to 8005 port: until you kill the process with a SIGKILL signal it won't release the port (and you are not allowed to start another tomcat on this port).
I am trying to deploy a simple WebService in Netbeans 6.7.1 and Glassfish v2.1.1 and am getting the following error. I am using GlassfishESBv2.2 and windows 7 machine. I have tried googling and implemented things as shown in
http://forums.netbeans.org/topic10055-0-asc-0.html . Still unable to deploy. Though the message says that application server is not started, from the Server tab , I am able to see a message which indicates Glassfish has started.Also , doing a netstat after trying to deploy returns this, which means that Glassfish is running.
C:>netstat -an | findstr "4848"
TCP 0.0.0.0:4848 0.0.0.0:0 LISTENING
I have been trying real hard to get this resolved. Any help is highly appreciated.
Error Message :
The Sun Java System Application Server could not start.
More information about the cause is in the Server log file.
Possible reasons include:
- IDE timeout: refresh the server node to see if it's running now.
- Port conflicts. (use netstat -a to detect possible port numbers already used by the operating system.)
- Incorrect server configuration (domain.xml to be corrected manually)
- Corrupted Deployed Applications preventing the server to start.(This can be seen in the server.log file. In this case, domain.xml needs to be modified).
- Invalid installation location.
C:\Users\xyz\Documents\NetBeansProjects\HWebService\nbproject\build-impl.xml:564: Deployment error:
The Sun Java System Application Server could not start.
More information about the cause is in the Server log file.
Possible reasons include:
- IDE timeout: refresh the server node to see if it's running now.
- Port conflicts. (use netstat -a to detect possible port numbers already used by the operating system.)
- Incorrect server configuration (domain.xml to be corrected manually)
- Corrupted Deployed Applications preventing the server to start.(This can be seen in the server.log file. In this case, domain.xml needs to be modified).
- Invalid installation location.
See the server log for details.
BUILD FAILED (total time: 29 seconds)
I've described the solution please visit here...
http://forums.netbeans.org/post-65058.html