We Keep Coding

JVM crash during Weblogic Server Start and huge error log

I am using Weblogic 10.3.6 and deployed few applications. I always do a clean stop start with the server. Suddenly one day when I am starting my server, it goes until 97% and then the JVM is getting crashed, and JAVA process stops and the server is unable to start from then. I tried to investigate from the WLS log file but it has nothing much except the JVM crash report. Also I saw there are many error logs which were mainly related to one of my application (like dispatcher servlet, struts2 error .. fileNotFound error etc).
Is it due to the huge application error getting piled up which is causing the JVM restart ? I have removed all the temp and cache. Restarted my machine but still unbale to start the weblogic server. Is there any way to start it ? I am not willing to delete the domain and create it again because there are so many configurations present.

I have removed the erroneous application from the list of deployed application from file config.xml inside domain_name/config folder. Then gave a server start and my WLS started fine. The issue was due to the faulty application. Thanks I did not have to delete the entire domain. But I could not establish as why there was a JVM crash and how shall I handle it from next time.

Mysterious Jboss shutdown

We have a several application deployed in Jboss server. From last few month its observed that jboss servers gets killed without logging any error or exception neither in any application log nor in Jboss server log. The crash is not periodic and happens randomly. The crash is very silent (similar to kill -9 )and its become extremely difficult to debug the root cause.From our analysis one thing sure its not out of memory issue.We analysed the syslog as well, nothing suspicious found. Is possible to force VM to generate error log before the process crash due to any reason.
Any help would be appreciated.

The JVM logs fatal errors to hs_err_<pid>.log files
You can change the location of the fatal log file using java -XX:ErrorFile=/var/log/java/java_error%p.log

I experienced similar problem using JBoss EAP 6 in Linux enviroment, when system memory is almost gone the S.O kill some process, normally the file hs_err_pidxxx.log with crash report are generated inside bin directory of JBoss. Other case that process is killed is when you are using JBoss with DC mode, when instance stay long time without communication with DC, the process is finalized, I dont know why.

Analyze stand alone Java Application locally for threads

Respected Experts,
I have a stand alone java application and want to monitor the threads created by it. I am planning to use a tool like JConsole or JVisualVM. However, I am not able to connect these tools locally to my Java Application.
I am using Windows machine. JConsole and Java program are running locally. I have tried to run Java application with following JMV arguments with no success:
java -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false LinkedListTest
When I try to connect using JConsole, my process id is greyed and following message appears:
Note: The management agent is not enabled on this process
I think I should be able to connect JConsole to a stand alone java application. Any thoughts what I am missing here
Thanks and Regards

Thanks for the inputs. I was able to solve the problem and the details are as follows:
As #Holger mentioned, both JConsole and JVisualVM can connect to the local Java Application running without any JMX arguments. The problem that I was facing was machine specific. I restarted the machine and deleted the following directory:
%TMP%\hsperfdata_User.Name
(I was not able to delete this directory without doing a restart)
Restarted JConsole/JVisualVM and was able to connect to local java processes using the process id.
In fact, the use of JVisualVM pointed me towards this fix. On start of JVisualVM, I got an error message stating something like local processes/applications can't be monitored. The message had a link to Troubleshooting guide. I am reproducing the relevant snippet:
Local Applications Cannot Be Monitored (Error Dialog On Startup)
Description: An error dialog saying that local applications cannot be monitored is shown >immediately after VisualVM startup. Locally running Java applications are displayed as Application> (pid ###).
Resolution: This can happen on Windows systems if the username contains capitalized letters. In >this case, username is UserName but the jvmstat directory created by JDK is >%TMP%\hsperfdata_username. To workaround the problem, exit all Java applications, delete the >%TMP%\hsperfdata_username directory and create new %TMP%\hsperfdata_UserName directory.
However, on my machine directory had the following format:
%TMP%\hsperfdata_User.Name
So, my recommendations are:
-check the name of the above mentioned directory for presence of camel case user name
-If not, follow the steps from the Trouble Shooting guide
-If the problem persists, delete the directory (may require machine reboot, as in my case)
-Restart JConsole/JVisualVM
Hopefully, the problem would be resolved.
Thanks and Regards

Tomcat shuts down automatically

I'm using Apache and Tomcat on a Windows server and since this morning, Tomcat stops working without any logs. It doesn't hang, it just shut down.
There's no log in Tomcat, the CPU/Memory are fines, there are no System.Exit in my code.
Anybody ever had this problem?
It happens at random, after 5-10 minutes. The application responds normally and sometime, boom.. stops working.
UPDATE : Still no clue. The Admin team will install the webapp on another box...

My script to start tomcat had last line tail -f catalina.out.
Sometime I did not kill this script, the shell then timed out and killed the script with all child processes, including tomcat.

This sounds like the JVM is crashing. Have you looked for a JVM crash log? It typically has a name like hs_err_pid*.log and is created in the JVM's working directory.
If you find a file like this and upload it, then we can probably help more.
Some questions:
Have you recently changed the version of Java you are using?
What is the exact version of Tomcat you are using?
Are you using Tomcat Native (the Apache Portable Runtime)?

Faced this issue recently.
Scenario : Tomcat started successfully but automatically gets shut down after 1 hour and sometimes this happened after 1 day and nothing is there in tomcat logs.
Issue : Actual issue was high memory usage and no free SWAP memory.
How I found the solution
If tomcat don't show any logs, then there must be something in system logs so, I checked /var/log/messages but since permission denied for me I tried /var/log/dmesg and got this
"Out of memory: Kill process 14606 (java) score 106 or sacrifice child".
In the output I noticed Swap Memory free 0 K. Ran top command to confirm the same. So, somehow there was a high memory usage which caused the OS to kill my tomcat process.
After spending hours finally got the reason.
ps -ef | grep tomcat showed that there were several tomcat processes running for the same application. It seems that, earlier tomcat shutdowns might not have taken successfully and the processes were not killed even after the shutdown due to some reason, which was causing the high memory usage.
So, killed all running tomcat processes using kill. SWAP memory got freed.
Started tomcat again, worked fine. :)

Recently I had this problem, If somebody faces the same issue in future I hope this will help.
Scenario: Tomcat shuts down without any logs or errors
Root Cause for my problem: synchronized method accessed from a task using TimerTask
I had a singleton class with a synchronized method accessed from various threads based on timer or user action
some times this method will take up to few minutes to complete. When TimerTask is waiting on this method for sometime (I guess timer is timed out /thread is killed or something is happening in the background) and the moment the lock on the method is released the tomcat is getting killed.
So I removed synchronized keyword and removed singleton and made some code changes for thread safety. Then the problem is gone.
How I found out: I had a log statement in the first line of synchronized method and everytime the tomcat shutdowns i found this message in the last few lines.
Regards,
Phanindra Kasturi

things to look for in debugging an issue like this:
Look at the logs directory ($TOMCAT_ROOT/logs) to make sure none of the log files have any stack traces
Look at the tomcat startup script to make check the location of the log files to see if the logs are not being written to another directory.
Another reason could be some other user/process could be issuing a kill -9 that could kill tomcat without giving it any chance to log errors.

another possibility is that some process was started this morning on the box that is binding to a port that your server requires.

Are your servlets or one of it's dependencies allowed to call System.exit()? (Not sure how locked down Tomcat VMs are in that sense)
I've had developers thinking it's ok to use exit(666); on detecting a non-invertable matrix (which isn't good, but sure as heck not fatal). Arrgh. Perhaps you have some similar culprit in your system?

I noticed CATALINA_OPTS in my path and that was set for a lower JVM size. Hence, the crash and no log trace of tomcat was found. The server automatically shutdown in less than 2 hrs.
check, CATALINA_OPTS or JAVA_OPTS -- these might have jvm settings. either increase them or comment them out and increase the swap memory.

“The Service on local computer started and then stopped ,Some services stop automatically if there are not in use by other services or programs.”
I gone through the problem i have tried so many ways to get out of the problem finally i got the solution as follows.
1) Click Run Command from start button.
2) Enter Services.msc then click OK,you will get all the services in your computer.
3) Select your service and right click on the service and select Properties
4) Goto Logon Properties and select Local System Account then click OK.
This will work.

Sometime it happens if some other program is running on the same port. For example Skype. Shut down that program before you start Tomcat.

try to clean your elipse projects because you could have tried to add another server which used port 8080 then when you try to execute the tomcat server externally that defaulty uses port 8080 the tomcat server automatically shutdowns after cleaning the project copy the new war file and paste it in bin it works fine
conclusion: when the server tries to use the port which has already been acquired you will see such type of issues.

Using the Netbeans 6.5 profiler to profile a remote web app?

I'm having trouble using the Netbeans 6.5 profiler to profile the memory usage of a java web app, remotely.
I have followed the steps in the tutorial, and was able to customize a Tomcat instance to start with the profiling agents. I can also attach the Netbeans profiler to the running server: for some seconds, the profiler shows graphs with the memory usage of the server.
However, I am not able to run my deployed web app. After a few seconds, the server is shut down gracefully as if it simply had reached the bottom of its method main (hardly the expected behavior of a web server).
During these seconds, the server is not responsible, and I can't connect to a web application in it, they are simply not deployed.
I do not want to import my app into a new Netbeans project just for profiling it. It's possible to profile a remote app without access to its source, right ? After all, for a few seconds, the server was running and being profiled.
I want to profile a web application in Netbeans 6.5. What did I forget ?

You can use VisualVM tool as well. Follow the below mentioned steps.
1) Run jstatd ( jstat demon process) ( Run on your remote machine where application is running)
jstatd -p 1098 -J-Djava.security.policy=tools.policy ( specify unused port ) if tools.policy file is not specified create it as follows.
tools.policy
grant codebase "file:${java.home}/../lib/tools.jar" {
permission java.security.AllPermission;
};
2) Also specify the following properties before you start your JVM.
#JMX property
JVM_ARGS="${JVM_ARGS} -Dcom.sun.management.jmxremote=true"
JVM_ARGS="${JVM_ARGS} -Dcom.sun.management.jmxremote.port=10992"
JVM_ARGS="${JVM_ARGS} -Dcom.sun.management.jmxremote.authenticate=false"
JVM_ARGS="${JVM_ARGS} -Dcom.sun.management.jmxremote.ssl=false"
3) Start the visual VM on local machine.
a. Under remote tab add remote node.
b. Wait for couple of minutes you will see all the core applications instance will be seen under remote tab.
c. Click on remote node then select Add JMX connections specify node:port (port mentioned above in point 2.)
d. Wait for few seconds you will see JMX connection under remote node.
4) After few seconds you will see new tab appearing under remote node. Double click on the remote_node:10992 you will see the profile table on right hand side.
You can see memory , cpu , thread etc details ..

Are profiling the entire webserver or did you use the profiling filter to profile only your package?
Reducing the interaction between the profiler and the app server itself can solve it.

Check your jvm version. The version used by netbeans should be same as that in tomcat.
Use default ports. If you have changed any ports ensure they are above 1024.
Try to profile the example application in tomcat (remove all other applications). If it is running sucessfully it means there is a memory constraint.
Profiling requires huge memory. use -xms1024m -xmx1024m for tomcat.
If every thing else fail. think of switching to jdk6 and tomcat 6.
After many trials I discoverd that there are lot of limitations in jdk5 and tomcat5. switching to tomcat6 and jdk6 solved the problem for me.
Profiling with netbeans does not work jrockit, ibm jdk etc..
--kiran.kumar