I have a Spring Boot executable application - It's REST webservice. Service is working well, but when I try to gracefully shutdown it (by issuing process interrupt - SIGINT), service "hangs" and is still running. Here is log after shutdown command (red square stop button in Intellij IDEA):
Disconnected from the target VM, address: '127.0.0.1:36656', transport: 'socket'
2016-10-28 11:15:26.369 INFO 5429 --- [ Thread-2] ationConfigEmbeddedWebApplicationContext : Closing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext#160ac7fb: startup date [Thu Oct 27 18:32:46 CEST 2016]; root of context hierarchy
Red stop button changes to skull (when clicked, SIGKILL is sent and process is immediatelly dead)
Apparently, something (resource?) is blocking closing of spring context and exiting application normal way.
My question is: How can I investigate what is wrong? What is the cause of blocking?
Dependencies of project:
spring-boot-starter-parent 1.4.1.RELEASE
spring-boot-starter-web
com.oracle:ojdbc6
com.thoughtworks.xstream:xstream
and some commons
Btw, service is running on Oracle Java "Java(TM) SE Runtime Environment (build 1.8.0_111-b14)" and on Ubuntu 16.04.1 LTS (and will run on production also)
I suspect xstream library, but I don't know how to find out. I would appreciate any suggestions.
Related
I am trying to set up a Zookeeper cluster for Pulsar. I am following the instructions here, but I keep failing.
In my setup, I have two nodes, that should be part of the cluster. Since I need to deploy bookie to the same nodes, I executed
$ PULSAR_EXTRA_OPTS="-Dstats_server_port=8001" bin/pulsar-daemon start zookeeper
to start zookeeper. Afterwards, I am trying to init the cluster using this command:
bin/pulsar initialize-cluster-metadata \
--cluster pulsar-cluster-1 \
--zookeeper 10.100.100.77:2181 \
--configuration-store 10.100.100.77:2181 \
--web-service-url http://10.100.100.77:8080 \
--broker-service-url pulsar://10.100.100.77:6650 \
But I keep getting this error:
17:12:24.146 [main-SendThread(10.100.100.77:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: 10.100.100.77/10.100.100.77:2181: Verbindungsaufbau abgelehnt
17:12:25.251 [main-SendThread(10.100.100.77:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 10.100.100.77/10.100.100.77:2181. Will not attempt to authenticate using SASL (unknown error)
I read here that I need to have an odd number of nodes, so I added a virtual machine on one of the nodes. When I start Zookeeper on it, it doesn't print an error message, but but shows:
$ PULSAR_EXTRA_OPTS="-Dstats_server_port=8001" bin/pulsar-daemon start zookeeper
doing start zookeeper ...
starting zookeeper, logging to /home/host1/apache-pulsar-2.4.0/logs/pulsar-zookeeper-host1-VirtualBox.log
OpenJDK 64-Bit Server VM warning: Option AggressiveOpts was deprecated in version 11.0 and will likely be removed in a future release.
[AppClassLoader#27c170f0] info AspectJ Weaver Version 1.9.2 built on Wednesday Oct 24, 2018 at 15:43:33 GMT
[AppClassLoader#27c170f0] info register classloader jdk.internal.loader.ClassLoaders$AppClassLoader#27c170f0
[AppClassLoader#27c170f0] info using configuration file:/home/host1/apache-pulsar-2.4.0/lib/org.apache.pulsar-pulsar-zookeeper-utils-2.4.0.jar!/META-INF/aop.xml
[AppClassLoader#27c170f0] info using configuration file:/home/host1/apache-pulsar-2.4.0/lib/org.apache.pulsar-pulsar-zookeeper-2.4.0.jar!/META-INF/aop.xml
[AppClassLoader#27c170f0] info register aspect org.apache.pulsar.zookeeper.SerializeUtilsAspect
[AppClassLoader#27c170f0] info register aspect org.apache.pulsar.broker.zookeeper.aspectj.ClientCnxnAspect
However the Zookeeper service is not started, even if the setup is very similar to its host and I can't make up why.
Any Ideas how I could proceed from here? Thanks in advance!
The first error you posted seems to indicate that the connection to 10.100.100.77:2181 is refused "Verbindungsaufbau abgelehnt", and therefore the ZK server isn't listening on that server and port. You should first confirm that ZK is up and running and check the ZK log for any errors.
HTH
I found the soulution. The original error was indeed caused by having an odd number of nodes. The third (virtual) one wouldn't start, because of a mislocation of Zookepers data-directory. Now that the third server started, also the configuration passed successfully.
I'm running my Spring Boot application with debug mode enabled. Like:
mvn -P dev spring-boot:run -Drun.jvmArguments="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005"
I can debug the application after that without any problem. But, after stopped (ctrl + c) and try to start again, I receive this error:
[INFO] Attaching agents: []
ERROR: transport error 202: bind failed: Address already in use
ERROR: JDWP Transport dt_socket failed to initialize, TRANSPORT_INIT(510)
JDWP exit error AGENT_ERROR_TRANSPORT_INIT(197): No transports initialized [debugInit.c:750]
So, every time that I need to start the application in debug mode after that, I need to kill the process using the port 5005.
I would like to know why the port 5005 is still opened after stop the application.
I'm using Cygwin64 on Windows 10, Maven 3.5.2, Java 8 and Spring Boot 1.5.9.RELEASE.
Thanks!
This is not a problem of Spring Boot, but a problem of non-cygwin applications running in cygwin. And your application is not only leaving port 5005 open, it just keeps running in the background, having port 8080 also opened, but as on a restart the debug port is first opened, this is what you see in the error message.
This was discussed in a SpringBoot issue back in 2014 https://github.com/spring-projects/spring-boot/issues/773. I cite from this issue (the link in the citation dates from 2006):
Anyone still following this bug, this isn't a Spring Boot bug at all. It's a known problem with non-Cygwin programs (Java in this case) running on pty-based terminals.
See this link for a full explanation why this can't be fixed: http://cygwin.com/ml/cygwin/2006-12/msg00151.html
So you can basically use one of the following approaches:
use the native windows terminal and setup your java and maven environment for that
run your maven command from within an IDE which can handle this - you might use this for debugging as well
add the Spring Actuator to your app and use the /shutdown endpoint to stop your application
I am new to Apache OFBiz. I have downloaded and installing the OFBiz by instructions. I have loaded the data successfully.
But when am running OFBiz using gradlew it stopped at birt-container. I have added the console information here.
Build process stopped at 91% with the message of "Started container birt-container" for 5 hours and I have stopped the process. Still no idea why it is stopping here.
2017-05-04 11:01:42,143 |main |CatalinaContainer |I| Connector HTTP/1.1 # 8443 - secure [org.apache.coyote.http11.Http11NioProtocol] started.
2017-05-04 11:01:42,143 |main |CatalinaContainer |I| Started Apache Tomcat/8.0.37
2017-05-04 11:01:42,143 |main |ContainerLoader |I| Started container catalina-container
2017-05-04 11:01:42,143 |main |ContainerLoader |I| Starting container birt-container
2017-05-04 11:01:42,143 |main |BirtContainer |I| Start BIRT container
2017-05-04 11:01:42,152 |main |BirtContainer |I| Startup BIRT platform
2017-05-04 11:01:43,034 |main |BirtContainer |I| Create factory object
2017-05-04 11:01:43,052 |main |BirtContainer |I| Create report engine
2017-05-04 11:01:43,130 |main |BirtContainer |I| BIRT supported formats: xlsx, postscript, docx, pptx, pdf, ppt, xls_spudsoft, doc, html,
odp, xls, odt, ods
2017-05-04 11:01:43,130 |main |ContainerLoader |I| Started container birt-container
> Building 91% > :ofbiz
The Gradle percentage is bit misleading here, it does not reach 100% as long as OFBiz is running. I think you just assumed after building the service will be up. Not like that. Once the Service is up it will be in 91%. But the Service has been started. You can access the E-Commerce and other services. Once you are running service get the following URL (ex).
localhost:8443/ap/control/main
You will be redirected to admin login page. Give user as "admin" and password as "ofbiz". Do the changes and test your data as you want in Accounting, Manufacturing, HR , etc..
Note: After completed your process stop the process using "stopofbiz" shell script file under tools folder. After stop the service you can able to see the build will be reach 100% in service console.
This is a duplicate to Installing OfBiz in debug mode hangs
I think OFBiz does not hang and that you should be able to access and work with it without problems. The Gradle percentage is bit misleading here, it does not reach 100% as long as OFBiz is running.
See also https://lists.apache.org/thread.html/26543a82adf1f8d9a8bec47975e8d87f750c2de2f2da7e242b04a72e#%3Cuser.ofbiz.apache.org%3E
Prepare OFBiz:
Clean all:
gradlew cleanAll loadDefault testIntegration
Note: Depending on your Internet connection speed it might take a long time for this step to complete if you are using OFBiz for the first time as it needs to download all dependencies. So please be patient!
MS Windows: gradlew cleanAll loadDefault
Unix-like OS: ./gradlew cleanAll loadDefault
Start OFBiz:
MS Windows: gradlew ofbiz
Unix-like OS: ./gradlew ofbiz
Reference: After download http://www.apache.org/dyn/closer.lua/ofbiz/apache-ofbiz-16.11.02.zip , unzip, look for
apache-ofbiz-16.11.02/tools/documentation/README.md.html
I have deployed my application in jboss-4.0.2RC1. It is working fine. But after some time unexpectedly jboss is getting shutdown.
Please find the below logs
2015-09-21 21:09:28,995 INFO [org.jboss.system.server.Server] JBoss SHUTDOWN: Undeploying all packages
2015-09-21 21:09:28,995 DEBUG [org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread] Notified to shutdown
2015-09-21 21:09:28,996 DEBUG [org.jboss.deployment.MainDeployer] Undeploying
I am not able to find the root cause?
Please suggest
What operating system is hosting your jboss? There are a number of reasons this could happen depending on your configuration. For instance, if you are running on Windows and running JBoss as a service, it will shutdown upon server logoff, unless you have added to your run.conf.bat something like:
set JAVA_OPTS=%JAVA_OPTS% -Xrs
Trying to diagnose some bizarre Tomcat 6 and/or JVM errors on a 64-bit linux machine:
OS VERSION : RED HAT ENTERPRISE LINUX SERVER RELEASE 6.3 (SANTIAGO)
SERVER MODEL : VMWARE VIRTUAL PLATFORM ()
I am running some application tests on virtual machines and facing a weird issue.
The problem occures on certain VMs where exactly 2 process are running: Tomcat Application server process (java process) and a monitor process, at one randomal point, after some random hours of execution the following message can be found on catalina.out (server log) of the VM that fails (the problem doesnt occurs always on same machine):
org.apache.coyote.http11.Http11Protocol pause
INFO: Pausing Coyote HTTP/1.1 on http-50100
INFO: Stopping service Catalina
org.apache.coyote.http11.Http11Protocol destroy
INFO: Stopping Coyote HTTP/1.1 on http-8080
After this moment the Tomcat process is shutted down and execution fails.
No picks can be seen on memory or CPU use.
No system messages can be seen under /var/logs.
At first stage we suspected that an external signal is killing our Tomcat process and we tried to use the -Xrs flag for JVM arguments of the process but issue still occurred.
The java flag (-xrs ) should prevent the Java runtime environment handling exception signals such SIGSEGV and SIGABRT not help
Before adding xrs flag we saw INFO: Pausing Coyote HTTP/1.1 on http-50100 error that happened because outcome signal
After adding `xrs flag the kettle/tomcat behaves the same way and crushes without any errors in log files.
Running starce command on relevant process gives us the following output:
Process 12917 attached - interrupt to quit
futex(0x7f6b9e8ab9d0, FUTEX_WAIT, 12918, NULLPANIC: attached pid 12917 exited with 143
We cannot run starce -f to include all threads because it causes our application to be stuck.
However, killing son process with kill command behaves similary - we see that parent JVM
exit with error code 143 and we see also the signal sent to the son process.
We are running also with java -XX:+HeapDumpOnOutOfMemoryError optind to create a heap dump for jvm crash due to out of memory error but no core dump or heap dumps are creted due to the crush.
Any ideas?
is it possible, that before the crash, there is some sort of memory depletion, so that the java process could be killed by the OOM.
Also, I found a link http://bugs.centos.org/view.php?id=4059 that describes a similar problem.