I have a spring boot java project, which has a open telemetry jar. The service is packaged as a docker image and it is deployed on Kubernetes.
When the service starts, this open telemetry jar tries to instantiate okHttpClient internally and fails. This instantiation in the open telemetry jar doesn't seem have retry limits, so it tries to load the class repeatedly. Eventually, the service shutsdown with OutOfMemoryError.
This behavior is sporadic. For eg - if there are multiple instances of the service running, occasionally 1 or 2 instances fail with this error. But when these instances are killed and restarted the issue doesn't come up again.
Also, the issue could not be replicated in the developer machines.
Can someone please give me pointers on how to debug this issue?
Related
I have created a Spring Boot microservice and hosted inside a Tomcat on a Linux machine.
Everything is inside the container and the container is inside the IBM cloud private platform.
Now the microservice should be running continuously.
But suppose for any reasons the microservice got stop or tomcat got crashed.
Is there any way we could restart the Tomcat server or microservice automatically without manual intervention?
Why are you deploying a Spring boot app in your local tomcat? By default Springboot comes with embedded Tomcat server in it, so if you just build and run the jar, a tomcat will be started with the service itself.You can also , configure the server type(tomcat or jetty) and other server properties in the application.yml file. More details here - https://www.springboottutorial.com/spring-boot-with-embedded-servers-tomcat-jetty
Coming to the second part, of the question,about how to make sure , that if one service crashes, a new instance should be started automatically, for this you might be needing to do some reading on container managers like dockerswarm or kubernetes, which support auto-scaling, and can take care of restarting services (pods) as and when required,they can even scale up, meaning increase the number of instances of a service if existing containers reach a resource usage threshold and then load balancing requests to that service through some discovery and registry client.I think this would be helpful to you - https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#restart-policy
If there is a OOME and the application crashes, what is the default behaviour of Elastic Beanstalk; does it restart the application?
We have also put in -XX:+ExitOnOutOfMemoryError JVM flag to ensure it doesn't stay in a stuck state.
Elastic Beanstalk won't restart anything for you. The EB config files only runs a set of commands. If it runs out of memory during the launch process the deployment will fail.
You should use the /health domain to determine if you your instance and application is running and is working correctly. Elastic Beanstalk (Elastic Load Balancer) checks this url every minute to verify. If it doesn't get response code 200 it will terminate the instance and launch a new one.
Another option would to setup a cron job that checks so that things are running correctly and do what is necessary for it work correctly again.
The bottom thing here though seems to be that you aren't using the correct instance type for your application. Either you should optimise your application to use less memory or use a bigger instance.
I am running a small webservice application with Spring Boot, Maven, Spring Data, Hibernate which is usually working fine. It has just a single webservice request mapping. The application is a standalone JAR and started in a shell with "screen".
Nearly once per day it just quits itself. There is nobody that forces the app to stop or so. It just says "Killed" in the shell. There is no exception, error or any other message. I already tried to activate logging in application.properties with:
logging.level.org.springframework.web=ERROR
logging.level.org.hibernate=ERROR
but that does not help. Also added exception handling to the webservice mapping trying to catch any exception thrown. But also nothing.
Does anyone know why the app could have been stopped or how I can display the problem?
Thanks in advance!
I encountered with the same problem and I had resolved it by creating swap area. I think it is a typical memory problem. I recommend you to create swap area if you are deploying the application in a linux environment. Check memory usage before..
This happens often when system kills your Java process because it started to consume a lot of memory.
Try to inspect your deployment target limitations and adjust heap size below that limit.
I have a web application running in Tomcat on Linux. My webapp uses a third party jar, and the jar uses a native library.
Whenever I redeploy my application, Tomcat notices the new war file and reloads my application. However, it apparently doesn't unload the previous version, and when I try to execute code that uses the jar, I get this error:
java.lang.UnsatisfiedLinkError: Native Library /var/cache/tomcat7/temp/libaocr_x64.so already loaded in another classloader
If I restart the entire Tomcat process, I can clear the error, but I would prefer not to do that.
What is causing Tomcat to not release the old version of the app?
I'm aware that this kind of problem is sometimes caused by connection pools that have threads running to monitor connections -- I have a connection in my app, but I don't think it's configured to have this kind of thread running.
Thanks,
Frank
JNI is so low-level that it is outside of Tomcats Container. Unfortunately You can not unload the librarys.
Try to put the jar into tomcat's endorsed-folder ($CATALINA_HOME/commons/endorsed) but it will be available for each app in the tomcat.
So we have a busy legacy web service that needs to be replaced by a new one. The legacy web service was deployed using a WAR file on an apache tomcat server. That is it was copied over into the web apps folder under tomcat and all went well. I have been delegated with the task to replace it and would like to do it ensuring
I have a back up of the old service
the service gets replaced by another WAR file with no down time
Again I know I am being overly cautious however it is production level and I would like everything to go smooth. Step by step instructions would help.
Make a test server
Read tutorials and play around with the test server until it goes smoothly
Replicate what you did on the test server on the prod server.
If this really is a "busy prod server" with "no down time", then you will have some kind of test server that you can get the configuration right on.
... with no down time
If you literally mean zero downtime, then you will need to replicate your webserver and implement some kind of front-end that can transparently switch request streams to different servers. You will also need to deal with session migration.
If you mean with minimal downtime, then most web containers support hot redeployment of webapps. However, this typically entails an automatic shutdown and restart of the webapp, which may take seconds or minutes, depending on the webapp. Furthermore there is a risk of significant memory leakage; e.g. of permgen space.
The fallback is a complete shutdown / restart of the web container.
And it goes without saying that you need:
A test server that replicates your production environment.
A rigorous procedure for checking that deployments to your test environment result in a fully functioning system.
A preplanned, tested and hopefully bomb-proof procedure for rolling back your production system in the event of a failed deployment.
All of this (especially rollback) gets a lot more complicated when you system includes other stuff apart from the webapp; e.g. databases.