If there is a OOME and the application crashes, what is the default behaviour of Elastic Beanstalk; does it restart the application?
We have also put in -XX:+ExitOnOutOfMemoryError JVM flag to ensure it doesn't stay in a stuck state.
Elastic Beanstalk won't restart anything for you. The EB config files only runs a set of commands. If it runs out of memory during the launch process the deployment will fail.
You should use the /health domain to determine if you your instance and application is running and is working correctly. Elastic Beanstalk (Elastic Load Balancer) checks this url every minute to verify. If it doesn't get response code 200 it will terminate the instance and launch a new one.
Another option would to setup a cron job that checks so that things are running correctly and do what is necessary for it work correctly again.
The bottom thing here though seems to be that you aren't using the correct instance type for your application. Either you should optimise your application to use less memory or use a bigger instance.
Related
My spring boot application becomes slow after 1-2 days when I deploy it on production server. I'm using AWS EC2 instance. In start the speed is fine, but after a couple of days I have to restart my instance to get back the desired performance. Any hint what might be wrong here?
Have you check for memory leakage in application as it is nothing to do with EC2 instance. As you mention it was working fine after restart.
It is not best practice to use embed server on production.
I would suggest you should use AWS Elastic Beanstalk service for deploying spring boot application, there is no additional charge on it.
Okay so after some analysis (Thread dumping of my tomcat server on production) I found out that there were some processes (code-smells) which were taking all of my CPU space, and hence my instance was becoming slow, and effecting the performance of my application overall.
I am running a small webservice application with Spring Boot, Maven, Spring Data, Hibernate which is usually working fine. It has just a single webservice request mapping. The application is a standalone JAR and started in a shell with "screen".
Nearly once per day it just quits itself. There is nobody that forces the app to stop or so. It just says "Killed" in the shell. There is no exception, error or any other message. I already tried to activate logging in application.properties with:
logging.level.org.springframework.web=ERROR
logging.level.org.hibernate=ERROR
but that does not help. Also added exception handling to the webservice mapping trying to catch any exception thrown. But also nothing.
Does anyone know why the app could have been stopped or how I can display the problem?
Thanks in advance!
I encountered with the same problem and I had resolved it by creating swap area. I think it is a typical memory problem. I recommend you to create swap area if you are deploying the application in a linux environment. Check memory usage before..
This happens often when system kills your Java process because it started to consume a lot of memory.
Try to inspect your deployment target limitations and adjust heap size below that limit.
Are there ways to update java class files in Tomcat without using Tomcat Manager and reloadable with saving uptime? Reloading application from Tomcat Manager takes about 15-30 seconds and it invoke locked up server. How to update a large application quickly on Tomcat?
Some possibilities
setup a cluster, have a loadbalancer for multiple servers in the background. For update you remove one server from the cluster, upgrade, add to the cluster again. Then continue until done with all servers
use a product like JRebel (development) or LiveRebel (production system). This enables you to hot-replace your code in the running instance for many usecases that required a plugin. This is a commercial option (well, running a cluster of multiple machines comes with some price as well)
Of course you can combine both options (and there are probably more that don't come to my mind right now)
It's all a question about your requirements of uptime, recovery times etc. While you're at it: Think of your database and other infrastructure as well as a possible cause for downtime.
We are deploying a java web application in Elastic Beanstalk. The Beanstalk by default provides Redhat instances with Tomcat7 & openjdk. But i want to customize the instances so that i can use Sunjdk.
Followed the steps as mentioned here, I launched a new instance from EC2 & installed sun jdk, tomcat7 and created a custom AMI of the instance and provided its ID in Custom AMI id field in edit Environment configuration in BeanStalk. But my Environment Health status is changing to RED, and the application is not deployed. Can you please suggest me a solution for this.
I have done exactly what you want to do, and am happy to report that it works. In my first attempts I encountered the same issue. Make sure that you:
Create your custom AMI from a running instance of Amazon's beanstalk AMI that you manually launch from EC2, NOT one that was launched from starting your application through beanstalk. This is very important, as Flashing a beanstalk launched EC2 instance won't work!!
install Sun's Java as per these instructions, to make sure all the environment and symbolic links are correct. https://forums.aws.amazon.com/message.jspa?messageID=251168#251168
be patient, as I found for some reason Tomcat takes longer to spin up than with the default OpenJDK config.
If you still can't get it, make sure you check the logs in /opt/tomcat7/logs/catalina.out - I hit a mysterious VerifyException that was resolved by uploading a new .jar to the environment, and I also ran out of permgen space almost immediately, which was resolved by upping the values in the 'Container' configuration area of the beanstalk 'edit configuration'.
So we have a busy legacy web service that needs to be replaced by a new one. The legacy web service was deployed using a WAR file on an apache tomcat server. That is it was copied over into the web apps folder under tomcat and all went well. I have been delegated with the task to replace it and would like to do it ensuring
I have a back up of the old service
the service gets replaced by another WAR file with no down time
Again I know I am being overly cautious however it is production level and I would like everything to go smooth. Step by step instructions would help.
Make a test server
Read tutorials and play around with the test server until it goes smoothly
Replicate what you did on the test server on the prod server.
If this really is a "busy prod server" with "no down time", then you will have some kind of test server that you can get the configuration right on.
... with no down time
If you literally mean zero downtime, then you will need to replicate your webserver and implement some kind of front-end that can transparently switch request streams to different servers. You will also need to deal with session migration.
If you mean with minimal downtime, then most web containers support hot redeployment of webapps. However, this typically entails an automatic shutdown and restart of the webapp, which may take seconds or minutes, depending on the webapp. Furthermore there is a risk of significant memory leakage; e.g. of permgen space.
The fallback is a complete shutdown / restart of the web container.
And it goes without saying that you need:
A test server that replicates your production environment.
A rigorous procedure for checking that deployments to your test environment result in a fully functioning system.
A preplanned, tested and hopefully bomb-proof procedure for rolling back your production system in the event of a failed deployment.
All of this (especially rollback) gets a lot more complicated when you system includes other stuff apart from the webapp; e.g. databases.