I am trying to build the very lightweight solution for zero downtime deployment for Java apps. For the sake of simplicity lets think that we have two servers. My solution is to use:
On the "front" -- some load balancer (software) - I am thinking about HAProxy here.
On the "back" - two servers, both running Tomcat with deployed application.
When we are about to deploy new release
We disable one of the servers with HAProxy, so only one server (let's call it server A, which is running old release) will be available.
Deploy new release on other server (let's call it server B), run production unit tests (in case we have them :-) and enable server B with HAProxy, disabling server A at the same time.
Now we have again only one server active (server B, with the new release). Deploy new release on server B, and re-enable it.
Any advises how to improve? How automate?
Any ready made solutions or do I have to end up with my own custom scripts?
Thanks!
I have found some interesting solutions from this article regarding Zero downtime. I would like to highlight only few solutions in that article.
1. A/B switch: ( Rolling upgrade + Fallback mechanism )
We should have a set of nodes in standing by mode. We will deploy the new version to those nodes and switch the traffic to them instantly. If we keep the old nodes in their original state, we could do instant rollback as well. A load balancer fronts the application and is responsible for this switch upon request.
cons: If you need X servers to run your application, yon need 2X servers with this approach.
2. Zero downtime
With this approach, we don’t keep a set of machines; rather, we delay the port binding. Shared resource acquisition is delayed until the application starts up. The ports are switched after the application starts, and the old version is also kept running (without an access point) to roll back instantly if needed.
3. Parallel deployment – Apache Tomcat: ( For web applications only)
Apache Tomcat has added the parallel deployment feature to their version 7 release. They let two versions of the application run at the same time and take the latest version as default.
4. Delayed port binding:
we propose here is the ability to start the server without binding the port and essentially without starting the connector. Later, a separate command will start and bind the connector. Version 2 of the software can be deployed while version 1 is running and already bound. When version 2 is started later, we can unbind version 1 and bind version 2. With this approach, the node is effectively offline only for a few seconds.
5. Advanced port binding:
By breaking the myth: ‘Address already in use’, *both old process & new process will bind to same port. SO_REUSEPORT option in ON mode lets two (or more) processes bind to the same port. Once the new process binds to the port, kill the old process.
The SO_REUSEPORT option address two issues:
The small glitch between the application version switching: The node can serve traffic all the time, effectively giving us zero downtime.
Improved scheduling:
In Summary:
By combining both late binding and port reuse, we can effectively achieve zero downtime. And if we keep the standby process around, we will be able to do an instant rollback as well.
Rolling upgrade is indeed a good solution, provided your load-balancer supports this option (server starvation).
Another solution is to use OSGi-enabled application servers, to hot-replace parts or whole of your application.
I would recommend the first one. SpringSource's AMS supervision console can take down a cluster of tcServer (a custom tomcat on steroids), and IIRC do the rolling upgrade automatically (but check the docs).
Have a look at the OSGi technology if you can accommodate an OSGi container since it provides good isolation and hot deployment for OSGi bundles. If you are using the Spring framework you could use Spring OSGi
LiveRebel provides the functionality for rolling restarts, provides CLI API and Hudson/Jenkins plugin for automation.
There is easy-deploy that does exactly that with Docker containers.
Deploy version 1
easy-deploy -p 80:80 -v some/path:other/path my-image:1
To deploy a new version just run the command with the updated tag name
easy-deploy -p 80:80 -v some/path:other/path my-image:2
Disclosure: I built this tool. I built it exactly because I couldn't find a simple solution for this problem.
Related
Are there ways to update java class files in Tomcat without using Tomcat Manager and reloadable with saving uptime? Reloading application from Tomcat Manager takes about 15-30 seconds and it invoke locked up server. How to update a large application quickly on Tomcat?
Some possibilities
setup a cluster, have a loadbalancer for multiple servers in the background. For update you remove one server from the cluster, upgrade, add to the cluster again. Then continue until done with all servers
use a product like JRebel (development) or LiveRebel (production system). This enables you to hot-replace your code in the running instance for many usecases that required a plugin. This is a commercial option (well, running a cluster of multiple machines comes with some price as well)
Of course you can combine both options (and there are probably more that don't come to my mind right now)
It's all a question about your requirements of uptime, recovery times etc. While you're at it: Think of your database and other infrastructure as well as a possible cause for downtime.
I am working at a startup, we are just about to roll out our first beta. Knowing that we will be having a good number of users, we want to have seamlessly deployment when we are adding new features.
I have worked with windows azure before, and I know they support seamless deployment, so I did some googling and cloudbees was the first result.
So the question is, with what we have now (geronimo server, rackspace hosting), is it possible to seamlessly redeploy a java web application? If so, how?
Are there other alternative solution, such as using another hosting provider or use a different web server? (Because it is a startup, it would be beneficial if the answer keeps scalability in mind)
If with a seamless redeploy, you mean an upgrade of your application without any downtime or restarting of your server, LiveRebel might be something to look at.
See http://zeroturnaround.com/liverebel
There are a lot of methods for doing this in the java world. If you don't use sessions (or use shared sessions between app servers) you can do a rolling stop/deploy/start of your appservers, taking 1 offline at a time and using a load balancer to ensure that traffic goes to the other servers.
I have heard Glassfish has such feature, the reference probably ment this (Glassfish 3.x redeploy command) : http://docs.oracle.com/cd/E19798-01/821-1758/6nmnj7q1h/index.html
What is the best way to manage deployments of a single web-app to multiple non-clustered tomcat instances.
My ideal solution will support:
A simple API - invoked with
groovy/ant/Rest or similar
Success/failure
notification for all nodes
Atomic
deployments - if the deployments
fails on any node it is rolled back.
We had over 100 clients each running on a dedicated tomcat instance across 5 servers where most would be updated to the latest release at the same time. In our case we used mapped network drives and some tricks with the CATALINA_BASE, but personally I think it may be easier to use WAR deployment via an ANT script to the Tomcat manager if you can get away with it.
For yours you could (at minimum) have a tomcat directory for each instance and each can use the same webapps dir via a network share. Upgrading would still require stopping each instance, updating the single shared dir then starting all instances.
You could also use the tomcat management console (via ANT or other automated process) to manage a scripted local installation or start/stop but this would be not be atomic.
You might want to look at using Tomcat manager from Ant. It might not have all you are asking for, but I guess you could script what you need.
So we have a busy legacy web service that needs to be replaced by a new one. The legacy web service was deployed using a WAR file on an apache tomcat server. That is it was copied over into the web apps folder under tomcat and all went well. I have been delegated with the task to replace it and would like to do it ensuring
I have a back up of the old service
the service gets replaced by another WAR file with no down time
Again I know I am being overly cautious however it is production level and I would like everything to go smooth. Step by step instructions would help.
Make a test server
Read tutorials and play around with the test server until it goes smoothly
Replicate what you did on the test server on the prod server.
If this really is a "busy prod server" with "no down time", then you will have some kind of test server that you can get the configuration right on.
... with no down time
If you literally mean zero downtime, then you will need to replicate your webserver and implement some kind of front-end that can transparently switch request streams to different servers. You will also need to deal with session migration.
If you mean with minimal downtime, then most web containers support hot redeployment of webapps. However, this typically entails an automatic shutdown and restart of the webapp, which may take seconds or minutes, depending on the webapp. Furthermore there is a risk of significant memory leakage; e.g. of permgen space.
The fallback is a complete shutdown / restart of the web container.
And it goes without saying that you need:
A test server that replicates your production environment.
A rigorous procedure for checking that deployments to your test environment result in a fully functioning system.
A preplanned, tested and hopefully bomb-proof procedure for rolling back your production system in the event of a failed deployment.
All of this (especially rollback) gets a lot more complicated when you system includes other stuff apart from the webapp; e.g. databases.
web application update best practice..
normally how you folk do web application update from large critical system? do u forward user to page stating upgrading in progress on another web server; quickly deploy new war and point back ?
can provide some guides..
If possible, run two tomcat servers, with an Apache web server in front, load balancing the requests between the two.
When it comes time to upgrade, tell Apache to use only Tomcat A, then upgrade Tomcat B, tell Apache to use Tomcat B, upgrade Tomcat A, then back to using both.
No major interruption of service, if you do it right, although without session replication you're going to get some intra-session disruption. It's not as easy as it sounds, though.
As always, the lengths to which you go to minimise disruption depends on how important that disruption is. You can always make it smoother, at more expense and effort.