I was wondering if there is a 'smooth way' of redeploying a Java WAR to a production server (no cluster, no OSGi)?
All I can come up with is stop server, update file, restart server. And 10 minutes beforehand I need to display a maintenance warning on the site.
What's your approach?
First, hot-deploy doesn't always work. We spent so much time to make sure every new module is loaded and decided it's not worth the trouble. So what you are doing may sound bad but it's the most reliable way to deploy a new WAR.
Our current approach is to use a switch with load-balancer in front of all servers. We run at least 2 instances of the application servers. When we shutdown one server for maintenance, the traffic automatically goes to the other one.
Some of the switches are really inexpensive. If you don't have enough load to justify a new box and your 2 instances can run on the same box.
In some circumstances, the switches can actually save money. For example, we have a SSL page that used to use 6 boxes and now it runs fine on 2 boxes with SSL acceleration in the switch.
You might have a look at JRebel, though I wouldn't use it in production. In production we do basically the same, though our boss keeps on dreaming of hot redeploys. Unfortunately they are supported mostly on paper - in most complex applications something always goes wrong with hot redeploys. The same is even truer for incremental hot redeploys...
Some application servers do support redeployment without interruption of service. This is at least true for WebLogic, see Using Production Redeployment to Update Applications. Note that this is not hot deploy (and I would NEVER use hot deploy with production servers).
Without application server support, I'm afraid you won't be able to do real "smooth" redeployments. If you want to minimize downtime, one approach is to deploy the new application in parallel (on the same server or another one) and to change the routing rules when done. But clients will loose their session.
Usually it's possible to optimize start-up time. Our web application starts with Jetty in 5-7 seconds. Other Java web servers are worse, because they start very slow.
Also, as I'm aware (not I did it), the front-end web server (such as apache, we use lighttpd) could be configured to hold request some period of time (up to 30 seconds on ours) while the Jetty is not ready. So, we just easily restart the Jetty while deploying, and users just have several seconds delay in worst case, which usually just looks like an Internet connection glitch.
Usually, mv old.war new.war and let the AS take it from there, although with very busy 24/7 services, I imagine this wouldn't be an option.
As ZZ Coder has already mentioned load balancer is a good solution especially for big deployments. For my own project I use reverse http proxy functionality of nginx web server. It redirects all http packets from a specified web context (seen from the Internet) to a server inside my network. Configuration is really easy:
location /best-app-ever/ {
proxy_pass host-address:8080/some-app-1.1
root /home/www/some-app-1.1
}
Switching version should be smooth as well. Assuming that you have already deployed new version of application just change nginx configuration file and apply changes:
location /best-app-ever/ {
proxy_pass host-address:8080/some-app-1.2
root /home/www/some-app-1.2
}
sudo nginx -t
sudo service nginx restart
Be warned that in case your web application is stateful and/or contains some running or scheduled processes, the deployment and undeployment might not be as smooth.
Related
We have several Java standalone applications (in form of Jar files) running on multiple servers. These applications mainly read and stream data between systems. We are using Java 8 mainly in our development. I was put in charge recently. My main function is to manage and maintain these apps.
Currently, I check these apps manually by accessing these servers, check if the app is running, and sometimes run some database queries to see if the app started pulling data. My problem is that in many cases, some of these apps fail and shutdown due to data issue or edge cases without anyone noticing. We need some monitoring and application recovery in place.
We don't have docker infrastructure in place. We plan to implement docker in the future, but for now this is not an option.
After research, the following are options I thought of or solutions I tried:
Have the apps create a socket client which sends a heartbeat to a monitoring app (which needs to be developed). I am keeping this as my last option.
I tried to use Eclipse Vertx to wrap the apps into Verticles. Then create a web view that can show me status and other info. After several tries, the apps fail to parse the data correctly (might be due to my lack of understanding to Vertx library).
Have a third party solution that does this, but I have no idea what solutions are out there. I am open for suggestions.
My requirements are:
Proper monitoring of the apps running and their status.
In case of failure, the app should start again while notifying the admin/developer.
I am willing to develop a solution or implement a third party one. I need you guidance on this.
Thank you.
You could use spring-boot-actuator (see health). It comes with a built-in endpoint that has some health checks(depending on your spring-boot project), but you can create your own as well.
Then, doing a http request to http://{host}:{port}/{context}/actuator/health (replace with yours), you could see those health checks status and also use the response status code to monitor your application.
Have you heard of Java Service Wrappers? Not a full management functionality, however it would monitor for JVM crashes and out of memory conditions and restart your application for sure. Alerting should also be possible.
There is a small comparison table here: https://yajsw.sourceforge.io/#mozTocId284533
So some basic monitoring and management is included already. If you need more, I suggest using JMX (https://www.oracle.com/java/technologies/javase/javamanagement.html) or Prometheus (https://prometheus.io/ and https://github.com/prometheus/client_java)
i'm trying to make a microservice architecture using spring cloud, for that I use config server, eureka and etc, also I exploit the docker to deploy my services. I use several machines for that. For redundancy and load balancing i'm gonna deploy one copy of each services into each machine, but i face a problem: some of these services must be working in one copy at the same time (e.g. monitoring of something which is executed by cron expression) That is to say I don't want to have several monitorings components to be run at same time, instead they have to be set up on each machine by rotation. (e.g. as here http://www.quartz-scheduler.org/documentation/quartz-..)
How could i do that the best way? What should i use for that?
thanks
So, I have a web client and an EJB timer, deployed seperately.
The workflow is as follows:
1) User accesses client.
2) User requests an action to take place which is known to be long-running, so we write the request to run this process in a database table.
3) TimerOne is checking this table every few seconds to see if there are any waiting tasks, so it finds the user's request and runs the task.
My problem is that in some environments in which our application is run, we are taking advantage of server clustering. When we do this, both the client and the EJB timer are deployed to each server in the cluster.
It is okay for the client to be deployed to multiple servers, as it helps with workload; however, having the timer run on multiple servers is an issue. When the user requests for a long-running task to be run, both timers grab the task at the same time from the database and start running it. As the long-running jobs usually write to the database, this scenario leads to collisions, among other issues.
My goal is to be able to deploy my EJB timer to both servers, but for there to be some state maintained across the cluster which can be used by the timers to decide whether they should pick up the task or if one of the other instances has already picked it up.
I tried using the database for this and tried file storage, but these are either too slow, or I could not come up with a bullet-proof workflow for synchronization.
Does anyone know of a good way to handle this problem? Is it even possible?
The solution should be able to run on a clustered WebLogic domain, a non-clustered WebLogic domain, a clustered Glassfish domain, and a non-clustered Glassfish domain.
I am open to changing the way this is done, if there is another, more elegent solution.
Thanks for any ideas!
Yes this is possible with clustered timers or a Weblogic Singleton Service (and has been asked a number of times here already). See the following:
Clustered timers:
https://blogs.oracle.com/muraliveligeti/entry/ejb_timer_ejb
http://shaoxiongyang.blogspot.com/2010/10/how-to-use-ejb-3-timer-in-weblogic-10.html
http://java.sys-con.com/node/43944
Singleton Services:
https://blogs.oracle.com/jamesbayer/entry/a_simple_job_scheduler_example
http://developsimpler.blogspot.com/2012/03/weblogic-clusters-and-singleton-service.html
I am open to changing the way this is done, if there is another, more elegent solution.
I know that your question is about a EJB Timer, but take in mind the following:
In my opinion, you have a requirement that need the advantage of asynchronous processing.
In earlier Java EE versions, one of the alternatives to achieve this kind of requirement was to use JMS which allows you to send a message that is processed later for a business layer component. Other possibility was the one that you have described, that required the use of EJB Timer. I think both cases were a workaround that filled a gap in the EE specification.
Since Java EE 6, you can define asynchronous services which allows you make asynchronous calls, avoiding to use features were thought for other purposes.
I am creating a Java service which will run within a web servlet container (probably Tomcat). One portion of the server will run on its own and will not be initiated by HTTP. I know that when an HTTP call causes an exception, the web container can call it again.
I want to be sure that the part of the server which runs continuously will continue to run, even if it fails. I will handle whichever failures I can manually, but if it all fails I want something to restart it all. Are there any tools that can accomplish this easily? I am already using Spring and Tomcat, so if those can provide it, that is ideal. If not, then how about a good design pattern?
Edit: To clarify, I have a web service which will run in Tomcat. I want to run a separate thread within that service and set it up such that when the thread ends or an un-handled exception occurs, Tomcat (or something else) detects the failure and restarts the web service. I know that typically web containers have threads start from some external call and thus handle failures from those threads. What I want is something which handles a background worker thread.
Not quite clear on the design you have in mind, but it seems to me you need some sort of health check.
You can implement such a mechanism in many ways e.g. open a socket from this process that runs all time and periodically send a message.
If there is no reply then the process failed.
You could restart tomcat or implement a mechanism to restart that process.
Can not tell you more details since you do not specify much on what you are trying to do.
UPDATE:
I think that you should use JMX. It is offered by Spring and Tomcat that you already use.
Just make the process you want to monitor a managed resource and another module can check if it is alive.
If you are running inside a Servlet then as per J2EE spec, you cannot restart the container but, you can use ScheduledExecutorService to continuously monitor that your service is running and if not, then re-start it.
EDIT. More details below
You can call isTerminated() to check if the service still running and add more tasks to it, if the queue is empty.
I may be misunderstanding your problem here, but you might be over-thinking it.
There's nothing stopping you from running multiple Tomcat instances on a single machine. You could then have Server A connect to Server B to pull down information (via a web service of your choosing). This would alleviate the need for an outage on server A to cause an outage on server B (which is what I'm assuming you're trying to avoid).
This is a common way to isolate production environments simply by binding to a separate port. If Tomcat doesn't fit the bill for the service you can always run the application as a service on [insert operating system of choice] and connect to it via a proprietary protocol. Your operating system can handle restarts in that case. Typically I think the multiple Tomcat containers is the easiest approach as it is simple to install and relatively easy to set up.
Good luck, it seems like a fun system administration problem. You also might be interested in checking out Quartz job scheduling as that might fit the bill for an intermittent service.
edit: a little more detail might provide some more detailed answers.
See this post. It's a simple tomcat-watchdog shell script.
I have heard that this is what JavaRebel does but is there any other good way to deploy a new version of an EAR while allowing users to remain active on the previous version? We use JBoss for the application server...
It's not what JavaRebel does. JavaRebel (according to description) hot-replaces the classes in memory. It's not acceptable in the case of existing connections to the system, since the updated classes may break the client's logic.
Once a company I was working for had a similar problem, and it was solved this way:
a smart router was used as a load-balancer
the new version was deployed to 50% of the nodes of the (new) cluster
new connections were delivered strictly to these updated nodes, old ones were balanced between old nodes
old nodes were took off-line (one-by-one, to keep number of clients per node within limits)
at the same time, new version was deployed to off-line "old" nodes and they were brought up as new nodes
due to EJB clustering, the sessions and beans were picked up by other old nodes
eventually (in a few hours), only one old node left, having a single instance of old version and all clients using old version were connected to it
when the last old client got disconnected, that node was too brought down
Now, I'm not a networking guy, and cannot give you many details (like what was the router hardware and such). My understanding this can be set up pretty easy, except, if I remember right, we had to setup an additional Weblogic domain to deploy new versions of the application (otherwise it would be conflicting with the old one on JNDI names).
Hope that helps.
P.S. Ichorus provided a comment saying that the app is deployed on clients' servers. So the router trick may be not feasible. Now, I see only one viable solution right now ( it's 21:52 now, I may overlook things :) ) --
Develop new version with "versioned" JNDI names; e.g. if Customer bean was under ejb/Customer in version 1, in version 2 it would be under ejb/Customer2
Have a business facade in the application with a stable basic interface (Factory-style) which, when asked for Customer bean, tries to find the highest-versioned JNDI name (not on every call, of course, can cache for a hour or so). That facade could (and should) be deployed as a separate application -- and never or very rarely updated
Now every new client would get access to the latest application deployed, and the applications won't conflict.
This approach takes a careful planning and testing, but should work IMHO.
I recently modified a few applications in a similar way to let them coexist in the same domain (before they used the same JNDI name for different data sources).
As I understand WebLogic has a feature called parallel deployment to eliminate downtime during EAR version upgrade. You can deploy the new version without stopping the existing application and once the new version deployed successfully you can switch over transparently from the old one to new one.
I am not sure if other application server supports this.
Ref: http://edocs.bea.com/wls/docs100/deployment/redeploy.html#wp1022490
Vladimir's suggestion around using a load balancer is a pretty sure way of achieving what you want. Keep in mind, it need not necessarily be a high-end hardware load balancer. Rather, if you front your JBoss server with a native web server (Apache or IIS) and mod_jk or mod_proxy, you can maintain one common web facade and implement the applicable loading and routing routines at EAR upgrade time.
//Nicholas
I think you might want to look into Spring using OSGI framework.
http://www.springframework.org/osgi