Continous Deployment for Java/JVM Web Application - java

My team and I would like to implement "Continuous Deployment" for our website. Continuous Deployment basically means deploying to production very frequently (multiple times a day). Supposedly Etsy does this all the time.
Our environment is Tomcat + Nginx. We already do Continuous deployment of any code change to our snapshot server (ie traditional continuous integration) using Hudson and a Hudson+Cargo plugin that hot deploys.
Surprisingly that works well (albeit over time we have to restart tomcat sometimes).
For production this is not going to work because we can't have the website down.
I have some ideas like having two web apps and redirecting while one is down.
Anybody have any ideas or has done this before in a real production environment?

From http://radar.oreilly.com/2009/03/continuous-deployment-5-eas.html:
Real-time alerting. No matter how good your deployment process, bugs can still get through. The most annoying variety are bugs that don’t manifest until hours or days after the code that caused them is deployed. To catch those nasty bugs, you need a monitoring platform that can let you know when things have gone awry, and get a human being involved in debugging them.
To effectively implement continuous deployment to production, you need good monitoring, otherwise you will not understand what is happening with your application.

I don't know WHY you think this is a good idea, but thats up to you.
I would use a balancer application with two hot systems, which can be found in tomcat itself, and then just stop the server before deployment, deploy, and restart the server. Leave a two minute window for each hot server, and you should be good.
EDIT: I wouldn't deploy EVERYTIME. We also are a small company with noch much QA (tm), but it is still one click in the build system to go live.

We use apache httpd 2.2 and mod_proxy for this
We then have 2 tomcats running, one on port 8080, and one on port 88.
The firewall prevents outside access to those ports, so only port 80 is open
Apache HTTPd is configured to listen on port 80
It is very simple to configure too.
This is a basic configuration (httpd.conf) that will work out of the box:
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
LoadModule proxy_http_module modules/mod_proxy_http.so
<Proxy balancer://mycluster>
BalancerMember http://localhost:8080
BalancerMember http://localhost:88 status=+H
</Proxy>
ProxyPass / balancer://mycluster/
ProxyPassReverse / balancer://mycluster/
The "+H" means that it is only used as a backup server, so when 8080 is unreachable, it will run on 88 until 8080 comes back online

Related

How to prevent HTTP 404 during deployment of ear in Wildfly/Jboss

We're using WildFly 10 as our application server and deploy via Docker (deployment in WF is ordinary hotdeployment). We're not using WildFly's clustering mechanisms but simply have load-balancers (HAProxys) in front.
The problem is that WF opens its HTTP port while the EAR deployment is still in progress. This (of course?) leads to HTTP 404 errors which we don't want to handle specifically in the LBs. This could lead to false negatives...
Is there a way to allow HTTP connections only after the EAR has started successfully?
Alternatively is it possible to replace the "404 because nothing is deployed here"-error with a "503 service unavailable"? This would much better express the problem and would be easy to handle externally...
You can set default-response-code for host you are running this on.
something along the lines:
<host name="default-host" alias="localhost" default-response-code="503">
or in cli:
/subsystem=undertow/server=default-server/host=default-host:write-attribute(name=default-response-code, value=503)
and similarly for any other host you might have.

How to configure Wildfly 9 to failover HTTP sessions

I'm really struggling to configure Wildfly 9 to cluster/failover its sessions...
I keep reading that wildfly in standalone-ha mode will automatically discover peers and automatically share sessions, but it's clearly not working for me.
I have setup 3 AWS EC2 servers which all have the same configuration. They all run the same versions of everything and have the same webapp .war file deployed to each of them. This webapp works fine, I can log in to the app which maintains a simple session variable to verify that I am logged in. I've launched each server with standalone-ha.xml configuration files but logging into one doesn't allow be to access the session in any of the others.
I've tried all the things I can think of, but don't know how to diagnose the issue as I don't know how the servers identify each other.
I've manually deployed the war file on each server but placing the file into .../standalone/deployments/
Each has a fully open firewall...
Oh - I set the muticast address on the command line to 230.0.0.4 (That number came from a guide, and I have literally no understanding of it) and each is bound (-b) to the internal IP of the server...
Any help appreciated...
First you must consider that in AWS EC2 multicast traffic is not allowed and thus MPING will not work.
See http://developer.jboss.org/wiki/JGroupsS3PING
An example how to implement S3Ping http://aws.typepad.com/awsaktuell/2013/10/elastic-jboss-as-7-clustering-in-aws-using-ec2-s3-elb-and-chef.html

WebSockets not working on OpenShift Tomcat 7.0 and WildFly Application Server 8.1.0.CR1

I 've spent the whole night trying to run a simple echo test on Tomcat 7.
The test I try to run runs locally as expected. But when deployed to the openshift the connection opens and then immediately closed. I've been looking for the right setup on the internet and found questions and answers for them for exactly the same problem but the solutions don't work for me.
The example of what I have as a test can be checked at http://pureikka-belgampaul.rhcloud.com/pi-8/index.html
At the bottom of the viewarea there's a debug console reporting the statsus of the websocket.
So I feel like I have not made a simple mistake in the setup and it runs locally with only the change of the host name and the port number for the websocket.
Could it be a routing trouble after the upgrade of the connection?
I deploy my war file directly to the webapps directory.
I also think I got websockets working at one moment but i'm not so sure anymore. I could be looking at the localhost after all.
Eventually I decided to try wildfly8 gear because it ships with support for websockets. I checked out the https://github.com/javaee-samples/javaee7-samples repo and deployed the chat demo to the local wildfly8 and the example worked from the first run as it is supposed. Great. But the same war file fails to work correctly on the openshift (http://wildfly8-belgampaul.rhcloud.com/chat3/)
I tried to use ports 80 or 8000 in both setups.
Need someone's help? advice or question to send me back to the right direction
Maybe someone has solved a similar issue?
Web Sockets on OpenShift work over ports 8000 for ws and 8443 for wss, it is not a "work around" it's the way websockets work on this system.
Here are the most useful links providing just enough light about the problem
https://www.openshift.com/blogs/paas-websockets (general info about websockets on openshift)
https://www.openshift.com/blogs/deploy-websocket-web-applications-with-jboss-wildfly (diy example)
to summarize:
websocket support out-of-the-box is the most reliable with node.js.

Amazon ec2 with windows 2008 server with TOMCAT

I just created a free ec2 account. I want to test few things but i want to make sure that i am using ONLY the free version.
I want to run a java application in windows 2008 R2 server with TOMCAT as web server. Ec2 by default doesn't have java SDK and TOMCAT installed.
SO here are the few queries
Will i be charged for installing TOMCAT and JAVA also for deploying my application?
How do i install TOMCAT and JAVA ? can someone provide a link?
I tried doing a remote desktop ut could not connect to the server. Looks like a firewall issue. But then how do i disable ?
Can't answer all your questions, but I'll take a shot.
If you want to watch your ec2 cost, setup a billing alert in AWS that will notify you when your account exceeds a certain threshold, and set your threshold very low if you are very concerend. You can always monitor your outstanding charges on your billing page on a daily basis if you are really concerned.
Pick a small or micro instance if you want to keep your costs low; micro instances I have found are mostly useless for any serious work, but they are very, very cheap and maybe ok for trying things out.
To install any software, you are going to need to get remote desktop working. Once the server is spun up, remote in and you can install anything you want; there is no extra cost to install software (except buying licenses if you need them, but that is not aws's concern, they won't be charging you for installing tomcat.
You need to open up the windows firewall on your 2008 server AND you need to make sure the AWS security group also allows incoming connections on port 3389 - its essentially there firewall that runs in addition to the EC2 instances firewall - BOTH need to be open. You can either modify the security group that was assigned to your instance, or create a new one and associated it with your instance.

Improving JBoss 5 slow startup

We upgraded from JBoss 4 (and JDK 5) to JBoss 5 (and JDK 6). The problem is that the start time has gone from 1.5 minutes (on JBoss 4) to more than 4 minutes.
18:53:35,444 INFO [ServerImpl] JBoss (Microcontainer) [5.1.0.GA (build: SVNTag=JBoss_5_1_0_GA date=200905221053)] Started in 3m:9s:262ms
It seems like the component that is taking JBoss the longest time to initialize is the JMX
18:50:41,926 INFO [LogNotificationListener] Adding notification listener for logging mbean "jboss.system:service=Logging,type=Log4jService" to server org.jboss.mx.server.MBeanServerImpl#1adc122[ defaultDomain='jboss' ]
18:52:38,797 INFO [JMXConnectorServerService] JMX Connector server: service:jmx:rmi://lharel2/jndi/rmi://lharel2:1090/jmxconnector
From the DEBUG server log, I get these lines at the problematic time:
2009-12-18 18:51:00,886 DEBUG [org.jboss.deployment.MappedReferenceMetaDataResolverDeployer] (main) vfsfile:/C:/QC/Views/QCDev/jboss-5.1.0.GA/server/default/deploy/jmx-console.war/ endpoint mappings:
2009-12-18 18:51:00,886 DEBUG [org.jboss.deployment.MappedReferenceMetaDataResolverDeployer] (main) Processing unit=jmx-console.war, structure: jmx-console.war
2009-12-18 18:52:35,209 DEBUG [org.jboss.deployment.OptAnnotationMetaDataDeployer] (main) Deployment is metadata-complete, skipping annotation processing, ejbJarMetaData=null, jbossWebMetaData=org.jboss.metadata.web.spec.Web23MetaData#1f, jbossClientMetaData=null, metaDataCompleteIsDefault=false
There is no EJB in the project.
The memory settings are:
-Xms128m -Xmx512m -XX:MaxPermSize=256m
Do you have any idea how JBoss start time can be improved?
Update: so far no luck, I tried shreeni's suggestion (changed the scanning xmls). The server is not running in debug mode so MicSim's suggestion is not relevant
A shot into the blue sky without more information
Network timeouts: 1,5 minutes of delay when deploying jmx-console.war may indicate a network timeout (e.g. 3 x 30 seconds). Try to start JBoss and bind it to a specific IP address using the -b command line argument or the jboss.bind.address system property. Also, try to make sure your host and DNS resolution settings on your system are correct.
JMX is also using RMI and you may want to set the RMI server host name as system property. On some Linux distributions, RMI has problems with looking up the correct hostname and jmx-console.war may try to connect to the 'wrong localhost'. The system property is java.rmi.server.hostname
System tracing: If that does not help, you may want to use strace to start the java process, so you can see the point where the system hangs (if it really does hang due to a network timeout or similar).
That is an awfully big gap in the logs. I suggest changing the log configuration to log everything at DEBUG level, rather than INFO. This will generate an awful lot more log entries, but hopefully will help you narrow it down.
The easiest way to do this is to set the -Djboss.server.log.threshold=DEBUG system property when you start JBoss
I had an issue like this but I found a good improvement by setting the initial and max heap size to same values, I mean:
-Xms512m -Xmx512m
With this, I improved from 4 to 2 minutes the starting time.
Your suspicion about the jmx-console can be misleading. There may be other components doing work in the background unrelated to the jmx-console. In my experience, we had an issue where a small war file appeared to take 3 minutes to load! It was innocent. The culprit was partly due to an EAR file with many wars and jars.
While I'm no expert, I would suggest the following:
Try turning up the logging to TRACE. By doing this, I witnessed one of the deployers (EJBDeployer, I think) unnecessarily scanning WARs in one of our EARs. I then manually disabled the scanning of those WAR files.
Run wireshark during the startup. I discovered some war files were hanging while waiting for a response from an external DTD request. Those websites were either now non-existent or would not properly serve the DTD files to java-based programs. I could speed it up by either having the programmers use local DTD files or mirroring those DTDs locally and having /etc/host loopback locally.
You could refer to this link to avoid unnecessary annotation scanning which could speedup your server start
See http://community.jboss.org/wiki/jboss5xtuningslimming especially the "Tuning" part.
I'm using JBoss 5.1.0 with a Macbook pro (2.26ghz 4gb) without applications it start in 54s
15:00:26,449 INFO [ServerImpl] JBoss (Microcontainer) [5.1.0.GA (build: SVNTag=JBoss_5_1_0_GA date=200905221634)] Started in 54s:720ms
I made a new configuration based on the "default". The JMS dataosurce points to a Postgres database instead of the "Hypersonic Database" (in memory database)
I suppose you are starting in debug mode. This mode can be up to 3 times slower than normal mode.
But there might be also a problem when switching from JDK5 to JDK6. I found this solution here on the net:
I've solved that. It's a debugging issue. I've changed my debug settings from:
wrapper.java.additional.26=-Xdebug
wrapper.java.additional.27=-Xnoagent
wrapper.java.additional.28=-Djava.compiler=NONE
wrapper.java.additional.27=-Xrunjdwp:transport=dt_socket,address=7199,server=y,suspend=n
to:
wrapper.java.additional.26=-Xdebug
wrapper.java.additional.27=-Xrunjdwp:transport=dt_socket,address=7199,server=y,suspend=n
and JBoss becomes fast again.
Hope this helps.
Do you need the JMX console application? Pragmatic thing would be to un-deploy it from the server, you could still use the jconsole or jvisualvm for basically the same thing.
Turn off annotation scanning and other features you don't need https://community.jboss.org/wiki/jboss5xtuningslimming
When you start/stop JBoss from eclipse, it does not clean up the tmp & work folders correctly. Setup an External Tool configuration and run a batch file to delete everything in tmp & work folders before each startup.
I was able to speedup running the "default" profile from 15/20 minutes to 5 minutes.

Categories