We are developing a spring based web application that needs to be deployed in various of environments:
1) developers local machines in order to develop new features \ fix bugs etc.
2) QA Env#1 - for the QA team to test our releases
3) QA Env#2 - for the QA team to test our releases
4) QA Env#3 - for the QA team to test our releases
5) Production
Now, each deployment for one of these environments requires to fill a properties file, this file contains many parameters :
1) tomcat ips
2) mysql ip
3) mongo ip
4) load balancer ip
5) ehcache multi cast ports and addresses.
6) many many more
Our question is:
Where this properties file should be defined? in our code base? (git, in a folder for each environment?) outside the our webapp project (this way, every environment should be updated with the correct properties file once, and then the deployment is straight forward with no configurations at all)?
Bear in mind we are working on a build\deploy machine that will automatically deploy our project to all of these envs, so this is something to keep in mind when deciding the correct way to handle this.
In general, I would not keep properties files in a repository, even though we are talking about a private one. Some reasons that come to my mind are:
Versioning is not needed. I personally see no point in keeping the version history of deployment or QA property files.
Customer privacy. You might not want to share deployment configuration with all people having access to the repository (developers, QA team, ...).
Preventing errors. Errors happens. A developer committing by mistake some changes to the deployment or QA property files might cause some serious troubles to your build/deploy chain.
What you can do is add to your repository some templates for the property files that contain a default configuration for your application (for instance the local developer configuration).
You can then have different property files for different environments that you distribute only within the involved teams (QA configuration to QA team, deployment configuration only for the deployed system).
Your build\deploy machine will most likely have access to all different property files and can retrieve the correct one depending on the target environment. With such configuration you have no benefit from keeping such files in the repository, it is enough to store them in a location available to people entitled to change them and to the build\deploy machine.
Java webapps provide a convenient way to make them run: It’s suficcient to drop the jar file into tomcat‘s webapps folder or upload it using the tomcat manager. If the jar file is named foo123.jar, the web application is soon accessible under http://<host>:8080/foo123/. However, in the majority of cases, there is a problem with the configuration: It’s a good practice to store data in a database, but where can I store the database connection parameters? Usually you have to adapt some server.xml or web.xml or other configuration file to put it there, but this hinders making use of the automatic deployment for such an application.
A “simple to use” web application should request its required configuration on the first run, like a “setup” screen, and then keep it in some place where it survives a restart of the servlet container. Of course, for database connection parameters, storing them in the database is not an option.
Following the specs, the servlet container has to provide a directory that a web application has write access to. It can be determined using:
File tempDir =
(File) session.getServletContext().getAttribute("javax.servlet.context.tempdir");
The content of this directory is bound to the ‘servlet context lifecycle’, if I got it right this means it is empty after a server restart. If that is true, it cannot be used for my purpose.
Does anybody know some kind of best practice for that? I don’t want to reinvent the wheel.
In lack of a better solution, I would implement it this way: As I said, if you make use of the easy deployment means described above, the context path is derived from the jar file name. So I could imagine to make use of this for the database connection as well. In simple terms: If the web application foo123 finds a MySQL connection on localhost:3306 (the MySQL default port) and can connect to it with username foo123 and password foo123 and has permissions to access a schema called foo123 it always uses that on restart.
What do you think of that?
You could just use a context.xml file. That will let you store the config files on a server-by-server basis and that means that you'll never have to put that information in the code itself.
This example seems to sum it up rather nicely.
We build 3-tier enterprise solutions that typically consists of several webapp and ejbjar modules that all talk to a db and have several external integration points.
Each module typically needs its own configurations that can change over the solution's life time.
Deploying it becomes a nightmare because now we have 18 property files that must be remembered to copied over and configured also setting up data-sources, queues, memory requirements etc.
I'm hopeful but not optimistic that there can be a better way.
Some options we've considered/used, each with it's pros and cons:
Use multiple maven projects and continuous integration (eg. hudson or jenkins) to build a configuration jar that includes all the property files for each environment (dev, qa, prod) and then bundle everything up as an EAR. But then things can't be easily changed in production when needed.
Put most of the settings in the DB and have a simple screen to modify it. Internally we can have a generic configuration service EJB that can read and modify the values. Each module can have a custom extended version that have specific getters and setter.
Version control all the property files then check it out on production and check it into a production branch after making changes.
With all of these you still need to configure data-sources and queues etc. in a container specific way :(
Сonsider binding a custom configuration object to JNDI. Then lookup this object in your apps to configure them. Benefits - you can use custom configuration object instead of rather generic Map or Properties.
Another way is to use JMX to configure applications you need. Benefits - you can bind objects you have to configure directly to MBean Server and then use such a well-known tools as jconsole or visualvm to configure components of your application.
Both ways support dynamic reconfiguration of your applications at runtime. I would prefer using JMX.
I've gone through several cycles of finding ways to do this. I still don't have a definite answer.
The last cycle ended up with a process based on properties files. The idea was that each server instance was configured with a single properties file that configured everything. That file was read by the startup scripts, to set memory parameters, by the app server, and by the application itself.
The key thing, though, was that this file was not managed directly. Rather, it was a product of the build process. We had a range of files for different purposes, kept in version control, and a build step which merged the appropriate ones. This lets you factor out commonalities that are shared along various axes.
For example, we had development, continuous integration, QA, UAT, staging, and production environments, each with its own database. Servers in different environments needed different database settings, but each server in a given environment used the same settings. So, there was something like a development-db.properties, qa-db.properties, and so on. In each environment, we had several kinds of servers - web servers, content management servers, batch process servers, etc. Each had JVM settings, for heap size and so on, that were different to other kinds of servers, but consistent between servers across environments. So, we had something like web-jvm.properties, cms-jvm.properties, batch-jvm.properties, and so on. We also had a way to have overrides for specific systems - production-cms-jvm.properties sort of thing. We also had a common.properties that set common properties, and sensible defaults which could be overridden where needed.
Our build process was actually a bit more complicated than just picking the right options from each set; we had a master file for each server in each environment which specified which other files to include. We allowed files to specify other files to include, so we could build a graph of imports to maximise reuse.
It ended up being quite complicated. Too complicated, i think. But it did work, and it did make it very, very easy to make changes affecting many servers in a controlled way. We even merged a set of input files from development, and another from operations, which contained sensitive information. It was a very flexible approach.
I know this has already been answered and my answer is not necessarily generic, but here's my take on things:
Note, here I'm only considering system/resource properties, not application settings. In my view, application settings (such as a payment threshold or other settings should be stored in a database, so that the system can be reconfigured without having to restart a service or cause downtime by re-deploying or re-reading a properties file).
For settings that impact on how different parts of a system connect with each other (such as web service endpoints, etc), I would make use of the JNDI tree.
Database connectivity and JMS connectivity would then be set-up using the Websphere console and can be managed by the Websphere administrators. These can also be created as JACL scripts which can be put into version control if necessary.
In addition to the JNDI resources, for additional properties, such as usernames for web service calls to a backend, etc, I would use Websphere "Name Space Bindings". These bindings can be edited using the Websphere console and accessed via JNDI using the "cell/persistent/mypassword" name.
So I could create the "mypassword" binding (a string), and the management for it falls to the Websphere admin (away from developer eyes or other people who should not have access to production systems), while the same EAR file can be used on dev, test, preproduction and production (which is preferable to have different EAR files for different systems, as the likelihood of other differences creeping in is reduced).
The Java code would then use a simple JNDI lookup (and possibly cache the value in memory).
Advantages over properties files:
Not having a "vulnerable" file that would need to be secured because system properties contain passwords.
Not having to add Java security policies to allow access to that file location
Advantages over database properties:
Not tied to having one database tied to an application server.
Hope that helps
Use multiple maven projects and continuous integration (eg. hudson or
jenkins) to build a configuration jar that includes all the property
files for each environment (dev, qa, prod) and then bundle everything
up as an EAR. But then things can't be easily changed in production
when needed.
I think the config should be in the database of the application instance. Your local machine config may be diffrent to dev and to QA, PROD , DR etc.
What you need is a way of getting the config out the database in a simple way.
I create a separate project with a provided dependency of Apache commons-configuration
It has many ways of storing data, but I like databases and the configurations lives in the database environment.
import javax.sql.DataSource;
import org.apache.commons.configuration.DatabaseConfiguration;
public class MYConfig extends DatabaseConfiguration {
public MYConfig(DataSource datasource) {
super(datasource, "TABLE_CONFIG", "PROP_KEY", "PROP_VALUE");
}
}
Put most of the settings in the DB and have a simple screen to modify
it. Internally we can have a generic configuration service EJB that
can read and modify the values. Each module can have a custom extended
version that have specific getters and setter.
Commons configurations as a simple API, you may then write the GUI as you wish.
You can do the interface in anyway you wish. Or as a quick win have no interface.
Version control all the property files then check it out on production
and check it into a production branch after making changes.
Version control is great. Add another DatabaseConfiguration using composition. The class you extends is the active config and the composed one being the audit. There is another constructor can can have a version. Just overload the right methods to get the desired effect.
import javax.sql.DataSource;
import org.apache.commons.configuration.DatabaseConfiguration;
public class MYConfig extends DatabaseConfiguration {
final DatabaseConfiguration audit;
public MYConfig(DataSource datasource) {
super(datasource, "TABLE_CONFIG", "PROP_KEY", "PROP_VALUE");
audit = new DatabaseConfiguration("TABLE_CONFIG_AUDIT", "PROP_KEY", "PROP_VALUE");
}
#Override
public void addProperty(String key, Object value) {
Object wasValue = super.getProperty(key);
super.addProperty(key, value);
audit.put(key,wasValue);//add version code
}
}
http://commons.apache.org/proper/commons-configuration/
User a simple database table (Section, Key, Value). Add "Version" if you need it, and wrap the entire thing in a simple ConfigurationService class with methods like getInt(String section, String key)
Not a lot of work, and it makes the application code very neat, and tweaking with the configuration very easy.
Interesting alternative config file format: write a scala trait. Your config file can then just be a scala file that you compile and evaluate when the server starts.
http://robey.lag.net//2012/03/26/why-config.html
There is a team develops enterprise application with web interface: java, tomcat, struts, mysql, REST and LDAP calls to external services and so on.
All configuration is stored in context.xml --tomcat specific file that contains variables available via servlet context and object available via JNDI resources.
Developers have no access to production and QA platforms (as it should be) so context.xml is managed by support/sysadmin team.
Each release has config-notes.txt with instructions like:
please add "userLimit" variable to context.xml with value "123", rename "DB" resource to "fooDB" and add new database connection to our new server (you should know url and credentials) named "barDb"
That is not good.
Here is my idea how to solve it.
Each release has special config file with required variable names, descriptions and default values (if any): even web.xml could be used.
Here is pseudo example:
foo=bar
userLimit=123
barDb=SET_MANUAL(connection to our new server)
And there is a special tool that support team runs against deployment artifact.
Look at it (text after ">" is typed by support guy):
Config for version 123 of artifact "mySever".
Enter your config file location> /opt/tomcat/context/myServer.xml
+"foo" value "bar" -- already exists and would not be changed
+"userLimit" value "123" -- adding new
+"barDb"(connection to our new server) please type> jdbc:mysql:host/db
Saving your file as /opt/tomcat/context/myServer.xml
Your environment is not configured to run myServer-123.
That will give us ability to deploy application on any environment and update configuration if needed.
Do you like my idea? What do you use for environment configuration management? Does there is ready-to-use tools for that?
There are plenty of different strategies. All of them are good and depends on what suit you best.
Build a single artifact and deploy configs to a separate location. The artifact could have placeholder variables and, on deployment, the config could be read in. Have a look at Springs property placeholder. It works fantastically for webapps that use Spring and doesn't involve getting ops involved.
Have an externalised property config that lives outside of the webapp. Keep the location constant and always read from the property config. Update the config at any stage and a restart will be up the new values.
If you are modifying the environment (i.e. application server being used or user/group permissions) look at using the above methods with puppet or chef. Also have a look at managing your config files with these tools.
As for the whole should devs be given access to prod, it really depends on a per company basis. For smaller companies where the dev is called every time there is a problem, regardless of whether that problem is server or application related, then obviously devs require access to the box.
DevOps is not about giving devs access to the box, its about giving devs the ability to use infrastructure as a service, the ability to spawn new instances with application X with config Y and to push their applications into environments without ops. In a large company like ours, what it allows is the ability for devs to manage the application they put on a server. Operations shouldn't care what version is on their, thats our job, their job is all about keeping the server up and running.
I strongly disagree with your remark that devs shouldn't have access to prod or staging environments. It's this kind of attitude that leads to teams working against each other instead of with eath other.
But to answer your question: you are thinking about what is typically called continuous integration ( http://en.wikipedia.org/wiki/Continuous_integration ) and moving towards devops. Ideally you should aim for the magic "1 click automated deployment". The guys from Flickr wrote a lot of blogs (and books) about how they achieved that.
Anyhow .. there's a lot of tools around that sector. You may want to have a look a things like Hudson/Jenkins or Puppet/Chef.
We have a build that can package war files for specific environments where all our property files are embedded in the archive (war file).
We are now about to build for production. My concern is the codebase will need to expose the production database password and although unlikely there is a risk where the production build profile could be run with a negative effect.
Options I thought of to negate this risk is to not store the production details in SVN and:
Have the administrators override system properties which are used to connect to the DB, or
Have the container manage the DB connection instead of c3p0, this way they can manage this configuration themselves.
Do you have any advice?
You should definitely not be putting the production DB username and password into your source control system. Your app should be getting its DB connection (eg a DataSource) using JNDI which is controlled/restricted by the admins on the production environment.
For example, if your app is deployed to Tomcat, you have the following in tomcat/conf/context.xml
<Resource name="jdbc/myDB"
auth="Container"
type="javax.sql.DataSource"
maxActive="20"
maxIdle="10"
maxWait="3000"
username="myusername"
password="mypassword"
driverClassName="com.mysql.jdbc.Driver"
url="jdbc:mysql://myhost:3306/myschema"
defaultAutoCommit="false"/>
..and the connection is obtained from java:/comp/env/jdbc/myDB without your app ever having to provide a username or password. The tomcat installation is protected on the prod servers by the admins, so it is unavailable to anyone without admin access on your prod server.
In production I favor the approach to not store credentials on property files at all. Instead I prefer the application server to supply the credentials using jndi.
If you are using Apache Tomcat, see for instance their jndi reference.
I have tried both having properties in the archive and outside of the archive, and having them outside of the archive is much easier to manage this kind of problem.
Some things to note:
To the extent that is possible, you can have defaults for properties, so that if the property is not found it will use the default (for example, using "localhost" as the default database connection URL).
You can still keep property files for non-production environments in source control alongside the code.
Using these two policies, developers can be responsible for managing non-production property files, which also therefore serve as examples to the production admin. It also keeps most of the properties centralized in source control, giving some of the benefits of keeping things centralized, while still decoupling the properties enough.
EDIT: Note that JNDI is an option, but architecturally it is the same as storing property files outside - you still need to take care to version these not have them be loose in different environments.