I have an application that consists of approx. 20 java components.
About half of the components are servers and the other half batch programs.
Almost all of them talk directly to an oracle database (jdbc via some of our infrastructure code jars) the other couple of components talk to some of the servers which talk to the database.
Anyway, each component is configured with numerous XML configuration files.
These are becoming almost impossible to maintain.
Some of the configuration is specific to a component others are similar (database URLs, connectors etc)
What is worse is that the application is not installed in many environments - in fact only about 10 environments (qa, dev, production etc etc).
But the people who own these environments don't seem able to maintain the configs correctly.
In particular whenever there is an upgrade there is invariably configuration errors.
I have even started checking in some of the environments configurations into SVN along with the code.
I tried an xml schema validator at one point (it consisted of defining the valid XML in .xsd files and then throwing an error if the schema rules were breached but that didnt work)
I'm thinking I am missing something basic here - perhaps there is a tool to manage this or perhaps I should be storing the configuration in the database.
The application was largely designed by a colleague but I feel myself that it's overly configurable - in fact many of the config actually refers to classes - i.e. one can choose handlers and parsers etc - the XML config almost looks like code.
Any advice greatly appreciated
Peter
Substituting XML for code is usually a bad idea; things that are declarative are probably OK, but things that are procedural probably aren't.
If all that configuration was defined in Java code, a lot of the upgrade issues would turn into compilation issues. The compiler would pick them out for you, and you could correct them.
So you've got a multi-part problem. You need to rationalize your configuration information into a set of partitions (per-component, per-installation, global). You need to try to verify configuration information at compile-time, where possible. And you need to write validation for the loaded configurations, to sanity check them.
To the extent possible, shift configuration relatively static stuff into Guice (at least, it's what I prefer). A lot of things happen in a nice, type-safe way with it.
Consider running a WebDAV server for each instance of the app, and storing configuration into it. Each can hit a simple URL to pull the current versions of the configuration files.
Or, stand up a lightweight XML database like BaseX with its REST capability, then store and load your configuration information there. Use JSLP or something like it to have your components find the central configuration repository.
An additional advantage to using an XML DB is that you'll be able to do a lot of sanity checking and updating by querying across the set of all configuration files. For example, if a given instance of the application should have the same JDBC parameters in each configuration file, a simple xquery will tell you if that's true.
If you don't have the ability to modify the applications that are pulling the configuration file (the config file format is fixed), then consider writing a query servlets for the XML database that assemble the required configuration information, from nested blocks or templates. That will allow you to figure out what's common between the configuration files and dynamically generate parameterized versions of those blocks.
Sounds like the key here is making incremental improvements. Allow the old way to configure, but have the configuration load look for a central config source first.
I don't think that the syntax of the configuration files is at the heart of the problem: using Java properties files instead of XML would leave you with exactly the same issues. There may be an issue that the configuration information is too dispersed - it's hard to tell. The main issue seems to be that the whole thing is too fragile - the application is too dependent on manual configuration, and it seems that the configuration for each environment needs to be different. You should try to focus on reducing the number of configuration parameters that need to be set to make the system work (without necessarily reducing the options available for diagnostics etc for use when they are really needed.), on having intelligent defaults and self-configuration. Perhaps even invest in creating an installation wizard.
As you have some Oracle databases handy why not store your configuration in there?
Then you only need one or two configuration parameters to point to an Oracle database suitable for that environment and download the rest of the configuration from the database.
The contents of the configuration table should be pretty static for any given environment so there should be no need to amend anything except the jdbc connection when you migrate your software through its life cycle.
Related
What is the best way to store parameters and data for an EE7 application. I have to provide the web applications with information like a member fee or similar data (which may/can be altered several times in a year). The owner of the application should also have a central place where these data are stored and an application to change them.
Thanks in advance for any input
Franz
This is one question we are currently struggling with as we re-architect some of our back-end systems here, and I do agree with the comment from #JB Nizet that it should be stored on the database, however I will try to add some additional considerations and options to help you make the decision that is right for you. The right option will depend on a few factors though.
If you are delivering source code and automation to build and deploy your software, the configuration can be stored in a source code repository (i.e. as YAML or XML) and bundled with your deployable during the build process. This is a bit archaic but certainly widely adopted practice and works well, for the most part.
If you are delivering deployable binaries, you have a couple of options.
First one is to have a predetermined place in the file system where your application will look for an "override" configuration file (i.e. home directory of the user used to run your application server). This way you can have your binary deployable file completely separate from your configuration, but you will still need to build some sort of automation and version control for that configuration file so that your customer can roll back versions if/when necessary. This can also be one or many configuration files (i.e. separate files for your app server, vs. the application itself).
The option we are contemplating currently is having a configuration database where all of our applications can query for their own configuration. This can either be a very simple or complex solution depending on your particular needs - for us these are internal applications and we manage the entire lifecycles ourselves, but we have a need to have a central repository since we have tens of services and applications running with a good number of common configuration keys, and updating these keys independently can be error prone.
We are looking at a few different solutions, but I would certainly not store the configuration in our main database as: 1) I don't think SQL is best repository for configuration, 2) I believe we can get better performance from NoSQL databases which can be critical if you need to load some of those configuration keys for every request.
MongoDB and CouchDB both come to mind as good candidates for storing the our configuration keys if you need clearly defined hierarchy for you options, whereas Redis or Memcached are great options if you just need a key-value storage for your configuration (faster than document based too). We will also likely build a small app to help up configure and version the configuration and push changes to existing/active servers, but we haven't spec'd out all the requirements for that.
There are also some OSS solutions that may work for you, although some of them add too much complexity for what we are trying to achieve at this point. If you are using springframework, take a look at the Spring Cloud Config Project, it is very interesting and worth looking into.
This is a very interesting discussion and I am very willing to continue it if you have more questions on how to achieve distributed configurations. Food for thought, here are some of my personal must haves and nice to haves for our new configuration architecture design:
Global configuration per environment (dev,staging,prod)
App specific configuration per environment (dev,staging,prod)
Auto-discovery (auto environment selection depending on requestor)
Access control and versioning
Ability to push updates live to different services
Roger,thanks a lot. Do you have an example for the version predetermined place in the file system"predetermined place in the file system"? Does it make sense to use a singleton which reads the configuration file (using Startup annotation) and provides then the configuration data? But this does not support a dynamic solution.kind regards Franz
We have a legacy application that uses Struts 1.2.9. The app is currently internationalized the standard way - .properties files for all UI labels, errors, messages, etc; <message-resouces> definition for each .properties file in struts-config.xml using default Factory & MessageResources definitions; <bean:message> usage in all JSPs. This has worked great till now, but for the fact that the application itself a framework for services used by a few hundred (yes 100's!) other applications internally.
We have a requirement to extend the i18n functionality as follows:
Define a custom directory for .properties files - so this would be outside the scope of the classpath; basically not inside the .war package. The idea is to support just message string changes w/o really having to redeploy the entire application.
This custom directory will also contain per supported applications messages - this could be just a subset of the existing ones or the entire set of resources tailored specifically to that application.
Custom way of supporting per request basis Locale setting - barring all other considerations (default stack, classpath/package lookups, etc.) this is analogous to the way I18nInterceptor works in Struts2 with the requestOnlyParameterName attribute set to true.
Yes, I do understand that a few 100 bundles loaded at the same time will be memory intensive, but that is acceptable in our case.
Any help is appreciated - be it direction, sample code, etc.
Note: I completely agree that moving onto a newer UI platform is probably the best solution. But we can't.
TIA.
I had a similar requirement in a spring project, not only for i18n, also web services endpoints and other kind of properties.
We accomplish that requirement by adding that directory, in which we place the properties files, into the classpath in the server start configuration file.
Tested and working in weblogic 11g (Preproduction and production) and in a tomcat server (development environment).
Hope helps
Currently my spring configurations are in a xml file (the traditionaly way).
One thing that i like about this is during deployment I can deploy a different version that has my production settings, or say in a test environment I can have test settings there.
I like the idea of having things configured in a class, but that will get compiled into my war and then it won't be as flexible.
Is there a way around this?
Java configuration is great and it has several advantages:
refactoring-friendly
type-safety
much more flexible (you can write any Java code, not being bound to XML semantics and capabilities).
I can deploy a different version that has my production settings, or say in a test environment I can have test settings there.
Investigate Spring #Profiles. They are orthogonal to your question (work both in XML and #Configuration) but are best suited in your situation.
Those are the only two ways available. If you don't want configuration baked in code, then you have to go with xml.
The spring reference manual includes a section on combining both Java and XML configuration. See http://static.springsource.org/spring/docs/3.1.x/spring-framework-reference/html/beans.html#beans-java-combining
If you tend to be more XML centric you can bootstrap your app using XML and then introduce Java config as needed in an ad-hoc fashion. This might be a good way to ease into it. You might decide to go Java config all the way.
In Java, is it a good practice to use annotations to configure an application rather than using XML files? I am more skeptical about it because, using annotations involves changing the java source files and it is as good as declaring constants in java files and then using those constants, whereas when we make the configurations using XML files, we can keep all the configuration changes away from java source files and keep the configurations in separate XML files, this approach sounds more neat. Also, when we need to make changes to configuration, we know which XML file to change rather than searching the java files for the annotations. Also, we can update an XML file in an EAR without compiling the code again, whereas if we make any change in an annotation, then we need to compile the code again.
Can anybody please throw some light on why should we use annotations and not XML files for configuration?
Which to use may vary depending on what's being configured, how the configuration is used, project/cultural conventions, etc. Good IDE support makes using either more convenient and reliable.
Personally, while I use both XML and annotations, I tend to prefer XML configuration for many tasks, particularly on larger projects. For example, with Spring, I prefer XML configuration: it's easier to manage the configuration itself, configuration changesets, and environment-specific changes (e.g., testing, server-based, etc.), when it's more localized.
For other configurations, annotations are often more appropriate and convenient. For quick projects with little or no domain class customization, Hibernate annotations may be more convenient.
Ultimately it's a matter of preference and convenience rather than a purely technical one. (Except when the XML and annotations support different features; sometimes one offers more-complete capabilities.)
I prefer annotations since my IDE can help me validate my configurations. Configurations stored in xml-files cannot be validated before runtime (I'm thinking mostly about spring and injections)
Also, I find that for anything more than a tiny project, a large xml-config is hard to maintain.
When working with annotations, you have to take care of only one place to configure your stuff (java code). When configuring with XML, many times a programmer can "forget" to configure a new property or class in the XML, and after the error must correct and restart, resulting in a waste of time.
I would say it very much depends on what you are configuring.
If configuration maybe or should be changed after deployment, then in most cases it is preferable to use XML (or other text-based format). This would include Hibernate server configuration, Tomcat/Jetty configuration, Log4j configuration, etc. The main advantage is flexibility.
For cases when configuration does not need change after deployment, configuration using annotations is preferable. Too many configuration files also complicate your software, so it's best to keep it to a minimum. Good examples would be of annotation-based configurations: Hibernate bean mapping configuration, Spring dependency injection, Guice, etc (some give you both options, but I would prefer annotations here). The advantage is better manageability, and compile-time checking for errors (this depends on the API, of course).
Personally, when I've tried to understand a new system, having the annotations right there with the code makes it easier to follow and comprehend. Hunting for references of the class in configuration files can be a little annoying.
UPDATE: See my blog post on this topic about a year after this was written: http://blog.ringerc.id.au/2012/07/java-ee-7-needs-improvements-in-app.html
... for references to the Java EE 7 planning discussion on this topic.
I've mostly finished writing a small Java EE 6 application, and am in the process of replacing the hard-coded preferences with a proper dynamic configuration interface.
I'm not sure how - or, more specifically, where - to store settings. There must be some obvious, "standard" way to do this that's expected to "just work" across various frameworks and containers, but for the life of me I cannot find it.
What I want is a simple way to load and store settings, one that works across different app servers and OSes, doesn't require any confguration by the user, and actually works properly. The Java Preferences API would be ideal - but seems broken under Glassfish 3.1.
Options for storing configuration would theoretically include:
Using context-parameters from the container environment
Storing them using the Java Preferences API
Reading/writing a properties file ... somewhere
Using JPA to store them in a JavaDB provided by the container
Putting it in a properties file that's loaded off the classpath
Using system properties to set configuration options, or path to a .properties file
This would seem to be a basic requirement that'd be well catered for in an environment where the container supposedly provides you with all the core services you might need - but all these approaches have issues.
A bug in glassfish renders (1) unworkable, and in any case the Glassfish web admin user interface lacks any way to configure context parameters, so you have to use `asadmin' and some less than lovely command line syntax to do it. Context parameters can only be accessed via the ServletContext - which isn't accessible in a consistent way between frameworks like JSF2, JAX-RS, and raw servlets - but at least Seam Servlet handles that.
What appears to be another bug in glassfish was a library version conflict between the deployed app and Glassfish breaks (2). The preferences backend fails to flush preferences to disk, so the stored preferences data is lost when the application server is restarted. The Java Preferences API also seems to be considered a J2SE/desktop thing, despite its inclusion in the Java EE 6 specs.
(3) might work - but there's no way to know where your app has read/write access on the file system and where it should look. You can't make this configurable, as it becomes a chicken-and-egg problem then. Various platform-specific guesses could be applied, but would break in the presence of a SecurityManager.
(4) would work, but it's nuking a fly. It requires that a JavaDB service be running and forces the user to make sure the JDBC and pool resources in the app server are configured properly. It's big and complicated for a simple job, and entity modelling isn't a lovely fit for preferences storage anyway, as it'll mostly land up being key/value structured.
(5) would work, but requires users to know where to put the config file where it'll be found under various different app servers. It also makes it hard for the app to provide any kind of configuration UI because it can't necessarily find the local path to the config file or open it for writing, especially in the presence of a SecurityManager.
(6) would also work, but forces the user to configure the configuration system before they can configure the application. Needless to say, that doesn't excite me, given how relatively complicated deploying the app and creating the resources already is for users who don't already know Glassfish/EE.
So ... how are you handling configuration and storage of options? Have you found a way that lets you "just do it" without the user having to configure anything to allow your app to store its configuration?
The problem with the preferences API was caused by the inclusion of jaxb and stax implementation jars on in the application's war, pulled in by jersey-json . With these excluded (as they're provided by the app server anyway) the preferences API resumed functioning correctly.
It looks like the prefs API with custom UI for setup appears to be the best way to go.
Though not the environment you spoke about: http://www.osgi.org/javadoc/r4v42/org/osgi/service/cm/ConfigurationAdmin.html