Suppose I've got a Java application, which uses properties files as its configuration. Now I'd like to make the application re-read the configuration without restart. I'll call such configuration "refreshable".
What is the easiest way to do that?
Suppose I am developing an application from scratch. How should I design it to make its configuration "refreshable"?
Using the commons-configuration is probably the easiest way to go. Here is an example of usage. There are many other features included, like hierarchical configuration (with overriding by defaults), JDBC sources, etc.
There must be some class say ConfigurationReader which has methods to read configurations and store them in-memory in some collection. Make sure that all the other classes uses the config information this in-memory map.
Now you can make a function call to re-read all the parameters and update the in-memory map, so that rest of the application will use the updated values from the map. This way you need to change code in entire application.
Related
What is the best way to store parameters and data for an EE7 application. I have to provide the web applications with information like a member fee or similar data (which may/can be altered several times in a year). The owner of the application should also have a central place where these data are stored and an application to change them.
Thanks in advance for any input
Franz
This is one question we are currently struggling with as we re-architect some of our back-end systems here, and I do agree with the comment from #JB Nizet that it should be stored on the database, however I will try to add some additional considerations and options to help you make the decision that is right for you. The right option will depend on a few factors though.
If you are delivering source code and automation to build and deploy your software, the configuration can be stored in a source code repository (i.e. as YAML or XML) and bundled with your deployable during the build process. This is a bit archaic but certainly widely adopted practice and works well, for the most part.
If you are delivering deployable binaries, you have a couple of options.
First one is to have a predetermined place in the file system where your application will look for an "override" configuration file (i.e. home directory of the user used to run your application server). This way you can have your binary deployable file completely separate from your configuration, but you will still need to build some sort of automation and version control for that configuration file so that your customer can roll back versions if/when necessary. This can also be one or many configuration files (i.e. separate files for your app server, vs. the application itself).
The option we are contemplating currently is having a configuration database where all of our applications can query for their own configuration. This can either be a very simple or complex solution depending on your particular needs - for us these are internal applications and we manage the entire lifecycles ourselves, but we have a need to have a central repository since we have tens of services and applications running with a good number of common configuration keys, and updating these keys independently can be error prone.
We are looking at a few different solutions, but I would certainly not store the configuration in our main database as: 1) I don't think SQL is best repository for configuration, 2) I believe we can get better performance from NoSQL databases which can be critical if you need to load some of those configuration keys for every request.
MongoDB and CouchDB both come to mind as good candidates for storing the our configuration keys if you need clearly defined hierarchy for you options, whereas Redis or Memcached are great options if you just need a key-value storage for your configuration (faster than document based too). We will also likely build a small app to help up configure and version the configuration and push changes to existing/active servers, but we haven't spec'd out all the requirements for that.
There are also some OSS solutions that may work for you, although some of them add too much complexity for what we are trying to achieve at this point. If you are using springframework, take a look at the Spring Cloud Config Project, it is very interesting and worth looking into.
This is a very interesting discussion and I am very willing to continue it if you have more questions on how to achieve distributed configurations. Food for thought, here are some of my personal must haves and nice to haves for our new configuration architecture design:
Global configuration per environment (dev,staging,prod)
App specific configuration per environment (dev,staging,prod)
Auto-discovery (auto environment selection depending on requestor)
Access control and versioning
Ability to push updates live to different services
Roger,thanks a lot. Do you have an example for the version predetermined place in the file system"predetermined place in the file system"? Does it make sense to use a singleton which reads the configuration file (using Startup annotation) and provides then the configuration data? But this does not support a dynamic solution.kind regards Franz
Disclaimer: I am a noob in Spring. What I am asking may be very "odd" as I don't even know what I don't know.
I am trying to create a batch data movement/manipulation tool (may I say a ETL tool) using Java. Someone suggested me to check out spring-batch which I really liked as it has many libraries for data reading/writing and processing.
But my trouble is- my data sources (flatfile or table) are not fixed. There is a fronted where user will select which flatfile or database table(s) they want to load and the program will automatically load that. This means, usual things like:
Source / target entity structures
source or target database URL/DSN
Job parameters etc.
are not pre-determined in my case. They are determined in runtime. But, so far, whatever spring-batch examples I have seen - they have configured these information in XML. I can't do that as that will make these information static.
My Question is - If I do not want to use Spring Container (and all its XML based bean configuration) but still want to use spring-batch to take advantage of it's batch processing libraries, will that be possible/viable?
No, you need to use the Spring Container for using spring batch and all its XML or annotation based bean configuration. However, what you are trying is achievable, you just need to find way to make it configurable by using parameters in Spring batch. You can take anyone example from internet and start working on it to make it configurable.
Like you can utilize file reader from Spring by simply writing custom mapper. You save the effort to create and maintain file reading logic.
You can have writer which can query which you create dynamically based on your table and file at run time.
Examples shows everything in xml for making simple to understand, how ever if you explore little bit almost everything can be done at runtime.
I started using dependency injection with roboguice and created an interface like DataProvider. I have an implementation which retrieves the data from some WebServer located in the WebServerDataProvider class. In Order to eliminate the waiting for the webserver i added a DummyDataProvider.
Where would i put such class? I don't like that it is in /src/main/java/my/package/providers/ since it is not real part of the application, but still i need it for development.
Typically you would use such a class in your unit tests. Roboguice works well with Robolectric , which allows you to mock things like http access. If you do that you would put your code in src/test/java/...
You could put it into the main project if you want to use it for fiddling around with the application without bothering the server each time and deactivate it with some constant for deployment, e.g.
if (DEBUG) {
setDataProvider(new MockDataProvider());
}
Proguard should be smart enough to remove this unused class if you remember to reset your variable (you might have to fiddle around with the settings there).
Currently my spring configurations are in a xml file (the traditionaly way).
One thing that i like about this is during deployment I can deploy a different version that has my production settings, or say in a test environment I can have test settings there.
I like the idea of having things configured in a class, but that will get compiled into my war and then it won't be as flexible.
Is there a way around this?
Java configuration is great and it has several advantages:
refactoring-friendly
type-safety
much more flexible (you can write any Java code, not being bound to XML semantics and capabilities).
I can deploy a different version that has my production settings, or say in a test environment I can have test settings there.
Investigate Spring #Profiles. They are orthogonal to your question (work both in XML and #Configuration) but are best suited in your situation.
Those are the only two ways available. If you don't want configuration baked in code, then you have to go with xml.
The spring reference manual includes a section on combining both Java and XML configuration. See http://static.springsource.org/spring/docs/3.1.x/spring-framework-reference/html/beans.html#beans-java-combining
If you tend to be more XML centric you can bootstrap your app using XML and then introduce Java config as needed in an ad-hoc fashion. This might be a good way to ease into it. You might decide to go Java config all the way.
I have an application that consists of approx. 20 java components.
About half of the components are servers and the other half batch programs.
Almost all of them talk directly to an oracle database (jdbc via some of our infrastructure code jars) the other couple of components talk to some of the servers which talk to the database.
Anyway, each component is configured with numerous XML configuration files.
These are becoming almost impossible to maintain.
Some of the configuration is specific to a component others are similar (database URLs, connectors etc)
What is worse is that the application is not installed in many environments - in fact only about 10 environments (qa, dev, production etc etc).
But the people who own these environments don't seem able to maintain the configs correctly.
In particular whenever there is an upgrade there is invariably configuration errors.
I have even started checking in some of the environments configurations into SVN along with the code.
I tried an xml schema validator at one point (it consisted of defining the valid XML in .xsd files and then throwing an error if the schema rules were breached but that didnt work)
I'm thinking I am missing something basic here - perhaps there is a tool to manage this or perhaps I should be storing the configuration in the database.
The application was largely designed by a colleague but I feel myself that it's overly configurable - in fact many of the config actually refers to classes - i.e. one can choose handlers and parsers etc - the XML config almost looks like code.
Any advice greatly appreciated
Peter
Substituting XML for code is usually a bad idea; things that are declarative are probably OK, but things that are procedural probably aren't.
If all that configuration was defined in Java code, a lot of the upgrade issues would turn into compilation issues. The compiler would pick them out for you, and you could correct them.
So you've got a multi-part problem. You need to rationalize your configuration information into a set of partitions (per-component, per-installation, global). You need to try to verify configuration information at compile-time, where possible. And you need to write validation for the loaded configurations, to sanity check them.
To the extent possible, shift configuration relatively static stuff into Guice (at least, it's what I prefer). A lot of things happen in a nice, type-safe way with it.
Consider running a WebDAV server for each instance of the app, and storing configuration into it. Each can hit a simple URL to pull the current versions of the configuration files.
Or, stand up a lightweight XML database like BaseX with its REST capability, then store and load your configuration information there. Use JSLP or something like it to have your components find the central configuration repository.
An additional advantage to using an XML DB is that you'll be able to do a lot of sanity checking and updating by querying across the set of all configuration files. For example, if a given instance of the application should have the same JDBC parameters in each configuration file, a simple xquery will tell you if that's true.
If you don't have the ability to modify the applications that are pulling the configuration file (the config file format is fixed), then consider writing a query servlets for the XML database that assemble the required configuration information, from nested blocks or templates. That will allow you to figure out what's common between the configuration files and dynamically generate parameterized versions of those blocks.
Sounds like the key here is making incremental improvements. Allow the old way to configure, but have the configuration load look for a central config source first.
I don't think that the syntax of the configuration files is at the heart of the problem: using Java properties files instead of XML would leave you with exactly the same issues. There may be an issue that the configuration information is too dispersed - it's hard to tell. The main issue seems to be that the whole thing is too fragile - the application is too dependent on manual configuration, and it seems that the configuration for each environment needs to be different. You should try to focus on reducing the number of configuration parameters that need to be set to make the system work (without necessarily reducing the options available for diagnostics etc for use when they are really needed.), on having intelligent defaults and self-configuration. Perhaps even invest in creating an installation wizard.
As you have some Oracle databases handy why not store your configuration in there?
Then you only need one or two configuration parameters to point to an Oracle database suitable for that environment and download the rest of the configuration from the database.
The contents of the configuration table should be pretty static for any given environment so there should be no need to amend anything except the jdbc connection when you migrate your software through its life cycle.