We Keep Coding

Late discovery of beans using CDI

I'm creating a Java web application, making use of CDI. My War includes various Jar files that are bean sources (including the META-INF/beans.xml file). My beans are automatically discovered when running inside a Java EE container like JBoss (I'm currently using JBoss AS 7.1.1). This much is working well.
I want to be able to extend the web application, ideally by allowing it to load classes from Jar files stored in a configurable location (so at a location specified by a system property). This too is fine, since I can use a ClassLoader to resolve classes and resources. What I'm missing is the ability to have CDI scan these external Jar files and include their beans.
This would allow my application to host plug-in functionality including new REST resources. I don't mind if I had to restart the application for it to pick up classes and resources contained within these external Jar files.
I can see no way of achieving this. Is this even something that should be attempted in this kind of environment?

I can see no way of achieving this.
You actually have two options:
Use the CDI extension mechanism to work with beans / bean-archives
at startup time and to modify them in the way you want. Have a look at the examples provided in the documentation, this should give you a start.
Work with the BeanManager at execution time. Have a look at this similar question.

multiple java servers and batch programs - XML configuration nightmare

I have an application that consists of approx. 20 java components.
About half of the components are servers and the other half batch programs.
Almost all of them talk directly to an oracle database (jdbc via some of our infrastructure code jars) the other couple of components talk to some of the servers which talk to the database.
Anyway, each component is configured with numerous XML configuration files.
These are becoming almost impossible to maintain.
Some of the configuration is specific to a component others are similar (database URLs, connectors etc)
What is worse is that the application is not installed in many environments - in fact only about 10 environments (qa, dev, production etc etc).
But the people who own these environments don't seem able to maintain the configs correctly.
In particular whenever there is an upgrade there is invariably configuration errors.
I have even started checking in some of the environments configurations into SVN along with the code.
I tried an xml schema validator at one point (it consisted of defining the valid XML in .xsd files and then throwing an error if the schema rules were breached but that didnt work)
I'm thinking I am missing something basic here - perhaps there is a tool to manage this or perhaps I should be storing the configuration in the database.
The application was largely designed by a colleague but I feel myself that it's overly configurable - in fact many of the config actually refers to classes - i.e. one can choose handlers and parsers etc - the XML config almost looks like code.
Any advice greatly appreciated
Peter

Substituting XML for code is usually a bad idea; things that are declarative are probably OK, but things that are procedural probably aren't.
If all that configuration was defined in Java code, a lot of the upgrade issues would turn into compilation issues. The compiler would pick them out for you, and you could correct them.
So you've got a multi-part problem. You need to rationalize your configuration information into a set of partitions (per-component, per-installation, global). You need to try to verify configuration information at compile-time, where possible. And you need to write validation for the loaded configurations, to sanity check them.
To the extent possible, shift configuration relatively static stuff into Guice (at least, it's what I prefer). A lot of things happen in a nice, type-safe way with it.
Consider running a WebDAV server for each instance of the app, and storing configuration into it. Each can hit a simple URL to pull the current versions of the configuration files.
Or, stand up a lightweight XML database like BaseX with its REST capability, then store and load your configuration information there. Use JSLP or something like it to have your components find the central configuration repository.
An additional advantage to using an XML DB is that you'll be able to do a lot of sanity checking and updating by querying across the set of all configuration files. For example, if a given instance of the application should have the same JDBC parameters in each configuration file, a simple xquery will tell you if that's true.
If you don't have the ability to modify the applications that are pulling the configuration file (the config file format is fixed), then consider writing a query servlets for the XML database that assemble the required configuration information, from nested blocks or templates. That will allow you to figure out what's common between the configuration files and dynamically generate parameterized versions of those blocks.
Sounds like the key here is making incremental improvements. Allow the old way to configure, but have the configuration load look for a central config source first.

I don't think that the syntax of the configuration files is at the heart of the problem: using Java properties files instead of XML would leave you with exactly the same issues. There may be an issue that the configuration information is too dispersed - it's hard to tell. The main issue seems to be that the whole thing is too fragile - the application is too dependent on manual configuration, and it seems that the configuration for each environment needs to be different. You should try to focus on reducing the number of configuration parameters that need to be set to make the system work (without necessarily reducing the options available for diagnostics etc for use when they are really needed.), on having intelligent defaults and self-configuration. Perhaps even invest in creating an installation wizard.

As you have some Oracle databases handy why not store your configuration in there?
Then you only need one or two configuration parameters to point to an Oracle database suitable for that environment and download the rest of the configuration from the database.
The contents of the configuration table should be pretty static for any given environment so there should be no need to amend anything except the jdbc connection when you migrate your software through its life cycle.

Is there something like the Weblogic filtering classloader for Websphere

I'd like to use something like the Filtering Classloader to prevent specific packages from creeping into the application context and becoming visible to Spring.
Changing the classloader order causes all sorts of nasty problems so I´d like to try this route.
Is it possible to achieve this with Websphere 6? If not, can I replace my own application classolader and implement the filter myself?

There is no such filtering mechanism in WebSphere, and there is no way to replace the application class loader. You'll have to use PARENT_LAST to override classes, sorry.

bkail's answer is right, WAS doesn't have such feature even in its latest public version (8.5.5).
I just created a RFE requesting such feature so whoever is interested in this, please vote for it which may increase the possibility of this being implemented:
http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=43936
(IBM ID required)
In the meantime, you may use isolated shared libraries to override any particular classes (the above mentioned class loading order control - like parent_last - is too rough as it affects the class loading order of the whole application or module)
Create a shared library with desired jars on the classpath, configure it as isolated shared library, reference it from the deployed application (or module).
See here for complete documentation
http://pic.dhe.ibm.com/infocenter/wasinfo/v8r5/topic/com.ibm.websphere.base.doc/ae/tcws_sharedlib.html

I was just about to post the same question. But the answer was quite unsatisfying. I however checked the request from Petr H at the IBM developerworks and IBM did implement this feature (Huge thanks Petr!):
"WebSphere Application Server V8.5.5.7 (=Fixpack 7) gained the ability to prevent packages from the server classloader being visible to applications. This was delivered in the document "ISOLATE DEPLOYED ARTIFACTS FROM OSS PACKAGES" and is documented in 'Isolating open source software packages'.
The supplied links describe the mechanism by configuring always-protected packages. You basically have to do the following:
Under Server Infrastructure on the server settings page in the administrative console, click Java and process management > Process definition.
Select Java virtual machine.
Define the following system properties in the JVM generic arguments section as follows:
-Dcom.ibm.ws.classloader.server.alwaysProtectedPackages=org.bouncycastle.
Please not that the final dot "." is really important otherwise everything will be ignored! Several packages can be added by comma ","
Click Apply, OK and save the changes. Make sure that a file synchronization is done before you restart the servers. Restart WebSphere Application Server for the changes to take effect.
Examine the native_stdout.log and find the system properties that are previously defined. For example, when you specify always-protected package org.bouncycastle., statements such as the following might appear:
ProtectionMetaData.clinit: system property: com.ibm.ws.classloader.server.alwaysProtectedPackages=org.bouncycastle.

Storing and editing configuration for Java EE applications

UPDATE: See my blog post on this topic about a year after this was written: http://blog.ringerc.id.au/2012/07/java-ee-7-needs-improvements-in-app.html
... for references to the Java EE 7 planning discussion on this topic.
I've mostly finished writing a small Java EE 6 application, and am in the process of replacing the hard-coded preferences with a proper dynamic configuration interface.
I'm not sure how - or, more specifically, where - to store settings. There must be some obvious, "standard" way to do this that's expected to "just work" across various frameworks and containers, but for the life of me I cannot find it.
What I want is a simple way to load and store settings, one that works across different app servers and OSes, doesn't require any confguration by the user, and actually works properly. The Java Preferences API would be ideal - but seems broken under Glassfish 3.1.
Options for storing configuration would theoretically include:
Using context-parameters from the container environment
Storing them using the Java Preferences API
Reading/writing a properties file ... somewhere
Using JPA to store them in a JavaDB provided by the container
Putting it in a properties file that's loaded off the classpath
Using system properties to set configuration options, or path to a .properties file
This would seem to be a basic requirement that'd be well catered for in an environment where the container supposedly provides you with all the core services you might need - but all these approaches have issues.
A bug in glassfish renders (1) unworkable, and in any case the Glassfish web admin user interface lacks any way to configure context parameters, so you have to use `asadmin' and some less than lovely command line syntax to do it. Context parameters can only be accessed via the ServletContext - which isn't accessible in a consistent way between frameworks like JSF2, JAX-RS, and raw servlets - but at least Seam Servlet handles that.
What appears to be another bug in glassfish was a library version conflict between the deployed app and Glassfish breaks (2). The preferences backend fails to flush preferences to disk, so the stored preferences data is lost when the application server is restarted. The Java Preferences API also seems to be considered a J2SE/desktop thing, despite its inclusion in the Java EE 6 specs.
(3) might work - but there's no way to know where your app has read/write access on the file system and where it should look. You can't make this configurable, as it becomes a chicken-and-egg problem then. Various platform-specific guesses could be applied, but would break in the presence of a SecurityManager.
(4) would work, but it's nuking a fly. It requires that a JavaDB service be running and forces the user to make sure the JDBC and pool resources in the app server are configured properly. It's big and complicated for a simple job, and entity modelling isn't a lovely fit for preferences storage anyway, as it'll mostly land up being key/value structured.
(5) would work, but requires users to know where to put the config file where it'll be found under various different app servers. It also makes it hard for the app to provide any kind of configuration UI because it can't necessarily find the local path to the config file or open it for writing, especially in the presence of a SecurityManager.
(6) would also work, but forces the user to configure the configuration system before they can configure the application. Needless to say, that doesn't excite me, given how relatively complicated deploying the app and creating the resources already is for users who don't already know Glassfish/EE.
So ... how are you handling configuration and storage of options? Have you found a way that lets you "just do it" without the user having to configure anything to allow your app to store its configuration?

The problem with the preferences API was caused by the inclusion of jaxb and stax implementation jars on in the application's war, pulled in by jersey-json . With these excluded (as they're provided by the app server anyway) the preferences API resumed functioning correctly.
It looks like the prefs API with custom UI for setup appears to be the best way to go.

Though not the environment you spoke about: http://www.osgi.org/javadoc/r4v42/org/osgi/service/cm/ConfigurationAdmin.html

Java Application/Thread Server

I am looking for something very close to an application server with these features:
it should handle a series of threads/daemons, allowing the user to start-stop-reload each one without affecting the others
it should keep libraries separated between different threads/daemons
it should allow to share some libraries
Currently we have some legacy code reinventing the wheel... and not a perflectly round-shaped one at that!
I thought to use Tomcat, but I don't need a web server, except maybe for the simple backoffice user interface (/manager/html).
Any suggestion? Is there a non-web application server, or is there a better alternative to Tomcat (more lightweight, for example, or easier to configure)? Thanks in advance.

Have you looked at OSGi ? You can load/unload bundles (basically .jar files with metadata) independently of each other, and optionally define dependencies between these (with a software lifecycle defined such that bundles are aware of other bundles being loaded/unloaded).

I have found the Jetty "contexts" concept very useful in handling applications (packaged as WAR's and with servlet context listeners), where the xml-file placed in contexts/ describe fully what you want to have started. When you remove the xml-file again, the thing described is stopped.
If you do not start a server connector you will just have a start-stop thing which sounds like what you are looking for.
Jetty can be made very small so the overhead is not bad.

You could consider Spring dmServer. It's a rather non-traditional appserver, with a very lightweight OSGi core (the web container is optional, for example), but it gives you classloader isolation and basic container services. It's not a JavaEE container, but comes with plug-in modules that are.
You're stlll going to have to do a lot of work yourself, but the basics of dmServer are very sound.

No one stops you from sending binary and text data instead of HTML-pages using http protocol. That is whats servlets are for. So I would use the tomcat server.