We Keep Coding

How can I define multiple jobs dynamically in Spring Batch?

I have an application that uses Spring Batch to define a preset number of jobs, which are currently all defined in the XML.
We add more jobs over time which requires updating the XML, however these jobs are always based on the same parent and can easily be predetermined using a simple SQL query.
So I've been trying to switch to use some combination of XML configuration and Java-based configuration but am quickly getting confused.
Even though we have many jobs, each job definition falls into essentially one of two categories. All of the jobs inherit from one or the other parent job and are effectively identical, besides having different names. The job name is used in the process to select different data from the database.
I've come up with some code much like the following but have run into problems getting it to work.
Full disclaimer that I'm also not entirely sure I'm going about this in the right way. More on that in a second; first, the code:
#Configuration
#EnableBatchProcessing
public class DynamicJobConfigurer extends DefaultBatchConfigurer implements InitializingBean {
#Autowired
private JobBuilderFactory jobBuilderFactory;
#Autowired
private JobRegistry jobRegistry;
#Autowired
private DataSource dataSource;
#Autowired
private CustomJobDefinitionService customJobDefinitionService;
private Flow injectedFlow1;
private Flow injectedFlow2;
public void setupJobs() throws DuplicateJobException {
List<JobDefinition> jobDefinitions = customJobDefinitionService.getAllJobDefinitions();
for (JobDefinition jobDefinition : jobDefinitions) {
Job job = null;
if (jobDefinition.getType() == 1) {
job = jobBuilderFactory.get(jobDefinition.getName())
.start(injectedFlow1).build()
.build();
} else if (jobDefinition.getType() == 2) {
job = jobBuilderFactory.get(jobDefinition.getName())
.start(injectedFlow2).build()
.build();
}
if (job != null) {
jobRegistry.register(new ReferenceJobFactory(job));
}
}
}
#Override
public void afterPropertiesSet() throws Exception {
setupJobs();
}
public void setInjectedFlow1(Flow injectedFlow1) {
this.injectedFlow1 = injectedFlow1;
}
public void setInjectedFlow2(Flow injectedFlow2) {
this.injectedFlow2 = injectedFlow2;
}
}
I have the flows that get injected defined in the XML, much like this:
<batch:flow id="injectedFlow1">
<batch:step id="InjectedFlow1.Step1" next="InjectedFlow1.Step2">
<batch:flow parent="InjectedFlow.Step1" />
</batch:step>
<batch:step id="InjectedFlow1.Step2">
<batch:flow parent="InjectedFlow.Step2" />
</batch:step>
</batch:flow>
So as you can see, I'm effectively kicking off the setupJobs() method (which is intended to dynamically create these job definitions) from the afterPropertiesSet() method of InitializingBean. I'm not sure that's right. It is running, but I'm not sure if there's a different entry point that's better intended for this purpose. Also I'm not sure what the point of the #Configuration annotation is to be honest.
The problem I'm currently running into is as soon as I call register() from JobRegistry, it throws the following IllegalStateException:
To use the default BatchConfigurer the context must contain no more than one DataSource, found 2.
Note: my project actually has two data sources defined. The first is the default dataSource bean which connects to the database that Spring Batch uses. The second data source is an external database, and this second one contains all the information I need to define my list of jobs. But the main one does use the default name "dataSource" so I'm not quite sure how else I can tell it to use that one.

First of all - I don't recommend using a combination of XML as well as Java Configuration. Use only one, preferably Java one as its not much of an effort to convert XML config to Java config. (Unless you have some very good reasons to do it - that you haven't explained)
I haven't used Spring Batch alone as I have always used it with Spring Boot and I have a project where I have defined multiple jobs and it always worked well for similar code that you have shown.
For your issue, there are some answers on SO like this OR this which are basically trying to say that you need to write your own BatchConfigurer and not rely on default one.
Now coming to solution using Spring Boot
With Spring Boot, You should try segregate job definitions and job executions.
You should first try to just define jobs and initialize Spring context without enabling jobs (spring.batch.job.enabled=false)
In your Spring Boot main method, when you start app with something like - SpringApplication.run(Application.class, args); ...you will get ApplicationContext ctx
Now you can get your relevant beans from this context & launch specif jobs by getting names from property or command line etc & using JobLauncher.run(...) method.
You can refer my this answer if willing to order job executions. You can also write job schedulers using Java.
Point being, you separate your job building / bean configurations & job execution concerns.
Challenge
Keeping multiple jobs in a single project can be challenging when you try to have different settings for each job as application.properties file is environment specific and not job specific i.e. spring boot properties will apply to all jobs.

In my particular case, the solution was to actually eliminate the #Configuration
and #EnableBatchProcessing annotations from my class above. Something about these caused it to try and use the DefaultBatchConfigurer which fails when you have more than one data source defined (even if you've identified them clearly with "dataSource" as the primary and some other name for the secondary).
The #Configuration class in particular wasn't necessary because all it really does is lets your class get auto-instantiated without having to define it as a bean in the app context. But since I was doing that anyway this one was superfluous.
One of the downsides of removing #EnableBatchProcessing was that I could no longer auto-wire the JobBuilderFactory bean. So instead I just had to do to create it:
JobRepositoryFactoryBean factory = new JobRepositoryFactoryBean();
factory.setDataSource(dataSource);
factory.setTransactionManager(transactionManager);
factory.afterPropertiesSet();
jobRepository = factory.getObject();
jobBuilderFactory = new JobBuilderFactory(jobRepository);
Then it seems I was on the right track already by using jobRegistry.register(...) to define my jobs. So essentially once I removed those annotations above everything started working. I'm going to mark Sabir's answer as the correct one however because it helped me out.

How can I ensure Spring constructs a bean once

I have a complex enterprise environment with multiple projects depending on each other. Configuration is handled by spring and includes crazy number imports and what not.
Looks like by default Spring is OK with constructing a bean with the same name more than once. Particularly, in case of multiple Spring contexts. Each context its own instance of the same singleton bean. Singleton is not really a singleton in Spring architects' minds...
Unfortunately, in my case there's a bean that can never be created more than once.
Is there a way to enforce Spring checking upon the bean whether it's been created already and not to try to call its constructor again?
Particularly, the bean creates ehcache's CacheManager inside and fails because CacheManager with the same name can't be created twice.
<bean id="cacheService" class="somepackage.CacheServiceImpl">
<constructor-arg index="0" value="/somepackage/ehcache.xml" />
</bean>
I don't have control over the CacheServiceImpl code. I can only change configuration around it.

Different Spring application contexts don't know about each other. As far as each context is concerned, your object is a singleton. However, it sounds like you don't want there to be multiple contexts.
How are you creating contexts? Your application should create a single instance of ApplicationContext (probably in main or somewhere nearby). Anything else that needs an application context should have it injected or be made ApplicationContextAware.
http://docs.spring.io/spring/docs/current/spring-framework-reference/html/beans.html#beans-factory-instantiation
You mention a "complex enterprise environment". What's worked best for us where I work is having a single project manage Spring. Assuming all your projects are being run in the same application (otherwise Spring can't really help you), there's likely some project that starts everything off. For us, that's the project that is built into a war and deployed to our server.

I figured it out finally. The reason for loading the context twice was a hidden error occurring during the first context loading. A long and boring explanation below.
Context is supposed to be loaded during first invocation of spring method DefaultTestContext.getApplicationContext().
There's a class DefaultCacheAwareContextLoaderDelegate, which is responsible for actual loading the context once and caching it for future uses. The method code is below:
#Override
public ApplicationContext loadContext(MergedContextConfiguration mergedContextConfiguration) {
synchronized (this.contextCache) {
ApplicationContext context = this.contextCache.get(mergedContextConfiguration);
if (context == null) {
try {
context = loadContextInternal(mergedContextConfiguration);
if (logger.isDebugEnabled()) {
logger.debug(String.format("Storing ApplicationContext in cache under key [%s]",
mergedContextConfiguration));
}
this.contextCache.put(mergedContextConfiguration, context);
}
catch (Exception ex) {
throw new IllegalStateException("Failed to load ApplicationContext", ex);
}
}
else {
if (logger.isDebugEnabled()) {
logger.debug(String.format("Retrieved ApplicationContext from cache with key [%s]",
mergedContextConfiguration));
}
}
this.contextCache.logStatistics();
return context;
}
}
Obviously, when a Throwable other than Exception gets thrown during loadContextInternal() invocation, the application context doesn't get put in the cache.
Apparently, even when a single JUnit test is run getApplicationContext() method is called more than once with an expectation that second time it won't need to be loaded again since already cached.
If an Error is thrown during first load it gets buried and JUnit/Spring continues with the test until it calls getApplicationContext() for the second time. That's where it catches the exception and crashes because ehcache's doesn't expect CacheManager to be initialized more than once.

What is the proper way to have a "singleton" in Spring 4?

I have a java file "DatabaseMan.java" that helps connect to a database and connects helper functions. How can I make it such that it is created once for the life of my spring application, and I can call of its methods "getAllRows" for example, in each of my other resource classes?
Should I be declaring a bean in my Application.java or using some sort of annotation on my "DatabaseMan" class to indicate that it is "injectable"/"resusable"?
I see the following Spring3 example:
http://www.mkyong.com/spring3/spring-3-javaconfig-example/
The issue is, do I have to include this within every single resource:
ApplicationContext context = new AnnotationConfigApplicationContext(AppConfig.class);
HelloWorld obj = (HelloWorld) context.getBean("helloBean");
obj.printHelloWorld("Spring3 Java Config");
Is there a better way to get to the "HelloWorld" with less code and more annotation in Spring 4?

Remember, the ApplicationContext is a container to manage all your beans and their inter-dependencies. It is the entry point to your application. Once you've set it up, all the managed objects are linked up and ready to go.
Is there a better way to get to the "HelloWorld" with less code and more annotation in Spring 4?
It depends where you want to get it. If you want to get it from outside the ApplicationContext, then you need to do what you did. If you want to get into another bean, just inject it and the ApplicationContext will do the rest.
#Component
class SomeOtherBean {
#Autowired
private HelloWorld helloWorldBean;
// do something with it
}

Spring container implementations

I keep on learning Spring and it is very difficult to figure out which implementation of ApplicationContext is intended for. I've standalone J2EE application and I don't interested in Web* or Portlet* implementations.
Can you provide me the brief list of possibilities (if isn't clear, see P.S. section of my question) and purposes of each implementation below:
ResourceAdapterApplicationContext
StaticApplicationContext
ClassPathXmlApplicationContext
FileSystemApplicationContext
P.S.
A don't ask you to provide me reference to the docs. For example:
ClassPathXmlApplicationContext Standalone XML application context,
taking the context definition files from the class path, interpreting
plain paths as class path resource names that include the package path
But from that definition its not clear that ClassPathXmlApplicationContext also implements AbstractRefreshableApplicationContext and can be used to change beans definition without stopping server.

I'm sorry you don't want references to the docs, but that's where all the information is.
StaticApplicationContext states
org.springframework.context.ApplicationContext implementation which
supports programmatic registration of beans and messages, rather than
reading bean definitions from external onfiguration sources. Mainly
useful for testing.
So you use it to register bean definitions directly
StaticApplicationContext context = new StaticApplicationContext();
context.registerBeanDefinition(beanName, beanDefinition);
This can be used in cases where your ApplicationContext needs to be dynamically changed. Note that you can pass a parent ApplicationContext to the StaticApplicationContext if you need both behaviors, ie. reading from XML/Java config and dynamically registering.
ClassPathXmlApplicationContext is one of the more common ApplicationContext implementations in my opinion. You simply point it to an XML (bean definition) resource on the classpath and it loads it up. The javadoc states
Useful for test harnesses as well as for application contexts embedded
within JARs.
You could therefore simply point to a resource on the classpath coming from a JAR and load that. It's simply enough to setup tests environments this way.
ClassPathXmlApplicationContext context = new ClassPathXmlApplicationContext("some-context.xml");
// boom you're ready to go
Note that Spring's JUnit support classes offer other (better) ways to setup testing environment.
But from that definition its not clear that
ClassPathXmlApplicationContext also implements
AbstractRefreshableApplicationContext and can be used to change beans
definition without stopping server.
That's what the javadoc is for.
FileSystemXmlApplicationContext is similar to the ClasspathXmlApplicationContext above, but it takes the configuration files from the file system instead of reading resources from the classpath.
ResourceAdapterApplicationContext states
org.springframework.context.ApplicationContext implementation for a
JCA ResourceAdapter. Needs to be initialized with the JCA
javax.resource.spi.BootstrapContext, passing it on to Spring-managed
beans that implement BootstrapContextAware.
I haven't worked with this one at all and I don't know where Resource Adapters are useful, but here are some more docs.

Just to add couple things to #Solitirios answer:
You forgot to mention several more context:
GenericApplicationContext
GenericXmlApplicationContext
AnnotationConfigApplicationContext
GenericWebApplicationContext
StaticWebApplicationContext
And many others.
In general, GenericApplicationContext is almost the same as StaticApplicationContext, the only difference between them in MessageSource support in StaticApplicationContext. Purpose for both of these classes is for small tests with tiny application context with couple beans.
GenericWebApplicationContext and StaticWebApplicationContext are also quite similar to each other, and typically they are used for emulation of Servlet container, e.g. tests or non-Servlet environment.
F.e. you can use something like this in your code (f.e. tests):
//create parent context
ApplicationContext xmlContext = new GenericXmlApplicationContext("classpath:/spring-*.xml");
//create mock servlet context
MockServletContext mockServletContext = new MockServletContext();
//create web context
GenericWebApplicationContext webContext = new GenericWebApplicationContext(mockServletContext);
//set attribute
mockServletContext.setAttribute(GenericWebApplicationContext.ROOT_WEB_APPLICATION_CONTEXT_ATTRIBUTE, webContext);
//set parent context
webContext.setParent(xmlContext);
//refresh context
webContext.refresh();
But there are couple contexts classes, which are worthy of attention. And considering your pre-requisites, I would choose one of them.
GenericXmlApplicationContext is very good alternative of ClassPathXmlApplicationContext and FileSystemXmlApplicationContext. Consider this example:
ApplicationContext context = new GenericXmlApplicationContext("classpath:some-context.xml");
is equivalent to
ApplicationContext context = new ClassPathXmlApplicationContext("some-context.xml");
or
ApplicationContext context = new GenericXmlApplicationContext("some-context.xml");
is equivalent to
ApplicationContext context = new FileSystemXmlApplicationContext("some-context.xml");
So GenericXmlApplicationContext looks more flexible.
AnnotationConfigApplicationContext is a context holder, if you don't want to keep your beans in XML-file.
//context creation
ApplicationContext context = new AnnotationConfigApplicationContext(MyConfig.class);
//context class
#Configuration
#ComponentScan("com.examples.services")
public class AppConfig {
#Bean
public DataSources dataSource() {
DataSource ds = new BasicDataSource();
//... init ds
return ds;
}
}
More information you can find here.

ApplicationContextInitializer in a non-web Spring Context?

I've created an ApplicationContextInitializer implementation to load properties from a custome source (ZooKeeper) and add them to the ApplicationContext's property sources list.
All the documentation I can find relates to Spring web-apps, but I want to use this in a standalone message-consuming application.
Is the right approach to instantiate my implementation, create the context, then pass the context to my implementation 'manually'? Or am I missing some automatic feature fo the framework that will apply my initializer to my context?

I have found it simple enough to implement the SpringMVC's strategy for Initializing a context by initializing with a blank context. In normal application contexts, there is nothing which uses an ApplicationContextInitializer, thus you must execute it on your own.
No problem, though since within a normal J2SE application given you have ownership of the context loader block, you will have access to every stage of the lifecycle.
// Create context, but dont initialize with configuration by calling
// the empty constructor. Instead, initialize it with the Context Initializer.
AnnotationConfigApplicationContext ctx = new AnnotationConfigApplicationContext();
MyAppContextInitializer initializer = new MyAppContextInitializer();
initializer.initialize( ctx );
// Now register with your standard context
ctx.register( com.my.classpath.StackOverflowConfiguration.class );
ctx.refresh()
// Get Beans as normal (e.g. Spring Batch)
JobLauncher launcher = context.getBean(JobLauncher.class);
I hope this helps!

If I understand the problem correctly, you can find the solution in Spring documentation Section 4. The IoC container
An example on how to start your app is here - 4.2.2 Instantiating a container
Also have a look at 5.7 Application contexts and Resource paths

Not sure about other versions, but in Spring 4:
AbstractApplicationContext ctx = new AnnotationConfigApplicationContext(yourConfig.class);