I'm using Spring Batch with #EnableBatchProcessing(modular = true)
The problem is that in this mode you have to explicitly declare which beans to initialize (i.e which classes Spring needs to scan)
Here's an exmaple:
#Configuration
#EnableBatchProcessing(modular = true)
public class ModularJobsConfig {
#Autowired
private AutomaticJobRegistrar registrar;
#PostConstruct
public void initialize() {
registrar.addApplicationContextFactory(new GenericApplicationContextFactory(
SomeJobConfig.class,
SomeJobTasklet.class,
SomeClassToDefineTaskExecutor.class
SomeClassToRunTheJob.class));
}
}
I can imagine that by the time I'll have several jobs, this configuration class will be bloated. How can I automate this?
It is worth mentioning that each job has its own package (e.g com.example.jobs.<job_name>) + they are defined in different maven-modules but I think it's irrelevant.
Further Clarification
I have a core module which contains the configuration above. Every job is defined in a separate maven module and it's been registered as a maven dependency in core.
Mainly for preventing naming clash, I'm using #EnableBatchProcessing(modular = true) and I'm registering the jobs with AutomaticJobRegistrar as you can see in the example code above.
Ideally, I'd like Spring to scan the maven dependency and to do it for me (i.e. defining GenericApplicationContextFactory)
Currently, it's cumbersome to add manually each and every class (in the example above: SomeJobConfig.class, SomeJobTasklet.class etc)
As a counter example, if I didn't use modular=true I could let Spring Batch to load all beans on its own, but then I'd have to make sure methods names are unique across all the modules.
Related
Team, I am working on an Spring boot application which uses lot of other third party libraries.
All those libraries are built on top of spring core.
The common issue that usually comes is DuplicateBeanException.
Let's say if the bean-name is same from different libraries, spring throws error while trying to start the server. As I do not have control over third party jars. I have to rename the beans in my application: below is the code snippet I wrote to handle this scenario. But this is not an efficient solution as in future there can be again some duplicate beans. How can I solve it in a generic way so that when ever any duplicate beans come, my application can handle and initialize those beans.
My code to handle duplicate bean :
#Configuration
public class ExternalBeanConfiguration {
#Bean(employeeLib1)
public Employee getEmployee() {
return new Employee();
}
#Bean(employeeLib2)
public com.another.library.Employee getEmployee() {
return new com.another.library.Employee();
}
}
When you configure where to scan the beans from the external libraries by #ComponentScan , you can specify a BeanNameGenerator for how the name of the detected beans are defined.
#Configuration
#ComponentScan(basePackages = { "com.foo.lib1" , "com.foo.lib2"}, nameGenerator = FullyQualifiedAnnotationBeanNameGenerator.class)
public class ExternalBeanConfiguration {
}
The FullyQualifiedAnnotationBeanNameGenerator is exactly for solving your problem which is mentioned in the javadoc as :
Favor this bean naming strategy over {#code
AnnotationBeanNameGenerator} if you run into naming conflicts due to
multiple autodetected components having the same non-qualified class
name (i.e., classes with identical names but residing in different
packages)
It will name the bean as the fully qualified class name as the default bean name such that even if different packages has a same class name , it will still have different bean name.
I have an application with some externalized configuration in the form of properties. I would like the application to react to a change of such properties without a restart or full context refresh.
I am not clear what my options are.
Artificial example: the application implements a service that receives requests and decides whether to queue or reject them. The maximum size of the queue is a property.
final int queueMaxSize = queueProperties.getMaxSize();
if (queue.size() >= queueMaxSize) { //reject }
Where QueueProperties is a class annotated with #ConfigurationProperties.
#ConfigurationProperties(prefix = "myapp.limits.queue")
#Getter
#Setter
public class QueueProperties {
public int maxSize = 10;
}
This works as far as allowing me to control behavior via system properties, profiles, etc.
However, I would like to be able to change this value without releasing/deploying/restarting/refreshing the application.
My application already uses Archaius.
If I push a new value for this property using our internal infrastructure, i can see the application Spring Environment does receive the new value.
(e.g., /admin/env reflects the value and changes dynamically).
The part I'm not clear on is: how to make my service react to the change of value in the environment?
I found two ways, but they seem hairy, I wonder if there are better options. I expected this to be a common problem with a first class solution in the Spring ecosystem.
Hacky solution #1:
#Bean
#Scope("prototype")
#ConfigurationProperties(prefix = "myapp.limits.queue")
QueueProperties queueProperties() {
return new QueueProperties();
}
And inject this into the service using the properties as Provider<QueueProperties> and use it as queuePropertiesProvider.get().getMaxSize().
This works but has a few side-effects I'm not a fan of:
ConfigurationProperties annotation moved from the class to the bean definition
A new QueueProperties object is created and bound to values for every request coming in
Provider might throw on get()
Invalid values are not detected until the first request comes in
Hacky solution #2:
Don't annotate my properties class with ConfigurationProperties, inject the Spring environment at construction time. Implement the getters as such:
int getMaxSize() {
return environment.getProperty("myapp.limits.queue", 10);
}
This also works ok in terms of behavior. However
- This is not annotated as property (unlike the rest of the properties classes in this large project, makes it harder to find)
- This class does not show up in /admin/configprops
Hacky solution #3:
Schedule a recurring task that uses Environment to update my singleton QueueProperties bean.
Any further ideas/suggestions/pointers?
Is there a canonical/recommended way to do this that does not have the shortcoming of my solutions above?
I am writing a library which will be used by spring-boot projects. I'd like to inject into the boot projects' SpringEnvironment a property source that I take from the Internet.
I tried the following
#Configuration
public class MyCustomConfiguration {
#Bean
BeanDefinedAbove above() { /* do some work */ }
#Bean
#ConditionalOnBean(BeanDefinedAbove.class)
SmartInitializingSingleton propertySourceSetting(ConfigurableEnvironment env, BeanDefinedAbove bean) {
return () -> {
PropertySource source = bean.getPropertySourceDownloadedFromTheInternet();
env.getPropertySources().addFirst(source);
}
}
}
In my clients' projects when I debug this code what happens is either one of the two:
above() is called
user's #Service or #Controller are called
propertySourceSetting(...) is called
OR
user's #Service or #Controller are called
above() is called
propertySourceSetting(...) is called
Depending whether or not my client's depend on my BeanDefinedAbove bean, which is normal as the #Service is depdent on the bean created in above().
I have also added the FQDN of my class to the EnableAutoConfiguration in the META-INF/spring.factories.
So how to ensure that the logic in propertySourceSetting(..) is called before users' #Service and #Controller
I'll provide you with three options.
Option 1: (THIS IS A BAD APPROACH, BUT A QUICK WORKAROUND)
Add #Lazy(false) annotation to both Beans. Spring will eagerly create those two beans, which they will probably be created before the other ones.
Why this is bad?
This does not ensure order. Spring decides the creation order based on dependencies and some other conditions. This is why it will "probably" work :)
Option 2: Create a main class to bootstrap Spring Boot Initialization (the old way of starting spring boot).
public static void main(String[] args) throws Exception {
SpringApplication application = new SpringApplication(MyApplication.class);
// ... add property source here before start
application.run(args)
}
You also need to specify main class in the manifest for Spring Boot like this: https://www.baeldung.com/spring-boot-main-class
In that main-class you would add your propertysource, kinda like this:
SomeClassThatRetrievesProperties propRetriever = new SomeClassThatRetrievesProperties ();
Map<String,String> properties = propRetriever.getAllPropertiesAsMap();
application.setDefaultProperties(properties);
Option 3: Create a CustomApplicationContext by extending WebApplicationContext and overriding getSpecificConfigurations() method.
This way you will have full control but we aware that you could break some important stuff.
I have an application that uses Spring Batch to define a preset number of jobs, which are currently all defined in the XML.
We add more jobs over time which requires updating the XML, however these jobs are always based on the same parent and can easily be predetermined using a simple SQL query.
So I've been trying to switch to use some combination of XML configuration and Java-based configuration but am quickly getting confused.
Even though we have many jobs, each job definition falls into essentially one of two categories. All of the jobs inherit from one or the other parent job and are effectively identical, besides having different names. The job name is used in the process to select different data from the database.
I've come up with some code much like the following but have run into problems getting it to work.
Full disclaimer that I'm also not entirely sure I'm going about this in the right way. More on that in a second; first, the code:
#Configuration
#EnableBatchProcessing
public class DynamicJobConfigurer extends DefaultBatchConfigurer implements InitializingBean {
#Autowired
private JobBuilderFactory jobBuilderFactory;
#Autowired
private JobRegistry jobRegistry;
#Autowired
private DataSource dataSource;
#Autowired
private CustomJobDefinitionService customJobDefinitionService;
private Flow injectedFlow1;
private Flow injectedFlow2;
public void setupJobs() throws DuplicateJobException {
List<JobDefinition> jobDefinitions = customJobDefinitionService.getAllJobDefinitions();
for (JobDefinition jobDefinition : jobDefinitions) {
Job job = null;
if (jobDefinition.getType() == 1) {
job = jobBuilderFactory.get(jobDefinition.getName())
.start(injectedFlow1).build()
.build();
} else if (jobDefinition.getType() == 2) {
job = jobBuilderFactory.get(jobDefinition.getName())
.start(injectedFlow2).build()
.build();
}
if (job != null) {
jobRegistry.register(new ReferenceJobFactory(job));
}
}
}
#Override
public void afterPropertiesSet() throws Exception {
setupJobs();
}
public void setInjectedFlow1(Flow injectedFlow1) {
this.injectedFlow1 = injectedFlow1;
}
public void setInjectedFlow2(Flow injectedFlow2) {
this.injectedFlow2 = injectedFlow2;
}
}
I have the flows that get injected defined in the XML, much like this:
<batch:flow id="injectedFlow1">
<batch:step id="InjectedFlow1.Step1" next="InjectedFlow1.Step2">
<batch:flow parent="InjectedFlow.Step1" />
</batch:step>
<batch:step id="InjectedFlow1.Step2">
<batch:flow parent="InjectedFlow.Step2" />
</batch:step>
</batch:flow>
So as you can see, I'm effectively kicking off the setupJobs() method (which is intended to dynamically create these job definitions) from the afterPropertiesSet() method of InitializingBean. I'm not sure that's right. It is running, but I'm not sure if there's a different entry point that's better intended for this purpose. Also I'm not sure what the point of the #Configuration annotation is to be honest.
The problem I'm currently running into is as soon as I call register() from JobRegistry, it throws the following IllegalStateException:
To use the default BatchConfigurer the context must contain no more than one DataSource, found 2.
Note: my project actually has two data sources defined. The first is the default dataSource bean which connects to the database that Spring Batch uses. The second data source is an external database, and this second one contains all the information I need to define my list of jobs. But the main one does use the default name "dataSource" so I'm not quite sure how else I can tell it to use that one.
First of all - I don't recommend using a combination of XML as well as Java Configuration. Use only one, preferably Java one as its not much of an effort to convert XML config to Java config. (Unless you have some very good reasons to do it - that you haven't explained)
I haven't used Spring Batch alone as I have always used it with Spring Boot and I have a project where I have defined multiple jobs and it always worked well for similar code that you have shown.
For your issue, there are some answers on SO like this OR this which are basically trying to say that you need to write your own BatchConfigurer and not rely on default one.
Now coming to solution using Spring Boot
With Spring Boot, You should try segregate job definitions and job executions.
You should first try to just define jobs and initialize Spring context without enabling jobs (spring.batch.job.enabled=false)
In your Spring Boot main method, when you start app with something like - SpringApplication.run(Application.class, args); ...you will get ApplicationContext ctx
Now you can get your relevant beans from this context & launch specif jobs by getting names from property or command line etc & using JobLauncher.run(...) method.
You can refer my this answer if willing to order job executions. You can also write job schedulers using Java.
Point being, you separate your job building / bean configurations & job execution concerns.
Challenge
Keeping multiple jobs in a single project can be challenging when you try to have different settings for each job as application.properties file is environment specific and not job specific i.e. spring boot properties will apply to all jobs.
In my particular case, the solution was to actually eliminate the #Configuration
and #EnableBatchProcessing annotations from my class above. Something about these caused it to try and use the DefaultBatchConfigurer which fails when you have more than one data source defined (even if you've identified them clearly with "dataSource" as the primary and some other name for the secondary).
The #Configuration class in particular wasn't necessary because all it really does is lets your class get auto-instantiated without having to define it as a bean in the app context. But since I was doing that anyway this one was superfluous.
One of the downsides of removing #EnableBatchProcessing was that I could no longer auto-wire the JobBuilderFactory bean. So instead I just had to do to create it:
JobRepositoryFactoryBean factory = new JobRepositoryFactoryBean();
factory.setDataSource(dataSource);
factory.setTransactionManager(transactionManager);
factory.afterPropertiesSet();
jobRepository = factory.getObject();
jobBuilderFactory = new JobBuilderFactory(jobRepository);
Then it seems I was on the right track already by using jobRegistry.register(...) to define my jobs. So essentially once I removed those annotations above everything started working. I'm going to mark Sabir's answer as the correct one however because it helped me out.
I've managed to avoid developing any xml files so far for my Spring Boot Maven project (apart from the pom) with them all being generated on compile and I was hoping to keep this way by defining my profiles within the run commands as specified here.
By simply using #ComponentScan my main class to enable the scanning of components and tagging my DAO as #Repository I have successfully managed to autowire my UserDAOmySQLImpl class (which inherits UserDAO).
#Autowired
private UserDAO userDAO;
Then looking forward to add a second DAO for when in Live where we use a db8 implementation I need the application to figure out which UserDAO implementation needs to be used. Profiles seemed to be the answer.
After some reading, it seemed that mostly I need to add in some sort of configuration classes and external xml to manage these profiles and components, though this seems messy and I am hoping unnecessary.
I have tagged by two implementations as so:
#Repository
#Profile("dev")
public class UserDAOmySQLImpl implements UserDAO {...
-
#Repository
#Profile("dev")
public class UserDAOdb8Impl implements UserDAO {...
And then set the active profile through the Maven goals as specified here in hope that this would be a nice clean solution so I could specify the active profile within the goal for our dev and live build scripts respectively.
My current goal for testing is:
spring-boot:run -Drun.profiles=dev
However, when building I receive an error that both beans are being picked up.
*No qualifying bean of type [com.***.studyplanner.integration.UserDAO] is defined: expected single matching bean but found 2: userDAOdb8Impl,userDAOmySQLImpl*
Questions
Is the profile set in the Maven Goal the one being checked against when using the #Profile tag?
Why is Spring picking up both implementations, when if the profile isn't being set properly surely neither implementation should be selected?
Any suggestions on a nice clean way to achieve what I'm looking for?
Bonus
I would really like to set the profile within the app, as it would be easy to simply check whether an environment file exists to decide which profile to use, though I'm struggling to replicate this (found within Spring Profiles Examples
public static void main(String[] args) {
AnnotationConfigApplicationContext context = new AnnotationConfigApplicationContext();
//Enable a "live" profile
context.getEnvironment().setActiveProfiles("live");
context.register(AppConfig.class);
context.refresh();
((ConfigurableApplicationContext) context).close();
}
in to the unusual main class in the application I am working on which I am hesitant to play around with.
public class StudyPlannerApplication {
...
public static void main(String[] args) {
SpringApplication.run(StudyPlannerApplication.class, args);
}
...
}
Any help would much appreciated.
Cheers,
Scott.
Silly me, proof reading the question I noticed that a bad copy & paste job meant that both DAO implementations had the #profile set to "Dev". After changing the db8 DAO to #Profile("live") the above works fine.
So to choose your repository based on profile is actually quite easy when using maven and spring boot.
1) Ensure your main class has the #ComponentScan annotation
#ComponentScan
public class StudyPlannerApplication {
2) Tag your components with the #Profile tags according to which profile you would like them sectected for
#Repository
#Profile("dev")
public class UserDAOdb8Impl implements UserDAO {...
-
#Repository
#Profile("live")
public class UserDAOdb8Impl implements UserDAO {...
3)
Send your profile in with your maven goal
spring-boot:run -Drun.profiles=dev
Hopefully this will help others, as I haven't seen a full example of using the autowiring profiles as I have done elsewhere on the web