When combining springboot (2.4.5) with springboot-starter-quartz dependency, I ran into memory leak problem when I try to force the Job.execute(JobExecutionContext context) to be #Transactional as I need to do some lazy loading in there, although the example I demonstrate the problem with is not doing any loading, but still creates the leak.
I followed the guidance at Baeldung spring quarz schedule, where the result of my implementation is the following class, which I create a trigger for & schedule it somewhere else in the application (irelevant where).
#Slf4j
#Component
public class FakeQuartzJob extends Job {
#Override
#Transactional
public void execute(JobExecutionContext context) {
log.info("I was called");
}
public static Trigger buildJobTrigger(JobDetail jobDetail) {
return TriggerBuilder.newTrigger()
.forJob(jobDetail)
.withIdentity(UUID.randomUUID().toString(), "fake-job-trigger")
.startAt(Date.from(Instant.now().plusSeconds(1)))
.withSchedule(SimpleScheduleBuilder.simpleSchedule()
.withMisfireHandlingInstructionFireNow())
.build();
}
public static JobDetail buildJobDetail() {
JobDataMap jobDataMap = new JobDataMap();
jobDataMap.put("someRandomData", new FakeDataClass());
return JobBuilder.newJob(FakeQuartzJob.class)
.withIdentity(UUID.randomUUID().toString(), "fake-job-detail")
.usingJobData(jobDataMap)
.build();
}
}
This results in creating transactional proxies for the FakeQuartzJob.execute method (expected with transactional) but when doing some heap investigation I found out it results in publishing this proxy into spring containers bean registry but it's never removed from the registry, see
These proxies are never garbage collected, which leads into heap memory being constantly eaten by this until it reaches the limit and application goes oom.
I tested different behaviors/combinations of this spring/quartz cooperation regarding this problem and
Theres no problem with #Autowire-ing services/repos beans into the job
Theres no problem #Autowiring and calling Job.execute if there's no #Transactional annotation
There is a problem using #Transactional in Job.execute(JobExecutionContext context)
What I expect and desperately need, is to have possibility to do hibernate lazy loading of entity relationships when executing quartz job, but for that I need to have a transaction support via #Transactional.
I do understand that I am scheduling this job, while it is annotated with #Component, so when the class is constructed, it becomes spring managed, but I do that to allow for autowiring.
What I absolutely do not understand is why spring does not free up the proxy classes and keeps holding references to them (class proxy is referenced by method proxy) see
Spring&Quartz question: Is there a way to allow for both autowiring and having possibility of doing #Transactional call in when executing the job?
Technical question: What could be the reason this is happening?
Related
I'm trying to use db-scheduler with Micronaut. Therefore, I created a #Singleton service where I inject the actual DataSource which is of type TransactionAwareDataSource. I then call a certain method to setup the scheduler which is something like:
#Transactional
public void createJob() {
RecurringTask<Void> hourlyTask = Tasks.recurring("my-hourly-task", FixedDelay.ofHours(1))
.execute((inst, ctx) -> {
System.out.println("Executed!");
});
final Scheduler scheduler = Scheduler
.create(dataSource)
.startTasks(hourlyTask)
.threads(5)
.build();
scheduler.start();
}
which, at "create" throws this exception:
io.micronaut.transaction.exceptions.NoTransactionException: No current transaction present. Consider declaring #Transactional on the surrounding method
at io.micronaut.transaction.jdbc.TransactionalConnectionInterceptor.intercept(TransactionalConnectionInterceptor.java:65)
at io.micronaut.aop.chain.MethodInterceptorChain.proceed(MethodInterceptorChain.java:96)
at io.micronaut.transaction.jdbc.TransactionalConnection$Intercepted.getMetaData(Unknown Source)
at com.github.kagkarlsson.scheduler.jdbc.AutodetectJdbcCustomization.<init>(AutodetectJdbcCustomization.java:40)
at com.github.kagkarlsson.scheduler.SchedulerBuilder.lambda$build$0(SchedulerBuilder.java:190)
at java.base/java.util.Optional.orElseGet(Optional.java:369)
Everywhere else in my app everything is working like it should, means, I can read and write to the DB via the repositories and #Transactional is working as well.
I'm not 100% sure where the problem is, but I guess it does have to do with placing the annotation. Which - in this case - is nothing I can really change. On the other hand, if I create the datasource manually, effectively bypassing micronaut, it's working.
BTW: the exception comes up within db-scheduler where the first call to the DB is made (c.getMetaData().getDatabaseProductName()).
Micronaut-Version: 2.3.4, Micronaut-Data: 2.2.4, everything setup properly.
Do you guys have any ideas how to solve this problem? Or is it even a bug?
Thanks!
So the problem is that Micronaut Data wraps the DataSource into a TransactionAwareDataSource, as you mentioned. Your library db-scheduler or mine JobRunr picks it up, and operates without the required annotations. The solution is to unwrap it before giving it to the db-scheduler or JobRunr:
Kotlin:
val unwrappedDataSource = (dataSource as DelegatingDataSource).targetDataSource
Java:
DataSource unwrappedDataSource = ((DelegatingDataSource) dataSource).targetDataSource
I am in a service, with a scheduled task, and I want to get an object from the database. It has EAGER associations, so the find method should get it totally.
#Service
public class CustomTask {
#Autowired
CustomRepository customRepository;
#Scheduled(fixedRate = 1000)
public void action() {
customRepository.find(1);
}
}
But here it doesn't work. The associations are null.
While inside a Spring Boot Controller, the repository method works perfectly.
Do you know I can get my whole object in this Scheduled method of a Service?
The scheduled task is called at the beginning of the app, at a moment where the environment may not be totally initialized.
With an initial delay to the task, I can access my whole object :
#Scheduled(initialDelay=10000, fixedRate = 1000)
NB : It is more a workaround than a fix.
My application loads a list of entities that should be processed. This happens in a class that uses a scheduler
#Component
class TaskScheduler {
#Autowired
private TaskRepository taskRepository;
#Autowired
private HandlingService handlingService;
#Scheduled(fixedRate = 15000)
#Transactional
public void triggerTransactionStatusChangeHandling() {
taskRepository.findByStatus(Status.OPEN).stream()
.forEach(handlingService::handle);
}
}
In my HandlingService processes each task in issolation using REQUIRES_NEW for propagation level.
#Component
class HandlingService {
#Transactional(propagation = Propagation.REQUIRES_NEW)
public void handle(Task task) {
try {
processTask(task); // here the actual processing would take place
task.setStatus(Status.PROCCESED);
} catch (RuntimeException e) {
task.setStatus(Status.ERROR);
}
}
}
The code works only because i started the parent transaction on TaskScheduler class. If i remove the #Transactional annotation the entities are not managed anymore and the update to the task entity is not propagated to the db.I don't find it natural to make the scheduled method transactional.
From what i see i have two options:
1. Keep code as it is today.
Maybe it`s just me and this is a correct aproach.
This varianthas the least trips to the database.
2. Remove the #Transactional annotation from the Scheduler, pass the id of the task and reload the task entity in the HandlingService.
#Component
class HandlingService {
#Autowired
private TaskRepository taskRepository;
#Transactional(propagation = Propagation.REQUIRES_NEW)
public void handle(Long taskId) {
Task task = taskRepository.findOne(taskId);
try {
processTask(task); // here the actual processing would take place
task.setStatus(Status.PROCCESED);
} catch (RuntimeException e) {
task.setStatus(Status.ERROR);
}
}
}
Has more trips to the database (one extra query/element)
Can be executed using #Async
Can you please offer your opinion on which is the correct way of tackling this kind of problems, maybe with another method that i didn't know about?
If your intention is to process each task in a separate transaction, then your first approach actually does not work because everything is committed at the end of the scheduler transaction.
The reason for that is that in the nested transactions Task instances are basically detached entities (Sessions started in the nested transactions are not aware of those instances). At the end of the scheduler transaction Hibernate performs dirty check on the managed instances and synchronizes changes with the database.
This approach is also very risky, because there may be troubles if you try to access an uninitialized proxy on a Task instance in the nested transaction. And there may be troubles if you change the Task object graph in the nested transaction by adding to it some other entity instance loaded in the nested transaction (because that instance will now be detached when the control returns to the scheduler transaction).
On the other hand, your second approach is correct and straightforward and helps avoid all of the above pitfalls. Only, I would read the ids and commit the transaction (there is no need to keep it suspended while the tasks are being processed). The easiest way to achieve it is to remove the Transactional annotation from the scheduler and make the repository method transactional (if it isn't transactional already).
If (and only if) the performance of the second approach is an issue, as you already mentioned you could go with asynchronous processing or even parallelize the processing to some degree. Also, you may want to take a look at extended sessions (conversations), maybe you could find it suitable for your use case.
The current code processes the task in the nested transaction, but updates the status of the task in the outer transaction (because the Task object is managed by the outer transaction). Because these are different transactions, it is possible that one succeeds while the other fails, leaving the database in an inconsistent state. In particular, with this code, completed tasks remain in status open if processing another task throws an exception, or the server is restarted before all tasks have been processed.
As your example shows, passing managed entities to another transaction makes it ambiguous which transaction should update these entities, and is therefore best avoided. Instead, you should be passing ids (or detached entities), and avoid unnecessary nesting of transactions.
Assuming that processTask(task); is a method in the HandlingService class (same as handle(task) method), then removing #Transactional in HandlingService won't work because of the natural behavior of Spring's dynamic proxy.
Quoting from spring.io forum:
When Spring loads your TestService it wrap it with a proxy. If you call a method of the TestService outside of the TestService, the proxy will be invoke instead and your transaction will be managed correctly. However if you call your transactional method in a method in the same object, you will not invoke the proxy but directly the target of the proxy and you won't execute the code wrapped around your service to manage transaction.
This is one of SO thread about this topic, and Here are some articles about this:
http://tutorials.jenkov.com/java-reflection/dynamic-proxies.html
http://tutorials.jenkov.com/java-persistence/advanced-connection-and-transaction-demarcation-and-propagation.html
http://blog.jhades.org/how-does-spring-transactional-really-work/
If you really don't like adding #Transaction annotation in your #Scheduled method, you could get transaction from EntityManager and manage transaction programmatically, for example:
UserTransaction tx = entityManager.getTransaction();
try {
processTask(task);
task.setStatus(Status.PROCCESED);
tx.commit();
} catch (Exception e) {
tx.rollback();
}
But I doubt that you will take this way (well, I wont). In the end,
Can you please offer your opinion on which is the correct way of tackling this kind of problems
There's no correct way in this case. My personal opinion is, annotation (for example, #Transactional) is just a marker, and you need a annotation processor (spring, in this case) to make #Transactional work. Annotation will have no impact at all without its processor.
I Will more worry about, for example, why I have processTask(task) and task.setStatus(Status.PROCESSED); live outside of processTask(task) if it looks like does the same thing, etc.
HTH.
I have a very weird situation, that has happened in several systems already. I am using Spring Boot and AspectJ CTW to #Autowired dependencies in some entities (instanciated outside the container).
The class that receives dependencies (an abstract entity) some times receive the dependency without applying the profile (configured by #ActiveProfile in my test class). It is not deterministic, since by changing how tests are executed different outputs can happen. To illustrate the situation with code:
The entity
#Configurable
public class AbstractMongoDocument<T> implements Persistable<T> {
#Transient
private transient MongoTemplate mongoTemplate;
//entity stuff
}
One of the failing tests
#RunWith(SpringJUnit4ClassRunner.class)
#SpringApplicationConfiguration(classes = LOVApplication.class)
#ActiveProfiles("local-test")
public class MyCrazyIntegrationTest {
#Test
public void filterByFieldsFullMatchShouldReturnResult() throws Exception {
//Given
Location l1 = new Location("name","code",new GeoJsonPoint(11,10));
l1.save(); //Hence the need of autowiring there.
//When: whatever
//Then: Some assertions
}
}
There are some facts that I find very disturbing here:
The dependency is always injected, but some times it apparently comes from an AppCtx with the default profile.
If it fails for 1 test in 1 class, it behaves the same for all the tests in that particular class.
It may or may not happen depending on how you execute that class (At the moment it fails only if I run all the tests, but succeed if I run that class in isolation, it also behaves differently in maven).
If I debug it, I can see that the dependency injected didn't get the proper profile. (I discovered that by injecting ApplicationContext and surprisingly discovering that it was a different object than the one I received in my tests).
What worries me the most, is that now I am not sure if this situation could also happen for non-test environments with for example a Production profile, which would imply a catastrophe.
I have tried to look for open bugs in Jira and I found nothing, so I don't discard I am misconfiguring something. Any help or ideas would be very much appreciated.
The behavior you are experiencing typically should not happen in a production deployment; however, it is a known issue with integration test suites that load multiple ApplicationContexts utilizing #Configurable and AspectJ load-time weaving.
For details, see the following issues in Spring's issue tracker:
https://jira.spring.io/browse/SPR-6353
https://jira.spring.io/browse/SPR-6121
I would like to have a (singleton) bean initialized only after it is actually used (instead of when it is only autowired). Let say I have a Client that I want to initialize only when I want to call any of its methods
#Component
#Lazy(true)
public class Client {
#PostConstruct
void init() {}
void action(){}
}
And I have a Service class that sometimes use it (and maybe sometimes not).
#Service
public class Service {
#Autowired
Client client;
void action1WithClient(){}
void action2WithClient(){}
void actionWithoutClient(){}
}
As it is now, the client is initialized right on the application startup without actually being used due to #Autowired and the fact that Service is eagerly loaded.
Currently only solution that comes to my mind is to do kind of double-checked locking with explicitly asking for Client bean from spring application context when someone tries to use (ie. without #Autowired) or (maybe even better) to do "manual" lazy loading inside the Client.
Question: Is there a "spring" way to postpone the initialization of client until any of its method is actually called (e.g. something like lazy loading is working for hibernate collections)?
I am using Spring 4.
Ah, ok, I should read the javadoc more properly... Solution seems to add anotation "each" autowired dependeny:
#Autowired
#Lazy
Client client;
Anyway - if anybody knows if it is possible to omit such declarations, since it might be error prone - one can easily forget to use #Lazy at each injection point.
Edit:
The easiest way is #ComponentScan(lazyInit = true, basePackages=...).
Previous answer:
There is http://docs.spring.io/spring/docs/3.0.x/javadoc-api/org/springframework/aop/target/LazyInitTargetSource.html
A bean wrapped with LazyInitTargetSource will not be created until the first actual usage.
How to nicely wrap most of your beans with this wrapper? One possible way is to create your own BeanFactoryPostProcessor...