I am wanting to stream some data out the database after a Spring Batch Tasklet step
public class TaskletUpdateAll implements Tasklet {
#Autowired
private AdRepository adRepository;
#Override
public RepeatStatus execute(StepContribution sc, ChunkContext cc) {
return RepeatStatus.FINISHED;
}
#AfterStep
#Transactional(readOnly = true)
public ExitStatus afterStepMethod(StepExecution stepExecution) {
try (Stream<Pair<String, String>> adStream = adRepository.streamAll();) {
adStream.forEach((Pair ad) -> {
// Process ad
});
}
return ExitStatus.COMPLETED;
}
}
Despite the fact the method that calls the stream is annotated with #Transactional(readOnly = true) it still returns the following error message:
Caused by: org.springframework.dao.InvalidDataAccessApiUsageException:
You're trying to execute a streaming query method without a
surrounding transaction that keeps the connection open so that the
Stream can actually be consumed. Make sure the code consuming the
stream uses #Transactional or any other way of declaring a (read-only)
transaction.
The AdRepository class follows the advice from here so the streamAll method looks like this:
#Repository
public interface AdRepository extends JpaRepository<Ad, Long> {
#QueryHints(value = #QueryHint(name = "HINT_FETCH_SIZE", value = "" + Integer.MIN_VALUE))
#Query(value = "SELECT href, provider FROM ad", nativeQuery = true)
Stream<Pair<String, String>> streamAll();
...
}
I have also made sure to include the #EnableTransactionManagement annotation in the configuration class for the Job as told to by the documentation.
...
#Configuration
#EnableTransactionManagement
public class MainTaskletsConfig {
...
Any idea if what I am doing is even possible and if it is what I am doing wrong?
Related
I am doing a work on arango db. Dose arangodb-spring-boot-starter has the transition and rollback support
I have tried #Transition annotation in the custom repo layer. added a error by custom error, the service has a functionality to create multiple document. I was expecting the rollback which is not happened.
This is the arango repository code.
public interface RelationRepository extends ArangoRepository<Relation, String> {
#Transactional
#Query("insert { _from: #from, _to: #to } into #collection return NEW")
Set<Relation> createEdge(#Param("from") String from,#Param("to") String to;}
This is the code snippet for the service
#Service
public class RelationService {
#Autowired
private RelationRepository relationRepository;
private final Logger log = LoggerFactory.getLogger(RelationService.class);
#Transactional(rollbackFor = SQLException.class)
public HashMap<String,String> demoRelation() {
relationRepository.createEdge("vertex1/121286","vertex2/167744","relation",
Instant.now().toEpochMilli(),Long.MAX_VALUE);
if(true)
throw new SQLException("custom exception to check rollback");
return null;
}
}
I was expecting the rollback, instead it is creating records
I need some help to find solution for my problem. I created 1 utility class and inject here some CrudRepositories. But repositoryies doesn't works fine here. They returns NullPointerException (Repositories works fine only in controllers).
Here is error and some code.
Error image
Once again, I note that such errors do not appear in the controller.
#Repository
public interface EventRepository extends CrudRepository<Event, Long> {
#Query(nativeQuery = true, value = "select * from events e order by e.id desc LIMIT 5")
List<Event> getEventsWithLimit();
}
#Service
public class CachedObject extends TimerTask {
#Autowired
EventRepository eventRepository;
#Autowired
MatchRepository matchRepository;
#Autowired
PlayerRepository playerRepository;
#Autowired
ImageRepository imageRepository;
List<Rank> ranking;
List<Image> image;
//Last 10 next Matches
List<Match> nextMatches;
//Last 10 results
List<Match> results;
List<PlayerOfTheWeek> playerOfTheWeek;
//Last 5 event
List<Event> events;
#Override
public void run() {
try{
refreshCache();
}
catch (Exception e){
e.printStackTrace();
}
}
public void refreshCache() throws Exception{
events = eventRepository.getEventsWithLimit();
image = imageRepository.getRandomImage();
results = matchRepository.getLastResult();
nextMatches = matchRepository.getLastMatches();
ranking = makeRanking();
...
}
...
}
Can you give me some tips for find solution guys?((
If you want to use solution TimerTask i think u need to create a constructor with needed autowired beans. But better solution in spring is to use #Scheduler annotation to periodically execute needed method. (more about Schedulers in spring https://spring.io/guides/gs/scheduling-tasks/)
I am using spring data/jpa to perform some database operations. I have a while loop which runs and successfully inserts data as it runs, but i also need an update operation to happen at the end of each run of a while loop. Here is basically what I have in a simple example. This is exactly the structure I am using.
Class doing all the operations:
#Component
public class MyClassImpl implements MyClass {
#Autowired
MyOtherClass myOtherClass;
#Override
public void run() {
while (expression) {
// get some data into and entity object
myOtherClass.insertMethod(entity);
myOtherClass.updateMethod(entityId);
}
}
}
my other class:
#Component
public class MyOtherClassImpl implements MyOtherClass {
#Override
JpaClass jpaClass;
#Override
#Transactional(propagation = Propagation.REQUIRES_NEW)
public void insertMethod(EntityObject entity) {
jpaClass.save(entity);
}
#Override
#Transactional(propagation = Propagation.REQUIRES_NEW)
public void updateMethod(String entityId) {
EntityObject entity = jpaClass.findById(entityId);
//change something on the entity
jpaClass.save(entity);
}
}
entity object:
public interface JpaClass extends JpaRepository<EntityObject, Long> {
EntityObject findById(String entityId);
}
the problem I am having is that the insert works just fine, but within the while loop I cannot get any updates to work like i have them. I have tried moving the logic around and putting the findById logic in a different method but cannot get it working. I am trying to update 1 row in a table which handles 1 value I then need to reference in the next run of the while loop.
so it goes:
get value
operate using value
update value
repeat
I set up the database config using spring #Configuration on a class which works fine for all transactions, for reference it is essentially set up like this:
#Configuration
#EnableTransactionManagement
#PropertySource(value = { "classpath:/${app.execution.environment}/application.properties" })
#EnableJpaRepositories(basePackages = "com.example", entityManagerFactoryRef = "mysqlEntityManager", transactionManagerRef = "mysqlTransactionManager")
public class MysqlHibernateConfig {
// all the needed beans here
}
Just to confirm as well, i ran this logic without the while loop and the data does update as expected, so the problem is somewhere in the database transaction, but I am stuck on how to resolve it.
This problem is caused by cache.
You can try like this.
in service class,
#Autowired
private EntityManager entityManager;
entityManager.clear();
in application.properties, you should set
spring.jpa.open-in-view = false
I am using spring MVC. From my controller, I am calling jobLauncher and in jobLauncher I am passing job parameters like below and I'm using annotations to enable configuration as below:
#Configuration
#EnableBatchProcessing
public class BatchConfiguration {
// read, write ,process and invoke job
}
JobParameters jobParameters = new JobParametersBuilder().addString("fileName", "xxxx.txt").toJobParameters();
stasrtjob = jobLauncher.run(job, jobParameters);
and here is my itemprocessor
public class DataItemProcessor implements ItemProcessor<InputData, OutPutData> {
public OutPutData process(final InputData inputData) throws Exception {
// i want to get job Parameters here ????
}
}
1) Put a scope annotation on your data processor i.e.
#Scope(value = "step")
2) Make a class instance in your data processor and inject the job parameter value by using value annotation :
#Value("#{jobParameters['fileName']}")
private String fileName;
Your final Data processor class will look like:
#Scope(value = "step")
public class DataItemProcessor implements ItemProcessor<InputData, OutPutData> {
#Value("#{jobParameters['fileName']}")
private String fileName;
public OutPutData process(final InputData inputData) throws Exception {
// i want to get job Parameters here ????
System.out.println("Job parameter:"+fileName);
}
public void setFileName(String fileName) {
this.fileName = fileName;
}
}
In case your data processor is not initialized as a bean, put a #Component annotation on it:
#Component("dataItemProcessor")
#Scope(value = "step")
public class DataItemProcessor implements ItemProcessor<InputData, OutPutData> {
A better solution (in my opinion) that avoids using Spring's hacky expression language (SpEL) is to autowire the StepExecution context into your processor using #BeforeStep.
In your processor, add something like:
#BeforeStep
public void beforeStep(final StepExecution stepExecution) {
JobParameters jobParameters = stepExecution.getJobParameters();
// Do stuff with job parameters, e.g. set class-scoped variables, etc.
}
The #BeforeStep annotation
Marks a method to be called before a Step is executed, which comes
after a StepExecution is created and persisted, but before the first
item is read.
I have written the in the process itself, rather then creating separate file using the lambda expression.
#Bean
#StepScope
public ItemProcessor<SampleTable, SampleTable> processor(#Value("#{jobParameters['eventName']}") String eventName) {
//return new RandomNumberProcessor();
return item -> {
SampleTable dataSample = new SampleTable();
if(data.contains(item)) {
return null;
}
else {
dataSample.setMobileNo(item.getMobileNo());
dataSample.setEventId(eventName);
return dataSample;
}
};
}
I have two services, like this (simplified code):
#Service
public class OuterService {
#Autowired
InnerService innerService;
#Transactional
public void doSomething() {
List<SomeEntity> list = entityRepo.findByWhatever(...);
for(SomeEntity listElement : list) {
innerService.processEntity(listElement);
}
}
}
#Service
public class InnerService {
#Transactional(propagation = Propagation.REQUIRES_NEW)
public void processEntity(Entity entity) {
// ...
StatusElement status = new StatusElement(...);
statusElementRepo.save(status);
}
}
The constructed StatusElement is now inserted by exiting InnerService.processEntity() and inserted again by exiting OuterService.doSomething().
If I change the #Transactional annotation of OuterService.doSomething() to #Transactional(readOnly = true), it is inserted just once.
Is it a problem with MySql (because it may not support nested transactions), do I need a special transaction manager, or is there something wrong with my code? TIA!
I solved it by using programmatically transactions using the PlatformTransactionManager.
see: http://docs.spring.io/spring/docs/current/spring-framework-reference/html/transaction.html#transaction-programmatic-ptm