While I have Java Batch jobs that read data, process it and store it in other places in the database, now I need a step to actually remove data from the database. All I need to run is a delete query via JPA.
The chunk based Reader/Processor/Writer pattern does not make sense here. But the Batchlet alternative is giving me a headache either. What did I do?
I created a Batchlet that gets invoked via CDI. At that moment it is easy to inject my JPA EntityManager. What is not easy is to run the update query. Code looks like this:
package ...;
import javax.batch.api.BatchProperty;
import javax.inject.Inject;
import javax.inject.Named;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
#Named("CleanerBatchlet")
public class CleanerBatchlet extends AbstractBatchlet {
public static final Logger log = LogManager.getLogger(CleanerBatchlet.class);
#PersistenceContext(unitName = "...")
private EntityManager entityManager;
#Inject
#BatchProperty(name = "technologyIds")
private String technologyIds;
private void clearQueue(long technologyId) {
//EntityManager entityManager = ...getEntityManager();
//entityManager.getTransaction().begin();
Query q = entityManager.createQuery("delete from Record r where r.technologyId=:technologyId");
q.setParameter("technologyId", technologyId);
int count = q.executeUpdate();
//entityManager.getTransaction().commit();
log.debug("Deleted {} entries from queue {}", count, technologyId);
//entityManager.close();
}
#Override
public String doProcess() throws Exception {
log.debug("doProcess()");
out.println("technologyIds=" + technologyIds);
log.info("technologyIds=" + technologyIds);
try {
String[] parts = technologyIds.split(",");
for (String part: parts) {
long technologyId = Long.parseLong(part);
clearQueue(technologyId);
}
} catch (NullPointerException | NumberFormatException e) {
throw new IllegalStateException("technologyIds must be set to a string of comma-separated numbers.", e);
}
return "COMPLETED";
}
}
As you can see some lines are commented out - these are the ones I am experimenting with.
So if I run the code as-is, I get an exception telling me that the update query requires a transaction. This is regardless of which of the two persistence units in my project I use (one is configured for JTA, the other is not).
javax.persistence.TransactionRequiredException: Executing an update/delete query
It also does not matter whether I uncomment the transaction handling code begin/commit. I still get the same error that a transaction is required to run the update query.
Even when I try to circumvent CDI and JTA completely by creating my own EntityManager via the Persistence API (and close it afterwards, respectively) I do get the very same exception.
So how can I run this delete query or other update queryies from within the batch job?
I'd suggest using plain jdbc to run this delete query, with either auto commit or manual transaction commit.
During the batchlet processing, the incoming transaction is suspended. So the entity manager does not have a transaction context.
Ultimately I made it work by following this tutorial: https://dzone.com/articles/resource-local-vs-jta-transaction-types-and-payara
and going for the Classic RESOURCE_LOCAL Application pattern.
It involves injecting the nonJTA EntityManagerFactory, using that to create the entitymanager and closing it after use. Of course the transaction has to be managed manually but after all now it works.
The essential excerpt of my code looke like this:
#PersistenceUnit(unitName = "...")
private EntityManagerFactory emf;
#Inject
#BatchProperty(name = "technologyIds")
private String technologyIds;
private void clearQueue(long technologyId) {
EntityManager entityManager = emf.createEntityManager();
entityManager.getTransaction().begin();
Query q = entityManager.createQuery("delete from Record r where r.technologyId=:technologyId");
q.setParameter("technologyId", technologyId);
q.executeUpdate();
entityManager.getTransaction().commit();
entityManager.close();
}
Related
I have the following scenario:
I have one controller which based on a path variable calls a different service.
In every service there is a transactional method where some import logic is happening(call one external api, get a csv file, parse it, convert it to entity and save it in database).
Additionally in every service I want to keep statistics of how many entities will be updated, inserted and deleted. For that reason I am using the org.hibernate.SessionFactory . One example of how I am using that is:
#Service
#Slf4j
public class MarketReportImporterImpl extends Support implements MarketReportImporter {
#Override
#Transactional
public void importMarketReports(ImporterLog importerLog) {
try {
String export = getCsvFile();
Session session = getCurrentSessionAndClearSessionFactoryStatistics();
// parse the csv and save the entities
flushSession(session);
setSuccessfulImport(session, importerLog);
} catch (Exception e) {
log.error("Failed to import market reports. Unable to parse export", e);
getTelemetryClient().trackException(e);
importerLogService.setFailedImport(importerLog, e.getMessage());
}
}
and the methods getCurrentSessionAndClearSessionFactoryStatistics() and setSuccessfulImport(session, importerLog); are in the Support class:
#Component
public abstract class Support {
private final ImporterLogService importerLogService;
#PersistenceContext
private EntityManager entityManager;
public Support(ImporterLogService importerLogService) {
this.importerLogService = importerLogService;
}
public void flushSession(Session session) {
session.flush();
}
public void setSuccessfulImport(Session session, ImporterLog importerLog) {
Statistics statistics = session.getSessionFactory().getStatistics();
int entityInsertCount = (int) statistics.getEntityInsertCount();
int entityDeleteCount = (int) statistics.getEntityDeleteCount();
int entityUpdateCount = (int) statistics.getEntityUpdateCount();
importerLogService.setSuccessfulImport(importerLog, entityUpdateCount, entityDeleteCount, entityInsertCount);
}
public Session getCurrentSessionAndClearSessionFactoryStatistics() {
Session session = entityManager.unwrap(Session.class);
SessionFactory sessionFactory = session.getSessionFactory();
sessionFactory.getStatistics().clear();
return session;
}
This works perfectly fine when calling it for one importer. But If from the frontend I am calling quickly two importers (meaning two threads run in parallel) there will be mix in the results. The session.getSessionFactory().getStatistics(); will have mix results from the first importer and from the second importer and I want to have a clear result only for the current session. For example service A and service B are running in parallel and in service A I am saving entities of type aa and in service B of type bb. In each service I want to know how many entities are saved, updated or deleted meaning in service A -> how many of type aa and in service B -> how many of type bb . As I understand every thread should open a session on it's own and then for every session I should get the correct sessionFactory and the correct results. But as it turns out this sessionFactory I guess (not sure in this statement) it belongs to every session and that is why I have mix results.
My question is if there is a way to separate somehow the sessionFactory and have clear vision of which entity how many times is saved,deleted,updated even in multithreaded environment.
If you want the statistics of a session, then call the getStatistics() method on session, which will give you the SessionStatistics. A SessionFactory should only exist once and statistics there are across all sessions.
I have some uncatchable bug in my work.
For example, I have code that looks like this:
#Entity
public class Message {
#Id
#GeneratedValue(strategy = SEQUENCE, generator = "message_generator")
private long id;
private long massMessageId;
}
public class MessageDTO {
public final long id;
public final long massMessageId;
}
#Transactional
#Service
public class ExtendedMessageService {
private MessageService messageService;
public MessageDTO createMessage(MessageCreateDTO createDTO) {
var messageDTO = messageService.create();
return messageService.linkMassMessage(messageDTO.id, createDTO.massMessageId);
}
}
#Transactional
#Service
public class MessageService {
private final MessageRepository repository;
private final ObjectMapper mapper;
public MessageDTO create() {
var message = new Message();
var savedMessage = repository.save(message);
return mapper.map(savedMessage, MessageDTO.class);
}
public MessageDTO linkMassMessage(long messageId, long massMessageId) {
var message = repository.findById(messageId)
.orElseThrow(() -> new ObjectNotFoundException("Message with id " + id + " was not found"));
return mapper.map(repository.save(message.setMassMessageId(massMessageId)), MessageDTO.class);
}
}
What will happen in this situation? I have some bugs, when repository.findById(id) can't find entity and throws exception.
And i have no reason, why this bug is only on prod (i tried to repeat it on dev and nothing succeeded)
And when i try to find the reason of it, i get a question:
"Can i save entity and get it in one transaction in Spring?"
How saving works
repository.save() doesn't save anything to database, this method puts entity to the session (persistent context) in memory.
flush step — on this step actual SQL insert happens. It can be invoked manually repository.saveAndFlush(), repository.flush(). Hibernate can do flush in the background, before operations that can use saved to the database value, like JPQL statements.
Also flush happens when the end of #Transactional boundary is reached.
What can be an issue
You are using incorrect method. This method from the old version of Spring data and it doesn't perform search in the database. You have to use findById() method instead.
Hibernate: findById vs getbyId
The most simple way, if you want to use id after save — flush the data immediately.
Entity entity = new Entity(some_information);
repository.saveAndFlush(entity);
Entity findedEntity = repository.findById(entity.getId())
.orElseThrow(() -> new RuntimeException("Can't find id=" + entity.getId()));
Hibernate will not necessary perform SQL select to get findedEntity. It can get it from the session, if it happens in the same #Transactional boundaries.
So if the above code resides in the method with #Transaction SQL will not performed. if there is not #Transaction SQL will be performed.
About this question
"Can Spring or Hibernate find not flushed entity in transaction context? Or there are some other ways to do it?"
Hibernate can't find not flushed entity. if id is autogenerated, Hibernate needs to perform SQL INSERT (flush) to get the id from a database. Another option to set up an id manually. Probably in this case it will be possible to get an entity from the persistent context.
I am updating my application from Spring Boot 1.4.5 / Hibernate 4.3.5 to Spring Boot 2.0.9 / Hibernate 5.2.18 and code that used to work in the previous configuration is no longer working.
The scenario is as follows:
Start a transaction by entering a method annotated with #Transactional
Hydrate the entity
Change the entity
Make another query
Detect a problem. As a result of this problem, determine that changes should not persist.
Evict the entity
Exit the method / transaction
With Hibernate 4.3.5, calling entityManager.detach() would prevent the changes from being persisted. However, with Hibernate 5.2.18, I'm finding that changes are persisted even with this call. I have also tried to evict() from the session and I have tried to clear() all entities from the session (just to see what would happen).
So I ask - is it possible to discard entity changes in Hibernate 5.2.18 the way that I was able to do in Hibernate 4.3.5?
The relevant code is below...
#Entity
public class Agreement {
private Long agreementId;
private Integer agreementStateId;
#Id
#Column(name = "agreement_id")
public Long getAgreementId() {
return agreementId;
}
public void setAgreementId(Long agreementId) {
this.agreementId = agreementId;
}
#Basic
#Column(name = "agreement_state_id", nullable = false)
public Integer getAgreementStateId() {
return agreementStateId;
}
public void setAgreementStateId(Integer agreementStateId) {
this.agreementStateId = agreementStateId;
}
}
#Component
public class Repo1 {
#PersistenceContext(unitName = "rights")
private EntityManager entityManager;
public void evict(Object entity) {
entityManager.detach(entity);
}
public Agreement getAgreement(Long agreementId) {
// Code to get entity is here.
// Agreement with an agreementStateId of 5 is returned.
}
public void anotherQuery() {
// Code to make another query is here.
}
}
#Component
public class Service1 {
#Autowired
Repo1 repo;
#Transactional
public void doSomething() {
Agreement agreement = repo.getAgreement(1L);
// Change agreementStateId. Very simple for purposes of example.
agreement.setAgreementStateId(100);
// Make another query
repo.anotherQuery();
// Detect a problem here. Simplified for purposes of example.
if (agreement.getAgreementStateId() == 100) {
repo.evict(agreement);
}
}
}
I have found the problem and it has nothing to do with evict(). It turns out that an additional query was causing the session to flush prior to the evict() call.
In general, the application uses QueryDSL to make queries. Queries made in this way did not result in the session flushing prior to making a query. However in this case, the query was created via Session.createSQLQuery(). This uses the FlushMode already assigned to the session which was FlushMode.AUTO.
I was able to prevent the flush by calling setHibernateFlushMode(FlushMode.COMMIT) on the query prior to making the query. This causes the session FlushMode to temporarily change until after the query has been run. After that, the evict() call worked as expected.
I am trying to implement logging in DB table using Spring AOP. By "logging in table" I mean to write in special log table information about record that was CREATED/UPDATED/DELETED in usual table for domain object.
I wrote some part of the code and all is working good except one thing - when transaction is rolled back then changes in log table still commit successfully. It's strange for me because in my AOP advice the same transaction is using as in my business and DAO layer. (From my AOP advice I called methods of special manager class with Transaction propagation MANDATORY and also I checked transaction name TransactionSynchronizationManager.getCurrentTransactionName() in business layer, dao layer and AOP advice and it is the same).
Does anyone tried to implement similar things in practice? Is it possible to use in AOP advice the same transaction as in the business layer and rollback changes made in AOP advice if some error in business layer occurs?
Thank you in advance for unswers.
EDIT
I want to clarify that problem with rollback occurs only for changes that was made from AOP advice. All changes that is made in DAO layer are rollbacked successfully. I mean that, for example, if some exception is thrown then changes made in DAO layer will be successfully rollbacked, but in log table information will be saved (commited). But I can't understand why it is like that because as I wrote above in AOP advice the same transaction is using.
EDIT 2
I checked with debugger the piece of the code where I am writting to the log table in AOP advice and it seems to me that JdbcTemplate's update method executes outside transaction because changes had been commited to the DB directly after execution of the statement and before transactional method was finished.
EDIT 3
I solved this problem. Actually, that was my stupid fault. I'm using MySQL. After creation of the log table I did't change DB engine and HeidySQL set MyIsam by default. But MyIsam doesn't support transaction so I changed DB engine to InnoDB (as for all other tables) and now all is working perfectly.
Thank you all for help and sorry for disturbing.
If someone is interested, here is simplified example that illustrate my approach.
Consider DAO class that has save method:
#Repository(value="jdbcUserDAO")
#Transactional(propagation=Propagation.SUPPORTS, readOnly=true, rollbackFor=Exception.class)
public class JdbcUserDAO implements UserDAO {
#Autowired
private JdbcTemplate jdbcTemplate;
#LoggedOperation(affectedRows = AffectedRows.ONE, loggedEntityClass = User.class, operationName = OperationName.CREATE)
#Transactional(propagation=Propagation.REQUIRED, readOnly=false, rollbackFor=Exception.class)
#Override
public User save(final User user) {
if (user == null || user.getRole() == null) {
throw new IllegalArgumentException("Input User object or nested Role object should not be null");
}
KeyHolder keyHolder = new GeneratedKeyHolder();
jdbcTemplate.update(new PreparedStatementCreator() {
#Override
public PreparedStatement createPreparedStatement(Connection connection)
throws SQLException {
PreparedStatement ps = connection.prepareStatement(SQL_INSERT_USER, new String[]{"ID"});
ps.setString(1, user.getUsername());
ps.setString(2, user.getPassword());
ps.setString(3, user.getFullName());
ps.setLong(4, user.getRole().getId());
ps.setString(5, user.geteMail());
return ps;
}
}, keyHolder);
user.setId((Long) keyHolder.getKey());
VacationDays vacationDays = user.getVacationDays();
vacationDays.setId(user.getId());
// Create related vacation days record.
vacationDaysDAO.save(vacationDays);
user.setVacationDays(vacationDays);
return user;
}
}
Here is how aspect looks like:
#Component
#Aspect
#Order(2)
public class DBLoggingAspect {
#Autowired
private DBLogManager dbLogManager;
#Around(value = "execution(* com.crediteuropebank.vacationsmanager.server.dao..*.*(..)) " +
"&& #annotation(loggedOperation)", argNames="loggedOperation")
public Object doOperation(final ProceedingJoinPoint joinPoint,
final LoggedOperation loggedOperation) throws Throwable {
Object[] arguments = joinPoint.getArgs();
/*
* This should be called before logging operation.
*/
Object retVal = joinPoint.proceed();
// Execute logging action
dbLogManager.logOperation(arguments,
loggedOperation);
return retVal;
}
}
And here is how my db log manager class LooksLike:
#Component("dbLogManager")
public class DBLogManager {
#Autowired
private JdbcTemplate jdbcTemplate;
#InjectLogger
private Logger logger;
#Transactional(rollbackFor={Exception.class}, propagation=Propagation.MANDATORY, readOnly=false)
public void logOperation(final Object[] inputArguments, final LoggedOperation loggedOperation) {
try {
/*
* Prepare query and array of the arguments
*/
jdbcTemplate.update(insertQuery.toString(),
insertedValues);
} catch (Exception e) {
StringBuilder sb = new StringBuilder();
// Prepare log string
logger.error(sb.toString(), e);
}
}
It could be to do with the order of the advice - you would want your #Transaction related advice to take effect around(or before and after) your logging related advice. If you are using Spring AOP you can probably control it using the order attribute of the advice - give your transaction related advice the highest precedence so that it executes last on the way out.
Nothing to do with AOP, set datasource property autocommit to false like :
<bean id="datasource" ...>
<property name="autoCommit" value="false/>
</bean>
If you are using xml configuration
When I run my unit tests in isolation they work fine ie. (omitted the asserts)
#Test
public void testSave()
{
EntityManagerHelper emh = new EntityManagerHelper();
LevelDAO dao = new LevelDAO();
Level l = new Level();
l.setName("aname");
emh.beginTransaction();
dao.save(l);
emh.commit();
}
then running this individual test below no problem
#Test
public void testUpdate()
{
EntityManagerHelper emh = new EntityManagerHelper();
LevelDAO dao = new LevelDAO();
Level l = new Level();
l.setName("bname");
l.setLevelid(1);
emh.beginTransaction();
dao.update(l);
emh.commit();
}
When they run at same time in sequence I recieve that error - Transaction is currently active. Is there a way to allow each unit test to run only after a transaction from previous piece of work is not active? Should I be looking at Spring instead?
Update
The EntityManagerHelper gains access to the persistence context like so
emf = Persistence.createEntityManagerFactory("bw_beta");
threadLocal = new ThreadLocal<EntityManager>();
which looks like the problem
So a hacky workaround was to use define locally ie.
EntityManagerFactory factory = Persistence.createEntityManagerFactory("bw_beta");
EntityManager entityManager = factory.createEntityManager();
entityManager.getTransaction().begin();
dao.save(l);
entityManager.persist(l);
entityManager.getTransaction().commit();
Pretty sure there's a better way - maybe using Spring?
Pretty sure there's a better way - maybe using Spring?
Yes, Spring cleans it up a lot and gives you control on what you'd like to run within a transaction without polluting the actual test.
With Spring, your tests would look something like this:
#RunWith(SpringJUnit4ClassRunner.class)
#ContextConfiguration({ "classpath:META-INF/conf/spring/application-context.xml",
"classpath:META-INF/conf/spring/test-datasource-spring-config.xml" })
#TransactionConfiguration(transactionManager="txMgr", defaultRollback=false)
public class LevelDaoTest {
#Resource( name="levelDao" )
LevelDao levelDao;
#Test
public void shouldSaveNewLevels() {
Level l = new Level();
l.setName("aname");
levelDao.save(l);
// assert
}
#Test
public void shouldUpdateExistingLevels() {
Level l = new Level(); // or I would assume, you'd read this level back from DB, or set a proper ID, so the DAO will know to update it.. But that is besides the point
l.setName("bname");
levelDao.update(l);
// assert
}
}
Take a look at Spring Documentation under Testing => Transaction Management to get more details.
P.S. From your example:
dao.save(l);
entityManager.persist(l);
Looks really strange, as usually you would encapsulate entityManager within a DAO, so all you'd need to do is dao.save(l)
For anyone that may be having this issue here is how I resolved it. I was doing multiple saves and I kept getting this error. You do not want to begin multiple transactions without checking if it is active.
if(!entityManager.getTransaction().isActive())
entityManager.getTransaction().begin();
dao.save(l);
entityManager.persist(l);
entityManager.getTransaction().commit();
I implemented a Singleton approach to handle it.