This question was asked a couple of years ago, but the answer didn't work for me. I have added the suggested annotations to the config and to the dao. I am sure that Template is actually connecting to the DB because I was getting appropriate errors when I had a column too small. The update call is doing a single row insert and it is returning 1 with no exceptions. Yet, when I check the database there is no data in it.
Any help would be appreciated.
Config:
#Configuration
#EnableTransactionManagement
#ComponentScan("com.XXX.query.repository")
public class SqlConfig {
#Autowired
private Environment env;
#Bean
public DataSource getDatasource() {
DriverManagerDataSource datasource = new DriverManagerDataSource();
datasource.setDriverClassName(env.getProperty("sql-datasource.driverClassName"));
datasource.setUrl(env.getProperty("sql-datasource.url"));
datasource.setUsername(env.getProperty("sql-datasource.username"));
datasource.setPassword(env.getProperty("sql-datasource.password"));
return datasource;
}
#Bean
public JdbcTemplate jdbcTemplate() {
return new JdbcTemplate(getDatasource());
}
DAO:
#Repository
public class SqlRepositoryImpl implements SqlRepository {
#Autowired
private JdbcTemplate template;
<snip>
#Transactional
#Override
public void addOrder(String foo, String bar, String bat, String cat) {
int i;
try {
// we may already have this data
System.out.println("here");
i = template.update(
"insert into someTable "
+ "(A, B, C, D)"
+ " values (?,?,?,?)",
foo, bar, bat, cat);
} catch (DataAccessException ex) {
checkForDupeKey(ex);
}
<snip>
There was a stupid mistake in the area that threw an exception and unwound the transaction. For some reason that exception never bubbled to the surface so I wasn't aware of it until I stepped through everything one line at a time. My apologies for wasting people's time.
Related
I'm using SimpleJdbcCall to make a call to two stored procedure in two different functions using JdbcTemplate. The first Stored procedure call get successfull, but second Stored procedure call just hangs up & following message appears in log & nothing goes forward:
2019-10-23 02:00:33,043 DEBUG [http-nio-8080-exec-13:org.springframework.jdbc.datasource.DataSourceUtils] - Fetching JDBC Connection from DataSource
Here is the source code for config & code part
DataSourceConfig.java
#Configuration
public class DataSourceConfig {
#Autowired
private DataSource dataSource;
#Bean
public JdbcClientDetailsService jdbcClientDetailsService() {
return new JdbcClientDetailsService(dataSource);
}
}
OAuth2AuthorizationServerConfigurer
#Configuration
#EnableAuthorizationServer
public class OAuth2AuthorizationServerConfigurer extends AuthorizationServerConfigurerAdapter {
#Autowired
private JdbcClientDetailsService jdbcClientDetailsService;
#Override
public void configure(ClientDetailsServiceConfigurer clients) throws Exception {
// clients.jdbc(dataSource);
clients.withClientDetails(jdbcClientDetailsService);
}
}
Class where two Stored Procedures are getting called & application hang's up at 2nd Stored Procedure call. (First stored procedure call is getting executed. If no data found in it's result, it will go into else block which is currently happening)
CompanyService.java
#Component
public class CompanyService {
#Autowired
private JdbcTemplate jdbcTemplate;
private static Logger logger = LoggerFactory.getLogger(CompanyService.class);
public CompanyInfo getCompanyDetails(Integer companyId){
SimpleJdbcCall simpleJdbcCall = new SimpleJdbcCall(jdbcTemplate);
simpleJdbcCall.withSchemaName("foo")
.withCatalogName("foo_pkg")
.withProcedureName("SELECT_COMPANY"); //select company Stored Procedure
SqlParameterSource in = new MapSqlParameterSource()
.addValue("companyId", companyId)
.addValue("name",null)
.addValue("uuid",null);
try {
Map<String, Object> out = simpleJdbcCall.execute(in);
return //some method to convert out into CompanyInfo
}catch (DataAccessException ex){
logger.error("getCompanyDetails"+ex.getMessage());
return null;
}
}
public CompanyInfo registerCompany(RegisterCompany registerCompany) {
CompanyInfo companyInfo = getCompanyDetails(registerCompany.getCompanyId());
if(companyInfo!=null){
//TODO: throw an exception
}
else{
SimpleJdbcCall mergeJdbcCall = new SimpleJdbcCall(jdbcTemplate);
mergeJdbcCall.withSchemaName("foo")
.withCatalogName("foo_pkg")
.withProcedureName("MERGE_COMPANY"); //Merge company Stored Procedure
SqlParameterSource in = new MapSqlParameterSource()
.addValue("companyId", registerCompany.getCompanyPartyClassId())
.addValue("name",getName())
.addValue("uuid",getUuid());
Map<String, Object> out = mergeJdbcCall.execute(in); //It's getting hang up at this level
}
return null;
}
}
What is the configuration I've missed here, which is causing an issue. I also went throug details of SimpleJdbcCall which describe
A SimpleJdbcCall is a multi-threaded, reusable object representing a call to a stored procedure or a stored function.
that's why I created two different object of it in class. Tried with defining it at class level as well, still the same issue.
Is it the case that SimpleJdbcCall works for one call in one class ? What are the other alternatives that I can use ? (Apart from PreparedStatement)
I am facing some issue with transactions configured in my code.
Below is the code with transactions which writes data to DB.
Writer.java
class Writer {
#Inject
private SomeDAO someDAO;
#Transactional(propagation = Propagation.REQUIRES_NEW)
public void write(){
this.batchWrite();
}
private void batchWrite () {
try {
someDAO.writeToTable1(true);
} catch(Exception ex) {
someDAO.writeToTable1(false);
}
someDAO.writeToTable2();
}
}
SomeDAO.java
class SomeDAO {
#Inject
private JdbcTemplate JdbcTemplate;
public void writeToTable1(boolean flag) {
// Writes data to table 1 using jdbcTemplate
jdbcTemplate.update();
}
pulic void writeToTable2() {
// Writes data to table 2 using jdbcTemplate
jdbcTemplate.update();
}
}
Here data is getting stored into table 1 properly but sometimes, table 2 is getting skipped.
I am not sure how this is happening as both the tables have been written within same transaction.
Either transaction is partially committing the data or partially rolling back.
I have doubt that in the SomeDAO class I am injecting JdbcTemplate object which is creating new connection instead of using existing connection of transaction.
Can anyone please help me here?
Try binding a Transaction Manager bean having your jdbcTemplate inside #Transactional:
//Create a bean
#Bean
public PlatformTransactionManager txnManager() throws Exception {
return new DataSourceTransactionManager(jdbcTemplate().getDataSource());
}
And then use this transaction manager in #Transactional("txnManager").
Below is the configuration of my spring batch job which takes records from DB, do some processing in item processor, updates status column and writes back to DB.
When I ran for 10k records, I could see its taking every record one by one and updating status in the same manner. Initially I was planning to use multithreading but it doesn't make any sense as my job runs once in a day with number of records ranging from 10 to 100k. ( Records are less than 5k in most of the days and a very few days in a year ( 5 to 10 days) it comes to 50k to 100k).
I don't want to add more cpus and getting charged by Kubernetes just for 10 days of an year. Now the problem is when I ran this job, it takes only 100 records that it runs every select query independently instead of taking 100 at a time. Also update is also one record at a time and it takes 10 mins to process 10k records which is really slow.
How can do a faster read, process and write? I can get rid of multithreading and have a bit more of CPU utilization once in a while. More information is given as comments in code.
#Configuration
#EnableBatchProcessing
public class BatchConfiguration extends DefaultBatchConfigurer{
public final static Logger logger = LoggerFactory.getLogger(BatchConfiguration.class);
#Autowired
JobBuilderFactory jobBuilderFactory;
#Autowired
StepBuilderFactory stepBuilderFactory;
#Autowired
MyRepository myRepository;
#Autowired
private EntityManagerFactory entityManagerFactory;
#Value("${chunk-size}")
private int chunkSize;
#Value("${max-threads}")
private int maxThreads;
private final DataSource dataSource;
/**
* #param dataSource
* Override to do not set datasource even if a datasource exist during intialization.
* Initialize will use a Map based JobRepository (instead of database) for Spring batch meta tables
*/
#Override
public void setDataSource(DataSource dataSource) {
}
#Override
public PlatformTransactionManager getTransactionManager() {
return jpaTransactionManager();
}
#Autowired
public BatchConfiguration(#Qualifier("dataSource") DataSource dataSource) {
this.dataSource = dataSource;
}
#Bean
public JpaTransactionManager jpaTransactionManager() {
final JpaTransactionManager transactionManager = new JpaTransactionManager();
transactionManager.setDataSource(dataSource);
return transactionManager;
}
#Bean
#StepScope
public JdbcPagingItemReader<ModelEntity> importReader() { // I tried using RepositoryItemReader but records were skipped by JPA hence I went for JdbcPagingItemReader
JdbcPagingItemReader<ModelEntity> reader = new JdbcPagingItemReader<ModelEntity>();
final SqlPagingQueryProviderFactoryBean sqlPagingQueryProviderFactoryBean = new SqlPagingQueryProviderFactoryBean();
sqlPagingQueryProviderFactoryBean.setDataSource( dataSource );
sqlPagingQueryProviderFactoryBean.setSelectClause( "SELECT *" );
sqlPagingQueryProviderFactoryBean.setFromClause( "FROM mytable" );
sqlPagingQueryProviderFactoryBean.setWhereClause( "WHERE STATUS = 'myvalue' " );
sqlPagingQueryProviderFactoryBean.setSortKey( "primarykey" );
try {
reader.setQueryProvider( sqlPagingQueryProviderFactoryBean.getObject() );
} catch (Exception e) {
e.printStackTrace();
}
reader.setDataSource( dataSource );
reader.setPageSize( chunkSize );
reader.setSaveState( Boolean.FALSE );
reader.setRowMapper( new BeanPropertyRowMapper<ModelEntity>(ModelEntity.class ) );
return reader;
}
#Bean
public ItemWriter<ModelEntity> databaseWriter() {
RepositoryItemWriter<ModelEntity> repositoryItemWriter=new RepositoryItemWriter<>();
repositoryItemWriter.setRepository(myRepository);
repositoryItemWriter.setMethodName("save");
return repositoryItemWriter;
}
#Bean
public Myprocessor myprocessor() {
return new Myprocessor();
}
#Bean
public JobExecutionListener jobExecutionListener() {
return new JobExecutionListener();
}
#Bean
public StepExecutionListener stepExecutionListener() {
return new StepExecutionListener();
}
#Bean
public ChunkExecutionListener chunkListener() {
return new ChunkExecutionListener();
}
#Bean
public TaskExecutor taskExecutor() {
SimpleAsyncTaskExecutor taskExecutor = new SimpleAsyncTaskExecutor();
taskExecutor.setConcurrencyLimit(maxThreads);
return taskExecutor;
}
#Bean
public Job processJob() {
return jobBuilderFactory.get("myjob")
.incrementer(new RunIdIncrementer())
.start(processStep())
.listener(jobExecutionListener())
.build();
}
#Bean
public Step processStep() {
return stepBuilderFactory.get("processStep")
.<ModelEntity,ModelEntity>chunk(chunkSize)
.reader(importReader())
.processor(myprocessor())
.writer(databaseWriter())
.taskExecutor(taskExecutor())
.listener(stepExecutionListener())
.listener(chunkListener())
.transactionManager(getTransactionManager())
.throttleLimit(maxThreads)
.build();
}
}
Repository that I am using is JpaRepository and code below. (Assuming save method of its parent class CrudRepository will do save)
public interface MyRepository extends JpaRepository<ModelEntity, BigInteger> {
}
Processor is as below
#Component
public class Myprocessor implements ItemProcessor<Myprocessor,Myprocessor> {
#Override
public ModelEntity process(ModelEntity modelEntity) throws Exception {
try {
// This is fast and working fine
if ((myProcessing)) {
modelEntity.setStatus(success);
} else {
modelEntity.setStatus(failed);
}
}
catch (Exception e){
logger.info( "Exception occurred while processing"+e );
}
return modelEntity;
}
// This is fast and working fine
public Boolean myProcessing(ModelEntity modelEntity){
//Processor Logic Here
return processingStatus;
}
}
Properties file below
logging.level.org.hibernate.SQL=DEBUG
logging.level.com.zaxxer.hikari.HikariConfig=DEBUG
logging.level.org.hibernate.type.descriptor.sql.BasicBinder=TRACE
logging.level.org.springframework.jdbc.core.JdbcTemplate=DEBUG
logging.level.org.springframework.jdbc.core.StatementCreatorUtils=TRACE
spring.datasource.type=com.zaxxer.hikari.HikariDataSource
spring.datasource.url=url
spring.datasource.username=username
spring.datasource.password=password
spring.jpa.hibernate.connection.provider_class
=org.hibernate.hikaricp.internal.HikariCPConnectionProvider
spring.jpa.database-platform=org.hibernate.dialect.Oracle10gDialect
spring.jpa.show-sql=false
spring.main.allow-bean-definition-overriding=true
spring.batch.initializer.enabled=false
spring.batch.job.enabled=false
spring.batch.initialize-schema=never
chunk-size=100
max-threads=5
You can enable JDBC batch processing for INSERT, UPDATE and DELETE statements with just one configuration property:
spring.jpa.properties.hibernate.jdbc.batch_size
It determines the number of updates that are sent to the database at one time for execution.
For details, see this link
Thank you all for the suggestions. I found the issue myself. I was using JdbcPagingItemReader and RepositoryItemWriter. The reader was working as expected, but the writer was triggering a select query for each record passed after processor. I believe reason behind is that the the record is persistent to JPA only after processor since the reader is not a standard JPA reader. I am not sure about it though. But changing the writer to JdbcBatchItemWriter fixed the issue.
I want to have a classic transactional behavior : create a, create b, if creation of b fail, rollback creation of a.
"Not so difficult, put an #Transactional spring annotation on your method."
So did I. But it doesn't work.
I'm using spring mvc 5.0.8 (Same version number for spring tx, context, in the below pom.xml). The database is mysql v.5.7.23
First, my config/code, then some redundant error many one get.
Here is a service-level method.
#Transactional(propagation = Propagation.REQUIRED, rollbackFor = Exception.class)
public B createB(B b) {
//some logic to create a, instance of A
aDao.saveA(a);
//some logic to create b, instance of B
return bDao.saveB(b);
}
Here is the datasource, jdbctemplate, and transaction manager config :
#Bean
public DataSource dataSource() {
DriverManagerDataSource dataSource = new DriverManagerDataSource();
dataSource.setDriverClassName(env.getRequiredProperty("jdbc.driverClassName"));
dataSource.setUrl(env.getRequiredProperty("jdbc.url"));
dataSource.setUsername(env.getRequiredProperty("jdbc.username"));
dataSource.setPassword(env.getRequiredProperty("jdbc.password"));
return dataSource;
}
#Bean
public JdbcTemplate jdbcTemplate(DataSource dataSource) {
JdbcTemplate jdbcTemplate = new JdbcTemplate(dataSource);
jdbcTemplate.setResultsMapCaseInsensitive(true);
return jdbcTemplate;
}
#Bean("transactionManager")
public PlatformTransactionManager getTransactionManager(DataSource dataSource) {
DataSourceTransactionManager dataSourceTransactionManager = new DataSourceTransactionManager();
dataSourceTransactionManager.setDataSource(dataSource);
return dataSourceTransactionManager;
}
Here is a part of my pom.xml :
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-context</artifactId>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-tx</artifactId>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-context</artifactId>
</dependency>
If you need more code, from Daos, or other, please ask.
Edit 1 : Add daos
#Repository
public class SqlADao implements ADao {
private static final String CREATE_A = "INSERT INTO a(id, param1,
param2) VALUES(?, ?, ?)";
#Autowired
private JdbcTemplate jdbcTemplate;
#Override
public Addressee saveA(A a) {
try {
String id = UUID.randomUUID().toString();
a.setId(id);
jdbcTemplate.update(CREATE_A, id, a.getParam1(),
addressee.getParam2());
return a;
} catch (Exception e) {
throw new DaoException("Exception while accessing data.", e);
}
}
}
#Repository
public class SqlBDao implements BDao {
private static final String CREATE_B = "INSERT INTO b(id, param1,
param2, param3, param4, param5, param6) VALUES(?, ?, ?, ?, ?, ?, ?)";
#Autowired
private JdbcTemplate jdbcTemplate;
#Override
public Account saveAccount(Account account) {
try {
String id = UUID.randomUUID().toString();
b.setId(id);
jdbcTemplate.update(CREATE_B, id, b.getParam1(),
Date.valueOf(b.getParam2LocalDate()), b.getParam3,
b.getParam4(), b.getParam5(),b.getParam6());
return b;
} catch (Exception e) {
throw new DaoException("Exception while accessing data.", e);
}
}
}
End of Edit 1
Now, possible answer found on many place on the internet, mostly stackoverflow :
Rollback work only for unchecked exceptions (=RuntimeException).
The BDao throws a custom RuntimeException. And I added the "rollbackFor = Exception.class" on the #Transactionnal definition.
Due to internal way #Transactional works (AOP), you have to call it from outside the method.
That is the case, the "createB" method is directly called from another class (at the moment, only from a controller".
You have to define a "transactionManager", with this precise name.
At first I had no transaction manager. When I realized, I felt a bit dumb. But even with one, with this precise name, it doesn't work.
I even found someone who needed to add the propagation=Propagation.REQUIRED, so I did too, but it doesn't change anything.
The Database have to support transactions, rollback...
It's a mysql with InnoDB engine on all table, which according to same answers supports it.
Now, I don't have any other idea, and can't found other ideas on internet, so here I am.
When using transactions make sure that you have enabled transactions using #EnableTransactionManagement on your #Configuration.
Without that annotation you are basicly running without transactions and that makes each operation run in its own transactions.
I've a Spring Batch process that read Report objects from a CSV and insert Analytic objects into a MySQL DB correctly, but the logical has changed for a more than one Analytics insert for each Report readed.
I'm new in Spring Batch and the actually process was very difficult for me, and I don't know how to do this change.
I haven't XML configuration, all is with annotations. Report and Analytics classes have a getter and a setter for two fields, adId and value. The new logic has seven values for an adId and I need to insert seven rows into table.
I hide, delete or supress some code that not contribute for the question.
Here is my BatchConfiguration.java:
#Configuration
#EnableBatchProcessingpublic
class BatchConfiguration {
#Autowired
private transient JobBuilderFactory jobBuilderFactory;
#Autowired
private transient StepBuilderFactory stepBuilderFactory;
#Autowired
private transient DataSource dataSource;
public FlatFileItemReader<Report> reader() {
// The reader from the CSV works fine.
}
#Bean
public JdbcBatchItemWriter<Analytic> writer() {
final JdbcBatchItemWriter<Analytic> writer = new JdbcBatchItemWriter<Analytic>();
writer.setItemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<Analytic>());
writer.setSql("INSERT INTO TABLE (ad_id, value) VALUES (:adId, :value)");
writer.setDataSource(dataSource);
return writer;
}
#Bean
public AnalyticItemProcessor processor() {
return new AnalyticItemProcessor();
}
#Bean
public Step step() {
return stepBuilderFactory.get("step1").<Report, Analytic> chunk(10000).reader(reader()).processor(processor()).writer(writer()).build();
}
#Bean
public Job process() {
final JobBuilder jobBuilder = jobBuilderFactory.get("process");
return jobBuilder.start(step()).build();
}
}
Then the AnalyticItemProcessor.java
public class AnalyticItemProcessor implements ItemProcessor<Report, Analytic> {
#Override
public Analytic process(final Report report) {
// Creates a new Analytic call BeanUtils.copyProperties(report, analytic) and returns analytic.
}
}
And the Process:
#SpringBootApplication
public class Process {
public static void main(String[] args) throws Exception {
SpringApplication.run(Process.class, args);
}
}
How can I do this change? Maybe with ItemPreparedStatementSetter or ItemSqlParameterSourceProvider? Thanks.
If I'm understanding your question correctly, you can use the CompositeItemWriter to wrap multiple JdbcBatchItemWriter instances (one per insert you need to accomplish). That would allow you to insert multiple rows per item. Otherwise, you'd need to write your own ItemWriter implementation.