I have below java code where i am trying to fetch data using select query and then import this data into json format.
The problem is currently i am getting error
ORA-02063: preceding line from ABSTP
; nested exception is java.sql.SQLException:
ORA-01555: snapshot too old: rollback segment number 14 with name "_SYSSMU14_1823253467$" too small
This error i believe is because of long running query. As i am not good in java so i would like to know is there any other process in java for transaction handling where i can distribute the transaction and run this query or is there any other way where i can handle such transactions in java code to avoid this issue?
#Service
public class InformerSupp {
public final static Logger log = LoggerFactory.getLogger(InformerSupp.class);
#Autowired
private NamedParameterJdbcTemplate NamedParameterJdbcTemplate;
#Autowired
private String queueName;
#Autowired
private JmsTemplate jmsTemplate;
private ObjectMapper mapper;
#PostConstruct
public void afterPropertiesSet() throws Exception {
mapper = new ObjectMapper();
}
public boolean transportData() {
final List<Map<String, Object>> maps = NamedParameterJdbcTemplate
.queryForList(format("select * from isi_trt c"),EMPTY_MAP);
for (Map<String, Object> entry : maps) {
String json = null;
try {
json = mapper.writeValueAsString(entry);
transportMessage(json);
} catch (JMSException e) {
log.error(String.format("Failed to create a JSON message : %s", entry), e);
return false;
} catch (JsonProcessingException e) {
log.error(String.format("Failed to transport message : %s to %s", json, queueName), e);
return false;
}
}
return true;
}
private void transportMessage(final String json) throws JMSException {
log.info(String.format("send message : %s ",json));
jmsTemplate.send(queueName, session -> {
TextMessage textMessage = session.createTextMessage();
int ccsid = _L.coalesce(((MQSession) session).getIntProperty(WMQ_QMGR_CCSID),0);
textMessage.setIntProperty(WMQ_CCSID, ccsid);
textMessage.setIntProperty(JMS_IBM_CHARACTER_SET, ccsid);
textMessage.setText(json);
return textMessage;
});
}
}
I know there are many examples over internet for the same problem. But what I am trying to get help is on architecture level.
I have a simple spring project where in I have one configuration class.I am trying to configure two datasources
(distDataSource, shipmentDataSource). I have two separate classes for two data sources (MyBatisDISTDataSource , MyBatisShipmentDataSource) mentioned below.
These two datasorces are working fine separately, but when I am trying to execute it together I get exception on console.
Console Exception Log
Exception in thread "main"
org.springframework.beans.factory.UnsatisfiedDependencyException:
Error creating bean with name 'distDAOImpl': Unsatisfied dependency
expressed through field 'distMapper'; nested exception is
org.springframework.beans.factory.UnsatisfiedDependencyException:
Error creating bean with name 'distMapper' defined in file
[D:\eclipse\ShippingModule\shipment-module-v2\CustEquip-CourierShipmentService-PickupSvc\target\classes\com\shipment\mapper\DistMapper.class]: Unsatisfied dependency expressed through bean property
'sqlSessionFactory'; nested exception is
org.springframework.beans.factory.NoUniqueBeanDefinitionException: No
qualifying bean of type 'org.apache.ibatis.session.SqlSessionFactory'
available: expected single matching bean but found 2:
distSqlSessionFactory,shipmentSqlSessionFactory at
org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:588)
at
org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88)
at
org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:366)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1264)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
at
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at
org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)
at
org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
at
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543)
at
org.springframework.context.annotation.AnnotationConfigApplicationContext.(AnnotationConfigApplicationContext.java:84)
at
com.telus.shipment.app.starter.SchedulePickup.main(SchedulePickup.java:11)
Caused by:
org.springframework.beans.factory.UnsatisfiedDependencyException:
Error creating bean with name 'distMapper' defined in file
[D:\eclipse\ShippingModule\shipment-module-v2\CustEquip-CourierShipmentService-PickupSvc\target\classes\com\shipment\mapper\DistMapper.class]: Unsatisfied dependency expressed through bean property
'sqlSessionFactory'; nested exception is
org.springframework.beans.factory.NoUniqueBeanDefinitionException: No
qualifying bean of type 'org.apache.ibatis.session.SqlSessionFactory'
available: expected single matching bean but found 2:
distSqlSessionFactory,shipmentSqlSessionFactory at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireByType(AbstractAutowireCapableBeanFactory.java:1357)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1249)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
at
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at
org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:208)
at
org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1138)
at
org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1066)
at
org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:585)
... 14 more Caused by:
org.springframework.beans.factory.NoUniqueBeanDefinitionException: No
qualifying bean of type 'org.apache.ibatis.session.SqlSessionFactory'
available: expected single matching bean but found 2:
distSqlSessionFactory,shipmentSqlSessionFactory at
org.springframework.beans.factory.config.DependencyDescriptor.resolveNotUnique(DependencyDescriptor.java:173)
at
org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1116)
at
org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1066)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireByType(AbstractAutowireCapableBeanFactory.java:1342)
... 25 more
Configuration Class
#Component
#Configuration
#Profile("local")
public class EnvironmentConfigLocal implements EnvironmentConfig {
#Autowired #Qualifier("DISTDataSource") private MyBatisDISTDataSource distDataSource;
#Autowired #Qualifier("ShipmentDataSource") private MyBatisShipmentDataSource shipmentDataSource;
#PostConstruct
public void postConstruct() {
System.out.println("Selected Profile : Local");
}
#Bean
public static PropertySourcesPlaceholderConfigurer dataProperties(final Environment environment) {
final PropertySourcesPlaceholderConfigurer propertySourcesPlaceholderConfigurer = new PropertySourcesPlaceholderConfigurer();
final YamlPropertiesFactoryBean yaml = new YamlPropertiesFactoryBean();
final SpringProfileDocumentMatcher matcher = new SpringProfileDocumentMatcher();
matcher.addActiveProfiles(environment.getActiveProfiles());
yaml.setDocumentMatchers(matcher);
yaml.setResources(new ClassPathResource("application.yaml"));
propertySourcesPlaceholderConfigurer.setProperties(yaml.getObject());
return propertySourcesPlaceholderConfigurer;
}
#Bean
public DataSourceTransactionManager distTransactionManager() throws SQLException {
return new DataSourceTransactionManager(distDataSource);
}
#Bean
public DataSourceTransactionManager shipmentTransactionManager() throws SQLException {
return new DataSourceTransactionManager(shipmentDataSource);
}
#Bean
#Override
public SqlSessionFactory distSqlSessionFactory() throws Exception {
SqlSessionFactoryBean distSessionFactory = new SqlSessionFactoryBean();
distSessionFactory.setDataSource(distDataSource);
return distSessionFactory.getObject();
}
#Bean
#Override
public SqlSessionFactory shipmentSqlSessionFactory() throws Exception {
SqlSessionFactoryBean shipmentSessionFactory = new SqlSessionFactoryBean();
shipmentSessionFactory.setDataSource(shipmentDataSource);
return shipmentSessionFactory.getObject();
}
}
MyBatisDISTDataSource
#Component
#Qualifier("DISTDataSource")
public class MyBatisDISTDataSource extends PooledDataSource {
#Value("${dist.db.poolMaximumActiveConnections}") int poolMaximumActiveConnections;
#Value("${dist.db.poolMaximumIdleConnections}") int poolMaximumIdleConnections;
public MyBatisDISTDataSource(
#Value("${dist.db.driver-class}") String driver,
#Value("${dist.db.url}") String url,
#Value("${dist.db.user}") String username,
#Value("${dist.db.password}") String password) {
super(driver, url, username, password);
System.out.println("DIST DB Attr: \n\t"
+driver+"\n\t"
+url+"\n\t"
+username+"\n\t"
+password+"\n\t");
}
#PostConstruct
private void setDataSourceProperties() {
this.setPoolMaximumActiveConnections(poolMaximumActiveConnections);
this.setPoolMaximumIdleConnections(poolMaximumIdleConnections);
}
}
MyBatisShipmentDataSource
#Component
#Qualifier("ShipmentDataSource")
public class MyBatisShipmentDataSource extends PooledDataSource {
#Value("${shipment.db.poolMaximumActiveConnections}") int poolMaximumActiveConnections;
#Value("${shipment.db.poolMaximumIdleConnections}") int poolMaximumIdleConnections;
public MyBatisShipmentDataSource(
#Value("${shipment.db.driver-class}") String driver,
#Value("${shipment.db.url}") String url,
#Value("${shipment.db.user}") String username,
#Value("${shipment.db.password}") String password) {
super(driver, url, username, password);
System.out.println("Shipment DB Attr: \n\t"
+driver+"\n\t"
+url+"\n\t"
+username+"\n\t"
+password+"\n\t");
}
#PostConstruct
private void setDataSourceProperties() {
this.setPoolMaximumActiveConnections(poolMaximumActiveConnections);
this.setPoolMaximumIdleConnections(poolMaximumIdleConnections);
}
}
DistMapper
#Mapper
#Component
public interface DistMapper {
#Select({"select * "
+ "from CONTACT_ADDRESS CA, ADDRESS A"
+ "where CONTACTING_ID = '10001134' "
+ "and PROVINCE_CD ='ON' "
+ "and STREET_NUMBER = '15'"})
#Results({#Result(column = "CONTACTING_ID", property = "contactingId", jdbcType = JdbcType.DECIMAL)})
public List<OutletAddress> findAddressByOutletId();
}
ShipmentMapper
#Mapper
#Component
public interface ShipmentMapper {
#Select({"select C.CONTACT_ID "
+ "from SHIPMENT_EVENT_TRACKING SE, SHIPMENT S, CONTACT_ADDR CA, CONTACT C "
+ "where SE.EVENT_CD = 'PICKUP' "
+ "and SE.SHIPMENT_ID = s.shipment_id "
+ "and S.SENDER_ADDR_ID = CA.CONTACT_ADDR_ID "
+ "and CA.CONTACT_ID = c.contact_id "
+ "and C.GROUP_CD = 'OT' "
+ "and SE.EVENT_OCCURRED_IND = 'N' "
+ "and S.CREATION_TS >= (select CURRENT_TIMESTAMP - interval '30' day from dual)"
+ "and S.SCHEDULE_PICKUP_IND = 'Y'"})
#Results({
#Result(column = "CONTACT_ID", property = "contactId", jdbcType = JdbcType.DECIMAL)})
public CopyOnWriteArrayList<EligibleShipment> findShipmentsByOutlet();
}
#Primary on one of the SqlSessionFactory bean solved my problem.
#Bean
#Primary
#Override
public SqlSessionFactory sqlSessionFactory2() throws Exception {
SqlSessionFactoryBean sessionFactory2 = new SqlSessionFactoryBean();
sessionFactory2.setDataSource(dataSource2);
return sessionFactory2.getObject();
}
With spring-mybatis you either register mappers one by one or you use mappers scanning.
If you are registering mappers manually make sure you pass correct SqlSessionFactory to every mapper. If you don't do this spring will try to autowire SqlSessionFactory and because you have two of them you would get the error that no single bean found.
If you are using scanning for mappers specify parameters for it so that appropriate mappers are using correct SqlSessionFactory.
One way to do that is by placing mappers that should use different DataSources to different packages. In this case you need to create two MapperScannerConfigurer beans like this (assuming that mappers are in com.mycompany.myapp.distMappersPackage and com.mycompany.myapp.shipmentMappersPackage):
#Bean
public MapperScannerConfigurer distMapperScannerConfigurer() throws Exception {
MapperScannerConfigurer configurer = new MapperScannerConfigurer();
configurer.setBasePackage("com.mycompany.myapp.distMappersPackage");
configurer.setSqlSessionFactoryBeanName("distSqlSessionFactory");
return configurer;
}
#Bean
public MapperScannerConfigurer shipmentMapperScannerConfigurer() throws Exception {
MapperScannerConfigurer configurer = new MapperScannerConfigurer();
configurer.setBasePackage("com.mycompany.myapp.shipmentMappersPackage");
configurer.setSqlSessionFactoryBeanName("shipmentSqlSessionFactory");
return configurer;
}
Alternatively, you can create two different annotation and use them on mappers like this:
#DistMapperMarker
public interface DistMapper {
...
}
#ShipmentMapperMarker
public interface ShipmentMapper {
...
}
In this case mappers can be in one package but you specify annotationClass on MapperScannerConfigurer:
#Bean
public MapperScannerConfigurer distMapperScannerConfigurer() throws Exception {
MapperScannerConfigurer configurer = new MapperScannerConfigurer();
configurer.setAnnotationClass(DistMapperMarker.class);
configurer.setSqlSessionFactoryBeanName("distSqlSessionFactory");
return configurer;
}
#Bean
public MapperScannerConfigurer shipmentMapperScannerConfigurer() throws Exception {
MapperScannerConfigurer configurer = new MapperScannerConfigurer();
configurer.setAnnotationClass(ShipmentMapperMarker.class);
configurer.setSqlSessionFactoryBeanName("shipmentSqlSessionFactory");
return configurer;
}
For java base spring configuration you can also try to use MapperScan and specify parameters as attributes to it. This would require you to split spring configuration into at least two classes. I haven't used this approach so I'm not sure if it work fine.
My Application will not start. gives me an error creating a bean with the name of my file. I've searched and found similar posts to this question but they didnt seem to pertain to my error so I am hoping someone can find what may be causing this file to fail. Also, I am at a cross between two methods. Do I use the FlatFileReader or MomgoItemreader? I just need to retrieve two things from the DB firstname and lastname. I need to read the db and then write it to a file which I havent done yet. Any help would be greatly appreciated.
***************************
APPLICATION FAILED TO START
***************************
Description:
Failed to configure a DataSource: 'url' attribute is not specified and no embedded datasource could be configured.
Reason: Failed to determine a suitable driver class
Here is my class
#Configuration
#EnableBatchProcessing
public class PaymentPortalJob {
private static final Logger LOG =
LoggerFactory.getLogger(PaymentPortalJob.class);
#SuppressWarnings("unused")
#Autowired
private JobBuilderFactory jobBuilderFactory;
#SuppressWarnings("unused")
#Autowired
private StepBuilderFactory stepBuilderFactory;
#Autowired
private PaymentAuditRepository paymentAuditRepository;
// #Bean
// public FlatFileItemReader<PaymentAudit> PaymentPortalReader() {
// LOG.info("Inside PaymentPortalReader Method {}", "");
// return new FlatFileItemReaderBuilder<PaymentAudit>
().name("PaymentPortalReader")
// .delimited().names(new String[] { "rxFname", "rxLname" })
// .fieldSetMapper(new BeanWrapperFieldSetMapper<PaymentAudit>
() {
// {
// setTargetType(PaymentAudit.class);
// }
// }).build();
//
// }
#Bean
public ItemReader<PaymentAudit> reader() {
LOG.info("inside of ItemReader");
MongoItemReader<PaymentAudit> reader = new MongoItemReader<PaymentAudit>();
try {
reader.setTemplate(mongoTemplate());
} catch (Exception e) {
LOG.error(e.toString());
}
reader.setCollection("local");
return reader();
}
#Bean
public ItemProcessor<PaymentAudit, PaymentAudit> processor() {
return new PaymentPortalNOSQLProcessor();
}
#Bean
public ItemWriter<PaymentAudit> writer() {
MongoItemWriter<PaymentAudit> writer = new MongoItemWriter<PaymentAudit>();
try {
writer.setTemplate(mongoTemplate());
} catch (Exception e) {
LOG.error(e.toString());
}
writer.setCollection("paymentPortal");
return writer;
}
#Bean
Job job(JobBuilderFactory jbf, StepBuilderFactory sbf, ItemReader<? extends PaymentAudit> ir,
ItemWriter<? super PaymentAudit> iw) {
Step s1 = sbf.get("file-db").<PaymentAudit, PaymentAudit>chunk(100).reader(ir).writer(iw).build();
return jbf.get("etl").incrementer(new RunIdIncrementer()).start(s1).build();
}
#Bean
public MongoDbFactory mongoDbFactory() throws Exception {
return new SimpleMongoDbFactory(new MongoClient(), "local");
}
#Bean
public MongoTemplate mongoTemplate() throws Exception {
MongoTemplate mongoTemplate = new MongoTemplate(mongoDbFactory());
return mongoTemplate;
}
}
appplication.yml
mongodb:
databasePort: xxxxx
databaseName: local
spring:
data:
mongodb:
host: localhost
port: xxxxx
profiles:
active: local
batch:
job:
enabled: true
Processor:
package com.spring_batchjob.job.item;
import org.springframework.batch.item.ItemProcessor;
import com.spring_batchjob.bean.PaymentAudit;
public class PaymentPortalNOSQLProcessor implements
ItemProcessor<PaymentAudit, PaymentAudit> {
#Override
public PaymentAudit process(PaymentAudit bean) throws Exception {
return bean;
}
}
repo:
package com.spring_batchjob.repository;
import java.time.LocalDateTime;
import java.util.List;
import org.springframework.data.mongodb.repository.MongoRepository;
import org.springframework.stereotype.Repository;
import com.spring_batchjob.bean.PaymentAudit;
#Repository
public interface PaymentAuditRepository extends
MongoRepository<PaymentAudit, String> {
// List<PaymentAudit> findByCreateDttmBetween(LocalDateTime createStart,
LocalDateTime createEnd);
List<PaymentAudit> findByRxFNameAndRxLName(String rxFName, String
rxLName);
}
Error after running app:
2018-10-26 13:26:34.256 WARN 16376 --- [ restartedMain]
ConfigServletWebServerApplicationContext : Exception encountered during
context initialization - cancelling refresh attempt:
org.springframework.beans.factory.UnsatisfiedDependencyException: Error
creating bean with name 'paymentPortalJob': Unsatisfied dependency expressed
through field 'jobBuilderFactory'; nested exception is
org.springframework.beans.factory.UnsatisfiedDependencyException: Error
creating bean with name
'org.springframework.batch.core.configuration.annotation.
SimpleBatchConfiguration': Unsatisfied dependency expressed through field
'dataSource'; nested exception is
org.springframework.beans.factory.BeanCreationException:
Error creating bean
with name 'dataSource' defined in class path resource
[org/springframework/boot/autoconfigure/jdbc/
DataSourceConfiguration$Hikari.class]: Bean instantiation via factory method
failed; nested exception is
org.springframework.beans.BeanInstantiationException: Failed to instantiate
[com.zaxxer.hikari.HikariDataSource]: Factory method 'dataSource' threw
exception; nested exception is
org.springframework.boot.autoconfigure.jdbc.
DataSourceProperties$DataSourceBeanCreationException: Failed to determine a
suitable driver class
using Spring batch , I have a class contains 2 methods. one of them has Transactional annotation.
When trying to run the batch I get an error :
Injection of autowired dependencies failed; nested exception is java.lang.IllegalArgumentException:com.sun.proxy.$Proxy60
context.xml :
<batch:job id="data-update" parent="raptorBaseJob">
<batch:step id="load-info" parent="raptorBaseStep">
<tasklet ref="loadInfoTasklet" />
<batch:next on="COMPLETED" to="update-info" />
</batch:step>
<batch:step id="update-info" parent="raptorBaseStep">
<tasklet ref="updateInfoTasklet" />
</batch:step>
</batch:job>
loadInfoTasklet.java :
#Component
public class LoadInfoTasklet implements Tasklet {
#Autowired
private InfoManager infoManager;
#Trace
#Override
public RepeatStatus execute(StepContribution stepContribution, ChunkContext chunkContext) throws Exception {
try {
infoManager.load();
} catch(Exception ex) {
stepContribution.setExitStatus(new ExitStatus("FAILED_LOADING"));
}
return RepeatStatus.FINISHED;
}
}
updateInfoTasklet.java :
#Component
public class UpdateInfoTasklet implements Tasklet{
#Autowired
private InfoManager infoManager;
#Trace
#Override
public RepeatStatus execute(StepContribution stepContribution, ChunkContext chunkContext) throws Exception {
try {
infoManager.update();
} catch (Exception ex) {
stepContribution.setExitStatus(new ExitStatus("FAILED"));
}
return RepeatStatus.FINISHED;
}
}
InfoManager.java :
#Component
public class InfoManager {
#Autowired
private NameInfo namesInfo;
protected List<String> names;
#Trace
public void load() throws Exception {
names = namesInfo.getNames();
}
#Trace
#Transactional(propagation=Propagation.REQUIRES_NEW)
public void update() {
System.out.println("Updating...");
}
}
Full stack error msg :
12:40:54.071 [main] ERROR o.s.b.c.l.s.CommandLineJobRunner - Job
Terminated in error: Error creating bean with name 'loadInfoTasklet':
Injection of autowired dependencies failed; nested exception is
org.springframework.beans.factory.BeanCreationException: Could not
autowire field: private InfoManager LoadInfoTasklet.infoManager;
nested exception is java.lang.IllegalArgumentException: Can not set
infoManager field LoadInfoTasklet.infoManager to
com.sun.proxy.$Proxy60
I have completed a "happy-path" (as below).
How I can advise a .transform call to have it invoke an error flow (via errorChannel) w/o interrupting the mainFlow?
Currently the mainFlow terminates on first failure occurrence in second .transform (when payload cannot be deserialized to type). My desired behavior is that I'd like to log and continue processing.
I've read about ExpressionEvaluatingRequestHandlerAdvice. Would I just add a second param to each .transform call like e -> e.advice(myAdviceBean) and declare such a bean with success and error channels? Assuming I'd need to break up my mainFlow to receive success from each transform.
On some commented direction I updated the original code sample. But I'm still having trouble taking this "all the way home".
2015-09-08 11:49:19,664 [pool-3-thread-1] org.springframework.integration.handler.ServiceActivatingHandler DEBUG handler 'ServiceActivator for [org.springframework.integration.dsl.support.BeanNameMessageProcessor#5f3839ad] (org.springframework.integration.handler.ServiceActivatingHandler#0)' produced no reply for request Message: ErrorMessage [payload=org.springframework.integration.handler.advice.ExpressionEvaluatingRequestHandlerAdvice$MessageHandlingExpressionEvaluatingAdviceException: Handler Failed; nested exception is org.springframework.integration.transformer.MessageTransformationException: failed to transform message; nested exception is com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "hasDoaCostPriceChanged" (class com.xxx.changehistory.jdbc.data.RatePlanLevelRestrictionLog), not marked as ignorable (18 known properties: "supplierUpdateDate", "fPLOSMaskArrival", "createDate", "endAllowed", "sellStateId", "ratePlanLevel", "ratePlanId", "startAllowed", "stayDate", "doaCostPriceChanged", "hotelId", "logActionTypeId" [truncated]])
at [Source: java.util.zip.GZIPInputStream#242017b8; line: 1, column: 32] (through reference chain: com.xxx.changehistory.jdbc.data.RatePlanLevelRestrictionLog["hasDoaCostPriceChanged"]), headers={id=c054d976-5750-827f-8894-51aba9655c77, timestamp=1441738159660}]
2015-09-08 11:49:19,664 [pool-3-thread-1] org.springframework.integration.channel.DirectChannel DEBUG postSend (sent=true) on channel 'errorChannel', message: ErrorMessage [payload=org.springframework.integration.handler.advice.ExpressionEvaluatingRequestHandlerAdvice$MessageHandlingExpressionEvaluatingAdviceException: Handler Failed; nested exception is org.springframework.integration.transformer.MessageTransformationException: failed to transform message; nested exception is com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "hasDoaCostPriceChanged" (class com.xxx.changehistory.jdbc.data.RatePlanLevelRestrictionLog), not marked as ignorable (18 known properties: "supplierUpdateDate", "fPLOSMaskArrival", "createDate", "endAllowed", "sellStateId", "ratePlanLevel", "ratePlanId", "startAllowed", "stayDate", "doaCostPriceChanged", "hotelId", "logActionTypeId" [truncated]])
at [Source: java.util.zip.GZIPInputStream#242017b8; line: 1, column: 32] (through reference chain: com.xxx.changehistory.jdbc.data.RatePlanLevelRestrictionLog["hasDoaCostPriceChanged"]), headers={id=c054d976-5750-827f-8894-51aba9655c77, timestamp=1441738159660}]
2015-09-08 11:49:19,664 [pool-3-thread-1] org.springframework.integration.channel.DirectChannel DEBUG preSend on channel 'mainFlow.channel#3', message: GenericMessage [payload=java.util.zip.GZIPInputStream#242017b8, headers={id=b80106f9-7f4c-1b92-6aca-6e73d3bf8792, timestamp=1441738159664}]
2015-09-08 11:49:19,664 [pool-3-thread-1] org.springframework.integration.aggregator.AggregatingMessageHandler DEBUG org.springframework.integration.aggregator.AggregatingMessageHandler#0 received message: GenericMessage [payload=java.util.zip.GZIPInputStream#242017b8, headers={id=b80106f9-7f4c-1b92-6aca-6e73d3bf8792, timestamp=1441738159664}]
2015-09-08 11:49:19,665 [pool-3-thread-1] org.springframework.integration.channel.DirectChannel DEBUG preSend on channel 'errorChannel', message: ErrorMessage [payload=org.springframework.messaging.MessageHandlingException: error occurred in message handler [org.springframework.integration.aggregator.AggregatingMessageHandler#0]; nested exception is java.lang.IllegalStateException: Null correlation not allowed. Maybe the CorrelationStrategy is failing?, headers={id=24e3a1c7-af6b-032c-6a29-b55031fba0d7, timestamp=1441738159665}]
2015-09-08 11:49:19,665 [pool-3-thread-1] org.springframework.integration.handler.ServiceActivatingHandler DEBUG ServiceActivator for [org.springframework.integration.dsl.support.BeanNameMessageProcessor#5f3839ad] (org.springframework.integration.handler.ServiceActivatingHandler#0) received message: ErrorMessage [payload=org.springframework.messaging.MessageHandlingException: error occurred in message handler [org.springframework.integration.aggregator.AggregatingMessageHandler#0]; nested exception is java.lang.IllegalStateException: Null correlation not allowed. Maybe the CorrelationStrategy is failing?, headers={id=24e3a1c7-af6b-032c-6a29-b55031fba0d7, timestamp=1441738159665}]
2015-09-08 11:49:19,665 [pool-3-thread-1] com.xxx.DataMigrationModule$ErrorService ERROR org.springframework.messaging.MessageHandlingException: error occurred in message handler [org.springframework.integration.aggregator.AggregatingMessageHandler#0]; nested exception is java.lang.IllegalStateException: Null correlation not allowed. Maybe the CorrelationStrategy is failing?
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:84)
at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:116)
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:101)
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:97)
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:287)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:245)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:115)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:45)
at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:95)
at org.springframework.integration.handler.AbstractMessageProducingHandler.sendOutput(AbstractMessageProducingHandler.java:231)
at org.springframework.integration.handler.AbstractMessageProducingHandler.produceOutput(AbstractMessageProducingHandler.java:154)
at org.springframework.integration.handler.AbstractMessageProducingHandler.sendOutputs(AbstractMessageProducingHandler.java:102)
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:105)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:78)
at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:116)
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:101)
at org.springframework.integration.dispatcher.UnicastingDispatcher.access$000(UnicastingDispatcher.java:48)
at org.springframework.integration.dispatcher.UnicastingDispatcher$1.run(UnicastingDispatcher.java:92)
at org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:52)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: Null correlation not allowed. Maybe the CorrelationStrategy is failing?
at org.springframework.util.Assert.state(Assert.java:385)
at org.springframework.integration.aggregator.AbstractCorrelatingMessageHandler.handleMessageInternal(AbstractCorrelatingMessageHandler.java:369)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:78)
... 22 more
UPDATED (09-08-2015)
code sample
#Bean
public IntegrationFlow mainFlow() {
// #formatter:off
return IntegrationFlows
.from(
amazonS3InboundSynchronizationMessageSource(),
e -> e.poller(p -> p.trigger(this::nextExecutionTime))
)
.transform(unzipTransformer())
.split(f -> new FileSplitter())
.channel(MessageChannels.executor(Executors.newCachedThreadPool()))
.transform(Transformers.fromJson(persistentType()), , e -> e.advice(handlingAdvice()))
// #see http://docs.spring.io/spring-integration/reference/html/messaging-routing-chapter.html#agg-and-group-to
.aggregate(a ->
a.releaseStrategy(g -> g.size() == persistenceBatchSize)
.expireGroupsUponCompletion(true)
.sendPartialResultOnExpiry(true)
.groupTimeoutExpression("size() ge 2 ? 10000 : -1")
, null
)
.handle(jdbcRepositoryHandler())
// TODO add advised PollableChannel to deal with possible persistence issue and retry with partial batch
.get();
// #formatter:on
}
#Bean
public ErrorService errorService() {
return new ErrorService();
}
#Bean
public MessageChannel customErrorChannel() {
return MessageChannels.direct().get();
}
#Bean
public IntegrationFlow customErrorFlow() {
// #formatter:off
return IntegrationFlows
.from(customErrorChannel())
.handle("errorService", "handleError")
.get();
// #formatter:on
}
#Bean
ExpressionEvaluatingRequestHandlerAdvice handlingAdvice() {
ExpressionEvaluatingRequestHandlerAdvice advice = new ExpressionEvaluatingRequestHandlerAdvice();
advice.setOnFailureExpression("payload");
advice.setFailureChannel(customErrorChannel());
advice.setReturnFailureExpressionResult(true);
advice.setTrapException(true);
return advice;
}
protected class ErrorService implements ErrorHandler {
private final Logger log = LoggerFactory.getLogger(getClass());
#Override
public void handleError(Throwable t) {
stopEndpoints(t);
}
private void stopEndpoints(Throwable t) {
log.error(ExceptionUtils.getStackTrace(t));
}
}
Turns out I had things wrong in a few places, like:
I had to autowire a Jackson2 ObjectMapper (that I get from Sprint Boot auto-config) and construct an instance of JsonObjectMapper to be added as second arg in Transformers.fromJson; made for more lenient unmarshalling to persistent type (stops UnrecognizedPropertyException); and thus waived need for ExpressionEvaluatingRequestHandlerAdvice
Choosing the proper variant of .split method in IntegrationFlowDefinition in order to employ the FileSplitter, otherwise you don't get this splitter rather a DefaultMessageSplitter which pre-maturely terminates flow after first record read from InputStream
Moved transform, aggregate, handle to a its own pubsub channel employing an async task executor
Still not 100% of what I need, but it's much further along.
See what I ended up w/ below...
#Configuration
#EnableIntegration
#IntegrationComponentScan
public class DataMigrationModule {
private final Logger log = LoggerFactory.getLogger(getClass());
#Value("${cloud.aws.credentials.accessKey}")
private String accessKey;
#Value("${cloud.aws.credentials.secretKey}")
private String secretKey;
#Value("${cloud.aws.s3.bucket}")
private String bucket;
#Value("${cloud.aws.s3.max-objects-per-batch:1024}")
private int maxObjectsPerBatch;
#Value("${cloud.aws.s3.accept-subfolders:false}")
private String acceptSubFolders;
#Value("${cloud.aws.s3.remote-directory}")
private String remoteDirectory;
#Value("${cloud.aws.s3.local-directory-ref:java.io.tmpdir}")
private String localDirectoryRef;
#Value("${cloud.aws.s3.local-subdirectory:target/s3-dump}")
private String localSubdirectory;
#Value("${cloud.aws.s3.filename-wildcard:}")
private String fileNameWildcard;
#Value("${app.persistent-type:}")
private String persistentType;
#Value("${app.repository-type:}")
private String repositoryType;
#Value("${app.persistence-batch-size:2500}")
private int persistenceBatchSize;
#Value("${app.persistence-batch-release-timeout-in-milliseconds:5000}")
private int persistenceBatchReleaseTimeoutMillis;
#Autowired
private ListableBeanFactory beanFactory;
#Autowired
private ObjectMapper objectMapper;
private final AtomicBoolean invoked = new AtomicBoolean();
private Class<?> repositoryType() {
try {
return Class.forName(repositoryType);
} catch (ClassNotFoundException cnfe) {
log.error("Unknown repository implementation!", cnfe);
System.exit(0);
}
return null;
}
private Class<?> persistentType() {
try {
return Class.forName(persistentType);
} catch (ClassNotFoundException cnfe) {
log.error("Unsupported type!", cnfe);
System.exit(0);
}
return null;
}
public Date nextExecutionTime(TriggerContext triggerContext) {
return this.invoked.getAndSet(true) ? null : new Date();
}
#Bean
public FileToInputStreamTransformer unzipTransformer() {
FileToInputStreamTransformer transformer = new FileToInputStreamTransformer();
transformer.setDeleteFiles(true);
return transformer;
}
#Bean
public MessageSource<?> amazonS3InboundSynchronizationMessageSource() {
AWSCredentials credentials = new BasicAWSCredentials(this.accessKey, this.secretKey);
AmazonS3InboundSynchronizationMessageSource messageSource = new AmazonS3InboundSynchronizationMessageSource();
messageSource.setCredentials(credentials);
messageSource.setBucket(bucket);
messageSource.setMaxObjectsPerBatch(maxObjectsPerBatch);
messageSource.setAcceptSubFolders(Boolean.valueOf(acceptSubFolders));
messageSource.setRemoteDirectory(remoteDirectory);
if (!fileNameWildcard.isEmpty()) {
messageSource.setFileNameWildcard(fileNameWildcard);
}
String directory = System.getProperty(localDirectoryRef);
if (!localSubdirectory.startsWith("/")) {
localSubdirectory = "/" + localSubdirectory;
}
if (!localSubdirectory.endsWith("/")) {
localSubdirectory = localSubdirectory + "/";
}
directory = directory + localSubdirectory;
FileUtils.mkdir(directory);
messageSource.setDirectory(new LiteralExpression(directory));
return messageSource;
}
#Bean
public IntegrationFlow mainFlow() {
// #formatter:off
return IntegrationFlows
.from(
amazonS3InboundSynchronizationMessageSource(),
e -> e.poller(p -> p.trigger(this::nextExecutionTime))
)
.transform(unzipTransformer())
.split(new FileSplitter(), null)
.publishSubscribeChannel(new SimpleAsyncTaskExecutor(), p -> p.subscribe(persistenceSubFlow()))
.get();
// #formatter:on
}
#Bean
public IntegrationFlow persistenceSubFlow() {
JsonObjectMapper<?, ?> jsonObjectMapper = new Jackson2JsonObjectMapper(objectMapper);
ReleaseStrategy releaseStrategy = new TimeoutCountSequenceSizeReleaseStrategy(persistenceBatchSize,
persistenceBatchReleaseTimeoutMillis);
// #formatter:off
return f -> f
.transform(Transformers.fromJson(persistentType(), jsonObjectMapper))
// #see http://docs.spring.io/spring-integration/reference/html/messaging-routing-chapter.html#agg-and-group-to
.aggregate(
a -> a
.releaseStrategy(releaseStrategy)
.correlationStrategy(m -> m.getHeaders().get("id"))
.expireGroupsUponCompletion(true)
.sendPartialResultOnExpiry(true)
, null
)
.handle(jdbcRepositoryHandler());
// #formatter:on
}
#Bean
public JdbcRepositoryHandler jdbcRepositoryHandler() {
return new JdbcRepositoryHandler(repositoryType(), beanFactory);
}
protected class JdbcRepositoryHandler extends AbstractMessageHandler {
#SuppressWarnings("rawtypes")
private Insertable repository;
public JdbcRepositoryHandler(Class<?> repositoryClass, ListableBeanFactory beanFactory) {
repository = (Insertable<?>) beanFactory.getBean(repositoryClass);
}
#Override
protected void handleMessageInternal(Message<?> message) {
repository.insert((List<?>) message.getPayload());
}
}
protected class FileToInputStreamTransformer extends AbstractFilePayloadTransformer<InputStream> {
#Override
protected InputStream transformFile(File payload) throws Exception {
return new GZIPInputStream(new FileInputStream(payload));
}
}
}
Yes, you are correct. To advice the handle() method of Transformer's MessageHandler you should use exactly that e.advice method of the second parameter of .transform() EIP-method. And yes: you should define ExpressionEvaluatingRequestHandlerAdvice bean for your purpose.
You can reuse that Advice bean for different goals to handle successes and failures the same manner.
UPDATE
Although it isn't clear to me how you'd like to continue the flow with the wrong message, but you you can use onFailureExpression and returnFailureExpressionResult=true of the ExpressionEvaluatingRequestHandlerAdvice to return something after the unzipErrorChannel().
BTW the failureChannel logic doesn't work without onFailureExpression:
if (this.onFailureExpression != null) {
Object evalResult = this.evaluateFailureExpression(message, actualException);
if (this.returnFailureExpressionResult) {
return evalResult;
}
}