I am using AbstractRoutingDataSource to create multitenancy in my application. I noticed that after a few redeployments of the webapp from my IDE I eventually get the MySQL error "Too many connections".
After further investigations, I found out that when I run the MySQL command show processlist;, I see that the open connection amount is increased by 10 after each deployment, which probably means that the connection pool is somehow still alive somewhere.
Before I used AbstractRoutingDataSource I used the default spring datasource configuration (using application.properties) and it worked fine.
Here's the Multitenant configuration class:
import org.apache.commons.io.IOUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.autoconfigure.jdbc.DataSourceBuilder;
import org.springframework.boot.autoconfigure.jdbc.DataSourceProperties;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import javax.sql.DataSource;
import java.io.IOException;
import java.io.InputStream;
import java.util.*;
import java.util.stream.Collectors;
/**
* Created by Alon Segal on 16/03/2017.
*/
#Configuration
public class MultitenantConfiguration {
#Autowired
private DataSourceProperties properties;
/**
* Defines the data source for the application
*
* #return
*/
#Bean
#ConfigurationProperties(
prefix = "spring.datasource"
)
public DataSource dataSource() {
//Creating datasources map "resolvedDataSources" here
MultitenantDataSource dataSource = new MultitenantDataSource();
dataSource.setDefaultTargetDataSource(defaultDataSource());
dataSource.setTargetDataSources(resolvedDataSources);
// Call this to finalize the initialization of the data source.
dataSource.afterPropertiesSet();
return dataSource;
}
/**
* Creates the default data source for the application
*
* #return
*/
private DataSource defaultDataSource() {
.
.
.
}
}
And the datasource class:
import org.springframework.jdbc.datasource.lookup.AbstractRoutingDataSource;
/**
* Created by Alon Segal on 16/03/2017.
*/
public class MultitenantDataSource extends AbstractRoutingDataSource {
#Override
protected Object determineCurrentLookupKey() {
return TenantContext.getCurrentTenant();
}
}
I also tried to use #Bean(destroyMethod = "close") but there is not close method defined on AbstractRoutingDataSource.
I searched everywhere but could not find and answer. Can someone help me understand what's preventing the connection pool from being released between redeployments?
Thanks in advance.
Ok so I eventually solved the issue by giving up on using Spring's AbstractRoutingDataSource and instead using Hibernate's mechanism for multitenancy based on the solution that can be found in this article.
Long story short
You need to do 3 steps:
Step 1: Create a CurrentTenantIdentifierResolver
#Component
public class TenantIdentifierResolver implements CurrentTenantIdentifierResolver {
#Override
public String resolveCurrentTenantIdentifier() {
String tenantId = TenantContext.getCurrentTenant();
if (tenantId != null) {
return tenantId;
}
return DEFAULT_TENANT_ID;
}
#Override
public boolean validateExistingCurrentSessions() {
return true;
}
}
Step 2: Create a MultiTenantConnectionProvider
#Component
public class MultiTenantConnectionProviderImpl implements MultiTenantConnectionProvider {
#Autowired
private DataSource dataSource;
#Override
public Connection getAnyConnection() throws SQLException {
return dataSource.getConnection();
}
#Override
public void releaseAnyConnection(Connection connection) throws SQLException {
connection.close();
}
#Override
public Connection getConnection(String tenantIdentifie) throws SQLException {
String tenantIdentifier = TenantContext.getCurrentTenant();
final Connection connection = getAnyConnection();
try {
if (tenantIdentifier != null) {
connection.createStatement().execute("USE " + tenantIdentifier);
} else {
connection.createStatement().execute("USE " + DEFAULT_TENANT_ID);
}
}
catch ( SQLException e ) {
throw new HibernateException(
"Could not alter JDBC connection to specified schema [" + tenantIdentifier + "]",
e
);
}
return connection;
}
#Override
public void releaseConnection(String tenantIdentifier, Connection connection) throws SQLException {
try {
connection.createStatement().execute( "USE " + DEFAULT_TENANT_ID );
}
catch ( SQLException e ) {
throw new HibernateException(
"Could not alter JDBC connection to specified schema [" + tenantIdentifier + "]",
e
);
}
connection.close();
}
#SuppressWarnings("rawtypes")
#Override
public boolean isUnwrappableAs(Class unwrapType) {
return false;
}
#Override
public <T> T unwrap(Class<T> unwrapType) {
return null;
}
#Override
public boolean supportsAggressiveRelease() {
return true;
}
}
Step 3: Wire it up
#Configuration
public class HibernateConfig {
#Autowired
private JpaProperties jpaProperties;
#Bean
public JpaVendorAdapter jpaVendorAdapter() {
return new HibernateJpaVendorAdapter();
}
#Bean
public LocalContainerEntityManagerFactoryBean entityManagerFactory(DataSource dataSource,
MultiTenantConnectionProvider multiTenantConnectionProviderImpl,
CurrentTenantIdentifierResolver currentTenantIdentifierResolverImpl) {
Map<String, Object> properties = new HashMap<>();
properties.putAll(jpaProperties.getHibernateProperties(dataSource));
properties.put(Environment.MULTI_TENANT, MultiTenancyStrategy.SCHEMA);
properties.put(Environment.MULTI_TENANT_CONNECTION_PROVIDER, multiTenantConnectionProviderImpl);
properties.put(Environment.MULTI_TENANT_IDENTIFIER_RESOLVER, currentTenantIdentifierResolverImpl);
LocalContainerEntityManagerFactoryBean em = new LocalContainerEntityManagerFactoryBean();
em.setDataSource(dataSource);
em.setPackagesToScan("com.autorni");
em.setJpaVendorAdapter(jpaVendorAdapter());
em.setJpaPropertyMap(properties);
return em;
}
}
Related
I have two database structures, example:
1- MAIN_DATABASE: USER PASSWORD
2: CUSTOMER DATABASE: CUSTOMER_A CUSTOMER_B CUSTOMER_C
I want to access the main database and after validating the data, redirect to the customer database.
I currently use spring and configure it in applicationContext.xml
Example:
<bean id = "encryptionPassword" class = "utils.EncryptionPasswordSpring" />
<bean id = "dataSource" class = "com.mchange.v2.c3p0.ComboPooledDataSource" destroy-method = "close">
<property name = "driverClass" value = "com.mysql.jdbc.Driver" />
<property name = "user" value = "user" />
<property name = "password" value = "123456" />
<property name = "jdbcUrl" value = "jdbc:mysql://localhost/testdb?useSSL = false" />
</bean>
Any example, suggestion? Thanks.
The below is my code for dynamic datasource with mybatis. one is main ds. another is read ds. Hope it useful to you.
use AbstractRoutingDataSource to define a DynamicDataSource
import java.util.Map;
import javax.sql.DataSource;
import org.springframework.jdbc.datasource.lookup.AbstractRoutingDataSource;
public class DynamicDataSource extends AbstractRoutingDataSource {
#Override
protected DataSource determineTargetDataSource() {
return super.determineTargetDataSource();
}
/**
*/
#Override
protected Object determineCurrentLookupKey() {
return DynamicDataSourceContextHolder.getDataSourceKey();
}
/**
* #param defaultDataSource
*/
public void setDefaultDataSource(Object defaultDataSource) {
super.setDefaultTargetDataSource(defaultDataSource);
}
/**
* #param dataSources
*/
public void setDataSources(Map<Object, Object> dataSources) {
super.setTargetDataSources(dataSources);
DynamicDataSourceContextHolder.addDataSourceKeys(dataSources.keySet());
}
}
use ThreadLocal to switch datasource in context
import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
import java.util.List;
public class DynamicDataSourceContextHolder {
private static final ThreadLocal<String> contextHolder = new ThreadLocal<String>() {
/**
make main as default ds
*/
#Override
protected String initialValue() {
return "main";
}
};
/**
*
*/
private static List<Object> dataSourceKeys = Collections.synchronizedList(new ArrayList<>());
/**
* switch ds
*
* #param key
*/
public static void setDataSourceKey(String key) {
contextHolder.set(key);
}
/**
* get ds
*
* #return
*/
public static String getDataSourceKey() {
return contextHolder.get();
}
/**
* reset ds
*/
public static void clearDataSourceKey() {
contextHolder.remove();
}
/**
* judge if ds existed
*
* #param key
* #return
*/
public static boolean containDataSourceKey(String key) {
return dataSourceKeys.contains(key);
}
/**
* add ds
*
* #param keys
* #return
*/
public static boolean addDataSourceKeys(Collection<? extends Object> keys) {
return dataSourceKeys.addAll(keys);
}
}
inject different datasource via application.yml or application.properties
import java.util.HashMap;
import java.util.Map;
import java.util.Properties;
import javax.sql.DataSource;
import org.apache.ibatis.plugin.Interceptor;
import org.apache.ibatis.session.SqlSessionFactory;
import org.mybatis.spring.SqlSessionFactoryBean;
import org.mybatis.spring.SqlSessionTemplate;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.autoconfigure.jdbc.DataSourceBuilder;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Primary;
import org.springframework.core.io.support.PathMatchingResourcePatternResolver;
import org.springframework.jdbc.datasource.DataSourceTransactionManager;
import org.springframework.transaction.PlatformTransactionManager;
#Configuration
public class DataSourceConfig {
#Primary
#Bean
#ConfigurationProperties("spring.datasource.main")
public DataSource main() {
return DataSourceBuilder.create().build();
}
#Bean
#ConfigurationProperties("spring.datasource.read")
public DataSource read() {
return DataSourceBuilder.create().build();
}
#Bean
public DataSource dynamicDataSource(
#Qualifier("main") DataSource main,
#Qualifier("read") DataSource read
) {
Map<Object, Object> targetDataSources = new HashMap<>(2);
targetDataSources.put("main", main);
targetDataSources.put("read", read);
DynamicDataSource dynamicDataSource = new DynamicDataSource();
dynamicDataSource.setDefaultTargetDataSource(main); //default
dynamicDataSource.setDataSources(targetDataSources);
return dynamicDataSource;
}
#Bean
public SqlSessionFactory sqlSessionFactory(
#Qualifier("dynamicDataSource") DataSource dynamicDataSource)
throws Exception {
SqlSessionFactoryBean bean = new SqlSessionFactoryBean();
bean.setDataSource(dynamicDataSource);
bean.setMapperLocations(new PathMatchingResourcePatternResolver().getResources("classpath*:mappings/**/*.xml"));
return bean.getObject();
}
#Bean(name = "sqlSessionTemplate")
public SqlSessionTemplate sqlSessionTemplate(
#Qualifier("sqlSessionFactory") SqlSessionFactory sqlSessionFactory)
throws Exception {
return new SqlSessionTemplate(sqlSessionFactory);
}
#Bean
public PlatformTransactionManager transactionManager(
#Qualifier("dynamicDataSource") DataSource dynamicDataSource
) {
return new DataSourceTransactionManager(dynamicDataSource);
}
}
define an AOP to control which Dao method use which datasource. Dao is interface to access DB via mybatis.
import org.aspectj.lang.JoinPoint;
import org.aspectj.lang.annotation.After;
import org.aspectj.lang.annotation.Aspect;
import org.aspectj.lang.annotation.Before;
import org.aspectj.lang.annotation.Pointcut;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.core.annotation.Order;
import org.springframework.stereotype.Component;
#Aspect
#Order(-1)
#Component
public class DynamicDataSourceAspect {
private final static Logger logger = LoggerFactory.getLogger(DynamicDataSourceAspect.class);
private final String[] QUERY_PREFIX = {
"select","get","find","query","quickGet"
};
#Pointcut("execution( * com.biz.dao..*.*(..))")
public void daoAspect() {
}
#Before("daoAspect()")
public void beforeDao(JoinPoint point) {
boolean isQueryMethod = isQueryMethod(point.getSignature().getName());
if (isQueryMethod) {
switchDataSource("read");
}
}
#After("daoAspect()")
public void afterDao(JoinPoint point) {
restoreDataSource();
}
//===============================private method
private void switchDataSource(String key) {
if (!DynamicDataSourceContextHolder.containDataSourceKey(key)) {
logger.debug("======>DataSource [{}] doesn't exist, use default DataSource [{}] " + key);
} else {
// switch ds
DynamicDataSourceContextHolder.setDataSourceKey(key);
logger.debug("======>Switch DataSource to " + DynamicDataSourceContextHolder.getDataSourceKey());
}
}
private void restoreDataSource() {
// reset to default ds
DynamicDataSourceContextHolder.clearDataSourceKey();
}
private boolean isQueryMethod(String methodName) {
for (String prefix : QUERY_PREFIX) {
if (methodName.startsWith(prefix)) {
return true;
}
}
return false;
}
}
Configure two beans with two different set of configs in app.props
For one configuration you can use this (beanName = dataSource1):
#Configuration
#EnableRetry
public class DataSourceConfiguration {
#Value("${datasource1.username}")
private String username;
#Value("${datasource1.password}")
private String password;
#Value("${datasource1.url}")
private String connection;
#Bean(name = "dataSource1")
#Primary
public DataSource mainDataSource() {
DriverManagerDataSource dataSource = new DriverManagerDataSource();
dataSource.setDriverClassName("com.mysql.cj.jdbc.Driver");
dataSource.setUrl(connection);
dataSource.setUsername(username);
dataSource.setPassword(password);
return dataSource;
}
}
I'm trying now to configure custom ConfigSource in my Quarkus App. Like in many other manuals i'm created my own DatabaseSourceConfig and implements org.eclipse.microprofile.config.spi.ConfigSource interface.I registered my ConfigSource in:
/META-INF/services/org.eclipse.microprofile.config.spi.ConfigSource
There is my ConfigSource:
public class DatabaseConfigSource implements ConfigSource {
private DataSource dataSource;
public DatabaseConfigSource() {
try {
dataSource = (DataSource) new InitialContext().lookup("openejb:Resource/config-source-database");
} catch (final NamingException e) {
throw new IllegalStateException(e);
}
}
#Override
public Map<String, String> getProperties() {
// Implementing Method
}
#Override
public String getValue(final String propertyName) {
// Implementing Method
}
#Override
public String getName() {
return DatabaseConfigSource.class.getSimpleName();
}
}
But this not working for Quarkus because of JNDI name. I need to use CDI. I was trying to use something like this:
#Inject
#io.quarkus.agroal.DataSource("my_connection")
AgroalDataSource usersDataSource;
and declare this connection in application.properties but it didn't help me. I'm getting all the time NULL Exception.
Maybe someone have ideas, how can i get DB connection there without to use JNDI namespace?
You can obtain the data source via
AgroalDataSource dataSource = Arc.container()
.instance(AgroalDataSource.class, new DataSource.DataSourceLiteral("my_connection"))
.get();
You'll need to do this somewhere else than the constructor though, I think, because the ConfigSource instance is created before CDI is fully booted. You can cache the obtained data source instance then to avoid executing this multiple times.
I found some answer myself, maybe it will be useful also for other ppl.
Like #Janmartiška said, CDI booted later, than ConfigSource, that's why i don't see any way to inject my connection via CDI.
I was created some HibernateUtil Class:
package org.myproject.config;
import java.util.Properties;
import org.hibernate.SessionFactory;
import org.hibernate.boot.registry.StandardServiceRegistryBuilder;
import org.hibernate.cfg.Configuration;
import org.hibernate.service.ServiceRegistry;
import org.myproject.entities.ConfigurationsEntity;
public class HibernateUtil {
private static SessionFactory sessionFactory;
private static SessionFactory buildSessionFactory() {
try {
Properties props = new Properties();
props.setProperty("hibernate.connection.url", "jdbc:mysql://[db-host]:[db-port]/db_name");
props.setProperty("hibernate.connection.driver_class", "com.mysql.cj.jdbc.Driver");
props.setProperty("hibernate.connection.username", "username");
props.setProperty("hibernate.connection.password", "password");
props.setProperty("hibernate.current_session_context_class", "thread");
props.setProperty("hibernate.dialect", "org.hibernate.dialect.MySQL8Dialect");
Configuration configuration = new Configuration();
configuration.addProperties(props);
configuration.addAnnotatedClass(ConfigurationsEntity.class);
System.out.println("Hibernate Configuration loaded");
ServiceRegistry serviceRegistry = new StandardServiceRegistryBuilder().applySettings(configuration.getProperties()).build();
System.out.println("Hibernate serviceRegistry created");
SessionFactory sessionFactory = configuration.buildSessionFactory(serviceRegistry);
return sessionFactory;
}
catch (Throwable ex) {
// Make sure you log the exception, as it might be swallowed
System.err.println("Initial SessionFactory creation failed." + ex);
throw new ExceptionInInitializerError(ex);
}
}
public static SessionFactory getSessionFactory() {
if(sessionFactory == null) sessionFactory = buildSessionFactory();
return sessionFactory;
}
}
than i used it in my SourceConfig:
package org.myproject.config;
import io.quarkus.runtime.annotations.RegisterForReflection;
import org.eclipse.microprofile.config.spi.ConfigSource;
import org.hibernate.Session;
import org.hibernate.SessionFactory;
import org.myproject.entities.ConfigurationsEntity;
import javax.persistence.NoResultException;
import javax.persistence.Query;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
#RegisterForReflection
public class DatabaseSourceConfig implements ConfigSource {
public SessionFactory sessionFactory;
public Session currentSession;
public DatabaseSourceConfig() {
sessionFactory = HibernateUtil.getSessionFactory();
this.checkFactoryConnection();
}
public void checkFactoryConnection() {
if (currentSession == null || (currentSession != null && !currentSession.isOpen())) {
try {
currentSession = sessionFactory.getCurrentSession();
} catch (NullPointerException e) {
currentSession = sessionFactory.openSession();
}
}
}
#Override
public Map<String, String> getProperties() {
// Implementing Method
}
#Override
public String getValue(String propertyName) {
this.checkFactoryConnection();
ConfigurationsEntity conf = new ConfigurationsEntity();
currentSession.beginTransaction();
try {
Query query = currentSession.createNamedQuery("Configuration.selectOne", ConfigurationsEntity.class);
query.setParameter("name", propertyName);
conf = (ConfigurationsEntity) query.getSingleResult();
currentSession.getTransaction().commit();
} catch (Exception ex) {
currentSession.getTransaction().rollback();
}
return conf.getValue();
}
#Override
public String getName() {
return DatabaseSourceConfig.class.getSimpleName();
}
}
Now i can use my ConfigSource in other classes like:
#Inject
#ConfigProperty(name = "[property-name-like-in-db]")
public String someProperty;
After my further research it was found that ConfigSource has no access to CDi and application.properties. That is why there is nothing left but to establish a connection to the database in the manner described above.
However, I did a little editing of the example. I cached properties from the database and created a #ApplicationScoped Bean that looks into the database once every 5 minutes to see whether one of properties "updated_at" has a timestamp later than others from Bean loaded properties.
However, I have to say that according to Quarkus and Apache developers - this violates “immutable deployment” and is not planned to change the application settings during runtime. So it depends on you whether you write it in the app or not.
I have 1 spring context named PersistenceJPAConfig. Now I want to configure a spring batch and for that I have added a new class with #Configuration annotation and #EnableBatchProcessing. After adding new configuration class, I got error trying to use repository methods: nested exception is javax.persistence.TransactionRequiredException: no transaction is in progress. I know this is because I have a parent spring context and a child context which means I will have 2 instance for every repository and every service. I have tried to exclude repository scanning and service scanning with:
#ComponentScan(useDefaultFilters = false,
excludeFilters = {#ComponentScan.Filter(Repository.class), #ComponentScan.Filter(Service.class), #ComponentScan.Filter(Configuration.class)})
but it's not working. The only solution until now is to move all the beans from the second configuration to the first one, but I don't want that. How to solve this conflict between the contexts?
Main context:
package com.netoptics.server;
import java.util.Properties;
import javax.persistence.EntityManagerFactory;
import javax.sql.DataSource;
import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.dao.annotation.PersistenceExceptionTranslationPostProcessor;
import org.springframework.data.jpa.repository.config.EnableJpaRepositories;
import org.springframework.jdbc.datasource.DriverManagerDataSource;
import org.springframework.orm.jpa.JpaTransactionManager;
import org.springframework.orm.jpa.JpaVendorAdapter;
import org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean;
import org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter;
import org.springframework.transaction.PlatformTransactionManager;
import org.springframework.transaction.annotation.EnableTransactionManagement;
import com.websilicon.security.SecurityGlobals;
import com.websilicon.util.AppConfig;
#Configuration
#EnableTransactionManagement
#EnableJpaRepositories({"com.whitelist.manager.repositories", "com.wsnms", "com.websilicon"})
public class PersistenceJPAConfig {
#Bean
public LocalContainerEntityManagerFactoryBean entityManagerFactory() {
LocalContainerEntityManagerFactoryBean entityManagerFactoryBean = new LocalContainerEntityManagerFactoryBean();
entityManagerFactoryBean.setDataSource(dataSource());
entityManagerFactoryBean.setPackagesToScan("com.whitelist.manager", "com.wsnms", "com.websilicon");
JpaVendorAdapter vendorAdapter = new HibernateJpaVendorAdapter();
entityManagerFactoryBean.setJpaVendorAdapter(vendorAdapter);
entityManagerFactoryBean.setJpaProperties(additionalProperties());
return entityManagerFactoryBean;
}
#Bean
public PlatformTransactionManager transactionManager(EntityManagerFactory entityManagerFactory) {
JpaTransactionManager transactionManager = new JpaTransactionManager();
transactionManager.setEntityManagerFactory(entityManagerFactory);
return transactionManager;
}
#Bean
public DataSource dataSource() {
String databaseDriver = AppConfig.getInstance().getString("dataDatabaseDriver", "");
String databaseUrl = AppConfig.getInstance().getString("dataDatabaseUrl", "");
String databaseUsername = AppConfig.getInstance().getString("dataDatabaseUsername", "");
String dataDatabasePassword = AppConfig.getInstance().getPassword("dataDatabasePassword", SecurityGlobals.KEY, "");
DriverManagerDataSource dataSource = new DriverManagerDataSource();
dataSource.setDriverClassName(databaseDriver);
dataSource.setUrl(databaseUrl);
dataSource.setUsername(databaseUsername);
dataSource.setPassword(dataDatabasePassword);
return dataSource;
}
#Bean
public PersistenceExceptionTranslationPostProcessor exceptionTranslation() {
return new PersistenceExceptionTranslationPostProcessor();
}
private Properties additionalProperties() {
Properties properties = new Properties();
properties.setProperty("hibernate.hbm2ddl.auto", "none");
properties.setProperty("hibernate.dialect", "org.hibernate.dialect.PostgreSQL82Dialect");
properties.setProperty("hibernate.show_sql", "false");
properties.setProperty("hibernate.jdbc.batch_size", "1000");
return properties;
}
}
Second context for configuring spring batch:
#Configuration
#EnableBatchProcessing
#ComponentScan(useDefaultFilters = false,
excludeFilters = {#ComponentScan.Filter(Repository.class), #ComponentScan.Filter(Service.class)})
public class SaveImsiCSVBatchConfig {
#Autowired
public JobBuilderFactory jobBuilderFactory;
#Autowired
public StepBuilderFactory stepBuilderFactory;
#Autowired
public DataSource dataSource;
#Autowired
private Environment environment;
#Autowired
private WmAdminImsisResourceHelper wmAdminImsisResourceHelper;
#Bean
public JobRepository jobRepository() {
MapJobRepositoryFactoryBean factoryBean = new MapJobRepositoryFactoryBean(new ResourcelessTransactionManager());
try {
return factoryBean.getObject();
} catch (Exception e) {
e.printStackTrace();
return null;
}
}
#Bean
public JobLauncher jobLauncher(JobRepository jobRepository) {
SimpleJobLauncher jobLauncher = new SimpleJobLauncher();
jobLauncher.setJobRepository(jobRepository);
return jobLauncher;
}
#Bean
#StepScope
public JdbcCursorItemReader<WmPushedImsi> reader(#Value("#{jobParameters['sortProperty']}") String sortProperty,
#Value("#{jobParameters['sortValue']}") String sortValue, #Value("#{jobParameters['username']}") String usernameFilter,
#Value("#{jobParameters['imsinumber']}") String imsiNumberFilter) {
JdbcCursorItemReader<WmPushedImsi> reader = new JdbcCursorItemReader<>();
reader.setDataSource(dataSource);
String sql =
"select us.username, wp.imsinumber, wp.startdate, wp.expiredate, case when wp.failedpushbatchid is not null or wp.faileddeletebatchid is not null then 'Yes' ELSE 'No' end as dirty from\n"
+ "wm_pushed_imsi wp INNER JOIN wm_admin_user wu on wp.userid = wu.id INNER JOIN users us on wu.userid = us.id";
if (usernameFilter != null && imsiNumberFilter != null) {
sql += " AND us.username LIKE '%" + usernameFilter + "%' AND wp.imsinumber LIKE '%" + imsiNumberFilter + "%'";
} else if (usernameFilter != null) {
sql += " AND us.username LIKE '%" + usernameFilter + "%'";
} else if (imsiNumberFilter != null) {
sql += " AND wp.imsinumber LIKE '%" + imsiNumberFilter + "%'";
}
if (sortProperty != null) {
sql += " order by " + sortProperty + " " + sortValue;
}
reader.setSql(sql);
reader.setRowMapper(new ImsiRowMapper());
return reader;
}
#Bean
public ImsiProcessor processor() {
return new ImsiProcessor();
}
#Bean
#StepScope
public FlatFileItemWriter<WmPushedImsi> writer(#Value("#{jobParameters['currentDate']}") Date currentDate) {
wmAdminImsisResourceHelper.createDirectoryForSavingCsv();
String fileName = wmAdminImsisResourceHelper.createFileNameForCsv(currentDate) + environment.getProperty("CSVEXTENSION");
String columnsTitle = Arrays.toString(new String[] {environment.getProperty("CSV_IMSINUMBER"), environment.getProperty("CSV_USERNAME"),
environment.getProperty("CSV_STARTDATE"), environment.getProperty("CSV_EXPIREDATE"), environment.getProperty("CSV_DIRTY")});
FlatFileItemWriter<WmPushedImsi> writer = new FlatFileItemWriter<>();
writer.setResource(new FileSystemResource(fileName));
writer.setHeaderCallback(writerHeader -> writerHeader.write(columnsTitle.substring(1, columnsTitle.length() - 1)));
writer.setLineAggregator(new DelimitedLineAggregator<>() {
{
setDelimiter(",");
setFieldExtractor(new BeanWrapperFieldExtractor<>() {
{
setNames(new String[] {WmPushedImsi_.IMSI_NUMBER, "username", WmPushedImsi_.START_DATE, WmPushedImsi_.EXPIRE_DATE, "dirty"});
}
});
}
});
return writer;
}
#Bean
public Step stepToCreateCsvFile() {
return stepBuilderFactory.get(Objects.requireNonNull(environment.getProperty("CSV_STEP_CREATE_FILE"))).<WmPushedImsi, WmPushedImsi>chunk(50000)
.reader(reader("", "", "", "")).processor(processor()).writer(writer(null)).build();
}
#Bean
public Step stepToDeleteFileAndCreateArchive() {
FileArchiveAndDeletingTasklet task = new FileArchiveAndDeletingTasklet();
task.setWmAdminImsisResourceHelper(wmAdminImsisResourceHelper);
task.setDateString(environment.getProperty("CSV_DATE"));
return stepBuilderFactory.get(Objects.requireNonNull(environment.getProperty("CSV_STEP_CREATE_ARCHIVE"))).tasklet(task).build();
}
#Bean
public Job exportImsiCSVJob() {
return jobBuilderFactory.get(Objects.requireNonNull(environment.getProperty("CSV_EXPORT_JOB"))).incrementer(new RunIdIncrementer())
.start(stepToCreateCsvFile()).next(stepToDeleteFileAndCreateArchive()).build();
}
#Bean
public JobRegistryBeanPostProcessor jobRegistryBeanPostProcessor(JobRegistry jobRegistry) {
JobRegistryBeanPostProcessor jobRegistryBeanPostProcessor = new JobRegistryBeanPostProcessor();
jobRegistryBeanPostProcessor.setJobRegistry(jobRegistry);
return jobRegistryBeanPostProcessor;
}
public class ImsiRowMapper implements RowMapper<WmPushedImsi> {
#Override
public WmPushedImsi mapRow(ResultSet rs, int rowNum) throws SQLException {
WmPushedImsi wmPushedImsi = new WmPushedImsi();
wmPushedImsi.setImsiNumber(rs.getString(WmPushedImsi_.IMSI_NUMBER));
wmPushedImsi.setUsername(rs.getString("username"));
wmPushedImsi.setStartDate(rs.getDate(WmPushedImsi_.START_DATE));
wmPushedImsi.setExpireDate(rs.getDate(WmPushedImsi_.EXPIRE_DATE));
wmPushedImsi.setDirty(rs.getString("dirty"));
return wmPushedImsi;
}
}
}
You are using the MapJobRepository with a ResourcelessTransactionManager. With this configuration, there are no transactions on Spring Batch side. Hence the error no transaction is in progress.
You need to configure a JobRepository with the transaction manager you defined in your PersistenceJPAConfig. To do this, you have to define a bean of type BatchConfigurer and override getTransactionManager. Here is an example:
#Bean
public BatchConfigurer batchConfigurer() {
return new DefaultBatchConfigurer() {
#Override
public PlatformTransactionManager getTransactionManager() {
return new MyTransactionManager();
}
};
}
For more details, please check the Java config section of the reference documentation. Please note that this requires Spring Batch v4.1+.
I'm trying to develop a Spring Boot(1.5) app that needs to listen to SQS queues from two different AWS accounts.
Is it possible to create a listener using JmsListener annotation? I've checked that the rights are correct, I'm able to get the queue url with getQueueUrl() and setting the right account id with setQueueOwnerAWSAccountId().
Below is the code I've used for the listener that's under main account. Trying to use that for the queue on the other account, gives error
HTTPStatusCode: 400 AmazonErrorCode: AWS.SimpleQueueService.NonExistentQueue
com.amazonaws.services.sqs.model.QueueDoesNotExistException: The specified queue does not exist for this wsdl version.
Queue reader class
#Service
public class QueueReader {
#JmsListener(destination = "queue-name")
public void messageReceived(#Payload String message) {
// message received
}
}
Queue config class
#Configuration
#EnableJms
public class QueueReaderConfig {
SQSConnectionFactory connectionFactory = SQSConnectionFactory.builder().withRegion(Region.getRegion(Regions.EU_WEST_1))
.withAWSCredentialsProvider(new DefaultAWSCredentialsProviderChain())
.build();
#Bean
public DefaultJmsListenerContainerFactory jmsListenerContainerFactory() {
DefaultJmsListenerContainerFactory factory =
new DefaultJmsListenerContainerFactory();
factory.setConnectionFactory(this.connectionFactory);
factory.setDestinationResolver(new DynamicDestinationResolver());
factory.setConcurrency("3-10");
factory.setSessionAcknowledgeMode(Session.CLIENT_ACKNOWLEDGE);
return factory;
}
#Bean
public JmsTemplate defaultJmsTemplate() {
return new JmsTemplate(this.connectionFactory);
}
}
I was stuck with the same issue. I found a workaround by creating custom DestinationResolver and setting it in the "DefaultJmsListenerContainerFactory" and "JmsTemplate".
Also, in the "CustomDynamicDestinationResolver" lookup the queue by ownerAccountId.
queue = ((SQSSession) session).createQueue(queueName, ownerAccountId);
Listen to queue by using the connection factory.
#JmsListener(destination = "MyQueue", containerFactory = "customJmsListenerContainerFactory")
public void process(String message) throws IOException {
A bit late but, I hope this helps someone like me looking for a solution.
Thanks,
Akshay
import com.amazon.sqs.javamessaging.ProviderConfiguration;
import com.amazon.sqs.javamessaging.SQSConnectionFactory;
import com.amazon.sqs.javamessaging.SQSSession;
import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.sqs.AmazonSQSClientBuilder;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.jms.config.DefaultJmsListenerContainerFactory;
import org.springframework.jms.core.JmsTemplate;
import org.springframework.jms.support.destination.DestinationResolver;
import org.springframework.util.Assert;
import javax.jms.*;
#Configuration
public class CustomJmsConfig {
private static final Logger LOGGER = LoggerFactory.getLogger(CustomJmsConfig.class);
#Value("${copies.processor.concurrency:5}")
private String concurrency;
#Value("${owner.account.id:1234}")
private String ownerAccountId;
SQSConnectionFactory customConnectionFactory =
new SQSConnectionFactory(
new ProviderConfiguration(),
AmazonSQSClientBuilder.standard().withRegion(Regions.EU_CENTRAL_1).withCredentials(new DefaultAWSCredentialsProviderChain())
);
#Bean
public DefaultJmsListenerContainerFactory customJmsListenerContainerFactory() {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
factory.setConnectionFactory(this.customConnectionFactory);
factory.setDestinationResolver(new CustomDynamicDestinationResolver(ownerAccountId));
factory.setConcurrency(concurrency);
factory.setSessionAcknowledgeMode(Session.CLIENT_ACKNOWLEDGE);
return factory;
}
#Bean
public JmsTemplate customJmsTemplate() {
JmsTemplate jmsTemplate = new JmsTemplate(this.customConnectionFactory);
jmsTemplate.setDestinationResolver(new CustomDynamicDestinationResolver(ownerAccountId));
return jmsTemplate;
}
public static class CustomDynamicDestinationResolver implements DestinationResolver {
private String ownerAccountId;
public CustomDynamicDestinationResolver(String ownerAccountId) {
this.ownerAccountId = ownerAccountId;
}
#Override
public Destination resolveDestinationName(Session session, String destinationName, boolean pubSubDomain) throws JMSException {
Assert.notNull(session, "Session must not be null");
Assert.notNull(destinationName, "Destination name must not be null");
if (pubSubDomain) {
return resolveTopic(session, destinationName);
} else {
return resolveQueue(session, destinationName);
}
}
protected Topic resolveTopic(Session session, String topicName) throws JMSException {
return session.createTopic(topicName);
}
protected Queue resolveQueue(Session session, String queueName) throws JMSException {
Queue queue;
LOGGER.info("Getting destination for libraryOwnerAccountId: {}, queueName: {}", libraryOwnerAccountId, queueName);
if (libraryOwnerAccountId != null && session instanceof SQSSession) {
queue = ((SQSSession) session).createQueue(queueName, ownerAccountId);
} else {
queue = session.createQueue(queueName);
}
return queue;
}
}
}
My solution is based on Akshay's response. I'm also using custom DestinationResolver. However my implementation doesn't need fixed ownerAccountId:
public class SqsDynamicDestinationResolver extends DynamicDestinationResolver {
#Override
protected Queue resolveQueue(Session session, String queueName) throws JMSException {
if (session instanceof SQSSession) {
SQSSession sqsSession = (SQSSession) session;
String[] parts = queueName.split(":");
if (parts.length == 2) {
return sqsSession.createQueue(parts[1], parts[0]);
}
}
return super.resolveQueue(session, queueName);
}
The resolver is then set in JmsTemplate and DefaultJmsListenerContainerFactory.
You can either specify queue name simply as "whatever_queue_name" or with owner account id delimited with colon, for example "0000216587:whatever_queue_name".
Please note, that this solution will not work, if the ownerAccountId is from different region! In that case you would need separate configurations.
These are my classes and configurations,
ServiceImpl.java
#Override
#Transactional(propagation = Propagation.REQUIRED,rollbackFor = Exception.class)
public InsertUpdateSuccessBean insertNewSiteEntry(SiteEntry newSiteBean) throws SQLException {
dao.insert1();
dao.insert2();
except();
return new InsertUpdateSuccessBean(true,LoggerConstants.NEW_SITE_ENTRY_SUCCESS);
}
private void except()throws SQLException{
throw new SQLException();
}
DAOImpl.java
#Repository
public class DAOImpl implements DAO
{
#Autowired
private DataSourceTransactionManager transManager;
#Autowired
private CommonDAOUtils commonUtils;
private static final Logger logger = Logger.getLogger(HunterDAOImpl.class) ;
#Override
public Integer insert1() throws SQLException {
Integer insertNumRows = 0;
Connection connectionObject = transManager.getDataSource().getConnection();
PreparedStatement psObject = connectionObject.prepareStatement(
SQLQuery );
insertNumRows = psObject.executeUpdate();
commonUtils.closer(null,psObject,connectionObject);
// Function to close open connection resultsets statement objects
return insertNumRows;
}
#Override
public Integer insert2() throws SQLException {
Integer insertNumRows = 0;
Connection connectionObject = transManager.getDataSource().getConnection();
PreparedStatement psObject = connectionObject.prepareStatement(
SQLQuery );
insertNumRows = psObject.executeUpdate();
commonUtils.closer(null,psObject,connectionObject);
// Function to close open connection resultsets statement objects
return insertNumRows;
}
}
AppConfig.java
import com.interceptors.AppInterceptor;
import com.utils.Constants;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor;
import org.springframework.beans.factory.annotation.Configurable;
import org.springframework.context.EnvironmentAware;
import org.springframework.context.annotation.*;
import org.springframework.context.support.PropertySourcesPlaceholderConfigurer;
import org.springframework.core.env.Environment;
import org.springframework.core.io.ClassPathResource;
import org.springframework.http.client.ClientHttpRequestFactory;
import org.springframework.http.client.HttpComponentsClientHttpRequestFactory;
import org.springframework.jdbc.datasource.DataSourceTransactionManager;
import org.springframework.jdbc.datasource.DriverManagerDataSource;
import org.springframework.stereotype.Component;
import org.springframework.transaction.annotation.EnableTransactionManagement;
import org.springframework.web.client.RestTemplate;
import org.springframework.web.servlet.config.annotation.DefaultServletHandlerConfigurer;
import org.springframework.web.servlet.config.annotation.EnableWebMvc;
import org.springframework.web.servlet.config.annotation.InterceptorRegistry;
import org.springframework.web.servlet.config.annotation.WebMvcConfigurerAdapter;
import org.springframework.web.servlet.view.InternalResourceViewResolver;
#Component
#Configuration
#EnableWebMvc
#Configurable
#ComponentScan(basePackages = {Constants.BASE_PACKAGE})
#EnableAspectJAutoProxy(proxyTargetClass = true)
#PropertySource(Constants.DB_PROPERTIES_PATH)
#EnableTransactionManagement
public class AppConfig extends WebMvcConfigurerAdapter implements EnvironmentAware{
#Autowired
Environment environmentObject;
#Bean
public static PropertySourcesPlaceholderConfigurer propertySourcesPlaceholderConfigurer(){
PropertySourcesPlaceholderConfigurer placeHolder =new PropertySourcesPlaceholderConfigurer();
placeHolder.setLocation(new ClassPathResource(Constants.PROP_FILE_NAME));
return placeHolder;
}
#Bean
public InternalResourceViewResolver viewResolver(){
InternalResourceViewResolver internalResourceViewResolver = new InternalResourceViewResolver();
internalResourceViewResolver.setPrefix(Constants.ROOT_PATH);
internalResourceViewResolver.setSuffix(Constants.JSP_DOT_EXTENSION);
return internalResourceViewResolver;
}
#Override
public void configureDefaultServletHandling(DefaultServletHandlerConfigurer configurer) {
configurer.enable();
}
#Bean
public DriverManagerDataSource dataSource(){
DriverManagerDataSource driverManagerDataSource = new DriverManagerDataSource();
driverManagerDataSource.setDriverClassName(environmentObject.getProperty(Constants.JDBC_DRIVERCLASS_NAME_PROP_FILE));
driverManagerDataSource.setUrl(environmentObject.getProperty(Constants.JDBC_URL_PROP_FILE));
driverManagerDataSource.setUsername(environmentObject.getProperty(Constants.JDBC_USERNAME_PROP_FILE));
driverManagerDataSource.setPassword(new String(Constants.PASSWORD));
return driverManagerDataSource;
}
#Bean
public AutowiredAnnotationBeanPostProcessor postProcessorBean(){
return new AutowiredAnnotationBeanPostProcessor();
}
private ClientHttpRequestFactory getClientHttpRequestFactory() {
int timeout = 5000;
HttpComponentsClientHttpRequestFactory clientHttpRequestFactory = new HttpComponentsClientHttpRequestFactory();
clientHttpRequestFactory.setConnectTimeout(timeout);
return clientHttpRequestFactory;
}
#Bean
public DataSourceTransactionManager transactionManager(){
final DataSourceTransactionManager transactionManager = new DataSourceTransactionManager(dataSource());
transactionManager.setRollbackOnCommitFailure(true);
return transactionManager;
}
}
And I also have an around advice for my project:
#Aspect
#Component
public class AroundAdvice
{
private static final Logger logger = Logger.getLogger(AroundAdvice.class) ;
// Add pointcuts here - package names here for around advice
#Around("execution(* com.beans.*.*(..)) || " +
"execution(* com.config.*.*(..)) || " +
"execution(* com.controller.*.*(..)) || " +
"execution(* com.dao.*.*(..)) || " +
"execution(* com.service.*.*(..)) || " +
"execution(* com.utils.*.*(..)) || "+
"execution(* com.interceptors.*.*(..))")
public Object aroundAdviceMethod(ProceedingJoinPoint proceedingObject)throws Throwable{
MethodSignature methodSignObject = (MethodSignature) proceedingObject.getSignature();
logger.debug(Constants.EXECUTING_METH_STR + methodSignObject.getMethod());
Object value = null;
try {
value = proceedingObject.proceed();
} catch (Exception e) {
e.printStackTrace();
logger.error(Constants.EXCEPTION_ASPECT_STR+methodSignObject.getMethod());
logger.error(Constants.EXCEPTION_MESSAGE,e);
throw e;
}
logger.debug(Constants.RETURN_STR+value);
return value;
}
}
On executing this flow, the inserts are successful, however when the exception is thrown, it is not rolling back. However, my logger reads that rolling back is initialized and done as follows,
14:11:51 DEBUG DataSourceTransactionManager:851 - Initiating transaction rollback
14:11:51 DEBUG DataSourceTransactionManager:325 - Rolling back JDBC transaction on Connection [org.postgresql.jdbc4.Jdbc4Connection#3b467b21]
14:11:51 DEBUG DataSourceTransactionManager:368 - Releasing JDBC Connection [org.postgresql.jdbc4.Jdbc4Connection#3b467b21] after transaction
Please let me know if I am missing something
The problem is your dao. You are opening connections yourself and therefor bypass all transaction management. Your dao is to complex just use a JdbcTemplate instead of your current code.
#Repository
public class DAOImpl implements DAO {
private static final Logger logger = Logger.getLogger(HunterDAOImpl.class) ;
private final JdbcTemplate jdbc;
public DAOImpl(DataSource dataSource) {
this.jdbc = new JdbcTemplate(dataSource);
}
#Override
public Integer insert1() throws SQLException {
return jdbc.update(SQLQuery);
}
#Override
public Integer insert2() throws SQLException {
return jdbc.update(SQLQuery);
}
}
This will do exactly the same as your code, with one main difference it will use the Connection opened when the transaction was started. Your sample used 3 separate connections and thus 3 individual transactions instead of 1 single transaction.