We Keep Coding

Ignite Performance: How to tune cache write performance for ignite thin client?

I am doing a simple POC to write a large volume of entries to ignite cache following client server mode, observed below during testing;
1) If thin client and server reside on same host , it takes around ~10 minutes to persist 1 million entries into two cache.
2) If thin client and server reside on different hosts , it takes around ~4 minutes to persist just 500 entries into two cache.This looks very bad.
I am not able to justify this significant delay in case2(mode we want to adopt for implementation) even if we take some network latency into account.I am wondering if its something to do with my cache configuration which is as below?
<bean id="grid.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="workDirectory" value="/path/"/>
<property name="activeOnStart" value="true"/>
<property name="autoActivationEnabled" value="true"/>
<property name="deploymentMode" value="SHARED"/>
<property name="igniteInstanceName" value="test"/>
<property name="dataStorageConfiguration">
<bean class="org.apache.ignite.configuration.DataStorageConfiguration">
<property name="defaultDataRegionConfiguration">
<bean class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="persistenceEnabled" value="true"/>
</bean>
</property>
<property name="storagePath" value="/path/"/>
</bean>
</property>
<!--
For better performance set this property to false in case
peer deployment is not used.
Default value is true.
-->
<property name="peerClassLoadingEnabled" value="false"/>
<property name="cacheConfiguration">
<!--
Specify list of cache configurations here. Any property from
CacheConfiguration interface can be configured here.
Note that absolutely all configuration properties are optional.
-->
<list>
<bean parent="cache-template">
<!-- Cache name is 'testcache1'. -->
<property name="name" value="testcache1"/>
</bean>
<bean parent="cache-template">
<!-- Cache name is 'testcache2'. -->
<property name="name" value="testcache2"/>
</bean>
</list>
</property>
</bean>
<!-- Template for all example cache configurations. -->
<bean id="cache-template" abstract="true" class="org.apache.ignite.configuration.CacheConfiguration">
<!-- REPLICATED cache mode. -->
<property name="cacheMode" value="REPLICATED"/>
<!-- Set synchronous rebalancing (default is asynchronous). -->
<property name="rebalanceMode" value="SYNC"/>
<!-- Set to FULL_SYNC for examples, default is PRIMARY_SYNC. -->
<property name="writeSynchronizationMode" value="FULL_SYNC"/>
<property name="atomicityMode" value="TRANSACTIONAL"/>
</bean>
Thin Client Code:
public class IgniteDataGridApplication {
static DateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSS");
private ClientCache<String, String> testcache1;
private ClientCache<String, String> testcache2;
public IgniteDataGridApplication() {
ClientConfiguration cfg = new ClientConfiguration().setAddresses("serverhostname.net:10800");
IgniteClient ignite = Ignition.startClient(cfg);
testcache1 = ignite.cache("testcache1");
testcache2 = ignite.cache("testcache2");
}
public static void main(String[] args) throws Exception {
IgniteDataGridApplication igniteDataGridApplication = new IgniteDataGridApplication();
igniteDataGridApplication.load();
}
private void load() throws Exception {
List<ThreadProducer> cacheMessages = new ArrayList<>();
for (int i = 1; i <= 1000000; i++) {
String testentry = i+"";
cacheMessages.add(new ThreadProducer("testKey" + i, testentry));
}
ExecutorService executorService = Executors.newFixedThreadPool(1000);
cacheMessages.forEach(executorService::submit);
executorService.shutdown();
executorService.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
}
class ThreadProducer implements Runnable {
private String String;
private String key;
public ThreadProducer(String key, String String) {
this.key = key;
this.String = String;
}
public void run() {
testcache1.putIfAbsent(key, String);
testcache2.putIfAbsent(key, String);
System.out.println("entry :: " + key + " :: " + sdf.format(Calendar.getInstance().getTime()));
}
}
}

Try profiling server nodes and the thin clients using JFR, JProfiler or another profiler of your choice to find a bottleneck that slows down the operation.
Make sure that in both cases the same number of nodes are present in the baseline topology. With improper configuration of the baseline topology data may be loaded only to one of the nodes.
You can try using API methods that load data in batches to improve the performance. ClientCache#putAll() is one of such methods.

Transaction don' work (Manager tx problem)

Good day!
My problem is when i transferred configuration for another DB from xml to #Bean my transactions is lost.... dont rollback and not work.
I see this when in DB after first insert created row (!), but in this method(transaction) start second insert i take Exception and row after first inset stay on DB!
This xml
<bean name="sqlSessionFactoryYarus" class="org.mybatis.spring.SqlSessionFactoryBean">
<property name="configLocation" value="/WEB-INF/MapperConfigYarus.xml" />
<property name="dataSource" ref="dataSourceYarus" />
</bean>
<bean class="org.mybatis.spring.mapper.MapperScannerConfigurer">
<property name="basePackage" value="ru.project.crm.mapper_yarus"/>
<property name="sqlSessionFactoryBeanName" value="sqlSessionFactoryYarus" />
</bean>
<bean id="transactionManagerYarus" class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
<property name="dataSource" ref="dataSourceYarus" />
<qualifier value="yarus"/>
</bean>
<tx:annotation-driven transaction-manager="transactionManagerYarus" />
this code (dataSource there is no this not to waste space)
#Component
#Scope("singleton")
#DependsOn("springApplicationContextHolder")
public class YarusConnectionConfig {
#Bean
public SqlSessionFactory sqlSessionFactoryYarus() throws Exception {
SqlSessionFactoryBean sqlSessionFactory = new SqlSessionFactoryBean();
sqlSessionFactory.setDataSource(dataSourceYarus());
sqlSessionFactory.setConfigLocation(new ClassPathResource("../MapperConfigYarus.xml"));
return sqlSessionFactory.getObject();
}
#Bean
public MapperScannerConfigurer yarusMapper() throws Exception {
MapperScannerConfigurer msc = new MapperScannerConfigurer();
msc.setSqlSessionFactoryBeanName("sqlSessionFactoryYarus");
msc.setBasePackage("ru.project.crm.mapper_yarus");
return msc;
}
#Bean
public DataSourceTransactionManager transactionManagerYarus() throws Exception {
DataSourceTransactionManager dataSourceTransactionManager = new DataSourceTransactionManager(dataSourceYarus());
return dataSourceTransactionManager;
}
}
And
All paces when i want to Transactional annotate #Transactional(value = "transactionManagerYarus")
And if i build project with xml Transactional works fine
BUT if build with #Bean its dont work...
Plesae Help me!
I use
1) Spring 4.3
2) MyBatis
3) Postgesql
4) Java 8

Also we find solution.
Problem was in dataSource
#Bean(destroyMethod = "close", name = "dataSourceYarus")
public ComboPooledDataSource dataSourceYarus() {
ComboPooledDataSource cpds = new ComboPooledDataSource();
//config connection
}
It is my bean, and i call this bean like method, for example
new DataSourceTransactionManager(dataSourceYarus());
I did not attach any importance to this, because in all example this is true.
BUT in xml configuration caused this like "Bean" on his name, i replace call in java-config to
new DataSourceTransactionManager(context.getBean("dataSourceYarus"))
and.... this work for me!
Because if call this method, every time creating new pool then transaction was over

Spring JDBC connection issue-Connections could not be acquired from the underlying database

I am stuck into a big trouble that i cannot connect the oracle 11g database form my Spring MVC application.
The error i am getting is
HTTP Status 500 - Request processing failed; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: Connections could not be acquired from the underlying database!
also,
in the stack trace i'm getting the error-
java.lang.ClassNotFoundException: oracle.jdbc.OracleDriver
If you can help me to resolve the issue, it will be a great help.
I am providing my configuration and coding details below:
Default-servlet.xml
<bean id="dataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource"
destroy-method="close">
<property name="driverClass" value="${jdbc.driverClassName}" />
<property name="jdbcUrl" value="${jdbc.url}" />
<property name="user" value="${jdbc.username}" />
<property name="password" value="${jdbc.password}" />
<property name="maxPoolSize" value="${jdbc.maxPoolSize}" />
<property name="minPoolSize" value="${jdbc.minPoolSize}" />
<property name="maxStatements" value="${jdbc.maxStatements}" />
<property name="testConnectionOnCheckout" value="${jdbc.testConnection}" />
</bean>
<bean id="userAuthenticationRepository"
class="com.era.repository.impl.UserAuthenticationRepositoryImpl">
<property name="dataSource" ref="dataSource" />
</bean>
UserAuthenticationRepositoryImpl.java
#Repository
public class UserAuthenticationRepositoryImpl implements UserAuthenticationRepository {
#Qualifier("dbDataSource")
private DataSource dataSource;
public void setDataSource(DataSource dataSource) {
this.dataSource = dataSource;
}
public User getUserAuthentication(User userToBeAuthenticated) {
// TODO Auto-generated method stub
String query = "select id, name, role from User where login =";
JdbcTemplate jdbcTemplate = new JdbcTemplate(dataSource);
StringBuilder queryString = new StringBuilder();
queryString.append(" SELECT ")
.append( "*" )
.append(" FROM table_name ")
.append(" WHERE login = ? ");
Object[] parameterList = { userToBeAuthenticated.getLogin() };
SqlRowSet dataRow = jdbcTemplate.queryForRowSet(queryString.toString(), parameterList);
if (dataRow.next()) {
System.out.println("Query executed successfully");
}
return null;
}

As you are using maven, note here you can't directly get Oracle driver jar to .m2 due to licence restriction, so you may need to manually download and place it to your repository.You may find this link helpful.

Hibernate Enable PostLoad Event

I'm currently trying to use the #PostLoad Annotation for some processing after my model has been loaded from the database. But at the moment it looks like that my method won't get triggered. I don't use the EntityManager so I'm looking for a way to enable this event bahavior.
My configuration looks like this:
<hibernate-configuration>
<session-factory>
<property name="hibernate.connection.provider_class">com.zaxxer.hikari.hibernate.HikariConnectionProvider
</property>
<property name="hibernate.connection.tinyInt1isBit">true</property>
<property name="hibernate.connection.driver_class">com.mysql.jdbc.Driver</property>
<property name="hibernate.connection.url">jdbc:mysql://localhost:3306/testing</property>
<property name="hibernate.connection.username">root</property>
<property name="hibernate.connection.password"></property>
<property name="hibernate.connection.pool_size">1</property>
<property name="hibernate.dialect">org.hibernate.dialect.MySQL5Dialect</property>
<property name="hibernate.hbm2ddl.auto">update</property>
<!-- OUTPUT STUFF -->
<property name="hibernate.show_sql">true</property>
<property name="hibernate.use_sql_comments">false</property>
<property name="hibernate.format_sql">false</property>
<!-- SESSION CONTEXT -->
<property name="hibernate.current_session_context_class">thread</property>
<!-- CONNECTION POOL hikariCP -->
<property name="hibernate.hikari.maximumPoolSize">25</property>
<property name="hibernate.hikari.idleTimeout">30000</property>
<property name="hibernate.hikari.dataSource.url">jdbc:mysql://localhost:3306/testing</property>
<property name="hibernate.hikari.dataSource.user">root</property>
<property name="hibernate.hikari.dataSource.password"></property>
<property name="hibernate.hikari.dataSource.cachePrepStmts">true</property>
<property name="hibernate.hikari.dataSource.prepStmtCacheSize">250</property>
<property name="hibernate.hikari.dataSource.prepStmtCacheSqlLimit">2048</property>
<property name="hibernate.hikari.dataSource.useServerPrepStmts">true</property>
<!-- Include the mapping entries -->
<mapping class="at.adtime.core.v1.model.User"/>
<mapping class="at.adtime.core.v1.model.Test"/>
</session-factory>
</hibernate-configuration>
Update:
I have a method called afterUserLoad which looks like this:
#PostLoad
public void afterUserLoad() {
ArrayList<String> computedIds = new ArrayList<String>();
for (Test test : this.Test) {
computedIds.add(test.getId());
}
this.setTestIds(computedIds);
}
It should load the Test List and put only the ids in an ArrayList.

#PostLoad handling is implemented at least since 5.2.17 version in Hibernate.

According with my research, #PostLoad, #PrePersist and other kinds of these JPA annotations do not work for Hibernate, instead of them, we can use Interceptors. I am using Hibernate-JPA and Spring with xml based configuration. With Spring I get a SessionFactory object, when i set the properties for this object I inject a MyInterceptor instance that extends EmptyInterceptor class, then you can overide several methods like postLoad, onSave, etc. When you implement this code, the overrided methods are executed according the life cycle of the operation. I have not tested the postLoad method because onSave method worked for me.
Update
Spring Java based configuration #JeffersonTavares:
Spring configuration class:
#Configuration
public class SpringConfig {
#Bean
public BasicDataSource basicDataSource(){ // org.apache.commons.dbcp2.BasicDataSource Object
BasicDataSource basicDataSource = new BasicDataSource();
basicDataSource.setDriverClassName("com.microsoft.sqlserver.jdbc.SQLServerDriver");
basicDataSource.setUrl("jdbc:sqlserver://172.125.25.7:1433;databaseName=LIST_FIRMAS;selectMethod=direct;");
basicDataSource.setUsername("user");
basicDataSource.setPassword("password");
return basicDataSource;
}
#Bean
public MyInterceptor myInterceptor(){ // Your interceptor
return new MyInterceptor();
}
#Bean(name = "sessionFactory")
public SessionFactory sessionFactory() throws IOException{
LocalSessionFactoryBean localSessionFactoryBean = new LocalSessionFactoryBean();
localSessionFactoryBean.setDataSource(basicDataSource()); // DataSource
localSessionFactoryBean.setAnnotatedClasses(
com.isapeg.timbrado.model.SalMinimo.class,
com.isapeg.timbrado.model.DeduccionCnom12.class,
com.isapeg.timbrado.model.PercepcionCnom12.class); // JPA entity classes ....
localSessionFactoryBean.setEntityInterceptor(myInterceptor()); // Setting your Interceptor
Properties properties = new Properties();
properties.put("hibernate.dialect", "org.hibernate.dialect.SQLServer2012Dialect");
properties.put("hibernate.current_session_context_class", "thread");
properties.put("hibernate.show_sql", "false");
localSessionFactoryBean.setHibernateProperties(properties); // Setting Hibernate Properties
localSessionFactoryBean.afterPropertiesSet(); // Session Factory initialization
return localSessionFactoryBean.getObject(); // Returning of SessionFactory Object
}
// other beans ....
}
Something to test session factory object:
ApplicationContext ctx = new AnnotationConfigApplicationContext(SpringConfig.class);
SessionFactory sessionFactory = (SessionFactory) ctx.getBean("sessionFactory");
if (sessionFactory == null) {
System.out.println("SessionFactory obj is null");
} else {
Session session = sessionFactory.openSession();
System.out.println("Session obj: " + session);
System.out.println("isOpen: " + session.isOpen() + " isConnected: " + session.isConnected());
session.close();
}
Update 2
How to implement an Interceptor here.

How to achieve synchronized thread in the spring MVC using solrj as a client connecting to the tomcat solr

guys,
At first, I show you the part of applicationContext.xml on ingerating HttpServer.
And I read the construcotr arguments from the solr.properties.
<!-- solr configuration-->
<bean id="solrHttpServer" class="com.augmentum.ksp.solr.server.SolrHttpServer" scope="prototype">
<constructor-arg value="${solr.url}" />
<constructor-arg value="${solr.socketTimeOut}" />
<constructor-arg value="${solr.connTimeOut}" />
<constructor-arg value="${solr.maxConnDefault}" />
<constructor-arg value="${solr.maxConnTotal}" />
<constructor-arg value="${solr.maxRetries}" />
<constructor-arg value="${solr.allowCompression}" />
<constructor-arg value="${solr.followRedirects}" />
</bean>
And following is our query method in DAO layer, we have many methods in service layer use the "queryCount" method at the same time.
#Repository
public class SolrBaseDaoImpl implements SolrBaseDao {
#Autowired
private SolrHttpServer solrHttpServer;
#Override
public synchronized int queryCount(String coreName, String queryExpression, Set<String> filterQuerys)
throws SolrServerException, IOException {
if (null == queryExpression || "".equals(queryExpression)) {
return 0;
}
String baseUrl = SolrUtil.getSolrBaseURL(solrHttpServer.getBaseURL());
solrHttpServer.setBaseURL(baseUrl + "/" + coreName);
SolrQuery query = new SolrQuery();
query.setQuery(queryExpression);
// set filter query
if (null != filterQuerys) {
for (String fq : filterQuerys) {
query.addFilterQuery(fq);
}
}
query.setStart(0);
query.setRows(Integer.MAX_VALUE);
QueryResponse rsp = solrHttpServer.query(query);
return rsp.getResults().size();
}
}
Thus, a problem occurred. Reported that "SolrRemoteException:undefined field *".
We think that the source of the error is that there is a lot of request access the "queryCount" method **at the same time, thus we add a "synchronized" decorateor to the method, so that the problem nerver occurred any more.
But, as we all know, when there is a lot of request access the method one by one, which perfom very slowly. That is terrible!
Could you tell me your soultion to the problem ? or the error is not caused by the thread synchronization？