We Keep Coding

Ignite Performance: How to tune cache write performance for ignite thin client?

I am doing a simple POC to write a large volume of entries to ignite cache following client server mode, observed below during testing;
1) If thin client and server reside on same host , it takes around ~10 minutes to persist 1 million entries into two cache.
2) If thin client and server reside on different hosts , it takes around ~4 minutes to persist just 500 entries into two cache.This looks very bad.
I am not able to justify this significant delay in case2(mode we want to adopt for implementation) even if we take some network latency into account.I am wondering if its something to do with my cache configuration which is as below?
<bean id="grid.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="workDirectory" value="/path/"/>
<property name="activeOnStart" value="true"/>
<property name="autoActivationEnabled" value="true"/>
<property name="deploymentMode" value="SHARED"/>
<property name="igniteInstanceName" value="test"/>
<property name="dataStorageConfiguration">
<bean class="org.apache.ignite.configuration.DataStorageConfiguration">
<property name="defaultDataRegionConfiguration">
<bean class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="persistenceEnabled" value="true"/>
</bean>
</property>
<property name="storagePath" value="/path/"/>
</bean>
</property>
<!--
For better performance set this property to false in case
peer deployment is not used.
Default value is true.
-->
<property name="peerClassLoadingEnabled" value="false"/>
<property name="cacheConfiguration">
<!--
Specify list of cache configurations here. Any property from
CacheConfiguration interface can be configured here.
Note that absolutely all configuration properties are optional.
-->
<list>
<bean parent="cache-template">
<!-- Cache name is 'testcache1'. -->
<property name="name" value="testcache1"/>
</bean>
<bean parent="cache-template">
<!-- Cache name is 'testcache2'. -->
<property name="name" value="testcache2"/>
</bean>
</list>
</property>
</bean>
<!-- Template for all example cache configurations. -->
<bean id="cache-template" abstract="true" class="org.apache.ignite.configuration.CacheConfiguration">
<!-- REPLICATED cache mode. -->
<property name="cacheMode" value="REPLICATED"/>
<!-- Set synchronous rebalancing (default is asynchronous). -->
<property name="rebalanceMode" value="SYNC"/>
<!-- Set to FULL_SYNC for examples, default is PRIMARY_SYNC. -->
<property name="writeSynchronizationMode" value="FULL_SYNC"/>
<property name="atomicityMode" value="TRANSACTIONAL"/>
</bean>
Thin Client Code:
public class IgniteDataGridApplication {
static DateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSS");
private ClientCache<String, String> testcache1;
private ClientCache<String, String> testcache2;
public IgniteDataGridApplication() {
ClientConfiguration cfg = new ClientConfiguration().setAddresses("serverhostname.net:10800");
IgniteClient ignite = Ignition.startClient(cfg);
testcache1 = ignite.cache("testcache1");
testcache2 = ignite.cache("testcache2");
}
public static void main(String[] args) throws Exception {
IgniteDataGridApplication igniteDataGridApplication = new IgniteDataGridApplication();
igniteDataGridApplication.load();
}
private void load() throws Exception {
List<ThreadProducer> cacheMessages = new ArrayList<>();
for (int i = 1; i <= 1000000; i++) {
String testentry = i+"";
cacheMessages.add(new ThreadProducer("testKey" + i, testentry));
}
ExecutorService executorService = Executors.newFixedThreadPool(1000);
cacheMessages.forEach(executorService::submit);
executorService.shutdown();
executorService.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
}
class ThreadProducer implements Runnable {
private String String;
private String key;
public ThreadProducer(String key, String String) {
this.key = key;
this.String = String;
}
public void run() {
testcache1.putIfAbsent(key, String);
testcache2.putIfAbsent(key, String);
System.out.println("entry :: " + key + " :: " + sdf.format(Calendar.getInstance().getTime()));
}
}
}

Try profiling server nodes and the thin clients using JFR, JProfiler or another profiler of your choice to find a bottleneck that slows down the operation.
Make sure that in both cases the same number of nodes are present in the baseline topology. With improper configuration of the baseline topology data may be loaded only to one of the nodes.
You can try using API methods that load data in batches to improve the performance. ClientCache#putAll() is one of such methods.

Not able to set property values to Spring-web.xml

I've a context listener where I'm loading all the properties. These properties I'm trying to set in my spring-web.xml, but it throws an exception
Because its not able to fetch and set the property to the xml
Here is my spring-web.xml
<bean id="dataSource"
class="org.springframework.jdbc.datasource.DriverManagerDataSource">
<property name="driverClassName" value="com.ibm.as400.access.AS400JDBCDriver" />
<property name="url" value="jdbc:as400://localhost/BB" />
<property name="username" value="{as400.username}" />
<property name="password" value="{as400.password}" />
</bean>
My class of loading properties
public class LoadProperties implements ServletContextListener {
private static Properties properties = null;
private static Logger logger = Logger.getLogger(LoadProperties .class);
#Override
public void contextDestroyed(ServletContextEvent arg0) { }
#Override
public void contextInitialized(ServletContextEvent arg0) {
properties = BBUtil.getProperties("datasource-cfg.properties");
for (String prop : properties.stringPropertyNames()) {
logger.info("Property Loaded :"+properties.getProperty(prop));
if (System.getProperty(prop) == null) {
System.setProperty(prop, properties.getProperty(prop));
}
}
}
}
This class is getting executed and setting the properties under System.
This is my properties file
as400.username=ROOT
as400.password=ROOT
How can I set the values into my spring-web.xml
Any ideas would be greatly appreciated.

we load it using context:property-placeholder
<context:property-placeholder location="classpath:db.properties" />
<bean id="datasource" class="org.springframework.jdbc.datasource.DriverManagerDataSource">
<property name="dbHost" value="${db.url}"/>
<property name="dbPort" value="${db.number}"/>
<property name="dbService" value="${db.name}"/>
<property name="dbUrl" value="${db.user}"/>
<property name="dbPassword" value="${db.password}"/>
</bean>
The properties file can also be loaded using prefixes such as http:,file:
The placeholder should be used with a $ sign as follows
<property name="username" value="${as400.username}" />
<property name="password" value="${as400.password}" />
ExtendedIllegalArgumentException essentially indicates that the argument passed is invalid.

Spring JDBC connection issue-Connections could not be acquired from the underlying database

I am stuck into a big trouble that i cannot connect the oracle 11g database form my Spring MVC application.
The error i am getting is
HTTP Status 500 - Request processing failed; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: Connections could not be acquired from the underlying database!
also,
in the stack trace i'm getting the error-
java.lang.ClassNotFoundException: oracle.jdbc.OracleDriver
If you can help me to resolve the issue, it will be a great help.
I am providing my configuration and coding details below:
Default-servlet.xml
<bean id="dataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource"
destroy-method="close">
<property name="driverClass" value="${jdbc.driverClassName}" />
<property name="jdbcUrl" value="${jdbc.url}" />
<property name="user" value="${jdbc.username}" />
<property name="password" value="${jdbc.password}" />
<property name="maxPoolSize" value="${jdbc.maxPoolSize}" />
<property name="minPoolSize" value="${jdbc.minPoolSize}" />
<property name="maxStatements" value="${jdbc.maxStatements}" />
<property name="testConnectionOnCheckout" value="${jdbc.testConnection}" />
</bean>
<bean id="userAuthenticationRepository"
class="com.era.repository.impl.UserAuthenticationRepositoryImpl">
<property name="dataSource" ref="dataSource" />
</bean>
UserAuthenticationRepositoryImpl.java
#Repository
public class UserAuthenticationRepositoryImpl implements UserAuthenticationRepository {
#Qualifier("dbDataSource")
private DataSource dataSource;
public void setDataSource(DataSource dataSource) {
this.dataSource = dataSource;
}
public User getUserAuthentication(User userToBeAuthenticated) {
// TODO Auto-generated method stub
String query = "select id, name, role from User where login =";
JdbcTemplate jdbcTemplate = new JdbcTemplate(dataSource);
StringBuilder queryString = new StringBuilder();
queryString.append(" SELECT ")
.append( "*" )
.append(" FROM table_name ")
.append(" WHERE login = ? ");
Object[] parameterList = { userToBeAuthenticated.getLogin() };
SqlRowSet dataRow = jdbcTemplate.queryForRowSet(queryString.toString(), parameterList);
if (dataRow.next()) {
System.out.println("Query executed successfully");
}
return null;
}

As you are using maven, note here you can't directly get Oracle driver jar to .m2 due to licence restriction, so you may need to manually download and place it to your repository.You may find this link helpful.

Spring MVC generate quoted field name json output

I am using spring 3.0 in my webapplication. I've got recently a problem in my application. I am using <mvc:annotation-drive/> tag in my spring-servlet.xml file but due to a requirement I've to remove this and place XML configuration instead.
But now my problem is it generates json output with quoted field names like if I return Boolean.TRUE I got "true" in the output. I want just true without quotes.
here is my XML configuration
<bean id="conversionService" class="org.springframework.format.support.FormattingConversionServiceFactoryBean" />
<bean id="pathMatcher" class="net.hrms.web.filters.CaseInsensitivePathMatcher" />
<bean name="handlerAdapter" class="org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter">
<property name="webBindingInitializer">
<bean class="org.springframework.web.bind.support.ConfigurableWebBindingInitializer">
<property name="conversionService" ref="conversionService"></property>
</bean>
</property>
<property name="messageConverters">
<list>
<ref bean="byteArrayConverter"/>
<ref bean="jaxbConverter"/>
<ref bean="jsonHttpMessageConverter"/>
<bean class="org.springframework.http.converter.StringHttpMessageConverter"></bean>
<bean class="org.springframework.http.converter.ResourceHttpMessageConverter"></bean>
<bean class="org.springframework.http.converter.xml.SourceHttpMessageConverter"></bean>
<bean class="org.springframework.http.converter.xml.XmlAwareFormHttpMessageConverter"></bean>
</list>
</property>
</bean>
<bean name="byteArrayConverter" class="org.springframework.http.converter.ByteArrayHttpMessageConverter"></bean>
<bean name="jaxbConverter" class="org.springframework.http.converter.xml.Jaxb2RootElementHttpMessageConverter"></bean>
<bean name="handlerMapping" class="org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping">
<property name="pathMatcher" ref="pathMatcher"></property>
</bean>
<bean class="org.springframework.web.servlet.view.InternalResourceViewResolver">
<property name="prefix" value="/WEB-INF/jsps/"/>
<property name="suffix" value=".jsp"/>
</bean>
<bean id="jsonHttpMessageConverter"
class="org.springframework.http.converter.json.MappingJacksonHttpMessageConverter">
<property name="prefixJson" value="false"/>
<property name="supportedMediaTypes" value="application/json"/>
</bean>
any help would be much appreciable.

If you are using the FlexJSON plugin, you can create custom output for JSON. I am sure you can do that in Jackson too, but I have never done it. There is loads of examples on the FlexJSON site.

What happens if you return just the primitive value of true (or Boolean.TRUE.booleanValue() rather than the wrapped object version of Boolean.TRUE?

I believe that true and false values are hardcoded in Jackson library (writeBoolean is called even for the boxed booleans):
private final static byte[] TRUE_BYTES = { 't', 'r', 'u', 'e' };
#Override
public void writeBoolean(boolean state)
throws IOException, JsonGenerationException
{
_verifyValueWrite("write boolean value");
if ((_outputTail + 5) >= _outputEnd) {
_flushBuffer();
}
byte[] keyword = state ? TRUE_BYTES : FALSE_BYTES;
int len = keyword.length;
System.arraycopy(keyword, 0, _outputBuffer, _outputTail, len);
_outputTail += len;
}
so Jackson will never return double-quoted "true" in field values(if its behavior is not heavily overriden, for example with codecs).
So please check that MappingJacksonHttpMessageConverter methods are being invoked, and conversion is not performed somewhere else.

That is the default behavior, if you want your boolean values unquoted use a JAXB ContextResolver
Something like this
import java.util.HashMap;
import java.util.HashSet;
import java.util.Map;
import javax.ws.rs.ext.ContextResolver;
import javax.ws.rs.ext.Provider;
import javax.xml.bind.JAXBContext;
import com.sun.jersey.api.json.JSONJAXBContext;
#Provider
public class JAXBContextResolver implements ContextResolver<JAXBContext> {
private JAXBContext context;
private Class[] types = {SomeClass.class}; //Add the classes processed by JAXB and exposing boolean properties
public JAXBContextResolver() throws Exception {
Map props = new HashMap<String, Object>();
props.put(JSONJAXBContext.JSON_NOTATION, JSONJAXBContext.JSONNotation.MAPPED);
props.put(JSONJAXBContext.JSON_ROOT_UNWRAPPING, Boolean.TRUE);
java.util.HashSet<String> eprops = new HashSet<String>();
eprops.add("someBooleanProperty"); //add properties you want unquoted
props.put(JSONJAXBContext.JSON_NON_STRINGS, eprops);
this.context = new JSONJAXBContext(types, props);
}
public JAXBContext getContext(Class<?> objectType) {
return (types[0].equals(objectType)) ? context : null;
}
}

hibernate always deletes all data from a table on its own

Hi i am developing a spring mvc app thats using hibernate to connect to a mysql database that stores files.
I have two methods. one that adds all files from a specific file path of my choosing and another method that invokes a query to return me a list of the files stored from mysql.
The issue is this. When i execute the first method on its own ie populating the database, it works fine i can see the contents of that table from mysql command line. however, when i then execute the query method right after populating it, the contents of that said table is completely gone instantly. Its as if hibernate only stored the data in the mysql temporarily or somewhere in mysql, it deleted data imediatly and doesnt keep it their.
this is the method that populated the table:
/**
* Test Method: ideal for another class to do this kind of work and this
* pass the FileObject into this class
*/
public void addSomeFiles() {
System.out.println("addSomeFiles");
File dir = new File(picturesPath);
String[] fileNames = dir.list();
for (int i = 0; i < fileNames.length; i++) {
System.out.println(fileNames[i]);
File file = new File(picturesPath + "\\" + fileNames[i]);
if (file.isFile()) {
FileObject fileO = contstructFileObject(file);
if (fileO == null) {
System.out.println("fileO is null!!!!!");
} else {
// addFile(fileO);
dbFileHelper.addFile(fileO);
}
}
}
System.out.println("//////////////");
// File file;
}
.........Hibernate template class........
public class DbFileHelper implements DbFileWrapper {
private HibernateTemplate hbTemplate;
//private static final String SQL_GET_FILE_LIST = "select filename, size, id, type from fileobject";
private static final String SQL_GET_FILE_LIST = "select new FileObject(filename, size, id, type) from FileObject";
public DbFileHelper() {
}
public void setHbTemplate(HibernateTemplate hbTemplate) {
System.out.println("setHbTemplate");
System.out.println("///////////////////");
System.out.println("///////////////////");
System.out.println("///////////////////");
this.hbTemplate = hbTemplate;
}
// ////////////////////////////////////////////////
#Override
public String addFile(FileObject file) {
// TODO Auto-generated method stub
System.out.println("addFile using hibernate");
if (hbTemplate == null) {
System.out.println("hbTemplate is null!! why?");
}
hbTemplate.saveOrUpdate(file);
hbTemplate.flush();
return "added succesfuly";
}
And here is the other method that makes the query:
........................
public JSONArray getFileList(String type){
return constructJsonArray(dbFileHelper.getFileList(ALL));
}
private JSONArray constructJsonArray(List<FileObject> fileList ){
JSONArray mJsonArray = new JSONArray();
for (int i = 0; i < fileList.size(); i++) {
System.out.println("fileName = " + fileList.get(i).getFilename() );
//mJson.put("Filename", fileList.get(i).getFileName() );
mJsonArray.add( new JSONObject().put("File ID", fileList.get(i).getId() ));
mJsonArray.add( new JSONObject().put("Filename", fileList.get(i).getFilename() ));
mJsonArray.add( new JSONObject().put("File type", fileList.get(i).getType()));
mJsonArray.add( new JSONObject().put("File Size", fileList.get(i).getSize()));
}
return mJsonArray;
}
..........hibernate Template class.......
private static final String SQL_GET_FILE_LIST = "select new FileObject(filename, size, id, type) from FileObject";
#Override
public List<FileObject> getFileList(String type) {
// TODO Auto-generated method stub
List<FileObject> files = hbTemplate.find(SQL_GET_FILE_LIST);
//hbTemplate.flush();
return files;
}
..........
Finally here is a print screen of what i originaly put inside my table but dissapears on its own:
http://img411.imageshack.us/img411/9553/filelisti.jpg
Am i missing something here?
edit: additional info.
my hbm.xml
<?xml version="1.0"?>
<!DOCTYPE hibernate-mapping PUBLIC
"-//Hibernate/Hibernate Mapping DTD 3.0//EN"
"http://hibernate.sourceforge.net/hibernate-mapping-3.0.dtd">
<hibernate-mapping package="com.kc.models.FileObject" >
<class name="com.kc.models.FileObject" table="fileobject">
<id name="id" column="ID">
<generator class="native" />
</id>
<property name="filename" type="string" column="FILENAME" />
<property name="type" type="string" column="TYPE" />
<property name="size" type="double" column="SIZE" />
<property name="file" type="blob" length="1000000000" column="FILE" />
</class>
</hibernate-mapping>
my controller:
#Override
public ModelAndView handleRequest(HttpServletRequest request,
HttpServletResponse response) throws Exception {
// TODO call a method that returns a list of Mobile Apps.
testAddingSomeFilesToDb();
return new ModelAndView("" + "testJsonResponse", "jsonArray",
getFileList() );
}
private void testAddingSomeFilesToDb() {
ctx = new ClassPathXmlApplicationContext("zang-file-service.xml");
FileHelper file = (FileHelper) ctx.getBean("fileHelper");
file.addSomeFiles();
}
/**
* Get file list from sql server based on type
* #return file list in json
*/
private JSONArray getFileList() {
// TODO: Get request parameter that states what type of file extensions
// the client wants to recieve
ctx = new ClassPathXmlApplicationContext("zang-file-service.xml");
FileHelper file = (FileHelper) ctx.getBean("fileHelper");
return file.getFileList("all");
}
Another edit:
my .xml file configuring the session factory and hibernate template
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:jee="http://www.springframework.org/schema/jee"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.5.xsd
http://www.springframework.org/schema/jee
http://www.springframework.org/schema/jee/spring-jee-2.0.xsd">
<!-- http://www.springframework.org/schema/jee http://www.springframework.org/schema/jee/spring-jee-2.0.xsd -->
<!-- Config properties files -->
<!-- Hibernate database stuff -->
<!-- <bean id="propertyConfigurer" class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations"> <list> <value>/properties/jdbc.properties</value>
</list> </property> </bean> -->
<!-- <bean id="dataSource1" class="org.springframework.jdbc.datasource.DriverManagerDataSource">
<property name="driverClassName" value="${database.driver}" /> <property
name="url" value="${database.url}" /> <property name="username" value="${database.user}"
/> <property name="password" value="${database.password}" /> </bean> -->
<bean id="dataSource1"
class="org.springframework.jdbc.datasource.DriverManagerDataSource">
<property name="driverClassName" value="com.mysql.jdbc.Driver" />
<property name="url" value="jdbc:mysql://localhost:3306/zangshop" />
<property name="username" value="root" />
<property name="password" value="password" />
</bean>
<!-- LocalSessionFactoryBean u need to put the hbm files in the WEB-INF/classes
root director -->
<bean id="sessionFactory"
class="org.springframework.orm.hibernate3.LocalSessionFactoryBean">
<property name="dataSource" ref="dataSource1"></property>
<property name="mappingResources">
<list>
<value>FileObject.hbm.xml</value>
</list>
</property>
<property name="hibernateProperties">
<props>
<prop key="hibernate.dialect">org.hibernate.dialect.MySQLDialect</prop>
<prop key="hibernate.show_sql">true</prop>
<prop key="hibernate.hbm2ddl.auto">create</prop>
</props>
</property>
</bean>
<bean id="hbTemplate" class="org.springframework.orm.hibernate3.HibernateTemplate">
<property name="sessionFactory" ref="sessionFactory"></property>
</bean>
<bean id="dbFileHelper" class="com.kc.models.DbFileHelper">
<property name="hbTemplate" ref="hbTemplate"></property>
</bean>
<bean id="fileHelper" class="com.kc.models.FileHelper">
<property name="dbFileHelper" ref="dbFileHelper"></property>
</bean>
</beans>

i have fixed the problem
i changed <prop key="hibernate.hbm2ddl.auto">create</prop>
to <prop key="hibernate.hbm2ddl.auto">update</prop> and it worked

Are you creating/destroying the SessionFactory between calls? Could you have the hbm2ddl.auto property set to create-drop?
Actually, can you show the Hibernate settings?
Reference
Hibernate Core Reference Guide
Table 3.7. Miscellaneous Properties

In my case also table was getting deleted automatically, following solution worked for me:
org.hibernate.dialect.MySQL8Dialect
Appending the version number with the MySQL Dialect.
Because commit was not getting executed earlier with org.hibernate.dialect.MySQLDialect.