We Keep Coding

Neo4j Causal Cluster Bolt Driver Performance Too Low

We are evaluating Neo4J Enterprise Edition Causal Cluster using Bolt Driver for Java.
We have 3 node Core Cluster.
The performance we saw is too low.
We are creating just 1 node with 2 property 10,00,000 times. When tracked, we are getting 300TPS (i.e. only 300 nodes are created per second).
OS is Linux, RHEL.
Each core is running with 32GB.
We were estimating close to 50,000 TPS for creation of just 1 node however it is only 300 TPS which is way way way too low.
I am sure we are missing something big.
This function is called 10,00,000 times by a thread pool of 64 threads.
Code Snippet:
#Override
public void createNode() throws InterruptedException {
try (Session session = RTNeo4j.getInstance().getWriteDriver().session(AccessMode.WRITE)) {
try (final Transaction tx = session.beginTransaction()) {
try {
tx.run("CREATE (a:Person {name: {name}, id: {id}})",
parameters("name", "king", "id", System.currentTimeMillis()));
tx.success();
} catch (Exception e) {
e.printStackTrace();
}
}
}
}
Appreciate quick help for evaluation.

You do not have to create each time a session within a method. Move the creation of the session outside method:
Session session = RTNeo4j.getInstance().getWriteDriver().session(AccessMode.WRITE)

SSH Server Identification never received - Handshake Deadlock [SSHJ]

We're having some trouble trying to implement a Pool of SftpConnections for our application.
We're currently using SSHJ (Schmizz) as the transport library, and facing an issue we simply cannot simulate in our development environment (but the error keeps showing randomly in production, sometimes after three days, sometimes after just 10 minutes).
The problem is, when trying to send a file via SFTP, the thread gets locked in the init method from schmizz' TransportImpl class:
#Override
public void init(String remoteHost, int remotePort, InputStream in, OutputStream out)
throws TransportException {
connInfo = new ConnInfo(remoteHost, remotePort, in, out);
try {
if (config.isWaitForServerIdentBeforeSendingClientIdent()) {
receiveServerIdent();
sendClientIdent();
} else {
sendClientIdent();
receiveServerIdent();
}
log.info("Server identity string: {}", serverID);
} catch (IOException e) {
throw new TransportException(e);
}
reader.start();
}
isWaitForServerIdentBeforeSendingClientIdent is FALSE for us, so first of all the client (we) send our identification, as appears in logs:
"Client identity String: blabla"
Then it's turn for the receiveServerIdent:
private void receiveServerIdent() throws IOException
{
final Buffer.PlainBuffer buf = new Buffer.PlainBuffer();
while ((serverID = readIdentification(buf)).isEmpty()) {
int b = connInfo.in.read();
if (b == -1)
throw new TransportException("Server closed connection during identification exchange");
buf.putByte((byte) b);
}
}
The thread never gets the control back, as the server never replies with its identity. Seems like the code is stuck in this While loop. No timeouts, or SSH exceptions are thrown, my client just keeps waiting forever, and the thread gets deadlocked.
This is the readIdentification method's impl:
private String readIdentification(Buffer.PlainBuffer buffer)
throws IOException {
String ident = new IdentificationStringParser(buffer, loggerFactory).parseIdentificationString();
if (ident.isEmpty()) {
return ident;
}
if (!ident.startsWith("SSH-2.0-") && !ident.startsWith("SSH-1.99-"))
throw new TransportException(DisconnectReason.PROTOCOL_VERSION_NOT_SUPPORTED,
"Server does not support SSHv2, identified as: " + ident);
return ident;
}
Seems like ConnectionInfo's inputstream never gets data to read, as if the server closed the connection (even if, as said earlier, no exception is thrown).
I've tried to simulate this error by saturating the negotiation, closing sockets while connecting, using conntrack to kill established connections while the handshake is being made, but with no luck at all, so any help would be HIGHLY appreciated.
: )

I bet following code creates a problem:
String ident = new IdentificationStringParser(buffer, loggerFactory).parseIdentificationString();
if (ident.isEmpty()) {
return ident;
}
If the IdentificationStringParser.parseIdentificationString() returns empty string, it will be returned to the caller method. The caller method will keep calling the while ((serverID = readIdentification(buf)).isEmpty()) since the string is always empty. The only way to break the loop would be if call to int b = connInfo.in.read(); returns -1... but if server keeps sending the data (or resending the data) this condition is never met.
If this is the case I would add some kind of artificial way to detect this like:
private String readIdentification(Buffer.PlainBuffer buffer, AtomicInteger numberOfAttempts)
throws IOException {
String ident = new IdentificationStringParser(buffer, loggerFactory).parseIdentificationString();
numberOfAttempts.incrementAndGet();
if (ident.isEmpty() && numberOfAttempts.intValue() < 1000) { // 1000
return ident;
} else if (numberOfAttempts.intValue() >= 1000) {
throw new TransportException("To many attempts to read the server ident").
}
if (!ident.startsWith("SSH-2.0-") && !ident.startsWith("SSH-1.99-"))
throw new TransportException(DisconnectReason.PROTOCOL_VERSION_NOT_SUPPORTED,
"Server does not support SSHv2, identified as: " + ident);
return ident;
}
This way you would at least confirm that this is the case and can dig further why .parseIdentificationString() returns empty string.

Faced a similar issue where we would see:
INFO [net.schmizz.sshj.transport.TransportImpl : pool-6-thread-2] - Client identity string: blablabla
INFO [net.schmizz.sshj.transport.TransportImpl : pool-6-thread-2] - Server identity string: blablabla
But on some occasions, there were no server response.
Our service would typically wake up and transfer several files simultaneously, one file per connection / thread.
The issue was in the sshd server config, we increased maxStartups from default value 10
(we noticed the problems started shortly after batch sizes increased to above 10)
Default in /etc/ssh/sshd_config:
MaxStartups 10:30:100
Changed to:
MaxStartups 30:30:100
MaxStartups
Specifies the maximum number of concurrent unauthenticated connections to the SSH daemon. Additional connections will be dropped until authentication succeeds or the LoginGraceTime expires for a connection. The default is 10:30:100. Alternatively, random early drop can be enabled by specifying the three colon separated values start:rate:full (e.g. "10:30:60"). sshd will refuse connection attempts with a probability of rate/100 (30%) if there are currently start (10) unauthenticated connections. The probability increases linearly and all connection attempts are refused if the number of unauthenticated connections reaches full (60).
If you cannot control the server, you might have to find a way to limit your concurrent connection attempts in your client code instead.

Tomcat DataSource - Max Active Connections

I set max active connections to 1 using the below code :
ConnectionPool initializePool(DataSource dataSource) {
if (!(org.apache.tomcat.jdbc.pool.DataSource.class.isInstance(dataSource))) {
return null;
}
org.apache.tomcat.jdbc.pool.DataSource tomcatDataSource = (org.apache.tomcat.jdbc.pool.DataSource) dataSource;
final String poolName = tomcatDataSource.getName();
try {
ConnectionPool pool = tomcatDataSource.createPool();
pool.getPoolProperties().setMaxActive(1);
pool.getPoolProperties().setInitialSize(1);
pool.getPoolProperties().setTestOnBorrow(true);
return pool;
} catch (SQLException e) {
logger.info(String.format(" !--! creation of pool failed for %s", poolName), e);
}
return null;
}
Now using threads , I have opened number of concurrent connections to DB. I also printed out the current active number of connections using the below listed code
System.out.println("Current Active Connections = " + ((org.apache.tomcat.jdbc.pool.DataSource) datasource).getActive());
System.out.println("Max Active Connections = " + ((org.apache.tomcat.jdbc.pool.DataSource) datasource).getMaxActive());
I see results similar to below. Active connections is being displayed as more than 1. However I want to restrict the max active connections to 1. Are there any other parameters that I need to set?
Current Active Connections = 9
Max Active Connections = 1
EDIT: However when I try with 15 or 20 as max active , it is always limited to 15 or 20 respectively.

Try with maxIdle and minIdle
ConnectionPool pool = tomcatDataSource.createPool();
pool.getPoolProperties().setMaxActive(1);
pool.getPoolProperties().setInitialSize(1);
pool.getPoolProperties().setMaxIdle(1);
pool.getPoolProperties().setMinIdle(1);
pool.getPoolProperties().setTestOnBorrow(true);

How to set time limit for creating HBase connection

I'm currently using HBase v0.98.6. I would like to check the current connection status from an external Java program. Right now, I'm doing something like this to check:
connectionSuccess = true;
try {
HConnection hConnection = createConnection(config);
} catch (Exception ex) {
connectionSuccess = false;
}
When the connection is working, this returns fairly quickly. The problem is when the connection is not working, and it takes 20 minutes for it to finally return connectionSuccess=false. Is there a way to reduce this time limit, as I'm just interested in getting the connection status at the current time?

The reason it takes so long is that by default if the connection fails it will retry multiple times (I think 6? don't quote me), and each connection attempt takes a while. Try a combination of these commands to limit time per connection before timeout, and number of permitted retry attempts.
hbase.client.retries.number = 3
hbase.client.pause = 1000
zookeeper.recovery.retry = 1 (i.e. no retry)
Credit to Lars from http://hadoop-hbase.blogspot.com/2012/09/hbase-client-timeouts.html

You can set the retries value to 1 to get the status of the connection at the current time.
Configuration conf = HBaseConfiguration.create();
conf.setInt("hbase.client.retries.number",1);
conf.setInt("zookeeper.recovery.retry",0);
Or you can use the below in-built HbaseAdmin method which does the same thing.
connectionSuccess = true;
try
{
HBaseAdmin.checkHBaseAvailable(config);
}
catch(MasterNotRunningException e)
{
connectionSuccess = false;
}

My org.apache.hadoop.conf.Configuration object contains following key value pairs:
Configuration conf = HBaseConfiguration.create();
//configuring timeout and retry parameters
conf.set("hbase.rpc.timeout", "10000");
conf.set("hbase.client.scanner.timeout.period", "10000");
conf.set("hbase.cells.scanned.per.heartbeat.check", "10000");
conf.set("zookeeper.session.timeout", "10000");
conf.set("phoenix.query.timeoutMs", "10000");
conf.set("phoenix.query.keepAliveMs", "10000");
conf.set("hbase.client.retries.number", "3");
conf.set("hbase.client.pause", "1000");
conf.set("zookeeper.recovery.retry", "1");

OpenLDAP 2.3/2.4 concurrency issue

Experiencing concurrency issues when authenticating and other read requests with Open LDAP (version tested 2.3.43 and 2.4.39)
When making 100 concurrent bind requests the test code takes around 150 milliseconds. Increasing this to 1000 concurrent requests sees the time taken increase to 9303 milliseconds.
So from x10 concurrent requests we are seeing a x62 increase in time taken.
Is this expected behaviour? Or is there something missing in our OpenLDAP server configuration/linux host configuration?
NOTE: We have run this test code against a Windows based Apache DS server 2.0.0 (same tree structure, etc) for comparison and against that server, the performance results where what we would normally expect (i.e. 100x takes ~80ms, 1000x takes ~400ms, 10,000x takes ~2700ms)
Settings in slapd.conf:
cachesize 100000
idlcachesize 300000
database bdb
suffix "dc=company,dc=com"
rootdn "uid=admin,ou=system"
rootpw secret
directory /var/lib/ldap
index objectClass eq,pres
index ou,cn,mail,surname,givenname eq,pres,sub
index uidNumber,gidNumber,loginShell eq,pres
index uid,memberUid eq,pres,sub
index nisMapName,nisMapEntry eq,pres,sub
sizelimit 100000
loglevel 256
Test code:
import java.util.ArrayList;
import javax.naming.NamingException;
import javax.naming.directory.DirContext;
import org.springframework.ldap.core.LdapTemplate;
import org.springframework.ldap.core.support.LdapContextSource;
public class DirectoryServiceMain {
public static void main(String[] args) {
int concurrentThreadCount = 100;
LdapContextSource ctx = new LdapContextSource();
ctx.setUrls(new String [] { "ldap://ldap1.dev.company.com:389/", "ldap://ldap1.dev.company.com:389/" });
ctx.setBase("dc=company,dc=com");
ctx.setUserDn("uid=admin,ou=system");
ctx.setPassword("secret");
ctx.setPooled(true);
ctx.setCacheEnvironmentProperties(false);
LdapTemplate template = new LdapTemplate();
template.setContextSource(ctx);
long startTime = System.currentTimeMillis();
ArrayList<Thread> threads = new ArrayList<>();
for(int i = 0; i < concurrentThreadCount; i++) {
Thread t = new Thread(
() -> {
DirContext context = template.getContextSource().getContext("uid=username,dc=users,uid=office,dc=suborganisations,uid=ABC,dc=organisations,dc=company,dc=com",
"password");
try {
context.close();
} catch(NamingException e) {}
});
t.start();
threads.add(t);
}
boolean alive = true;
while(alive) {
alive = false;
for(Thread t : threads) {
if(t.isAlive()) {
alive = true;
try {Thread.sleep(10);} catch(InterruptedException e) {}
}
}
}
long endTime = System.currentTimeMillis();
System.out.println("Total time: " + (endTime - startTime));
}
}
ulimit -n
131072
* UPDATE *
If a slight delay (e.g. Thread.sleep(1)) is added after each t.start(), then processing time of n concurrent threads drops considerably.

A longer answer is if you are using BDB as the database then you will likely see linear scaling problems above a certain number of concurrent requests. BDB has its own db_config file that you can configure to provide better performance characteristics. You could also consider change to MDB which was specifically written for open ldap and has better linear scaling with minimal configuration.
You should also consider limiting the number of concurrent connection made by setting the jndi ldap connection pool sizes against the LDAPContextSource:
Map<String, Object> map = new HashMap<>();
map.put("com.sun.jndi.ldap.connect.pool.initsize", 2);
map.put("com.sun.jndi.ldap.connect.pool.maxsize", 2);
map.put("com.sun.jndi.ldap.connect.pool.prefsize", 2);
ctx.setBaseEnvironmentProperties(map);