We Keep Coding

Neo4j: unit testing the bolt driver properly

I am adding the Neo4j Bolt driver to my application just following the http://neo4j.com/developer/java/:
import org.neo4j.driver.v1.*;
Driver driver = GraphDatabase.driver( "bolt://localhost", AuthTokens.basic( "neo4j", "neo4j" ) );
Session session = driver.session();
session.run( "CREATE (a:Person {name:'Arthur', title:'King'})" );
StatementResult result = session.run( "MATCH (a:Person) WHERE a.name = 'Arthur' RETURN a.name AS name, a.title AS title" );
while ( result.hasNext() )
{
Record record = result.next();
System.out.println( record.get( "title" ).asString() + " " + record.get("name").asString() );
}
session.close();
driver.close();
However, always from the official documentation unit testing is made using:
GraphDatabaseService db = new TestGraphDatabaseFactory()
.newImpermanentDatabaseBuilder()
So if I want to test in some way the code above, I have to replace the GraphDatabase.driver( "bolt://localhost",...) with the GraphDatabaseService from the test. How can I do that? I cannot extract any sort of in-memory driver from there as far as I can see.

The Neo4j JDBC has a class called Neo4jBoltRule for unit testing. It is a junit rule starting/stopping an impermanent database together with some configuration to start bolt.
The rule class uses dynamic port assignment to prevent test failure due to running multiple tests in parallel (think of your CI infrastructure).
An example of a unit test using that rule class is available at https://github.com/neo4j-contrib/neo4j-jdbc/blob/master/neo4j-jdbc-bolt/src/test/java/org/neo4j/jdbc/bolt/SampleIT.java

An easy way now is to pull neo4j-harness, and use their built-in Neo4jRule as follows:
import static org.neo4j.graphdb.factory.GraphDatabaseSettings.boltConnector;
// [...]
#Rule public Neo4jRule graphDb = new Neo4jRule()
.withConfig(boltConnector("0").address, "localhost:" + findFreePort());
Where findFreePort implementation can be as simple as:
private static int findFreePort() {
try (ServerSocket socket = new ServerSocket(0)) {
return socket.getLocalPort();
} catch (IOException e) {
throw new RuntimeException(e.getMessage(), e);
}
}
As the Javadoc of ServerSocket explains:
A port number of 0 means that the port number is automatically allocated, typically from an ephemeral port range. This port number can then be retrieved by calling getLocalPort.
Moreover, the socket is closed before the port value is returned, so there are great chances the returned port will still be available upon return (the window of opportunity for the port to be allocated again in between is small - the computation of the window size is left as an exercise to the reader).
Et voilà !

Detect open transaction not yet committed in JDBC connection

How do I detect if a transaction remains open, still pending on a COMMIT or ROLLBACK on a JDBC Connection?
I'm getting my Connection objects via a connection pool. So I want to check the state of the connection before using it.
Using Postgres 9.x and Java 8.

I'm not aware of any way to detect current transaction status on a Connection using only standard JDBC API methods.
However, for PostgreSQL specifically, there is AbstractJdbc2Connection.getTransactionState(), which you can compare against the constant ProtocolConnection.TRANSACTION_IDLE. PostgreSQL's JDBC4 Connection extends this class so you should be able to cast your Connection to get access to this property.
That constant is one of three values defined in the pgjdbc driver source code:
/**
* Constant returned by {#link #getTransactionState} indicating that no
* transaction is currently open.
*/
static final int TRANSACTION_IDLE = 0;
/**
* Constant returned by {#link #getTransactionState} indicating that a
* transaction is currently open.
*/
static final int TRANSACTION_OPEN = 1;
/**
* Constant returned by {#link #getTransactionState} indicating that a
* transaction is currently open, but it has seen errors and will
* refuse subsequent queries until a ROLLBACK.
*/
static final int TRANSACTION_FAILED = 2;

As I understand, you use plain JDBC and this is why you have this problem. Because you told about the Tomcat's JDBC Connection Pool, you could use JDBCInterceptor.invoke(), where you could track what happens to each Connection. More details here.

The accepted Answer by heenenee is correct.
Example Code
This Answer posts the source code of a helper class. This source code is based on the ideas if that accepted Answer.
package com.powerwrangler.util;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.UUID;
import javax.naming.Context;
import javax.naming.InitialContext;
import javax.naming.NamingException;
import javax.sql.DataSource;
import org.slf4j.LoggerFactory;
/**
*
* Help with database chores.
*
* © 2015 Basil Bourque
* This source code available under terms of the ISC License. http://opensource.org/licenses/ISC
*
* #author Basil Bourque.
*
*/
public class DatabaseHelper
{
static final org.slf4j.Logger logger = LoggerFactory.getLogger( DatabaseHelper.class );
public enum TransactionState
{
IDLE,
OPEN,
FAILED;
}
/**
* If using the Postgres database, and the official "org.postgresql" JDBC driver, get the current state of the
* current transaction held by a Connection. Translate that state to a convenient Enum value.
*
* #param connArg
*
* #return DatabaseHelper.TransactionState
*/
public DatabaseHelper.TransactionState transactionStateOfConnection ( Connection connArg ) {
// This code is specific to Postgres.
// For more info, see this page on StackOverflow: https://stackoverflow.com/q/31754214/642706
// Verify arguments.
if ( connArg == null ) {
logger.error( "Received null argument for Connection object. Message # 6b814e3c-80e3-4145-9648-390b5315243e." );
}
DatabaseHelper.TransactionState stateEnum = null; // Return-value.
Connection conn = connArg; // Transfer argument to local variable.
// See if this is a pooled connection.
// If pooled, we need to extract the real connection wrapped inside.
// Class doc: http://docs.oracle.com/javase/8/docs/api/javax/sql/PooledConnection.html
// I learned of this via the "Getting the actual JDBC connection" section of the "Tomcat JDBC Connection Pool" project.
// Tomcat doc: https://tomcat.apache.org/tomcat-8.0-doc/jdbc-pool.html#Getting_the_actual_JDBC_connection
if ( conn instanceof javax.sql.PooledConnection ) {
javax.sql.PooledConnection pooledConnection = ( javax.sql.PooledConnection ) conn;
try { // Can throw java.sql.SQLException. So using a Try-Catch.
// Conceptually we are extracting a wrapped Connection from with in a PooledConnection. Reality is more complicated.
// From class doc: Creates and returns a Connection object that is a handle for the physical connection that this PooledConnection object represents.
conn = pooledConnection.getConnection();
} catch ( SQLException ex ) {
// We could just as well throw this SQLException up the call chain. But I chose to swallow it here. --Basil Bourque
logger.error( "Failed to extract the real Connection from its wrappings in a PooledConnection. Message # ea59e3a3-e128-4386-949e-a70d90e1c19e." );
return null; // Bail-out.
}
}
// First verify safe to cast.
if ( conn instanceof org.postgresql.jdbc2.AbstractJdbc2Connection ) {
// Cast from a generalized JDBC Connection to one specific to our expected Postgres JDBC driver.
org.postgresql.jdbc2.AbstractJdbc2Connection aj2c = ( org.postgresql.jdbc2.AbstractJdbc2Connection ) conn; // Cast to our Postgres-specific Connection.
// This `getTransactionState` method is specific to the Postgres JDBC driver, not general JDBC.
int txnState = aj2c.getTransactionState();
// We compare that state’s `int` value by comparing to constants defined in this source code:
// https://github.com/pgjdbc/pgjdbc/blob/master/org/postgresql/core/ProtocolConnection.java#L27
switch ( txnState ) {
case org.postgresql.core.ProtocolConnection.TRANSACTION_IDLE:
stateEnum = DatabaseHelper.TransactionState.IDLE;
break;
case org.postgresql.core.ProtocolConnection.TRANSACTION_OPEN:
stateEnum = DatabaseHelper.TransactionState.OPEN;
break;
case org.postgresql.core.ProtocolConnection.TRANSACTION_FAILED:
stateEnum = DatabaseHelper.TransactionState.FAILED;
break;
default:
// No code needed.
// Go with return value having defaulted to null.
break;
}
} else {
logger.error( "The 'transactionStateOfConnection' method was passed Connection that was not an instance of org.postgresql.jdbc2.AbstractJdbc2Connection. Perhaps some unexpected JDBC driver is in use. Message # 354076b1-ba44-49c7-b987-d30d76367d7c." );
return null;
}
return stateEnum;
}
public Boolean isTransactionState_Idle ( Connection connArg ) {
Boolean b = this.transactionStateOfConnection( connArg ).equals( DatabaseHelper.TransactionState.IDLE );
return b;
}
public Boolean isTransactionState_Open ( Connection conn ) {
Boolean b = this.transactionStateOfConnection( conn ).equals( DatabaseHelper.TransactionState.OPEN );
return b;
}
public Boolean isTransactionState_Failed ( Connection conn ) {
Boolean b = this.transactionStateOfConnection( conn ).equals( DatabaseHelper.TransactionState.FAILED );
return b;
}
}
Example usage:
if ( new DatabaseHelper().isTransactionState_Failed( connArg ) ) {
logger.error( "JDBC transaction state is Failed. Expected to be Open. Cannot process source row UUID: {}. Message # 9e633f31-9b5a-47bb-bbf8-96b1d77de561." , uuidOfSourceRowArg );
return null; // Bail-out.
}
Include JDBC Driver In Project But Omit From Build
The challenge with this code is that at compile-time we must address classes specific to a specific JDBC driver rather than generalized JDBC interfaces.
You might think, “Well enough, just add the JDBC driver jar file to the project”. But, no, in a web app Servlet environment we must not include the JDBC driver in our build (our WAR file/folder). In a web app, technical issues mean we should deposit our JDBC driver with the Servlet container. For me that means Apache Tomcat where we place the JDBC driver jar file into Tomcat’s own /lib folder rather than within our web app’s WAR file/folder.
So how to include the JDBC driver jar in our project during compile-time while excluding from the build of our WAR file? See this Question, Include a library while programming & compiling, but exclude from build, in NetBeans Maven-based project. The solution in Maven is the scope tag with a value of provided.
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<version>9.4-1201-jdbc41</version>
<scope>provided</scope>
</dependency>

You can retrieve txId from postgres select txid_current() and write it to log. This number is differ for different transactions.

I managed to do something with Statement.getUpdateCount() .
The idea was that after each statement execution, I verify if updateCount > 0 .
If it's true and that autocommit is disabled, it means the connection of that statement will need a commit or a rollback before being closed.
By wrapping Datasource, Connection, Statement, PreparedStatement, CallableStatement, it's possible to implement this verification at each call of execute(), executeUpdate(), executeBatch(), store the stack trace and the flag in the Connection wrapper.
In connection close() you can then show the last statement execution with the stack , then rollback and throw an exception.
However I am not sure of the overhead of getUpdateCount(), and if it doesn't mess with the results.
But the integration test cases are working to far.
We could check if getUpdateCount() >-1, but it would break ode that might already avoid commit if nothing was updated.

Scala - Cassandra: cluster read fails with error "Can't use this Cluster instance because it was previously closed"

I'm getting this error when reading from a table in a 5 node cluster using datastax drivers.
2015-02-19 03:24:09,908 ERROR [akka.actor.default-dispatcher-9] OneForOneStrategy akka://user/HealthServiceChecker-49e686b9-e189-48e3-9aeb-a574c875a8ab Can't use this Cluster instance because it was previously closed
java.lang.IllegalStateException: Can't use this Cluster instance because it was previously closed
at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1128) ~[cassandra-driver-core-2.0.4.jar:na]
at com.datastax.driver.core.Cluster.init(Cluster.java:149) ~[cassandra-driver-core-2.0.4.jar:na]
at com.datastax.driver.core.Cluster.connect(Cluster.java:225) ~[cassandra-driver-core-2.0.4.jar:na]
at com.datastax.driver.core.Cluster.connect(Cluster.java:258) ~[cassandra-driver-core-2.0.4.jar:na]
I am able to connect using cqlsh and perform read operations.
Any clue what could be the problem here?
settings:
Consistency Level: ONE
keyspace replication strategy:
'class': 'NetworkTopologyStrategy',
'DC2': '1',
'DC1': '1'
cassandra version: 2.0.6
The code managing cassandra sessions is central and it is;
trait ConfigCassandraCluster
extends CassandraCluster
{
def cassandraConf: CassandraConfig
lazy val port = cassandraConf.port
lazy val host = cassandraConf.host
lazy val cluster: Cluster =
Cluster.builder()
.addContactPoints(host)
.withReconnectionPolicy(new ExponentialReconnectionPolicy(100, 30000))
.withPort(port)
.withSocketOptions(new SocketOptions().setKeepAlive(true))
.build()
lazy val keyspace = cassandraConf.keyspace
private lazy val casSession = cluster.connect(keyspace)
val session = new SessionProvider(casSession)
}
class SessionProvider(casSession: => Session) extends Logging {
var lastSuccessful: Long = 0
var firstSuccessful: Long = -1
def apply[T](fn: Session => T): T = {
val result = retry(fn, 15)
if(firstSuccessful < 0)
firstSuccessful = System.currentTimeMillis()
lastSuccessful = System.currentTimeMillis()
result
}
private def retry[T](fn: Session => T, remainingAttempts: Int): T = {
//retry logic
}

The problem is, cluster.connect(keyspace) will close the cluster itself if it experiences NoHostAvailableException. Due to that during retry logic, you are experiencing IllegalStateException.
Have a look at Cluster init() method and you will understand more.
The solution for your problem would be, in the retry logic, do Cluster.builder.addContactPoint(node).build.connect(keyspace). This will enable to have a new cluster object while you retry.

Search your code for session.close().
You are closing your connection somewhere as stated in the comments. Once a session is closed, it can't be used again. Instead of closing connections, pool them to allow for re-use.

Get the number of open connections in mongoDB using java

My program requires a large number of connections to be open (Mongo). I get the error :
Too many connections open, can't open anymore
after 819 connections. I already know we can increase this limit. But that's not what I have in mind. I'm thinking of closing the MongoClient object, and then creating a new one again after 800 connections.
My thinking is that with a new mongoClient object all the connections will be closed and when I start/create it again, the connections will be opened again until 800.
Thus not giving the error. (Let me know if this approach is totally wrong/ won't give the required results.)
For this I need to know the number of connections opened ATM. Is there any way to get this information using java?

You can get connection information by using the db.serverStatus() command. It has a connections subdocument which contains the total/available connections information.
For more information :
Documentation of server status
Details of connections block

Check the number of MongoDB connections using MongoDB Scala driver:
Create a MongoDB client:
import org.mongodb.scala._
import scala.collection.JavaConverters._
import scala.concurrent.Await
import scala.concurrent.duration._
import scala.util.{Failure, Success, Try}
// To directly connect to the default server localhost on port 27017
val mongodbClient: MongoClient = MongoClient()
// Use a Connection String
val mongodbClient: MongoClient = MongoClient("mongodb://localhost")
// or provide custom MongoClientSettings
val settings: MongoClientSettings = MongoClientSettings.builder()
.applyToClusterSettings(b => b.hosts(List(new ServerAddress("localhost")).asJava).
.build()
val mongodbClient: MongoClient = MongoClient(settings)
Call getNoOfMongodbConnection by passing mongodbClient:
val result = getNoOfMongodbConnection(mongodbClient)
Method to get the number of connections(current, available and total)
def getNoOfMongodbConnection(mongodbClient: MongoClient) = {
val adminDatabase = mongodbClient.getDatabase("admin")
val serverStatus = adminDatabase.runCommand(Document("serverStatus" -> 1)).toFuture()
Try {
Await.result(serverStatus, 10 seconds)
} match {
case Success(x) => {
val connection = x.get("connections")
logger.info("Number of mongodb connection:--> " + connection)
connection
}
case Failure(ex) => {
logger.error("Got error while getting the number of Mongodb connection:---> " + ex.printStackTrace())
None
}
}
}

Cannot connect multiple times to mysql

I am writing an own databse in scala. To verify my results are correct, I check with a MySQL inside of a specs2 specification. I get the right result and everything is just fine. But if I run the test again without any changes, I get a SQLException: No suitable driver found for jdbc:mysql://localhost:3306/DBNAME?user=DBUSER (null:-1). Why is the driver not loaded again?
Edit
import java.sql.{ Connection, DriverManager, ResultSet }
import org.specs2.mutable.Specification
// SDDB imports ...
class DBValidationSpec extends Specification {
"SDDB and MySQl" should {
// SDDB
// ...
// JDBC
val connectionString = "jdbc:mysql://localhost:3306/sddb_test?user=root"
val query = """SELECT content, SUM( duration ) duration
FROM test
WHERE times
BETWEEN '2011-12-08'
AND '2011-12-09'
GROUP BY content"""
classOf[com.mysql.jdbc.Driver]
"give the same result" in {
// ...
//sddbResult
lazy val conn = DriverManager.getConnection(connectionString)
try{
val rs = conn.createStatement().executeQuery(query)
var mysqlResult = Map[List[String], Int]()
while (rs.next) {
mysqlResult += (rs.getString("content") :: Nil) -> rs.getInt("duration")
}
sddbResult == mysqlResult && sddbResult.size == 478 must beTrue
} finally {
conn.close()
}
}
}
}
I left out some parts of my code because they don't belong to the question.
Edit #2
The problem became even weirder. I added a second testcase. The testcase uses the same connectionString. The Exception was only raised once. The second test succeeded. I added sequential to my test definition and saw that only the first executed test raises the Exception. Afterwards I traced the classLoader to check if it is the same one. It is.
I did the following workaround:
trait PreExecuting extends Before {
override def before {
var conn: Option[Connection] = None
try {
val connectionString = "jdbc:mysql://localhost:3306/sddb_test?user=root"
conn = Some(DriverManager.getConnection(connectionString))
} catch {
case _ =>
} finally {
conn map (_.close())
}
}
}
I don't get the Exception any more because I suppress it by using the PreExecution tait. But I still wonder what is going wrong here.

I cannot pin down the error, to the following, but at least better also close the result set and statement.
val stmt = conn.createStatement()
val rs = stmt.executeQuery(query)
var mysqlResult = Map[List[String], Int]()
while (rs.next) {
mysqlResult += (rs.getString("content") :: Nil) -> rs.getInt("duration")
}
sddbResult == mysqlResult && sddbResult.size == 478 must beTrue
rs.close()
stmt.close()

It's seems to be a problem with the Driver registration, the Driver has to be registered some like this...
DriverManager.registerDriver(new com.mysql.jdbc.Driver());
or some like this...
DriverManager.registerDriver(new DriverWrapper((Driver) Class.forName(props.getProperty("dbg.driver"), true, gcloader).newInstance()));
before use getConnection. I hope this help.

The driver is only loaded once.
No suitable driver usually means that the connection URL syntax is incorrect.