We Keep Coding

Neo4j: unit testing the bolt driver properly

I am adding the Neo4j Bolt driver to my application just following the http://neo4j.com/developer/java/:
import org.neo4j.driver.v1.*;
Driver driver = GraphDatabase.driver( "bolt://localhost", AuthTokens.basic( "neo4j", "neo4j" ) );
Session session = driver.session();
session.run( "CREATE (a:Person {name:'Arthur', title:'King'})" );
StatementResult result = session.run( "MATCH (a:Person) WHERE a.name = 'Arthur' RETURN a.name AS name, a.title AS title" );
while ( result.hasNext() )
{
Record record = result.next();
System.out.println( record.get( "title" ).asString() + " " + record.get("name").asString() );
}
session.close();
driver.close();
However, always from the official documentation unit testing is made using:
GraphDatabaseService db = new TestGraphDatabaseFactory()
.newImpermanentDatabaseBuilder()
So if I want to test in some way the code above, I have to replace the GraphDatabase.driver( "bolt://localhost",...) with the GraphDatabaseService from the test. How can I do that? I cannot extract any sort of in-memory driver from there as far as I can see.

The Neo4j JDBC has a class called Neo4jBoltRule for unit testing. It is a junit rule starting/stopping an impermanent database together with some configuration to start bolt.
The rule class uses dynamic port assignment to prevent test failure due to running multiple tests in parallel (think of your CI infrastructure).
An example of a unit test using that rule class is available at https://github.com/neo4j-contrib/neo4j-jdbc/blob/master/neo4j-jdbc-bolt/src/test/java/org/neo4j/jdbc/bolt/SampleIT.java

An easy way now is to pull neo4j-harness, and use their built-in Neo4jRule as follows:
import static org.neo4j.graphdb.factory.GraphDatabaseSettings.boltConnector;
// [...]
#Rule public Neo4jRule graphDb = new Neo4jRule()
.withConfig(boltConnector("0").address, "localhost:" + findFreePort());
Where findFreePort implementation can be as simple as:
private static int findFreePort() {
try (ServerSocket socket = new ServerSocket(0)) {
return socket.getLocalPort();
} catch (IOException e) {
throw new RuntimeException(e.getMessage(), e);
}
}
As the Javadoc of ServerSocket explains:
A port number of 0 means that the port number is automatically allocated, typically from an ephemeral port range. This port number can then be retrieved by calling getLocalPort.
Moreover, the socket is closed before the port value is returned, so there are great chances the returned port will still be available upon return (the window of opportunity for the port to be allocated again in between is small - the computation of the window size is left as an exercise to the reader).
Et voilà !

VSAM file locking when writing to it using Java JDBC

This is my first time trying to read and write to a VSAM file. What I did was:
Created a Map for the File using VSE Navigator
Added the Java beans VSE Connector library to my eclipse Java project
Use the code show below to Write and Read to the KSDS file.
Reading the file is not a problem but when I tried to write to the file it only works if I go on the mainframe and close the File before running my java program but it locks the file for like an hour. You cannot open the file on the mainframe or do anything to it.
Anybody can help with this problem. Is there a special setting that I need to set up for the file on the mainframe ? Why do you first need to close the file on CICS to be able to write to it ? And why does it locks the file after writing to it ?
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.sql.*;
public class testVSAM {
public static void main(String argv[]){
Integer test = Integer.valueOf(2893);
String vsamCatalog = "VSESP.USER.CATALOG";
String FlightCluster = "FLIGHT.ORDERING.FLIGHTS";
String FlightMapName = "FLIGHT.TEST2.MAP";
try{
String ipAddr = "10.1.1.1";
String userID = "USER1";
String password = "PASSWORD";
java.sql.Connection jdbcCon;
java.sql.Driver jdbcDriver = (java.sql.Driver) Class.forName(
"com.ibm.vse.jdbc.VsamJdbcDriver").newInstance();
// Build the URL to use to connect
String url = "jdbc:vsam:"+ipAddr;
// Assign properties for the driver
java.util.Properties prop = new java.util.Properties();
prop.put("port", test);
prop.put("user", userID);
prop.put("password", password);
// Connect to the driver
jdbcCon = DriverManager.getConnection(url,prop);
try {
java.sql.PreparedStatement pstmt = jdbcCon.prepareStatement(
"INSERT INTO "+vsamCatalog+"\\"+FlightCluster+"\\"+FlightMapName+
" (RS_SERIAL1,RS_SERIAL2,RS_QTY1,RS_QTY2,RS_UPDATE,RS_UPTIME,RS_EMPNO,RS_PRINTFLAG,"+
"RS_PART_S,RS_PART_IN_A_P,RS_FILLER)"+" VALUES(?,?,?,?,?,?,?,?,?,?,?)");
//pstmt.setString(1, "12345678901234567890123003");
pstmt.setString(1, "1234567890");
pstmt.setString(2,"1234567890123");
pstmt.setInt(3,00);
pstmt.setInt(4,003);
pstmt.setString(5,"151209");
pstmt.setString(6, "094435");
pstmt.setString(7,"09932");
pstmt.setString(8,"P");
pstmt.setString(9,"Y");
pstmt.setString(10,"Y");
pstmt.setString(11," ");
// Execute the query
int num = pstmt.executeUpdate();
System.out.println(num);
pstmt.close();
}
catch (SQLException t)
{
System.out.println(t.toString());
}
try
{
// Get a statement
java.sql.Statement stmt = jdbcCon.createStatement();
// Execute the query ...
java.sql.ResultSet rs = stmt.executeQuery(
"SELECT * FROM "+vsamCatalog+"\\"+FlightCluster+"\\"+FlightMapName);
while (rs.next())
{
System.out.println(rs.getString("RS_SERIAL1") + " " + rs.getString("RS_SERIAL2")+ " " + rs.getString("RS_UPTIME")+ " " + rs.getString("RS_UPDATE"));
}
rs.close();
stmt.close();
}
catch (SQLException t)
{
}
}
catch (Exception e)
{
// do something appropriate with the exception, *at least*:
e.printStackTrace();
}
}
}
Note: the OS is z/VSE

The short answer to your original question is that KSDS VSAM is not a DBMS.
As you have discovered, you can define the VSAM file such that you can update it both from batch and from CICS, but as #BillWoodger points out, you must serialize your updates yourself.
Another approach would be to do all updates from the CICS region, and have your Java application send a REST or SOAP or MQ message to CICS to request its updates. This does require there be a CICS program to catch the requests from the Java application and perform the updates.

The IBM Mainframe under z/VSE has different partitions that run different jobs. For example partition F7 CICS, partition F8 Batch Jobs, ETC.
When you define a new VSAM file you have to set the SHAREOPTIONS of the file. When I define the file I set the SHAREOPTIONS (2 3). 2 Means that only one partition can write to the file.
So when the batch program (in a different partition to the CICS partition) which is called from Java was trying to write to the file it was not able to write to the file unless I close the file in CICS first.
To fix it I REDEFINE the CICS file with SHAREOPTIONS (4 3). 4 Means that multiple partitions of the Mainframe can write to it. Fixing the problem
Below is a part of the definition code where you set the SHAREOPTION:
* $$ JOB JNM=DEFFI,CLASS=9,DISP=D,PRI=9
* $$ LST CLASS=X,DISP=H,PRI=2,REMOTE=0,USER=JAVI
// JOB DEFFI
// EXEC IDCAMS,SIZE=AUTO
DEFINE CLUSTER -
( -
NAME (FLIGHT.ORDERING.FLIGHTS) -
RECORDS (2000 1000) -
INDEXED -
KEYS (26 0) -
RECORDSIZE (128 128) -
SHAREOPTIONS (4 3) -
VOLUMES (SYSWKE) -
) -
.
.
.

How to use JCo connection without creating *.JcoDestination file

I'm trying to connect to SAP ECC 6.0 using JCo. I'm following this tutorial. However, there is a Note saying:
For this example the destination configuration is stored in a file that is called by the program. In practice you should avoid this for security reasons.
And that is reasonable and understood. But, there is no explenation how to set up secure destination provider.
I found solution in this thread that created custom implementation of DestinationDataProvider and that works on my local machine. But when I deploy it on Portal I get an error saying that there is already registered DestinationDataProvider.
So my question is:
How to store destination data in SAP Java EE application?
Here is my code to further clarify what I'm trying to do.
public static void main(String... args) throws JCoException {
CustomDestinationProviderMap provider = new CustomDestinationProviderMap();
com.sap.conn.jco.ext.Environment.registerDestinationDataProvider(provider);
Properties connectProperties = new Properties();
connectProperties.setProperty(DestinationDataProvider.JCO_ASHOST, "host.sap.my.domain.com");
connectProperties.setProperty(DestinationDataProvider.JCO_SYSNR, "00");
connectProperties.setProperty(DestinationDataProvider.JCO_CLIENT, "100");
connectProperties.setProperty(DestinationDataProvider.JCO_USER, "user");
connectProperties.setProperty(DestinationDataProvider.JCO_PASSWD, "password");
connectProperties.setProperty(DestinationDataProvider.JCO_LANG, "en");
provider.addDestination(DESTINATION_NAME1, connectProperties);
connect();
}
public static void connect() throws JCoException {
String FUNCTION_NAME = "BAPI_EMPLOYEE_GETDATA";
JCoDestination destination = JCoDestinationManager.getDestination(DESTINATION_NAME1);
JCoContext.begin(destination);
JCoFunction function = destination.getRepository().getFunction(FUNCTION_NAME);
if (function == null) {
throw new RuntimeException(FUNCTION_NAME + " not found in SAP.");
}
//function.getImportParameterList().setValue("EMPLOYEE_ID", "48");
function.getImportParameterList().setValue("FSTNAME_M", "ANAKIN");
function.getImportParameterList().setValue("LASTNAME_M", "SKYWALKER");
try {
function.execute(destination);
} catch (AbapException e) {
System.out.println(e.toString());
return;
}
JCoTable table = function.getTableParameterList().getTable("PERSONAL_DATA");
for (int i = 0; i < table.getNumRows(); i++) {
table.setRow(i);
System.out.println(table.getString("PERNO") + '\t' + table.getString("FIRSTNAME") + '\t' + table.getString("LAST_NAME")
+'\t' + table.getString("BIRTHDATE")+'\t' + table.getString("GENDER"));
}
JCoContext.end(destination);
}

Ok, so I got this up and going and thought I'd share my research.
You need to add your own destination in Portal. To achieve that you need to go to NetWeaver Administrator, located at: host:port/nwa. So it'll be something like sapportal.your.domain.com:50000/nwa.
Then you go to Configuration-> Infrastructure-> Destinations and add your destination there. You can leave empty most of the fields like Message Server. The important part is Destination name as it is how you will retrieve it and destination type which should be set to RFC Destination in my case. Try pinging your newly created destination to check if its up and going.
Finally you should be able to get destination by simply calling: JCoDestination destination = JCoDestinationManager.getDestination(DESTINATION_NAME); as it is added to your Portal environment and managed from there.

Take a look at the CustomDestinationDataProvider in the JCo examples of the Jco connector download. The important parts are:
static class MyDestinationDataProvider implements DestinationDataProvider
...
com.sap.conn.jco.ext.Environment.registerDestinationDataProvider(new MyDestinationDataProvider());
Then you can simply do:
instance = JCoDestinationManager.getDestination(DESTINATION_NAME);
Btw. you may also want to check out http://hibersap.org/ as they provide nice ways to store the config as well.

Cannot connect multiple times to mysql

I am writing an own databse in scala. To verify my results are correct, I check with a MySQL inside of a specs2 specification. I get the right result and everything is just fine. But if I run the test again without any changes, I get a SQLException: No suitable driver found for jdbc:mysql://localhost:3306/DBNAME?user=DBUSER (null:-1). Why is the driver not loaded again?
Edit
import java.sql.{ Connection, DriverManager, ResultSet }
import org.specs2.mutable.Specification
// SDDB imports ...
class DBValidationSpec extends Specification {
"SDDB and MySQl" should {
// SDDB
// ...
// JDBC
val connectionString = "jdbc:mysql://localhost:3306/sddb_test?user=root"
val query = """SELECT content, SUM( duration ) duration
FROM test
WHERE times
BETWEEN '2011-12-08'
AND '2011-12-09'
GROUP BY content"""
classOf[com.mysql.jdbc.Driver]
"give the same result" in {
// ...
//sddbResult
lazy val conn = DriverManager.getConnection(connectionString)
try{
val rs = conn.createStatement().executeQuery(query)
var mysqlResult = Map[List[String], Int]()
while (rs.next) {
mysqlResult += (rs.getString("content") :: Nil) -> rs.getInt("duration")
}
sddbResult == mysqlResult && sddbResult.size == 478 must beTrue
} finally {
conn.close()
}
}
}
}
I left out some parts of my code because they don't belong to the question.
Edit #2
The problem became even weirder. I added a second testcase. The testcase uses the same connectionString. The Exception was only raised once. The second test succeeded. I added sequential to my test definition and saw that only the first executed test raises the Exception. Afterwards I traced the classLoader to check if it is the same one. It is.
I did the following workaround:
trait PreExecuting extends Before {
override def before {
var conn: Option[Connection] = None
try {
val connectionString = "jdbc:mysql://localhost:3306/sddb_test?user=root"
conn = Some(DriverManager.getConnection(connectionString))
} catch {
case _ =>
} finally {
conn map (_.close())
}
}
}
I don't get the Exception any more because I suppress it by using the PreExecution tait. But I still wonder what is going wrong here.

I cannot pin down the error, to the following, but at least better also close the result set and statement.
val stmt = conn.createStatement()
val rs = stmt.executeQuery(query)
var mysqlResult = Map[List[String], Int]()
while (rs.next) {
mysqlResult += (rs.getString("content") :: Nil) -> rs.getInt("duration")
}
sddbResult == mysqlResult && sddbResult.size == 478 must beTrue
rs.close()
stmt.close()

It's seems to be a problem with the Driver registration, the Driver has to be registered some like this...
DriverManager.registerDriver(new com.mysql.jdbc.Driver());
or some like this...
DriverManager.registerDriver(new DriverWrapper((Driver) Class.forName(props.getProperty("dbg.driver"), true, gcloader).newInstance()));
before use getConnection. I hope this help.

The driver is only loaded once.
No suitable driver usually means that the connection URL syntax is incorrect.

How to Execute SQL Script File in Java?

I want to execute an SQL script file in Java without reading the entire file content into a big query and executing it.
Is there any other standard way?

There is great way of executing SQL scripts from Java without reading them yourself as long as you don't mind having a dependency on Ant. In my opinion such a dependency is very well justified in your case. Here is sample code, where SQLExec class lives in ant.jar:
private void executeSql(String sqlFilePath) {
final class SqlExecuter extends SQLExec {
public SqlExecuter() {
Project project = new Project();
project.init();
setProject(project);
setTaskType("sql");
setTaskName("sql");
}
}
SqlExecuter executer = new SqlExecuter();
executer.setSrc(new File(sqlFilePath));
executer.setDriver(args.getDriver());
executer.setPassword(args.getPwd());
executer.setUserid(args.getUser());
executer.setUrl(args.getUrl());
executer.execute();
}

There is no portable way of doing that. You can execute a native client as an external program to do that though:
import java.io.*;
public class CmdExec {
public static void main(String argv[]) {
try {
String line;
Process p = Runtime.getRuntime().exec
("psql -U username -d dbname -h serverhost -f scripfile.sql");
BufferedReader input =
new BufferedReader
(new InputStreamReader(p.getInputStream()));
while ((line = input.readLine()) != null) {
System.out.println(line);
}
input.close();
}
catch (Exception err) {
err.printStackTrace();
}
}
}
Code sample was extracted from here and modified to answer question assuming that the user wants to execute a PostgreSQL script file.

Flyway library is really good for this:
Flyway flyway = new Flyway();
flyway.setDataSource(dbConfig.getUrl(), dbConfig.getUsername(), dbConfig.getPassword());
flyway.setLocations("classpath:db/scripts");
flyway.clean();
flyway.migrate();
This scans the locations for scripts and runs them in order. Scripts can be versioned with V01__name.sql so if just the migrate is called then only those not already run will be run. Uses a table called 'schema_version' to keep track of things. But can do other things too, see the docs: flyway.
The clean call isn't required, but useful to start from a clean DB.
Also, be aware of the location (default is "classpath:db/migration"), there is no space after the ':', that one caught me out.

No, you must read the file, split it into separate queries and then execute them individually (or using the batch API of JDBC).
One of the reasons is that every database defines their own way to separate SQL statements (some use ;, others /, some allow both or even to define your own separator).

You cannot do using JDBC as it does not support . Work around would be including iBatis iBATIS is a persistence framework and call the Scriptrunner constructor as shown in iBatis documentation .
Its not good to include a heavy weight persistence framework like ibatis in order to run a simple sql scripts any ways which you can do using command line
$ mysql -u root -p db_name < test.sql

Since JDBC doesn't support this option the best way to solve this question is executing command lines via the Java Program. Bellow is an example to postgresql:
private void executeSqlFile() {
try {
Runtime rt = Runtime.getRuntime();
String executeSqlCommand = "psql -U (user) -h (domain) -f (script_name) (dbName)";
Process pr = rt.exec();
int exitVal = pr.waitFor();
System.out.println("Exited with error code " + exitVal);
} catch (Exception e) {
System.out.println(e.toString());
}
}

The Apache iBatis solution worked like a charm.
The script example I used was exactly the script I was running from MySql workbench.
There is an article with examples here:
https://www.tutorialspoint.com/how-to-run-sql-script-using-jdbc#:~:text=You%20can%20execute%20.,to%20pass%20a%20connection%20object.&text=Register%20the%20MySQL%20JDBC%20Driver,method%20of%20the%20DriverManager%20class.
This is what I did:
pom.xml dependency
<!-- IBATIS SQL Script runner from Apache (https://mvnrepository.com/artifact/org.apache.ibatis/ibatis-core) -->
<dependency>
<groupId>org.apache.ibatis</groupId>
<artifactId>ibatis-core</artifactId>
<version>3.0</version>
</dependency>
Code to execute script:
import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.io.Reader;
import java.sql.Connection;
import org.apache.ibatis.jdbc.ScriptRunner;
import lombok.extern.slf4j.Slf4j;
#Slf4j
public class SqlScriptExecutor {
public static void executeSqlScript(File file, Connection conn) throws Exception {
Reader reader = new BufferedReader(new FileReader(file));
log.info("Running script from file: " + file.getCanonicalPath());
ScriptRunner sr = new ScriptRunner(conn);
sr.setAutoCommit(true);
sr.setStopOnError(true);
sr.runScript(reader);
log.info("Done.");
}
}

For my simple project the user should be able to select SQL-files which get executed.
As I was not happy with the other answers and I am using Flyway anyway I took a closer look at the Flyway code. DefaultSqlScriptExecutor is doing the actual execution, so I tried to figure out how to create an instance of DefaultSqlScriptExecutor.
Basically the following snippet loads a String splits it into the single statements and executes one by one.
Flyway also provides other LoadableResources than StringResource e.g. FileSystemResource. But I have not taken a closer look at them.
As DefaultSqlScriptExecutor and the other classes are not officially documented by Flyway use the code-snippet with care.
public static void execSqlQueries(String sqlQueries, Configuration flyWayConf) throws SQLException {
// create dependencies FlyWay needs to execute the SQL queries
JdbcConnectionFactory jdbcConnectionFactory = new JdbcConnectionFactory(flyWayConf.getDataSource(),
flyWayConf.getConnectRetries(),
null);
DatabaseType databaseType = jdbcConnectionFactory.getDatabaseType();
ParsingContext parsingContext = new ParsingContext();
SqlScriptFactory sqlScriptFactory = databaseType.createSqlScriptFactory(flyWayConf, parsingContext);
Connection conn = flyWayConf.getDataSource().getConnection();
JdbcTemplate jdbcTemp = new JdbcTemplate(conn);
ResourceProvider resProv = flyWayConf.getResourceProvider();
DefaultSqlScriptExecutor scriptExec = new DefaultSqlScriptExecutor(jdbcTemp, null, false, false, false, null);
// Prepare and execute the actual queries
StringResource sqlRes = new StringResource(sqlQueries);
SqlScript sqlScript = sqlScriptFactory.createSqlScript(sqlRes, true, resProv);
scriptExec.execute(sqlScript);
}

The simplest external tool that I found that is also portable is jisql - https://www.xigole.com/software/jisql/jisql.jsp .
You would run it as:
java -classpath lib/jisql.jar:\
lib/jopt-simple-3.2.jar:\
lib/javacsv.jar:\
/home/scott/postgresql/postgresql-8.4-701.jdbc4.jar
com.xigole.util.sql.Jisql -user scott -password blah \
-driver postgresql \
-cstring jdbc:postgresql://localhost:5432/scott -c \; \
-query "select * from test;"

JDBC does not support this option (although a specific DB driver may offer this).
Anyway, there should not be a problem with loading all file contents into memory.

Try this code:
String strProc =
"DECLARE \n" +
" sys_date DATE;"+
"" +
"BEGIN\n" +
"" +
" SELECT SYSDATE INTO sys_date FROM dual;\n" +
"" +
"END;\n";
try{
DriverManager.registerDriver ( new oracle.jdbc.driver.OracleDriver () );
Connection connection = DriverManager.getConnection ("jdbc:oracle:thin:#your_db_IP:1521:your_db_SID","user","password");
PreparedStatement psProcToexecute = connection.prepareStatement(strProc);
psProcToexecute.execute();
}catch (Exception e) {
System.out.println(e.toString());
}

If you use Spring you can use DataSourceInitializer:
#Bean
public DataSourceInitializer dataSourceInitializer(#Qualifier("dataSource") final DataSource dataSource) {
ResourceDatabasePopulator resourceDatabasePopulator = new ResourceDatabasePopulator();
resourceDatabasePopulator.addScript(new ClassPathResource("/data.sql"));
DataSourceInitializer dataSourceInitializer = new DataSourceInitializer();
dataSourceInitializer.setDataSource(dataSource);
dataSourceInitializer.setDatabasePopulator(resourceDatabasePopulator);
return dataSourceInitializer;
}
Used to set up a database during initialization and clean up a
database during destruction.
https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/jdbc/datasource/init/DataSourceInitializer.html