We Keep Coding

Jooq Converter Cast Exceptions

I'm currently trying to store encrypted data in some of the columns of a Postgres DB. After receiving helpful feedback from this question: client-side-encryption-with-java-and-postgres-database I am using converters/bindings to implement transparent encryption in the JDBC layer.
Right now I'm trying to insert a BigDecimal[][][] into a Postgres DB column of type bytea.
The insertion works but the problem is that the encryption code I've added in the converters/binding doesn't seem to run. Unfortunately, when I check the database I'm seeing an unencrypted 3D matrix. (FYI my encryption utility code is tested and does work)
To test, I put my encryption code in the DAO layer and the BigDecimal[][][] matrix does get encrypted on DB inserts. Although I could do this it defeats the purpose of using converters/bindings for encryption.
So my question:
With the code I provided below am I doing anything wrong that is preventing the encryption code in my converter/binding to be run? I thought after a Prepared Statement is executed the converter is the next step but maybe not? I have a lack of knowledge on just when the converter/binding code gets called in the whole JOOQ flow so any insight is much appreciated! Thanks :D
First I'm using a PreparedStatment in a DAO to execute the insert query.
I can't show the full code but basically for the stmt I'm setting the BigDecimal[][][] as an object parameter:
private Result executeInsert(BigDecimal[][][] valueToEncrypt, String insertSql) {
try (Connection conn = config.connectionProvider().acquire();
PreparedStatement stmt = conn.prepareStatement(insertSql)) {
// Get a human readable version of the 3d matrix to insert into the db.
PostgresReadableArray humanReadableMatrix = getPostgresReadableArray(valueToEncrypt)
stmt.setObject(parameterIndex++, humanReadableMatrix, Types.OTHER);
ResultSet res = stmt.executeQuery();
}
...
}
I am currently attaching the binding to a codegen xml file here:
<forcedType>
<userType>
java.math.BigDecimal[][][]
</userType>
<binding>com.myapp.EncryptionBinding</binding>
<includeExpression>matrix_column</includeExpression>
<includeTypes>bytea</includeTypes>
</forcedType>
Here is my binding class EncryptionBinding:
public class EncryptionBinding implements Binding<byte[], BigDecimal[][][]> {
#Override
public Converter<byte[], BigDecimal[][][]> converter() {
return new MatrixConverter();
}
// Rending a bind variable for the binding context's value and casting it to the json type
#Override
public void sql(BindingSQLContext<BigDecimal[][][]> ctx) throws SQLException {
}
// Registering VARCHAR types for JDBC CallableStatement OUT parameters
#Override
public void register(BindingRegisterContext<BigDecimal[][][]> ctx) throws SQLException {
ctx.statement().registerOutParameter(ctx.index(), Types.VARCHAR);
}
// Converting the BigDecimal[][][] to a Encrypted value and setting that on a JDBC PreparedStatement
#Override
public void set(BindingSetStatementContext<BigDecimal[][][]> ctx) throws SQLException {
ctx.statement().setBytes(ctx.index(), ctx.convert(converter()).value());
}
...
Here is my converter class MatrixConverter used in the above EncryptionBinding class:
public class MatrixConverter extends AbstractConverter<byte[], BigDecimal[][][]> {
private static final Logger logger = LoggerFactory.getLogger(MatrixConverter.class);
public MatrixConverter() {
super(byte[].class, BigDecimal[][][].class);
}
#Override
public BigDecimal[][][] from(byte[] databaseObject) {
return EncryptionUtils.decrypt(databaseObject);
}
#Override
public byte[] to(BigDecimal[][][] userObject) {
return EncryptionUtils.encrypt(JsonUtils.toJson(userObject));
}
}

Unit tests for one main method

I have a method. What is the best practice to unit testing?
You can make a mock Connection and ResultSet and pass as a method parameter as an object but I find it stupid, unprofessional solution.
public static void playA() {
MyObject object = check("cat");
// some stuff...
}
private static MyObject check(String s) throws SQLException {
// some stuff checking string
MyObject o = methodA(s);
// some stuff doing on the object
return o;
}
private static MyObject methodA(String s) throws SQLException {
Connection c = MySettings.getConnection();
PreparedStatement ps = c.prepareStatement("select * from Objects where type = ?");
ps.setString(1, s);
ResultSet resultSet = ps.executeQuery();
while (resultSet.next()) {
// mapping
return object;
}
return null;
}

My take is that it should be according the level of abstraction you want to test. Mocking Connection and ResultSet can be fine if you want to check the the logic of your SQL for example.
If you want to be precise testing the database connection layer you can use tools for unit testing database like http://dbunit.sourceforge.net/
Also try to have your code as most Object Oriented as possible. that would help you in testing ( although I know it's probably just an example )

Convert java.sql.ResultSet to stream in java 8 [duplicate]

I have few tables with big amount of data (about 100 million records). So I can't store this data in memory but I would like to stream this result set using java.util.stream class and pass this stream to another class. I read about Stream.of and Stream.Builder operators but they are buffered streams in memory. So is there any way to resolve this question?
UPDATE #1
Okay I googled and found jooq library. I'm not sure but looks like it could be applicable to my test case. To summarize I have few tables with big amount of data. I would like to stream my resultset and transfer this stream to another method. Something like this:
// why return Stream<String>? Because my result set has String type
private Stream<Record> writeTableToStream(DataSource dataSource, String table) {
Stream<Record> record = null;
try (Connection connection = dataSource.getConnection()) {
String sql = "select * from " + table;
try (PreparedStatement pSt = connection.prepareStatement(sql)) {
connection.setAutoCommit(false);
pSt.setFetchSize(5000);
ResultSet resultSet = pSt.executeQuery();
//
record = DSL.using(connection)
.fetch(resultSet).stream();
}
} catch (SQLException sqlEx) {
logger.error(sqlEx);
}
return record;
}
Could please someone advise, am I on correct way? Thanks.
UPDATE #2
I made some experiment on jooq and could say now that above decision is not suitable for me. This code record = DSL.using(connection).fetch(resultSet).stream(); takes too much time

The first thing you have to understand is that code like
try (Connection connection = dataSource.getConnection()) {
…
try (PreparedStatement pSt = connection.prepareStatement(sql)) {
…
return stream;
}
}
does not work as by the time you leave the try blocks, the resources are closed while the processing of the Stream hasn’t even started.
The resource management construct “try with resources” works for resources used within a block scope inside a method but you are creating a factory method returning a resource. Therefore you have to ensure that the closing of the returned stream will close the resources and the caller is responsible for closing the Stream.
Further, you need a function which produces an item out of a single line from the ResultSet. Supposing, you have a method like
Record createRecord(ResultSet rs) {
…
}
you may create a Stream<Record> basically like
Stream<Record> stream = StreamSupport.stream(new Spliterators.AbstractSpliterator<Record>(
Long.MAX_VALUE,Spliterator.ORDERED) {
#Override
public boolean tryAdvance(Consumer<? super Record> action) {
if(!resultSet.next()) return false;
action.accept(createRecord(resultSet));
return true;
}
}, false);
But to do it correctly you have to incorporate the exception handling and closing of resources. You can use Stream.onClose to register an action that will be performed when the Stream gets closed, but it has to be a Runnable which can not throw checked exceptions. Similarly the tryAdvance method is not allowed to throw checked exceptions. And since we can’t simply nest try(…) blocks here, the program logic of suppression exceptions thrown in close, when there is already a pending exception, doesn’t come for free.
To help us here, we introduce a new type which can wrap closing operations which may throw checked exceptions and deliver them wrapped in an unchecked exception. By implementing AutoCloseable itself, it can utilize the try(…) construct to chain close operations safely:
interface UncheckedCloseable extends Runnable, AutoCloseable {
default void run() {
try { close(); } catch(Exception ex) { throw new RuntimeException(ex); }
}
static UncheckedCloseable wrap(AutoCloseable c) {
return c::close;
}
default UncheckedCloseable nest(AutoCloseable c) {
return ()->{ try(UncheckedCloseable c1=this) { c.close(); } };
}
}
With this, the entire operation becomes:
private Stream<Record> tableAsStream(DataSource dataSource, String table)
throws SQLException {
UncheckedCloseable close=null;
try {
Connection connection = dataSource.getConnection();
close=UncheckedCloseable.wrap(connection);
String sql = "select * from " + table;
PreparedStatement pSt = connection.prepareStatement(sql);
close=close.nest(pSt);
connection.setAutoCommit(false);
pSt.setFetchSize(5000);
ResultSet resultSet = pSt.executeQuery();
close=close.nest(resultSet);
return StreamSupport.stream(new Spliterators.AbstractSpliterator<Record>(
Long.MAX_VALUE,Spliterator.ORDERED) {
#Override
public boolean tryAdvance(Consumer<? super Record> action) {
try {
if(!resultSet.next()) return false;
action.accept(createRecord(resultSet));
return true;
} catch(SQLException ex) {
throw new RuntimeException(ex);
}
}
}, false).onClose(close);
} catch(SQLException sqlEx) {
if(close!=null)
try { close.close(); } catch(Exception ex) { sqlEx.addSuppressed(ex); }
throw sqlEx;
}
}
This method wraps the necessary close operation for all resources, Connection, Statement and ResultSet within one instance of the utility class described above. If an exception happens during the initialization, the close operation is performed immediately and the exception is delivered to the caller. If the stream construction succeeds, the close operation is registered via onClose.
Therefore the caller has to ensure proper closing like
try(Stream<Record> s=tableAsStream(dataSource, table)) {
// stream operation
}
Note that also the delivery of an SQLException via RuntimeException has been added to the tryAdvance method. Therefore you may now add throws SQLException to the createRecord method without problems.

jOOQ
I'm going to answer the jOOQ part of your question. As of jOOQ 3.8, there have now been quite a few additional features related to combining jOOQ with Stream. Other usages are also documented on this jOOQ page.
Your suggested usage:
You tried this:
Stream<Record> stream = DSL.using(connection).fetch(resultSet).stream();
Indeed, this doesn't work well for large result sets because fetch(ResultSet) fetches the entire result set into memory and then calls Collection.stream() on it.
Better (lazy) usage:
Instead, you could write this:
try (Stream<Record> stream = DSL.using(connection).fetchStream(resultSet)) {
...
}
... which is essentially convenience for this:
try (Cursor<Record> cursor = DSL.using(connection).fetchLazy(resultSet)) {
Stream<Record> stream = cursor.stream();
...
}
See also DSLContext.fetchStream(ResultSet)
Of course, you could also let jOOQ execute your SQL string, rather than wrestling with JDBC:
try (Stream<Record> stream =
DSL.using(dataSource)
.resultQuery("select * from {0}", DSL.name(table)) // Prevent SQL injection
.fetchSize(5000)
.fetchStream()) {
...
}
The dreaded SELECT *
As was criticised in the comments, their jOOQ usage seemed slow because of how jOOQ eagerly fetches LOB data into memory despite using fetchLazy(). The word "lazy" corresponds to fetching records lazily (one by one), not fetching column data lazily. A record is completely fetched in one go, assuming you actually want to project the entire row.
If you don't need some heavy rows, don't project them! SELECT * is almost always a bad idea in SQL. Drawbacks:
It causes a lot more I/O and memory overhead in the database server, the network, and the client.
It prevents covering index usage
It prevents join elimination transformations
More info in this blog post here.
On try-with-resources usage
Do note that a Stream produced by jOOQ is "resourceful", i.e. it contains a reference to an open ResultSet (and PreparedStatement). So, if you really want to return that stream outside of your method, make sure it is closed properly!

I'm not aware of any well-known library that will do it for you.
That said, this article shows how to wrap the resultset with an Iterator (ResultSetIterator) and pass it as the first parameter to Spliterators.spliteratorUnknownSize() in order to create a Spliterator.
The Spliterator can then be used by StreamSupport in order to create a Stream on top of it.
Their suggested implementation of ResultSetIterator class:
public class ResultSetIterator implements Iterator {
private ResultSet rs;
private PreparedStatement ps;
private Connection connection;
private String sql;
public ResultSetIterator(Connection connection, String sql) {
assert connection != null;
assert sql != null;
this.connection = connection;
this.sql = sql;
}
public void init() {
try {
ps = connection.prepareStatement(sql);
rs = ps.executeQuery();
} catch (SQLException e) {
close();
throw new DataAccessException(e);
}
}
#Override
public boolean hasNext() {
if (ps == null) {
init();
}
try {
boolean hasMore = rs.next();
if (!hasMore) {
close();
}
return hasMore;
} catch (SQLException e) {
close();
throw new DataAccessException(e);
}
}
private void close() {
try {
rs.close();
try {
ps.close();
} catch (SQLException e) {
//nothing we can do here
}
} catch (SQLException e) {
//nothing we can do here
}
}
#Override
public Tuple next() {
try {
return SQL.rowAsTuple(sql, rs);
} catch (DataAccessException e) {
close();
throw e;
}
}
}
and then:
public static Stream stream(final Connection connection,
final String sql,
final Object... parms) {
return StreamSupport
.stream(Spliterators.spliteratorUnknownSize(
new ResultSetIterator(connection, sql), 0), false);
}

Here is the simplest sample by abacus-jdbc.
final DataSource ds = JdbcUtil.createDataSource(url, user, password);
final SQLExecutor sqlExecutor = new SQLExecutor(ds);
sqlExecutor.stream(sql, parameters).filter(...).map(...).collect(...) // lazy execution&loading and auto-close Statement/Connection
Or:
JdbcUtil.prepareQuery(ds, sql)
.stream(ResultRecord.class) // or RowMapper.MAP/...
.filter(...).map(...).collect(...) // lazy execution&loading and auto-close Statement/Connection
This is totally lazy loading and auto-closure. The records will loaded from db by fetch size (default if not specified) and the Statement and Connection will automatically closed after the result/records are collected.
Disclosure： I'm the developer of AbacusUtil.

Using my library it would be done like this:
attach maven dependency:
<dependency>
<groupId>com.github.buckelieg</groupId>
<artifactId>db-fn</artifactId>
<version>0.3.4</version>
</dependency>
use library in code:
Function<Stream<I>, O> processor = stream -> //process input stream
try (DB db = new DB("jdbc:postgresql://host:port/database?user=user&password=pass")) {
processor.apply(
db.select("SELECT * FROM my_table t1 JOIN my_table t2 ON t1.id = t2.id")
.fetchSize(5000)
.execute(rs -> /*ResultSet mapper*/)
);
}
See more here

Some common module called Tools of a Ujorm framework offers a simple solution using the RowIterator class.
Example of use:
PreparedStatement ps = dbConnection.prepareStatement("SELECT * FROM myTable");
new RowIterator(ps).toStream().forEach((RsConsumer)(resultSet) -> {
int value = resultSet.getInt(1);
});
Maven dependency on the Tools library (50KB):
<dependency>
<groupId>org.ujorm</groupId>
<artifactId>ujo-tools</artifactId>
<version>1.93</version>
</dependency>
See jUnit test for more information.

I just did the summary to provide the real example about how to stream ResultSet and do the simple SQL query without using 3rd
click here for detail
Blockquote: Java 8 provided the Stream family and easy operation of it. The way of pipeline usage made the code clear and smart.
However, ResultSet is still go with very legacy way to process. Per actual ResultSet usage, it is really helpful if converted as Stream.
....
StreamUtils.uncheckedConsumer is required to convert the the SQLException to runtimeException to make the Lamda clear.

How to get parameters from PreparedStatement?

I'm writing generic logger for SQLException and I'd like to get parameters that were passed into PreparedStatement, how to do it ? I was able to get the count of them.
ParameterMetaData metaData = query.getParameterMetaData();
parameterCount = metaData.getParameterCount();

Short answer: You can't.
Long answer: All JDBC drivers will keep the parameter values somewhere but there is no standard way to get them.
If you want to print them for debugging or similar purposes, you have several options:
Create a pass-through JDBC driver (use p6spy or log4jdbc as a basis) which keeps copies of the parameters and offers a public API to read them.
Use Java Reflection API (Field.setAccessible(true) is your friend) to read the private data structures of the JDBC drivers. That's my preferred approach. I have a factory which delegates to DB specific implementations that can decode the parameters and that allows me to read the parameters via getObject(int column).
File a bug report and ask that the exceptions are improved. Especially Oracle is really stingy when it comes to tell you what's wrong.

Solution 1: Subclass
Simply create a custom implementation of a PreparedStatement which delegates all calls to the original prepared statement, only adding callbacks in the setObject, etc. methods. Example:
public PreparedStatement prepareStatement(String sql) {
final PreparedStatement delegate = conn.prepareStatement(sql);
return new PreparedStatement() {
// TODO: much more methods to delegate
#Override
public void setString(int parameterIndex, String x) throws SQLException {
// TODO: remember value of X
delegate.setString(parameterIndex, x);
}
};
}
If you want to save parameters and get them later, there are many solutions, but I prefer creating a new class like ParameterAwarePreparedStatement which has the parameters in a map. The structure could be similar to this:
public class ParameterAwarePreparedStatement implements PreparedStatement {
private final PreparedStatement delegate;
private final Map<Integer,Object> parameters;
public ParameterAwarePreparedStatement(PreparedStatement delegate) {
this.delegate = delegate;
this.parameters = new HashMap<>();
}
public Map<Integer,Object> getParameters() {
return Collections.unmodifiableMap(parameters);
}
// TODO: many methods to delegate
#Override
public void setString(int parameterIndex, String x) throws SQLException {
delegate.setString(parameterIndex, x);
parameters.put(parameterIndex, x);
}
}
Solution 2: Dynamic proxy
This second solution is shorter, but seems more hacky.
You can create a dynamic proxy by calling a factory method on java.lang.reflect.Proxy and delegate all calls on the original instance. Example:
public PreparedStatement prepareStatement(String sql) {
final PreparedStatement ps = conn.prepareStatement(sql);
final PreparedStatement psProxy = (PreparedStatement) Proxy.newProxyInstance(ClassLoader.getSystemClassLoader(), new Class<?>[]{PreparedStatement.class}, new InvocationHandler() {
#Override
public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
if (method.getName().equals("setLong")) {
// ... your code here ...
}
// this invokes the default call
return method.invoke(ps, args);
}
});
return psProxy;
}
Then you intercept the setObject, etc. calls by looking at method names and looking to the second method arguments for your values.

This article, from Boulder, ahtoulgh DB 2 "specific", gives a complete example of ParameterMetadata usage.

Java: Prepare a statement without a connection

I'm trying to generate some sql files in my java application.
The application will not execute any sql statements, just generate a file with sql statements and save it.
I'd like to use the java.sql.PreparedStatement to create my statements so that i don't have to validate every string etc. with my own methods.
Is there a way to use the PreparedStatement without the calling java.sql.Connection.prepareStatement(String) function, because I don't have a java.sql.Connection?

Take a look at this Java library: http://openhms.sourceforge.net/sqlbuilder/

I'm guessing that until you've got a sql connection, the parser won't know what rules to apply. I'm guessing that it's actually the SQL driver or even server that's compiling the sql statement.
Assuming your sql is simple enough, then how about using a cheap connection, like, say a sqlite connection.
SQLite will create a new database on the fly if the database you're attempting to connect to does not exist.
public Connection connectToDatabase() {
// connect to the database (creates new if not found)
try {
Class.forName("org.sqlite.JDBC");
conn = DriverManager.getConnection("jdbc:sqlite:mydatabase.db");
// initialise the tables if necessary
this.createDatabase(conn);
}
catch (java.lang.ClassNotFoundException e) {
System.out.println(e.getMessage());
}
catch (java.sql.SQLException e) {
System.out.println(e.getMessage());
}
return conn;
}

Not really. Preparing a statement in most cases means that it will be compiled by DBMS which is "hard" without connection.
http://java.sun.com/docs/books/tutorial/jdbc/basics/prepared.html

This is a dastardly devious problem, thankfully it's pretty easy to cope with:
public class PreparedStatementBuilder
{
private String sql; // the sql to be executed
public PreparedStatementBuilder(final String sql) { this.sql = sql; }
protected void preparePrepared(final PreparedStatement preparedStatement)
throws SQLException
{
// this virtual method lets us declare how, when we do generate our
// PreparedStatement, we want it to be setup.
// note that at the time this method is overridden, the
// PreparedStatement has not yet been created.
}
public PreparedStatement build(final Connection conn)
throws SQLException
{
// fetch the PreparedStatement
final PreparedStatement returnable = conn.prepareStatement(sql);
// perform our setup directives
preparePrepared(returnable);
return returnable;
}
}
To use, just write an anonymous class that overrides void preparePrepared(PreparedStatement):
final String sql = "SELECT * FROM FOO WHERE USER = ?";
PreparedStatementBuilder psBuilder = new PreparedStatementBuilder(sql){
#Override
protected void preparePrepared(PreparedStatement preparedStatement)
throws SQLException
{
preparedStatement.setString(1, "randal");
}};
return obtainResultSet(psBuilder);
Presto! You now have a way to work with a PreparedStatement without yet having built it. Here's an example showing the minimal boilerplate you'd otherwise have to copy paste to kingdom come, every time you wanted to write a different statement:
public ResultSet obtainResultSet(final PreparedStatementBuilder builder)
throws SQLException {
final Connection conn = this.connectionSource.getConnection();
try
{
// your "virtual" preparePrepared is called here, doing the work
// you've laid out for your PreparedStatement now that it's time
// to actually build it.
return builder.build(conn).executeQuery();
}
finally
{
try { conn.close(); }
catch (SQLException e) { log.error("f7u12!", e); }
}
}
You really really don't want to be copy pasting that everywhere, do you?

Try implementing PreparedStatement.
Example : class YourOwnClass implements PreparedStatement {
// 1. Do implement all the methods ,
2. Get the minimal logic to implement from OraclePreparedStatement(classes12.jar) or
sun.jdbc.odbc.JdbcOdbcCallableStatement
}