We Keep Coding

Stream very large table via spring-data-jdbc Resultset.TYPE_FORWARD_ONLY READ_ONLY

Hi I would like to stream a very large table spring-data-jdbc. For this purpose
I have set my connection to READ_ONLY I have declared in my repository a method that looks in the following way:
PackageRepository extends Repository<Package,String> {
Stream<Package> findAll();
}
My expectation here would be that the resultset would be of type FORWARD_ONLY and this method will not block indefinatly untill all results are recieved from the database.
Here I would make a comparison with Spring Data JPA where the Stream methods are not blocking and the content of the database is fetched in portions depending on the fetch size.
Have I missed some configuration ? How can I achieve this behaviour with spring-data-jdbc ?
UPDATE: I will put the question in a different form. How can I achieve with spring-data-jdbs the equivalent of:
template.query(new PreparedStatementCreator() {
#Override
public PreparedStatement createPreparedStatement(Connection con) throws SQLException {
PreparedStatement statement = con.prepareStatement("select * from MYTABLE with UR",ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY);
statement.setFetchSize(150000);
return statement;
}
}, new RowCallbackHandler() {
#Override
public void processRow(ResultSet rs) throws SQLException {
// do my processing here
}
});

Just adding thesetFetchSize(Integer.MIN_VALUE) before querying, the queryForStream indeed gives us a stream which load records one by one rather than eagerly load all records into memroy in one shot.
namedTemplate.getJdbcTemplate().setFetchSize(Integer.MIN_VALUE);
Stream<LargeEntity> entities = namedTemplate.queryForStream(sql, params, rowMapper);
dependencies:
spring framework 5.3+
mysql-connector-java 8.0.x (or mariadb-java-client 2.7.x)

Convert methods to a generic method in DAO layer

Currently I'm having some methods in my DAO layer with multiple select queries. What I was thinking is to have a generic method for all these three methods so that it can be used for further also. Here are my methods.
public List<Customer> findAll(){
String sql = "SELECT * FROM CUSTOMER";
List<Customer> customers = getJdbcTemplate().query(sql,
new BeanPropertyRowMapper(Customer.class));
return customers;
}
For finding phone numbers of a customer.
public List<Phone> findPhoneNumbers(int custId){
String sql = "SELECT * FROM PHONE WHERE CUST_ID="+custId;
List<Phone> phoneNumbers = getJdbcTemplate().query(sql,
new BeanPropertyRowMapper(Phone.class));
return phoneNumbers;
}
and so on.
Can these methods can be converted in a single generic method, so that it can be called from my service layer. Any suggestions or ideas would be greatly appreciated.

You can use a generic method and let the caller specify the class.
It's easy for findAll
public <T> List<T> findAll(Class<T> entityClass, String tableName){
String sql = "SELECT * FROM " + tableName;
return getJdbcTemplate().query(sql,
new BeanPropertyRowMapper(entityClass));
return phoneNumbers;
}
You can further improve this by using class metadata, such as specifying an annotations:
#Target(ElementType.Type)
public #interface MappedTable {
String tableName();
}
//apply the annotation to bean classes:
#MappedTable(tableName="CUSTOMERS")
public class Customer {}
Which will allow your findAll method to look like:
public <T> List<T> findAll(Class<T> entityClass){
String sql = "SELECT * FROM " + entityClass
.getAnnotation(MappedTable.class).tableName();
return getJdbcTemplate().query(sql,
new BeanPropertyRowMapper(entityClass));
return phoneNumbers;
}
Implementing findByXyZ will be trickier, but you can use a similar approach that either takes the full query from the caller or that uses additional annotation-based metadata.
Note that the complexity of this kind of code grows very fast. That's why it's a good idea to consider using ORM tools instead of reinventing them (the above is just an idea for simple cases as the one in the question).

public <T> List<T> findAll(String sql, Class<T> clazz) {
return getJdbcTemplate().query(sql, new BeanPropertyRowMapper(clazz));
}
Then you call
String sql = "SELECT * FROM CUSTOMER";
List<Customer> customers = findAll(sql, Customer.class);
Or
String sql = "SELECT * FROM PHONE WHERE CUST_ID="+custId;
List<Phone> phoneNumbers = findAll(sql, Phone.class);
The problem is that you have to construct sql statement before calling the method.
Maybe you want an android sqldatabase-like interface and construct sql statement in the method.
public <T> List<T> findAll(String table, String[] columns, String selection, String[] selectionArgs, String groupBy, String having, String orderBy, Class<T> clazz) {
String sql = // construct sql statement
return getJdbcTemplate().query(sql, new BeanPropertyRowMapper(clazz));
}
I'm not familiar with sql statement, you may find something in the Android SDK SQLiteQueryBuilder.buildQueryString(...).
If you really want to return a single item not a list with one item, you need another method.
public <T> T findOne(String sql, Class<T> clazz) {
// your sql statement should contains something like "limit 1"
List<T> result = findAll(sql, clazz);
return result.isEmpty() ? null : result.get(0);
}
If you need one method but you require to return List<T> for multiple data and T for single data. Try this:
public <T> Object findAll(String sql, Class<T> clazz, bool one) {
List<T> all = getJdbcTemplate().query(sql, new BeanPropertyRowMapper(clazz));
return one ? all.get(0) : all;
}
But I don't really suggest this approach. I don't think it is necessary to return a single item rather than a list with a single item.

You can use inheritance to achieve this
1) class Customer
2)class PhoneNumber extends Customer
3) class Address extends Customer
4) Declaring methods as
public List<T> findPhoneNumbers(Customer customer){
if(customer instanceOf PhoneNumber) {sql ="get phone number query ...."}
else if(customer instanceOf Address) {sql ="get address query ...."}
else {sql = get customer query}
}
Now in the calling method(ServiceLayer), you can pass appropriate object to choose query and get result

Custom mapper class is not getting the result?

I am writing a DAO application and I want to get zone data. I have written my own mapper class. I am unable to get values. It is throwing an exception.
Mapper class
public class zoneMapper implements RowMapper {
#Override
public Object mapRow(ResultSet resultSet, int rowNum) throws SQLException {
ZoneBean zone=new ZoneBean();
zone.setZoneId(resultSet.getInt("ZONE_ID"));
zone.setZoneName(resultSet.getString("ZONE_NAME"));
return zone;
}
dao call
List<ZoneBean> zoneList=new ArrayList<ZoneBean>();
try {
JdbcTemplate jdbcTemplate = new JdbcTemplate(dataSource);
zoneList = (List<ZoneBean>) jdbcTemplate.queryForObject(
queriesConstants.GET_ZONES, new zoneMapper());
}catch(Exception e){
System.out.println("getZones "+e.getMessage());
//logger.error("getZones "+e.getMessage());
}
**Exception is**
getZones Incorrect result size: expected 1, actual 17

getZones Incorrect result size: expected 1, actual 17
You retrieve multiple elements when your execute your query.
public <T> T queryForObject(String sql, RowMapper<T> rowMapper) is therefore not suitable for your need because it executes a query given static SQL, mapping a single result row to a Java object via a RowMapper.
You should use rather this method :
public <T> List<T> query(String sql, RowMapper<T> rowMapper)which executes a query given static SQL, mapping each row to a Java object via a RowMapper.

JDBI, retrieve data with sql query into customized object(constructor) instead of Map

So when we use JDBI to query from database, it is getting it into a Map<String, Object> type.
I want to get it as my customized object (constructor) instead of Map<String, Object>.
DBI dbi = establishConnection(url, userName, passWord);
Handle handle = dbi.open();
List<Map<String, Object>> rs = handle.select("select * from sometable");
Instead I want to use:
List<customizedObject> rs = handle.select("select * from sometable");
Where customizedObject class is an object that contains all the column properties with it.
Is there any way to do this? I found some relative documentation, but I cannot really understand the implementation.
http://jdbi.org/sql_object_api_queries/

Please also see the previous page in the documentation that shows how to link your Handle or DBI with the mappers.
Essentially, you need a mapper to convert the ResultSet to the desired object and an interface to refer to the mapper.
Let's assume a minimal example. First the mapper needs to be provided:
public class CustomizedObjectMapper implements ResultSetMapper<customizedObject> {
#Override
public customizedObject map(int index, ResultSet r, StatementContext ctx)
throws SQLException {
return new customizedObject(r.getString("uuid"), r.getString("other_column"));
}
}
Then we need an interface to define which query provides the data that is passed to the mapper class. One result row leads to one invocation of CustomizedObjectMapper.map(...):
#RegisterMapper(CustomizeObjectMapper.class)
public interface CustomizeObjectQuery {
#SqlQuery("Select uuid, other_column from schema.relation")
List<customizedObject> get();
}
Finally, the objects can be retrieved: List<customizedObject> test = dbi.open(CustomizeObjectQuery.class).get().
Your can also put the components together on an individual basis like so and omit the interface:
dbi.open().createQuery("Select uuid, other_colum from schema.relation").map(new EventMapper()).list()

How to get parameters from PreparedStatement?

I'm writing generic logger for SQLException and I'd like to get parameters that were passed into PreparedStatement, how to do it ? I was able to get the count of them.
ParameterMetaData metaData = query.getParameterMetaData();
parameterCount = metaData.getParameterCount();

Short answer: You can't.
Long answer: All JDBC drivers will keep the parameter values somewhere but there is no standard way to get them.
If you want to print them for debugging or similar purposes, you have several options:
Create a pass-through JDBC driver (use p6spy or log4jdbc as a basis) which keeps copies of the parameters and offers a public API to read them.
Use Java Reflection API (Field.setAccessible(true) is your friend) to read the private data structures of the JDBC drivers. That's my preferred approach. I have a factory which delegates to DB specific implementations that can decode the parameters and that allows me to read the parameters via getObject(int column).
File a bug report and ask that the exceptions are improved. Especially Oracle is really stingy when it comes to tell you what's wrong.

Solution 1: Subclass
Simply create a custom implementation of a PreparedStatement which delegates all calls to the original prepared statement, only adding callbacks in the setObject, etc. methods. Example:
public PreparedStatement prepareStatement(String sql) {
final PreparedStatement delegate = conn.prepareStatement(sql);
return new PreparedStatement() {
// TODO: much more methods to delegate
#Override
public void setString(int parameterIndex, String x) throws SQLException {
// TODO: remember value of X
delegate.setString(parameterIndex, x);
}
};
}
If you want to save parameters and get them later, there are many solutions, but I prefer creating a new class like ParameterAwarePreparedStatement which has the parameters in a map. The structure could be similar to this:
public class ParameterAwarePreparedStatement implements PreparedStatement {
private final PreparedStatement delegate;
private final Map<Integer,Object> parameters;
public ParameterAwarePreparedStatement(PreparedStatement delegate) {
this.delegate = delegate;
this.parameters = new HashMap<>();
}
public Map<Integer,Object> getParameters() {
return Collections.unmodifiableMap(parameters);
}
// TODO: many methods to delegate
#Override
public void setString(int parameterIndex, String x) throws SQLException {
delegate.setString(parameterIndex, x);
parameters.put(parameterIndex, x);
}
}
Solution 2: Dynamic proxy
This second solution is shorter, but seems more hacky.
You can create a dynamic proxy by calling a factory method on java.lang.reflect.Proxy and delegate all calls on the original instance. Example:
public PreparedStatement prepareStatement(String sql) {
final PreparedStatement ps = conn.prepareStatement(sql);
final PreparedStatement psProxy = (PreparedStatement) Proxy.newProxyInstance(ClassLoader.getSystemClassLoader(), new Class<?>[]{PreparedStatement.class}, new InvocationHandler() {
#Override
public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
if (method.getName().equals("setLong")) {
// ... your code here ...
}
// this invokes the default call
return method.invoke(ps, args);
}
});
return psProxy;
}
Then you intercept the setObject, etc. calls by looking at method names and looking to the second method arguments for your values.

This article, from Boulder, ahtoulgh DB 2 "specific", gives a complete example of ParameterMetadata usage.