Using prepared statements with JDBCTemplate - java

I'm using the JDBC template and want to read from a database using prepared statements. I iterate over many lines in a .csv file, and on every line I execute some SQL select queries with corresponding values.
I want to speed up my reading from the database but I don't know how to get the JDBC template to work with prepared statements.
There is the PreparedStatementCreator and the PreparedStatementSetter. As in this example both of them are created with anonymous inner classes.
But inside the PreparedStatementSetter class I don't have access to the values I want to set in the prepared statement.
Since I'm iterating through a .csv file, I can't hard code them as a String because I don't know them.
I also can't pass them to the PreparedStatementSetter because there are no arguments for the constructor. And setting my values to final would be dumb too.
I was used to the creation of prepared statements being fairly simple. Something like
PreparedStatement updateSales = con.prepareStatement(
"UPDATE COFFEES SET SALES = ? WHERE COF_NAME LIKE ? ");
updateSales.setInt(1, 75);
updateSales.setString(2, "Colombian");
updateSales.executeUpdate():
as in this Java tutorial.

By default, the JDBCTemplate does its own PreparedStatement internally, if you just use the .update(String sql, Object ... args) form. Spring, and your database, will manage the compiled query for you, so you don't have to worry about opening, closing, resource protection, etc. One of the saving graces of Spring. A link to Spring 2.5's documentation on this. Hope it makes things clearer. Also, statement caching can be done at the JDBC level, as in the case of at least some of Oracle's JDBC drivers.
That will go into a lot more detail than I can competently.

class Main {
public static void main(String args[]) throws Exception {
ApplicationContext ac = new
ClassPathXmlApplicationContext("context.xml", Main.class);
DataSource dataSource = (DataSource) ac.getBean("dataSource");
// DataSource mysqlDataSource = (DataSource) ac.getBean("mysqlDataSource");
JdbcTemplate jdbcTemplate = new JdbcTemplate(dataSource);
String prasobhName =
jdbcTemplate.query(
"select first_name from customer where last_name like ?",
new PreparedStatementSetter() {
public void setValues(PreparedStatement preparedStatement) throws
SQLException {
preparedStatement.setString(1, "nair%");
}
},
new ResultSetExtractor<Long>() {
public Long extractData(ResultSet resultSet) throws SQLException,
DataAccessException {
if (resultSet.next()) {
return resultSet.getLong(1);
}
return null;
}
}
);
System.out.println(machaceksName);
}
}

Try the following:
PreparedStatementCreator creator = new PreparedStatementCreator() {
#Override
public PreparedStatement createPreparedStatement(Connection con) throws SQLException {
PreparedStatement updateSales = con.prepareStatement(
"UPDATE COFFEES SET SALES = ? WHERE COF_NAME LIKE ? ");
updateSales.setInt(1, 75);
updateSales.setString(2, "Colombian");
return updateSales;
}
};

I'd factor out the prepared statement handling to at least a method. In this case, because there are no results it is fairly simple (and assuming that the connection is an instance variable that doesn't change):
private PreparedStatement updateSales;
public void updateSales(int sales, String cof_name) throws SQLException {
if (updateSales == null) {
updateSales = con.prepareStatement(
"UPDATE COFFEES SET SALES = ? WHERE COF_NAME LIKE ?");
}
updateSales.setInt(1, sales);
updateSales.setString(2, cof_name);
updateSales.executeUpdate();
}
At that point, it is then just a matter of calling:
updateSales(75, "Colombian");
Which is pretty simple to integrate with other things, yes? And if you call the method many times, the update will only be constructed once and that will make things much faster. Well, assuming you don't do crazy things like doing each update in its own transaction...
Note that the types are fixed. This is because for any particular query/update, they should be fixed so as to allow the database to do its job efficiently. If you're just pulling arbitrary strings from a CSV file, pass them in as strings. There's also no locking; far better to keep individual connections to being used from a single thread instead.

I've tried a select statement now with a PreparedStatement, but it turned out that it was not faster than the Jdbc template. Maybe, as mezmo suggested, it automatically creates prepared statements.
Anyway, the reason for my sql SELECTs being so slow was another one. In the WHERE clause I always used the operator LIKE, when all I wanted to do was finding an exact match. As I've found out LIKE searches for a pattern and therefore is pretty slow.
I'm using the operator = now and it's much faster.

Related

array not being filled

This should have at least 3 entries in the array when I view it at a later stage, but it only displays one. I believe this is thee problematic method, any advice?
String[] getKidsNamebyCid(int cid) {
String[] out = new String[20];
try {
String qry = "SELECT KIDSNAME FROM TBLKIDS WHERE CID = ?";//setting query command
ps = connect.prepareStatement(qry);//preparing statement
ps.setInt(1, cid);//setting CID
ps.executeQuery();//running command
int i = 0;
while (ps.getResultSet().next()) {
out[i] = ps.getResultSet().getString("KIDSNAME");
i++;
}
} catch (SQLException se) {
se.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
}
return out;
}
The getResultSet() call isn't a getter. That method does things to the DB, and you can't just repeatedly call it; the first time you get a ResultSet object (which you need to close), the second time everything is reset. So don't; you need to call getResultSet() once.
How do I know? By reading. Straight from getResultSet() documentation:
This method should be called only once per result.
Also this code is littered with severe code issues more generally focussed around resources. Resources are objects which aren't -just- a bunch of bits in memory, they represent (and hold open) a 'resource'. In the case of DBs, it's connections to the DB engine. You can't just make resources, you have to always put them in 'guarding' blocks that guarantee the resources are cleaned up. As a consequence, you never want them to be a field unless there's no other way (and then the thing they are a field inside of becomes a resource).
So, the fact that your PreparedStatement is a field? No good. The fact that you call .getResource just like that, unguarded? No good.
Finally, your exception handling is silly. The default act when facing checked exceptions is to just add them to your throws clause. If you can't do that, the right move is throw new RuntimeException("uncaught", e);, not what you did.
executeQuery already returns a resultset. Generally, never call getResultSet*.
Finally, arrays are more or less obsolete; you want collections.
Putting it all together:
// delete your 'ps' field!
List<String> getKidsNamebyCid(int cid) throws SQLException {
var out = new ArrayList<String>();
String qry = "SELECT KIDSNAME FROM TBLKIDS WHERE CID = ?";
try (PreparedStatement ps = connect.prepareStatement(qry)) {
ps.setInt(1, cid);
try (ResultSet rs = ps.executeQuery()) {
while (rs.next()) out.add(rs.getString("KIDSNAME"));
}
}
return out;
}
*) The PreparedStatement API is extremely unfortunate. The way you interact with a PS is wildly different vs. a Statement (which you should rarely use; you can't add user input to a plain jane Statement), and yet because reasons and history PreparedStatement extends Statement. That means a whole bevy of methods are in PreparedStatements that you should never call. That's unfortunate. There are 2 things to learn here: [1] Java does not break backwards compatibility, even if that means some of the APIs are horrible, and [2] JDBC is not really meant for 'human consumption'. We don't program our CPUs in machine code either, yet machine code exists and will continue to. We use 'machine code' as glue; something library and language developers use as common tongue. So it is with JDBC: It's not meant for you. Use a library with a nice API, like JDBI. This probably goes beyond what your high school curriculum wants, but hey. There's not much to say except: It's on your curriculum and teacher that they're making you work with outmoded tools 'real'** developers don't use.
**) 'real' in the sense of: Is writing code that powers apps that get a lot of dollars and/or eyeballs.
You need to learn how PreparedStatement actually works. I highly recommend you follow a tutorial to learn how to use it, then follow the pattern for your own code.
But it's also all in the documentation, so let be quote the various relevant pieces.
The javadoc of executeQuery() says:
Executes the SQL query in this PreparedStatement object and returns the ResultSet object generated by the query.
The code in the question is already wrong at this point, since it **ignores the return value of the executeQuery() call.
In addition, the javadoc of getResultSet() says:
Retrieves the current result as a ResultSet object. This method should be called only once per result.
The code in the question is even more wrong at this point, since it calls getResultSet() repeatedly in a loop.
If you had read the javadoc of the methods you're using, it would have been obvious that something is wrong. As already stated, going through a tutorial would have shown how to do this right. Actually, any web search for examples of executing a query using JDBC would show that.
For extra background information for how it works, the javadoc of execute() says:
Executes the SQL statement in this PreparedStatement object, which may be any kind of SQL statement. Some prepared statements return multiple results; the execute method handles these complex statements as well as the simpler form of statements handled by the methods executeQuery and executeUpdate.
The execute method returns a boolean to indicate the form of the first result. You must call either the method getResultSet or getUpdateCount to retrieve the result; you must call getMoreResults to move to any subsequent result(s).
The javadoc of getMoreResults() says:
Moves to this Statement object's next result, returns true if it is a ResultSet object, and implicitly closes any current ResultSet object(s) obtained with the method getResultSet.
The "return multiple results" is not talking about multiple rows from a single query, but about multiple results from multiple queries. It generally requires the execution of a stored procedure, or a block of SQL code, for this to happen.
This is how to correctly get the multiple rows from the execution of a single SELECT statement:
String qry = "SELECT KIDSNAME FROM TBLKIDS WHERE CID = ?";
try (PreparedStatement ps = connect.prepareStatement(qry)) {
ps.setInt(1, cid);//setting CID
try (ResultSet rs = ps.executeQuery()) {
int i = 0;
while (rs.next()) {
out[i] = rs.getString("KIDSNAME");
i++;
}
}
}
If the SQL code in question had returned multiple result sets, you would do it this way:
try (PreparedStatement ps = connect.prepareStatement(qry)) {
// call ps.setXxx() methods here
boolean isResultSet = ps.execute();
while (isResultSet) {
try (ResultSet rs = ps.getResultSet()) {
int i = 0;
while (rs.next()) {
// call rs.getXxx() methods here
i++;
}
}
isResultSet = ps.getMoreResults();
}
}
That is better written using for loops, to keep the loop logic together:
try (PreparedStatement ps = connect.prepareStatement(qry)) {
// call ps.setXxx() methods here
for (boolean isResultSet = ps.execute(); isResultSet; isResultSet = ps.getMoreResults()) {
try (ResultSet rs = ps.getResultSet()) {
for (int i = 0; rs.next(); i++) {
// call rs.getXxx() methods here
}
}
}
}

How to pass query param to REST API to do a IN query in DB [duplicate]

This question already has answers here:
PreparedStatement IN clause alternatives?
(33 answers)
Closed 5 years ago.
Say that I have a query of the form
SELECT * FROM MYTABLE WHERE MYCOL in (?)
And I want to parameterize the arguments to in.
Is there a straightforward way to do this in Java with JDBC, in a way that could work on multiple databases without modifying the SQL itself?
The closest question I've found had to do with C#, I'm wondering if there is something different for Java/JDBC.
There's indeed no straightforward way to do this in JDBC. Some JDBC drivers seem to support PreparedStatement#setArray() on the IN clause. I am only not sure which ones that are.
You could just use a helper method with String#join() and Collections#nCopies() to generate the placeholders for IN clause and another helper method to set all the values in a loop with PreparedStatement#setObject().
public static String preparePlaceHolders(int length) {
return String.join(",", Collections.nCopies(length, "?"));
}
public static void setValues(PreparedStatement preparedStatement, Object... values) throws SQLException {
for (int i = 0; i < values.length; i++) {
preparedStatement.setObject(i + 1, values[i]);
}
}
Here's how you could use it:
private static final String SQL_FIND = "SELECT id, name, value FROM entity WHERE id IN (%s)";
public List<Entity> find(Set<Long> ids) throws SQLException {
List<Entity> entities = new ArrayList<Entity>();
String sql = String.format(SQL_FIND, preparePlaceHolders(ids.size()));
try (
Connection connection = dataSource.getConnection();
PreparedStatement statement = connection.prepareStatement(sql);
) {
setValues(statement, ids.toArray());
try (ResultSet resultSet = statement.executeQuery()) {
while (resultSet.next()) {
entities.add(map(resultSet));
}
}
}
return entities;
}
private static Entity map(ResultSet resultSet) throws SQLException {
Enitity entity = new Entity();
entity.setId(resultSet.getLong("id"));
entity.setName(resultSet.getString("name"));
entity.setValue(resultSet.getInt("value"));
return entity;
}
Note that some databases have a limit of allowable amount of values in the IN clause. Oracle for example has this limit on 1000 items.
Since nobody answer the case for a large IN clause (more than 100) I'll throw my solution to this problem which works nicely for JDBC. In short I replace the IN with a INNER JOIN on a tmp table.
What I do is make what I call a batch ids table and depending on the RDBMS I may make that a tmp table or in memory table.
The table has two columns. One column with the id from the IN Clause and another column with a batch id that I generate on the fly.
SELECT * FROM MYTABLE M INNER JOIN IDTABLE T ON T.MYCOL = M.MYCOL WHERE T.BATCH = ?
Before you select you shove your ids into the table with a given batch id.
Then you just replace your original queries IN clause with a INNER JOIN matching on your ids table WHERE batch_id equals your current batch. After your done your delete the entries for you batch.
The standard way to do this is (if you are using Spring JDBC) is to use the org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate class.
Using this class, it is possible to define a List as your SQL parameter and use the NamedParameterJdbcTemplate to replace a named parameter. For example:
public List<MyObject> getDatabaseObjects(List<String> params) {
NamedParameterJdbcTemplate jdbcTemplate = new NamedParameterJdbcTemplate(dataSource);
String sql = "select * from my_table where my_col in (:params)";
List<MyObject> result = jdbcTemplate.query(sql, Collections.singletonMap("params", params), myRowMapper);
return result;
}
I solved this by constructing the SQL string with as many ? as I have values to look for.
SELECT * FROM MYTABLE WHERE MYCOL in (?,?,?,?)
First I searched for an array type I can pass into the statement, but all JDBC array types are vendor specific. So I stayed with the multiple ?.
I got the answer from docs.spring(19.7.3)
The SQL standard allows for selecting rows based on an expression that includes a variable list of values. A typical example would be select * from T_ACTOR where id in (1, 2, 3). This variable list is not directly supported for prepared statements by the JDBC standard; you cannot declare a variable number of placeholders. You need a number of variations with the desired number of placeholders prepared, or you need to generate the SQL string dynamically once you know how many placeholders are required. The named parameter support provided in the NamedParameterJdbcTemplate and JdbcTemplate takes the latter approach. Pass in the values as a java.util.List of primitive objects. This list will be used to insert the required placeholders and pass in the values during the statement execution.
Hope this can help you.
AFAIK, there is no standard support in JDBC for handling Collections as parameters. It would be great if you could just pass in a List and that would be expanded.
Spring's JDBC access supports passing collections as parameters. You could look at how this is done for inspiration on coding this securely.
See Auto-expanding collections as JDBC parameters
(The article first discusses Hibernate, then goes on to discuss JDBC.)
See my trial and It success,It is said that the list size has potential limitation.
List l = Arrays.asList(new Integer[]{12496,12497,12498,12499});
Map param = Collections.singletonMap("goodsid",l);
NamedParameterJdbcTemplate namedParameterJdbcTemplate = new NamedParameterJdbcTemplate(getJdbcTemplate().getDataSource());
String sql = "SELECT bg.goodsid FROM beiker_goods bg WHERE bg.goodsid in(:goodsid)";
List<Long> list = namedParameterJdbcTemplate.queryForList(sql, param2, Long.class);
There are different alternative approaches that we can use.
Execute Single Queries - slow and not recommended
Using Stored Procedure - database specific
Creating PreparedStatement Query dynamically - good performance but loose benefits of caching and needs recompilation
Using NULL in PreparedStatement Query - I think this is a good approach with optimal performance.
Check more details about these here.
sormula makes this simple (see Example 4):
ArrayList<Integer> partNumbers = new ArrayList<Integer>();
partNumbers.add(999);
partNumbers.add(777);
partNumbers.add(1234);
// set up
Database database = new Database(getConnection());
Table<Inventory> inventoryTable = database.getTable(Inventory.class);
// select operation for list "...WHERE PARTNUMBER IN (?, ?, ?)..."
for (Inventory inventory: inventoryTable.
selectAllWhere("partNumberIn", partNumbers))
{
System.out.println(inventory.getPartNumber());
}
One way i can think of is to use the java.sql.PreparedStatement and a bit of jury rigging
PreparedStatement preparedStmt = conn.prepareStatement("SELECT * FROM MYTABLE WHERE MYCOL in (?)");
... and then ...
preparedStmt.setString(1, [your stringged params]);
http://java.sun.com/docs/books/tutorial/jdbc/basics/prepared.html

Why Spring's jdbcTemplate.batchUpdate() so slow?

I'm trying to find the faster way to do batch insert.
I tried to insert several batches with jdbcTemplate.update(String sql), where
sql was builded by StringBuilder and looks like:
INSERT INTO TABLE(x, y, i) VALUES(1,2,3), (1,2,3), ... , (1,2,3)
Batch size was exactly 1000. I inserted nearly 100 batches.
I checked the time using StopWatch and found out insert time:
min[38ms], avg[50ms], max[190ms] per batch
I was glad but I wanted to make my code better.
After that, I tried to use jdbcTemplate.batchUpdate in way like:
jdbcTemplate.batchUpdate(sql, new BatchPreparedStatementSetter() {
#Override
public void setValues(PreparedStatement ps, int i) throws SQLException {
// ...
}
#Override
public int getBatchSize() {
return 1000;
}
});
where sql was look like
INSERT INTO TABLE(x, y, i) VALUES(1,2,3);
and I was disappointed! jdbcTemplate executed every single insert of 1000 lines batch in separated way. I loked at mysql_log and found there a thousand inserts.
I checked the time using StopWatch and found out insert time:
min[900ms], avg[1100ms], max[2000ms] per Batch
So, can anybody explain to me, why jdbcTemplate doing separated inserts in this method? Why method's name is batchUpdate?
Or may be I am using this method in wrong way?
These parameters in the JDBC connection URL can make a big difference in the speed of batched statements --- in my experience, they speed things up:
?useServerPrepStmts=false&rewriteBatchedStatements=true
See: JDBC batch insert performance
I found a major improvement setting the argTypes array in the call.
In my case, with Spring 4.1.4 and Oracle 12c, for insertion of 5000 rows with 35 fields:
jdbcTemplate.batchUpdate(insert, parameters); // Take 7 seconds
jdbcTemplate.batchUpdate(insert, parameters, argTypes); // Take 0.08 seconds!!!
The argTypes param is an int array where you set each field in this way:
int[] argTypes = new int[35];
argTypes[0] = Types.VARCHAR;
argTypes[1] = Types.VARCHAR;
argTypes[2] = Types.VARCHAR;
argTypes[3] = Types.DECIMAL;
argTypes[4] = Types.TIMESTAMP;
.....
I debugged org\springframework\jdbc\core\JdbcTemplate.java and found that most of the time was consumed trying to know the nature of each field, and this was made for each record.
Hope this helps !
I have also faced the same issue with Spring JDBC template. Probably with Spring Batch the statement was executed and committed on every insert or on chunks, that slowed things down.
I have replaced the jdbcTemplate.batchUpdate() code with original JDBC batch insertion code and found the Major performance improvement.
DataSource ds = jdbcTemplate.getDataSource();
Connection connection = ds.getConnection();
connection.setAutoCommit(false);
String sql = "insert into employee (name, city, phone) values (?, ?, ?)";
PreparedStatement ps = connection.prepareStatement(sql);
final int batchSize = 1000;
int count = 0;
for (Employee employee: employees) {
ps.setString(1, employee.getName());
ps.setString(2, employee.getCity());
ps.setString(3, employee.getPhone());
ps.addBatch();
++count;
if(count % batchSize == 0 || count == employees.size()) {
ps.executeBatch();
ps.clearBatch();
}
}
connection.commit();
ps.close();
Check this link as well
JDBC batch insert performance
Simply use transaction. Add #Transactional on method.
Be sure to declare the correct TX manager if using several datasources #Transactional("dsTxManager"). I have a case where inserting 60000 records. It takes about 15s. No other tweak:
#Transactional("myDataSourceTxManager")
public void save(...) {
...
jdbcTemplate.batchUpdate(query, new BatchPreparedStatementSetter() {
#Override
public void setValues(PreparedStatement ps, int i) throws SQLException {
...
}
#Override
public int getBatchSize() {
if(data == null){
return 0;
}
return data.size();
}
});
}
Change your sql insert to INSERT INTO TABLE(x, y, i) VALUES(1,2,3). The framework creates a loop for you.
For example:
public void insertBatch(final List<Customer> customers){
String sql = "INSERT INTO CUSTOMER " +
"(CUST_ID, NAME, AGE) VALUES (?, ?, ?)";
getJdbcTemplate().batchUpdate(sql, new BatchPreparedStatementSetter() {
#Override
public void setValues(PreparedStatement ps, int i) throws SQLException {
Customer customer = customers.get(i);
ps.setLong(1, customer.getCustId());
ps.setString(2, customer.getName());
ps.setInt(3, customer.getAge() );
}
#Override
public int getBatchSize() {
return customers.size();
}
});
}
IF you have something like this. Spring will do something like:
for(int i = 0; i < getBatchSize(); i++){
execute the prepared statement with the parameters for the current iteration
}
The framework first creates PreparedStatement from the query (the sql variable) then the setValues method is called and the statement is executed. that is repeated as much times as you specify in the getBatchSize() method. So the right way to write the insert statement is with only one values clause.
You can take a look at http://docs.spring.io/spring/docs/3.0.x/reference/jdbc.html
I had also some bad time with Spring JDBC batch template. In my case, it would be, like, insane to use pure JDBC, so instead I used NamedParameterJdbcTemplate. This was a must have in my project. But it was way slow to insert hundreds os thousands of lines in the database.
To see what was going on, I've sampled it with VisualVM during the batch update and, voilà:
What was slowing the process was that, while setting the parameters, Spring JDBC was querying the database to know the metadata each parameter. And seemed to me that it was querying the database for each parameter for each line every time. So I just taught Spring to ignore the parameter types (as it is warned in the Spring documentation about batch operating a list of objects):
#Bean(name = "named-jdbc-tenant")
public synchronized NamedParameterJdbcTemplate getNamedJdbcTemplate(#Autowired TenantRoutingDataSource tenantDataSource) {
System.setProperty("spring.jdbc.getParameterType.ignore", "true");
return new NamedParameterJdbcTemplate(tenantDataSource);
}
Note: the system property must be set before creating the JDBC Template object. It would be possible to just set in the application.properties, but this solved and I've never after touched this again
I don't know if this will work for you, but here's a Spring-free way that I ended up using. It was significantly faster than the various Spring methods I tried. I even tried using the JDBC template batch update method the other answer describes, but even that was slower than I wanted. I'm not sure what the deal was and the Internets didn't have many answers either. I suspected it had to do with how commits were being handled.
This approach is just straight JDBC using the java.sql packages and PreparedStatement's batch interface. This was the fastest way that I could get 24M records into a MySQL DB.
I more or less just built up collections of "record" objects and then called the below code in a method that batch inserted all the records. The loop that built the collections was responsible for managing the batch size.
I was trying to insert 24M records into a MySQL DB and it was going ~200 records per second using Spring batch. When I switched to this method, it went up to ~2500 records per second. so my 24M record load went from a theoretical 1.5 days to about 2.5 hours.
First create a connection...
Connection conn = null;
try{
Class.forName("com.mysql.jdbc.Driver");
conn = DriverManager.getConnection(connectionUrl, username, password);
}catch(SQLException e){}catch(ClassNotFoundException e){}
Then create a prepared statement and load it with batches of values for insert, and then execute as a single batch insert...
PreparedStatement ps = null;
try{
conn.setAutoCommit(false);
ps = conn.prepareStatement(sql); // INSERT INTO TABLE(x, y, i) VALUES(1,2,3)
for(MyRecord record : records){
try{
ps.setString(1, record.getX());
ps.setString(2, record.getY());
ps.setString(3, record.getI());
ps.addBatch();
} catch (Exception e){
ps.clearParameters();
logger.warn("Skipping record...", e);
}
}
ps.executeBatch();
conn.commit();
} catch (SQLException e){
} finally {
if(null != ps){
try {ps.close();} catch (SQLException e){}
}
}
Obviously I've removed error handling and the query and Record object is notional and whatnot.
Edit:
Since your original question was comparing the insert into foobar values (?,?,?), (?,?,?)...(?,?,?) method to Spring batch, here's a more direct response to that:
It looks like your original method is likely the fastest way to do bulk data loads into MySQL without using something like the "LOAD DATA INFILE" approach. A quote from the MysQL docs (http://dev.mysql.com/doc/refman/5.0/en/insert-speed.html):
If you are inserting many rows from the same client at the same time,
use INSERT statements with multiple VALUES lists to insert several
rows at a time. This is considerably faster (many times faster in some
cases) than using separate single-row INSERT statements.
You could modify the Spring JDBC Template batchUpdate method to do an insert with multiple VALUES specified per 'setValues' call, but you'd have to manually keep track of the index values as you iterate over the set of things being inserted. And you'd run into a nasty edge case at the end when the total number of things being inserted isn't a multiple of the number of VALUES lists you have in your prepared statement.
If you use the approach I outline, you could do the same thing (use a prepared statement with multiple VALUES lists) and then when you get to that edge case at the end, it's a little easier to deal with because you can build and execute one last statement with exactly the right number of VALUES lists. It's a bit hacky, but most optimized things are.
Solution given by #Rakesh worked for me.
Significant improvement in performance. Earlier time was 8 min, with this solution taking less than 2 min.
DataSource ds = jdbcTemplate.getDataSource();
Connection connection = ds.getConnection();
connection.setAutoCommit(false);
String sql = "insert into employee (name, city, phone) values (?, ?, ?)";
PreparedStatement ps = connection.prepareStatement(sql);
final int batchSize = 1000;
int count = 0;
for (Employee employee: employees) {
ps.setString(1, employee.getName());
ps.setString(2, employee.getCity());
ps.setString(3, employee.getPhone());
ps.addBatch();
++count;
if(count % batchSize == 0 || count == employees.size()) {
ps.executeBatch();
ps.clearBatch();
}
}
connection.commit();
ps.close();
Encountered some serious performance issue with JdbcBatchItemWriter.write() (link) from Spring Batch and find out the write logic delegates to JdbcTemplate.batchUpdate() eventually.
Adding a Java system properties of spring.jdbc.getParameterType.ignore=true fixed the performance issue entirely ( from 200 records per second to ~ 5000 ).
The patch was tested working on both Postgresql and MsSql (might not be dialect specific)
... and ironically, Spring documented this behaviour under a "note" section link
In such a scenario, with automatic setting of values on an underlying PreparedStatement, the corresponding JDBC type for each value needs to be derived from the given Java type. While this usually works well, there is a potential for issues (for example, with Map-contained null values). Spring, by default, calls ParameterMetaData.getParameterType in such a case, which can be expensive with your JDBC driver. You should use a recent driver version and consider setting the spring.jdbc.getParameterType.ignore property to true (as a JVM system property or in a spring.properties file in the root of your classpath) if you encounter a performance issue — for example, as reported on Oracle 12c (SPR-16139).
Alternatively, you might consider specifying the corresponding JDBC
types explicitly, either through a 'BatchPreparedStatementSetter' (as
shown earlier), through an explicit type array given to a
'List<Object[]>' based call, through 'registerSqlType' calls on a
custom 'MapSqlParameterSource' instance, or through a
'BeanPropertySqlParameterSource' that derives the SQL type from the
Java-declared property type even for a null value.

Java create PreparedStatement with no SQL content

Is it possible to create a PreparedStatement in java without setting the initial SQL query?
Example code:
#Override
public List<AccountBean> search(AccountConstraint... c) {
if (c.length == 0) {
throw new IllegalArgumentException("dao.AccountDAO.search: c.length == 0");
}
try {
List<AccountBean> beans = new ArrayList<>();
for (AccountConstraint ac : c) {
PreparedStatement ps = connection.prepareStatement(null);
QueryBuilder queryBuilder = new QueryBuilder(ps, "SELECT * FROM accounts");
queryBuilder.add(ac.getAccountIdConstraint());
queryBuilder.add(ac.getUsernameConstraint());
queryBuilder.add(ac.getPasswordConstraint());
queryBuilder.add(ac.getEmailConstraint());
//INSERT QUERY INTO PS
ResultSet rs = ps.executeQuery();
while (rs.next()) {
beans.add(new AccountBean(rs));
}
}
return beans;
} catch (SQLException ex) {
throw new RuntimeException(ex);
}
}
The trick is in QueryBuilder, this class is responsible for building parts of a query based on the initial SELECT part and then adds respective WHERE and AND clauses.
However to ensure that all data is safe, the actual arguments must also be put in the PreparedStatement, hence why it is being passed to the QueryBuilder.
Every QueryBuilder.add() adds some arguments into the PreparedStatement and appends a specific string to the end of the query.
I think some workarounds are possible, such as instead of giving a PreparedStatement to the QueryBuilder you would give a List<Object> and then you would write a custom function that puts them in the PreparedStatement later on.
But what are your thoughts, suggestions on this?
Regards.
Solution added
Few key changes first:
QueryBuilder now implements the Builder pattern properly.
QueryBuilder.add() accepts multiple Constraints at once.
AccountConstraint can give an array that gives all Constraints now.
#Override
public List<AccountBean> search(AccountConstraint... c) {
if (c.length == 0) {
throw new IllegalArgumentException("dao.AccountDAO.search: c.length == 0");
}
try {
List<AccountBean> beans = new ArrayList<>();
for (AccountConstraint ac : c) {
try (PreparedStatement ps = new QueryBuilder("SELECT * FROM accounts").add(ac.getConstraints()).build();ResultSet rs = ps.executeQuery()) {
while (rs.next()) {
beans.add(new AccountBean(rs));
}
}
}
return beans;
} catch (SQLException ex) {
throw new RuntimeException(ex);
}
}
ps. I get two statements in one try{ } because of the try-with-resources.
Preparing a statement means compiling it so you can efficiently execute it many times with different arguments. So, no it does not make sense to compile a query before it is defined.
As I understand, you want to use the Java compiler to assist you in dynamically defining the query. Why don't you just create the prepared statement in a compile() method, thus, as the result of your builder. Also, your code becomes more readable and more resembles a declarative query if you use the builder pattern such that each call to add() returns this. Then you can write your query like this:
PreparedStatement ps = new QueryBuilder()
.select("*")
.from("accounts")
.where()
.add(yourConstraint())
...
.compile();
However, you must create the prepared statement before the loop. Otherwise, if you keep a reference to the builder and call compile() in your loop you will get a new prepared statement on every call. So you won't get the benefit of reusing a precompiled query. In the loop you only assign values to the variables in your prepared statement.
You can't modify the prepared statement via the API after you crate it. You can't create it without an SQL statement either.
Why not create the query separately and then bind the parameters? You can use a Map to hold the parameter placeholders and their values so they can be set to the prepared statement.
Although I'd just use the Spring's JDBC templates to get the same thing done more quickly.
How to improve your SQL query builder
If you look at how popular query builders like jOOQ and others do it, the idea is that you separate your concerns more thoroughly. You should have:
An expression tree representation of your SQL statement (and ideally that doesn't directly operate on strings)
A way to construct that expression tree conveniently, e.g. by using a DSL
Some sort of execution lifecycle management that generates the SQL string, prepares the statement, binds the variables, etc.
Or in code (jOOQ example, but this could also apply to your own query builder):
Result<?> result =
// This constructs the expression tree through the jOOQ DSL
ctx.selectFrom(ACCOUNTS)
.where(ac.getAccountIdConstraint())
.and(ac.getUsernameConstraint())
.and(ac.getPasswordConstraint())
.and(ac.getEmailConstraint())
// This internally creates a PreparedStatement, binds variables, executes it, and maps results
.fetch();
Of course, your AccountConstraint.getXYZConstraint() methods would not return SQL string snippets, but again expression tree elements. In the case of jOOQ, this would be a Condition
(Disclaimer: I work for the vendor of jOOQ)
How to improve your SQL performance
I've noticed that you run N queries for N AccountConstraint values, and you mix the results in a way that it doesn't matter which AccountConstraint value produced which AccountBean. I strongly suggest you move that loop into the generated SQL query, as you're going to get much faster results on pretty much every database. I've blogged about this here.

Create PreparedStatement from statement?

This is going to sound a little bit weird but I want to create a PreparedStatement
after I have a Statement . This is because my project is almost 50% , and until now
I've used a Statement .
In my Database class I have a constructor that connects to mysql every time that I need to use the Mysql server :
public class Database
{
private Statement m_statement = null;
private Connection m_connection = null;
private PreparedStatement m_prepared = null; // that one was added just now
private static final String ACCOUNTS_TABLE = "CheckingAccountsTable";
private static final String PERSONNEL_TABLE = "PersonnelTable";
// private static final String
public Database() throws ClassNotFoundException
{
try
{
Class.forName("com.mysql.jdbc.Driver");
this.m_connection = DriverManager.getConnection("jdbc:mysql://localhost","root","root");
this.m_statement = this.m_connection.createStatement();
}
catch (SQLException s)
{
System.out.println("SQL statement is not executed! 2!");
}
}
}
// from here I have something like 20 methods that are based on the Statement
Changing everything would require a lot time that I don't have . So ,my question is :
Can I use the Statement that I created with the PreparedStatement ? or create PreparedStatement based on Statement ?
Thanks
This is not possible. Statement and PreparedStatement represent different things.
Statement is used to call different SQL queries, you create a Statement instance without knowing what SQL query you are going to use.
PreparedStatement is given a SQL query at the time it is created, and the driver tries to precompile the query, if possible. You can fill in different parameters and use it repeatedly, but you cannot change the query, it always uses the one given at its creation time.
So I'm afraid you'll have to refactor your code, there is no simple way how to switch between these two.
In most cases, using PreparedStatement is much preferred, as it allows you pass parameters to a SQL query in a safe way, without risking SQL injection.

Categories