I have 200K rows to be inserted in one single database table. I tried to use jdbcTemplate.batchUpdate in spring in order to do insertion 10,000 per batch. However, this process consumes too much time (7 mins for 200K rows). So on database side, I check the number of rows inserted by select count(*) from table_X. I found the number of rows increased slightly instead of 10K expected. Can anyone explain what's reason or is it something which should be configurated on Database side ?
PS: I am using sybase ....
There are lot of approaches available on the web.
Performance directly depends on the
Code you have written
JDBC driver you are using
database server and number of connection you are using
table indexes leads to slowness for insertion
Without looking at your code anyone can guess, but no one can find the exact solution.
Approach 1
//insert batch example
public void insertBatch(final List<Customer> customers){
String sql = "INSERT INTO CUSTOMER " +
"(CUST_ID, NAME, AGE) VALUES (?, ?, ?)";
getJdbcTemplate().batchUpdate(sql, new BatchPreparedStatementSetter() {
#Override
public void setValues(PreparedStatement ps, int i) throws SQLException {
Customer customer = customers.get(i);
ps.setLong(1, customer.getCustId());
ps.setString(2, customer.getName());
ps.setInt(3, customer.getAge() );
}
#Override
public int getBatchSize() {
return customers.size();
}
});
}
reference
https://www.mkyong.com/spring/spring-jdbctemplate-batchupdate-example/
http://docs.spring.io/spring-framework/docs/3.0.0.M4/reference/html/ch12s04.html
Approach 2.1
//insert batch example
public void insertBatch(final List<Customer> customers){
String sql = "INSERT INTO CUSTOMER " +
"(CUST_ID, NAME, AGE) VALUES (?, ?, ?)";
List<Object[]> parameters = new ArrayList<Object[]>();
for (Customer cust : customers) {
parameters.add(new Object[] {cust.getCustId(),
cust.getName(), cust.getAge()}
);
}
getSimpleJdbcTemplate().batchUpdate(sql, parameters);
}
Alternatively, you can execute the SQL directly.
//insert batch example with SQL
public void insertBatchSQL(final String sql){
getJdbcTemplate().batchUpdate(new String[]{sql});
}
reference
https://www.mkyong.com/spring/spring-simplejdbctemplate-batchupdate-example/
Approach 2.2
public class JdbcActorDao implements ActorDao {
private SimpleJdbcTemplate simpleJdbcTemplate;
public void setDataSource(DataSource dataSource) {
this.simpleJdbcTemplate = new SimpleJdbcTemplate(dataSource);
}
public int[] batchUpdate(final List<Actor> actors) {
SqlParameterSource[] batch = SqlParameterSourceUtils.createBatch(actors.toArray());
int[] updateCounts = simpleJdbcTemplate.batchUpdate(
"update t_actor set first_name = :firstName, last_name = :lastName where id = :id",
batch);
return updateCounts;
}
// ... additional methods
}
Approach 2.3
public class JdbcActorDao implements ActorDao {
private SimpleJdbcTemplate simpleJdbcTemplate;
public void setDataSource(DataSource dataSource) {
this.simpleJdbcTemplate = new SimpleJdbcTemplate(dataSource);
}
public int[] batchUpdate(final List<Actor> actors) {
List<Object[]> batch = new ArrayList<Object[]>();
for (Actor actor : actors) {
Object[] values = new Object[] {
actor.getFirstName(),
actor.getLastName(),
actor.getId()};
batch.add(values);
}
int[] updateCounts = simpleJdbcTemplate.batchUpdate(
"update t_actor set first_name = ?, last_name = ? where id = ?",
batch);
return updateCounts;
}
// ... additional methods
}
Approach 3 :JDBC
dbConnection.setAutoCommit(false);//commit trasaction manually
String insertTableSQL = "INSERT INTO DBUSER"
+ "(USER_ID, USERNAME, CREATED_BY, CREATED_DATE) VALUES"
+ "(?,?,?,?)";
PreparedStatement = dbConnection.prepareStatement(insertTableSQL);
preparedStatement.setInt(1, 101);
preparedStatement.setString(2, "mkyong101");
preparedStatement.setString(3, "system");
preparedStatement.setTimestamp(4, getCurrentTimeStamp());
preparedStatement.addBatch();
preparedStatement.setInt(1, 102);
preparedStatement.setString(2, "mkyong102");
preparedStatement.setString(3, "system");
preparedStatement.setTimestamp(4, getCurrentTimeStamp());
preparedStatement.addBatch();
preparedStatement.executeBatch();
dbConnection.commit();
reference
https://www.mkyong.com/jdbc/jdbc-preparedstatement-example-batch-update/
/*Happy Coding*/
Try setting below for connection string - useServerPrepStmts=false&rewriteBatchedStatements=true. Have not tried but its from my bookmarks. You can search on these lines..
Connection c = DriverManager.getConnection("jdbc:<db>://host:<port>/db?useServerPrepStmts=false&rewriteBatchedStatements=true", "username", "password");
For us moving the code to a wrapper class and annotating the batch insert method with #Transactional did solve the problem.
Related
I want to test how JDBC transactions work. Particularly, I want to see a read of uncommitted data. I've written one integration test in spring boot environment using a locally installed PostgreSQL database.
I'm trying to insert a row into a table, read it from one transaction, then update from another transaction without committing it, and read it again hoping it would change.
Table for the test (DDL):
create table users
(
id integer default nextval('user_id_sequence'::regclass) not null
constraint users_pkey
primary key,
first_name varchar(255) not null,
second_name varchar(255) not null,
email varchar(255)
);
alter table users
owner to postgres;
The test:
public void testHealthCheck() throws SQLException {
Connection zeroConnection = dataSource.getConnection();
Integer insertedUserId = insertUserSilently(zeroConnection, new User()
.setFirstName("John")
.setSecondName("Doe")
.setEmail("johndoe#gmail.com"));
zeroConnection.close();
Connection firstConnection = dataSource.getConnection();
firstConnection.setTransactionIsolation(Connection.TRANSACTION_READ_UNCOMMITTED);
firstConnection.setAutoCommit(false);
Connection secondConnection = dataSource.getConnection();
secondConnection.setTransactionIsolation(Connection.TRANSACTION_READ_UNCOMMITTED);
secondConnection.setAutoCommit(false);
List<User> users = getAllUsersSilently(firstConnection);
log.info("Got users: {}", silentToJsonString(users));
PersistenceUtils.updateUserEmailSilently(secondConnection, insertedUserId, "johndoe#yahoo.com");
users = getAllUsersSilently(firstConnection);
log.info("Got users: {}", silentToJsonString(users));
secondConnection.rollback();
secondConnection.close();
users = getAllUsersSilently(firstConnection);
log.info("Got users: {}", silentToJsonString(users));
firstConnection.close();
}
Utility class:
private static final String INSERT_USER_SQL = "insert into users(first_name, second_name, email) values (?, ?, ?)";
private static final String UPDATE_USER_SQL = "update users set email = ? where id = ?;";
private static final String SELECT_ALL_USERS_SQL = "select * from users";
public static List<User> extractUsersSilently(ResultSet resultSet) {
List<User> resultList = newArrayList();
try {
while (resultSet.next()) {
Integer id = resultSet.getInt(1);
String firstName = resultSet.getString(2);
String secondName = resultSet.getString(3);
String email = resultSet.getString(4);
resultList.add(new User(id, firstName, secondName, email));
}
} catch (SQLException e) {
log.error("Error while extracting result set", e);
return emptyList();
}
return resultList;
}
public static Integer insertUserSilently(Connection connection, User user) {
try {
PreparedStatement insertStatement = connection.prepareStatement(INSERT_USER_SQL, Statement.RETURN_GENERATED_KEYS);
insertStatement.setString(1, user.getFirstName());
insertStatement.setString(2, user.getSecondName());
insertStatement.setString(3, user.getEmail());
insertStatement.execute();
ResultSet resultSet = insertStatement.getGeneratedKeys();
resultSet.next();
return resultSet.getInt(1);
} catch (Exception exception) {
log.error(format("Exception while inserting user %s", user), exception);
return -1;
}
}
public static List<User> getAllUsersSilently(Connection connection) {
try {
PreparedStatement selectStatement = connection.prepareStatement(SELECT_ALL_USERS_SQL);
selectStatement.execute();
return extractUsersSilently(selectStatement.getResultSet());
} catch (Exception exception) {
log.error("Exception while getting all users", exception);
return Collections.emptyList();
}
}
public static void updateUserEmailSilently(Connection connection, Integer userId, String userEmail) {
try {
PreparedStatement updateStatement = connection.prepareStatement(UPDATE_USER_SQL);
updateStatement.setString(1, userEmail);
updateStatement.setInt(2, userId);
updateStatement.execute();
} catch (Exception exception) {
log.error(format("Exception while updating user %d", userId), exception);
}
}
}
Actual results are (you have to clear table manually before the test):
Got users:
[{"id":55,"firstName":"John","secondName":"Doe","email":"johndoe#gmail.com"}]
Got users:
[{"id":55,"firstName":"John","secondName":"Doe","email":"johndoe#gmail.com"}]
Got users:
[{"id":55,"firstName":"John","secondName":"Doe","email":"johndoe#gmail.com"}]
Although second read should've seen uncommitted change to email.
Cannot read uncommitted data in Postgres
See section 13.2. Transaction Isolation of the PostgreSQL documentation:
In PostgreSQL, you can request any of the four standard transaction isolation levels, but internally only three distinct isolation levels are implemented, i.e. PostgreSQL's Read Uncommitted mode behaves like Read Committed. This is because it is the only sensible way to map the standard isolation levels to PostgreSQL's multiversion concurrency control architecture.
This means that if you want to test TRANSACTION_READ_UNCOMMITTED, you need a DBMS other than PostgreSQL.
I work with hibernate3 and didn't use JPA
I have a procedure in oracle which return 2 out parameter
For test I execute this procedure in oracle with this query.
declare
req_type number;
req_seq number;
begin
insert_req(1111,req_type,req_seq);
dbms_output.put_line('req_type='||req_type);
dbms_output.put_line('req_seq='||req_seq);
end;
Now I want to call this procedure using hibernate
I try with native query without success using this code :
public void insertReq(String numEmp) {
int req_type ;
int req_seq;
String sql = " insert_req(1111,:in1,:in2) ";
SQLQuery query = session.createSQLQuery(sql);
query.setParameter("in1", req_type);
query.setParameter("in2", req_seq);
List results = query.list();
System.out.println(req_type);
System.out.println(req_seq);
}
when I have a function I can run it using hibernate using this code as an example :
public void insertOrder(String numEmp) {
String query = "call insert_order(" + numEmp + ",50)";
SQLQuery sqlQuery = this.getSession().createSQLQuery(query);
sqlQuery.executeUpdate();
}
but the problem is how to call procedure with 2 out parameter using hibernate.
you have to use CallableStatement and registerOutParameter.
you can get a connection from your hibernate session and create the callablestatement.
hibernate does not provide a mecanism to deal with this (at least as i know).
i hope that helps.
Try this and let me know.
EntityManager em = emf.createEntityManager();
em.getTransaction().begin();
Session session = em.unwrap(Session.class);
session.doWork(new Work() {
#Override
public void execute(Connection con) throws SQLException {
// do something useful
try (CallableStatement stmt = con.prepareCall("{call my_sp()}")) {
stmt.execute();
}
}
});
em.close();
Best regards.
I am using Spring 3.1.2,am using PreparedStatementSetter for Select, update and insert, but how can I use for Delete query?
jdbcTemplate.update() can be used for delete's
e.g.
JdbcTemplate jdbcTemplate = new JdbcTemplate(dataSource);
jdbcTemplate.update("delete from my_table where value=? ", new PreparedStatementSetter() {
public void setValues(PreparedStatement ps) throws SQLException {
ps.setLong(1, 2L);//or setString or whatever
}
});
you can also execute a delete without the PreparedStatementSetter as shown here: http://www.java2s.com/Code/Java/Spring/UseJdbcTemplateToExecuteDeleteStatementWithParameter.htm
You can use PreparedStatement for deletion like this
String sql = "DELETE FROM MY_TABLE WHERE ID = ?";
PreparedStatement ps = yourDBConnection.prepareStatement(sql);
ps.setInt(1, 1001);
// execute delete SQL stetement
ps.executeUpdate();
Newbie programmer here. Upon doing mvn tomcat:run I get the following error:
SEVERE: Servlet.service() for servlet appServlet threw exception
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'values (?, ?)' at line 1
The code in question is as follows:
public void create(User user) {
this.jdbcTemplate.update("INSERT INTO xyz.user(user_name, user_password values (?, ?)");
user.getUserName(); user.getId();
}
public void delete(User user) {
this.jdbcTemplate.update("DELETE FROM xyz.user WHERE id = ?");
}
public void update(User user) {
this.jdbcTemplate.update(
"UPDATE xyz.user SET UserName = ? password = ? WHERE id = ?");
Googled - couldn't find a solution for (?, ?) scenarios. Pls. help - Thx in advance :)
Here's the complete code (almost) - I am doing something wrong but can't figure out what.
public User find(String login) {
System.out.println("Trying to find the user...." + login);
User user = this.jdbcTemplate.queryForObject(
"select * from xyz where user_name = ?",
new Object[]{login},
new RowMapper<User>() {
public User mapRow(ResultSet rs, int rowNum) throws SQLException {
User user = new User();
user.setId(Long.valueOf(rs.getInt(1)));
user.setUserName(rs.getString(2));
user.setPassword(rs.getString(3));
return user;
}
});
System.out.println("Found user..." + user);
return user;
}
public void create(User user) {
this.jdbcTemplate.update("INSERT INTO ibstechc_dev.user(user_name, user_password) VALUES (?,?)");
user.getUserName(); user.getId() ;
}
public void delete(User user) {
this.jdbcTemplate.update("DELETE FROM xyz WHERE id = ?");
// TODO Auto-generated method stub
}
public void update(User user) {
this.jdbcTemplate.update(
"UPDATE xyz SET user_name = ?, user_password = ? WHERE id = ?");
// TODO Auto-generated method stub
}
}
I am stuck with the same error - tomcat:run throws the following -
SEVERE: Servlet.service() for servlet appServlet threw exception
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '?,?)' at line 1
Use this code:
this.jdbcTemplate.update("INSERT INTO xyz.user(user_name, user_password) values (?, ?)");
There was an issue with your SQL statement. To give you perspective you were trying to do:
INSERT INTO xyz.user(user_name, user_password values ('testuser','testpass'))
instead of
INSERT INTO xyz.user(user_name, user_password) values ('testuser','testpass'))
Hope this makes sense?
I think here is the sql syntax problem:
INSERT INTO xyz.user(user_name, user_password values (?, ?)
replace this by
INSERT INTO xyz.user(user_name, user_password) values (?, ?);
There is a sql syntax error in update
public void update(User user) {
this.jdbcTemplate.update(
"UPDATE xyz.user SET UserName = ? password = ? WHERE id = ?");
this is not the way to update,give a comma(,) like this
public void update(User user) {
this.jdbcTemplate.update(
"UPDATE xyz.user SET UserName = ? ,password = ? WHERE id = ?");
your error is this statement
this.jdbcTemplate.update("INSERT INTO xyz.user(user_name, user_password values (?, ?)");
It should be
this.jdbcTemplate.update("INSERT INTO xyz.user(user_name, user_password) values (?, ?)");
Hope it helps
Cheers
Try this:-
this.jdbcTemplate.update("INSERT INTO xyz.user(user_name, user_password) values (?, ?)");
instead of
this.jdbcTemplate.update("INSERT INTO xyz.user(user_name, user_password values (?, ?)");
Is it possible to execute stored procedure using Spring JDBC API in batch mode?
I tried to use Spring JdbcTemplate's method batchUpdate and it doesn't work.
I use Spring 3.0 and Oracle JDBC driver version 11.2.0.3
JdbcTemplate template = new JdbcTemplate(dataSource);
template.batchUpdate("BEGIN SCHEMA.PERSON_UPDATE(I_N_PERSON_ID=> ? ,I_S_NAME=> ? ,J_N_TID=> ?);END;",
new BatchPreparedStatementSetter() {
public void setValues(PreparedStatement ps, int i) throws SQLException {
OracleCallableStatement cs = (OracleCallableStatement) ps;
cs.setString(1, persons.get(i).getName());
cs.setLong(2, persons.get(i).getPersonId());
cs.registerOutParameter(3, OracleTypes.NUMBER);
}
public int getBatchSize() {
return persons.size();
}}
);
Thanks
There is a mistake in implementation of the method setValues. You create instance of OracleCallableStatement but you have to register parameters in PreparedStatement which given in method argement like ps
JdbcTemplate template = new JdbcTemplate(dataSource);
template.batchUpdate("{call SCHEMA.PERSON_UPDATE(?, ?)}",
new BatchPreparedStatementSetter() {
public void setValues(PreparedStatement ps, int i) throws SQLException {
ps.setString(1, persons.get(i).getName());
ps.setLong(2, persons.get(i).getPersonId());
}
public int getBatchSize() {
return persons.size();
}
}
);
But there is a problem. PreparedStatement doesn't work with out parameters.