I am using MySQL Database. The following piece creates a record and gets the id from the created record:
insertStmt = connection
.prepareStatement("INSERT INTO bugs (summary, status, report_date) VALUES (?, ?, ? )");
//...
insertStmt.executeUpdate();
idQuery = connection.prepareStatement("SELECT LAST_INSERT_ID()");
rs = idQuery.executeQuery();
if (rs != null) {
rs.next();
return new Long(rs.getLong(1)).toString();
}
Now, if two threads execute this and their execution is interleaved, say, the first thread inserts the record followed by the insertion by the second thread, after which the first thread calls last_insert_id() which will be incorrect for this thread as the second thread has already inserted a record.
This might be overcome using synchronization, however. Is there a way we can execute the two statements in a single database call?
LAST_INSERT_ID works per-connection, and as your question states you can have a race condition if two statements in two threads use the same connection.
You have two ways around this:
1: Use a separate connection per thread (not easy, but this is really the best option for scaling and sense; use connection pooling)
2: Use the form of executeUpdate that records the auto-generated key in the same API call, allowing you to read it back later using getGeneratedKeys so that you don't have to use LAST_INSERT_ID in a second query, so avoiding the race condition. There's a similar form of prepareStatement that you can use in prepared statements.
Option 2 is probably what you want in the short term. The link in option 2 goes straight to that API. This link is a MySQL article outlining how to use it.
According to https://dev.mysql.com/doc/refman/5.7/en/connector-j-reference-configuration-properties.html, you should be able to add ?allowMultiQueries=true to your JDBC connection string. Then you would be able to pass multiple statements, separated by semicolons, in Statement#execute(String sql) calls.
Edit: or, use a stored procedure that does what you want. Or, as you said, synchronize the Java code.
You can try using a Multiquery, combined the Insert and the Select Last_INSERT_ID() in the same string.
1) prepare the connection for using the multiquery:
"jdbc:mysql://"+host+"/"+database+"?allowMultiQueries=true"
2) Combine The Insert Query with the Select:
multiQuerySqlString="INSERT INTO bugs (summary, status, report_date) VALUES (1, 2, 3 ); SELECT LAST_INSERT_ID()"
3) esecute the query and expecting multiple result sets:
boolean isResultSet = statement.execute();
ResultSet res = statement.getResultSet();
if isResultSet = statement.getMoreResults();
// Second ReulstSet object
res = cs.getResultSet();
I hope it works
If you have to do this all on a single connection you can ask the driver to return the generated ID:
insertStmt = connection.prepareStatement("...",PreparedStatement.RETURN_GENERATED_KEYS );
insertStmt.executeUpdate();
ResultSet rs = insertStatement.getGeneratedKeys();
Long id = null;
if (rs != null)
{
rs.next();
id = rs.getLong(1);
}
connection.commit();
return id;
Depending on the driver you might need a different prepareStatement() call that takes the column names as the second parameter:
insertStmt = connection.prepareStatement("INSERT ", new String[] {"ID"});
But even in with the above code you should be doing the concurrent inserts on different physical connections to be able to properly control your transactions.
Related
What is the difference between statement.executeQuery and statement.getResultSet(). I believe both will return ResultSet for a select statement but are there any specific criteria when we should use which of them.
In general you should use executeQuery if you know you are executing a select statement. The getResultSet() method by itself does not execute the statement.
The getResultSet is intended to be used in combination with execute. The execute methods are intended for use with unknown statement types, or statements that can produce multiple results (that is 0 or more update counts or result sets).
So in short: you should normally use executeQuery.
A simple example when you should use execute if the code doesn't know what query it is going to execute (an update, a query, or something more complex), for example when executing user provided queries.
Another example are SQL Server stored procedures, which can return multiple update counts and result sets.
A generic way of processing a result of execute is:
boolean isResultSet = statement.execute(sql);
while (true) {
if (isResultSet) {
try (ResultSet rs = statement.getResultSet()) {
// do something with result set
}
} else {
int updateCount = statement.getUpdateCount();
if (updateCount == -1) {
// -1 signals no more results
break;
}
// do something with update count
}
// move to next result
isResultSet = statement.getMoreResults();
}
This ensures that all* results get processed.
*: This example ignores exception results for systems (like SQL Server) that allow multiple exceptions to be interleaved with result sets and update counts, see How to get *everything* back from a stored procedure using JDBC for a more thorough example
Check out the JavaDoc for these methods. getResultSet can return null but executeQuery never return null.
There are also more constraints. For example executeQuery cannot be called on a PreparedStatement or CallableStatement.
String insert1 = "INSERT INTO Table1(Col1, col2, col3)"
+ "VALUES(?,?,?)";
String insert2 = "INSERT INTO Table2(Colx, coly)"
+ "VALUES(?,?)";
Connection conn = aConn;
PreparedStatement ps = null;
try {
ps = conn.prepareStatement(insert1);
// ps.addBatch(insert2);
I'm trying to Insert Data into multiple tables at a time, and it seems like addBatch(String sql) is not defined for PreparedStatement.
Is there any alternate way?
First of all, a PreparedStatement is used to cache a single SQL statement. This has the advantage that the driver/database might optimize the statement since it is expecting many of them and since it is a parameterized statement. If you want to use it for two different SQL statements you need two PreparedStatements.
In order to add rows to the statement you need to set your parameters using set*(1,...), set*(2,...), set*(3,...), etc. and then call addBatch() (no arguments!). Finally you submit the batch of statements using executeBatch().
I need to insert a couple hundreds of millions of records into the mysql db. I'm batch inserting it 1 million at a time. Please see my code below. It seems to be slow. Is there any way to optimize it?
try {
// Disable auto-commit
connection.setAutoCommit(false);
// Create a prepared statement
String sql = "INSERT INTO mytable (xxx), VALUES(?)";
PreparedStatement pstmt = connection.prepareStatement(sql);
Object[] vals=set.toArray();
for (int i=0; i<vals.length; i++) {
pstmt.setString(1, vals[i].toString());
pstmt.addBatch();
}
// Execute the batch
int [] updateCounts = pstmt.executeBatch();
System.out.append("inserted "+updateCounts.length);
I had a similar performance issue with mysql and solved it by setting the useServerPrepStmts and the rewriteBatchedStatements properties in the connection url.
Connection c = DriverManager.getConnection("jdbc:mysql://host:3306/db?useServerPrepStmts=false&rewriteBatchedStatements=true", "username", "password");
I'd like to expand on Bertil's answer, as I've been experimenting with the connection URL parameters.
rewriteBatchedStatements=true is the important parameter. useServerPrepStmts is already false by default, and even changing it to true doesn't make much difference in terms of batch insert performance.
Now I think is the time to write how rewriteBatchedStatements=true improves the performance so dramatically. It does so by rewriting of prepared statements for INSERT into multi-value inserts when executeBatch() (Source). That means that instead of sending the following n INSERT statements to the mysql server each time executeBatch() is called :
INSERT INTO X VALUES (A1,B1,C1)
INSERT INTO X VALUES (A2,B2,C2)
...
INSERT INTO X VALUES (An,Bn,Cn)
It would send a single INSERT statement :
INSERT INTO X VALUES (A1,B1,C1),(A2,B2,C2),...,(An,Bn,Cn)
You can observe it by toggling on the mysql logging (by SET global general_log = 1) which would log into a file each statement sent to the mysql server.
You can insert multiple rows with one insert statement, doing a few thousands at a time can greatly speed things up, that is, instead of doing e.g. 3 inserts of the form INSERT INTO tbl_name (a,b,c) VALUES(1,2,3); , you do INSERT INTO tbl_name (a,b,c) VALUES(1,2,3),(1,2,3),(1,2,3); (It might be JDBC .addBatch() does similar optimization now - though the mysql addBatch used to be entierly un-optimized and just issuing individual queries anyhow - I don't know if that's still the case with recent drivers)
If you really need speed, load your data from a comma separated file with LOAD DATA INFILE , we get around 7-8 times speedup doing that vs doing tens of millions of inserts.
If:
It's a new table, or the amount to be inserted is greater then the already inserted data
There are indexes on the table
You do not need other access to the table during the insert
Then ALTER TABLE tbl_name DISABLE KEYS can greatly improve the speed of your inserts. When you're done, run ALTER TABLE tbl_name ENABLE KEYS to start building the indexes, which can take a while, but not nearly as long as doing it for every insert.
You may try using DDBulkLoad object.
// Get a DDBulkLoad object
DDBulkLoad bulkLoad = DDBulkLoadFactory.getInstance(connection);
bulkLoad.setTableName(“mytable”);
bulkLoad.load(“data.csv”);
try {
// Disable auto-commit
connection.setAutoCommit(false);
int maxInsertBatch = 10000;
// Create a prepared statement
String sql = "INSERT INTO mytable (xxx), VALUES(?)";
PreparedStatement pstmt = connection.prepareStatement(sql);
Object[] vals=set.toArray();
int count = 1;
for (int i=0; i<vals.length; i++) {
pstmt.setString(1, vals[i].toString());
pstmt.addBatch();
if(count%maxInsertBatch == 0){
pstmt.executeBatch();
}
count++;
}
// Execute the batch
pstmt.executeBatch();
System.out.append("inserted "+count);
so I have a software which basically downloads 1.5K game server address from my MySQL db. It then pings all of them and then upload the information such as online players back to the database. The process looks like this:
Download server address
Ping the servers and get information
Upload information back to the database
So far I have been able to solve the part where it download the server host name and pings them but the problem arises when updating the servers.
To update I thought about using a for loop to construct one BIG string of many update statements and execute it at once but this is prone to sql injections. So idealy one would want to use prepared statements.
The SQL update statement i'm using is:
UPDATE serverlist SET `onlineplayers` = '3', maxplayers = '10',
name = 'A game server' WHERE `ip` = 'xxx.xxx.xxx.xxx' AND `port` = 1234;
So my question is: How can i execute all the 1.5K updates statements using parameterized queries?
If you google for "jdbc bulk update" you'll get lots of results like this one or this one.
The latter has an example like this:
try {
...
connection con.setAutoCommit(false);
PreparedStatement prepStmt = con.prepareStatement(
"UPDATE DEPT SET MGRNO=? WHERE DEPTNO=?");
prepStmt.setString(1,mgrnum1);
prepStmt.setString(2,deptnum1);
prepStmt.addBatch();
prepStmt.setString(1,mgrnum2);
prepStmt.setString(2,deptnum2);
prepStmt.addBatch();
int [] numUpdates=prepStmt.executeBatch();
for (int i=0; i < numUpdates.length; i++) {
if (numUpdates[i] == -2)
System.out.println("Execution " + i +
": unknown number of rows updated");
else
System.out.println("Execution " + i +
"successful: " numUpdates[i] + " rows updated");
}
con.commit();
} catch(BatchUpdateException b) {
// process BatchUpdateException
}
Sounds like you want to do a batch SQL update. Prepared statements are your friend. Here's an example of using prepared statements in batch:
http://www.mkyong.com/jdbc/jdbc-preparedstatement-example-batch-update/
Using prepared statements makes setting parameters easier and it allows the DB to efficiently perform multiple updates. Executing multiple SQL strings would work but would be inefficient since each SQL string would be sent to the DBMS, parsed, compiled, then executed. With prepared statements the SQL is parsed and compiled once then reused for future updates with different parameters.
Another important step that you should be aware about during MySQL batch update / insert is JDBC Connection propertie rewriteBatchedStatements=true ( false by default ). Without it batch mode is useless.
It cost me 1 day to "fix bug" till I found out this.
When you have small number of lines and close client-to-DB location ( 1ms ping ) , you even can't realize that you in "fake batch mode" , but when I switch environment to remote client ( ping=100ms ) and 100k lines to update , it would take 4hours of "batch mode update" with default rewriteBatchedStatements=false and just 2minutes with rewriteBatchedStatements=true
Create a prepared statement:
String sql = "update serverlist SET onlineplayers = ?, maxplayers = ?, name = ? where ip = ? and port = ?";
PreparedStatement stmt = connection.prepareStatement(sql);
Then loop through your list, and at each iteration, do
stmt.setInt(1, onlinePlayers);
stmt.setInt(2, maxPlayers);
stmt.setString(3, name);
stmt.setString(4, ip);
stmt.setInt(5, port);
stmt.executeUpdate();
For better performance, you could also use batch updates.
Read the JDBC tutorial.
I want to batch up multiple types of database calls in one PreparedStatement. Is this possible?
Is there anyway to do something like
PreparedStatement pstmt = connection.prepareStatement("?");
where the ? can either be INSERT INTO MY_TABLE VALUES(1,2,3,4) or it could be UPDATE MY_TABLE, SET MY_VAL='1' WHERE MY_VAL IS NULL
Or do I always need to specify a table and action for my prepared statement?
Java will not allow you add only ? in preparedstatement string parameter, as it expects the ? for the place holder only for the parameters to the give SQL.
For your case, you may have to have 2 prepared statement objects, and in loop through, you can make a decision which one to call. So it would be something like below:
PreparedStatement insertPstmt = connection.prepareStatement("INSERT INTO MY_TABLE VALUES(?,?,?,?)");
PreparedStatement updatePstmt = connection.prepareStatement("UPDATE MY_TABLE, SET MY_VAL=? WHERE MY_VAL IS NULL");
While (<condition>) {
If (<insert condition>) {
// use insert pstmt and add batch
} else {
// use update pstmt and add batch
}
}
insertPstmt.executeBatch();
updatePstmt.executeBatch();
if you have any insert , which has dependency on the update, you might execute the batches accordingly. This will make sure that the update will work correctly. I would think of executing insert first, as they might not depend on update.
On a PreparedStatement, after binding the variables for the first execution, call
pstmt.addBatch();
then bind the variables for the next, and each time calling addBatch().
Then, when you're done adding batches you execute the bacth by alling
pstmt.executeBatch();
See :
http://docs.oracle.com/javase/7/docs/api/java/sql/PreparedStatement.html#addBatch%28%29
and
http://docs.oracle.com/javase/7/docs/api/java/sql/Statement.html#executeBatch%28%29
BTW : injecting the entire statment into a variable won't work. This batch mechanism exists to reuse the same statement binding different variables each time.
Insert and Update commands don't return any data that has to be processed. If you want to do just things like in your examples, you can simply run a non executing query command and provide a concatenated string of all your sql strings separated by a semicolon.
"INSERT INTO MY_TABLE VALUES(1,2,3,4)" + ";" +"UPDATE MY_TABLE, SET MY_VAL='1' WHERE MY_VAL IS NULL" + ";" +...
You don't need to prepare the statement in that case and also wouldn't receive any performance gain by doing so.