I have a csv file with 80 000 rows,
each rows have: cost;date (123.232;30/12/2008)
I have to insert all cost data into tables names as a date in parametr second for example:
123.232 cost will be row in a "30/12/2008" table
and i have so many rows like this..
Now my program looks like:
Now i have to declare SQL query in for loop because i need "date" parameter,
my question - how to draw a "pStatement = connection.prepareStatement("INSER...." line away from for loop ? ofcourse with getting "date" parameter
Why i need that?- because now addingBatch doesn't work and now add to database only last row.
If i will move pStatement.executeBatch() inside for loop- then that will not work like a batch addingbut like normally each adding.
I'm using batch adding because i need fast working my application.
All advice will be wanted
Database database = new Database();
Connection connection = database.GetConnection();
PreparedStatement pStatement = null;
for(int x=0; x<=allRowsInCSVFile.size()-1; x++){
Rows row = allRows.get(x);
pStatement = connection.prepareStatement("INSERT INTO \""+ row.getDate() +"\" (cost) VALUES (?);");
pStatement.setLong(1, row.getCost());
pStatement.addBatch();
}
pStatement.executeBatch();
connection.close();
I think you should split the allRowsInCSVFile to multiple lists each for one date then you can draw the prepared statement out of the loop (sort of). It will not be as you exactly want, but it will a batch for each date. I think that will be a compromise that you have to do.
not sure how good this is so ill post as community wiki
Object obj1 = new Object();
PreparedStatement pStatement =
connection.prepareStatement("insert into " + obj1.toString() );
while(true)
{
obj1.setSomeValue
}
You could use just jdbcTemplate.batchUpdate. See an example here: http://www.mkyong.com/spring/spring-jdbctemplate-batchupdate-example/
This allows you to use different sql but still execute in batch.
With preparedstatement you would have to group together all inserts with same date values and prepare different statements for each of those dates.
Related
Is there a way to enter recurring data into a set database fields while using the prepared statement (pst)? I’m using the Derby DB.
I have an Excel spreadsheet that I get data from. Each line has 5 entry fields: Line item description, cost, quantity, extended cost, and price. I want to enter that into the derby db using just the five entries but with multiple lines each having five entries off the spreadsheet. The database has the following fields:
MATLINEITEM varchar 200
MATLINEITEMCOST varchar 7
MATLINEITEMQTY varchar 7
MATLINEITEMXCOST varchar 7
MATLINEITEMPRICE varchar 7
Using the standard INSERT with “ ?” approach, I would need to have an entry for each line and each spreadsheet field. If I have 10 lines, then I would need 50 “prepared statement” lines. I want to just use a loop to read each line and the five entries assigned to that line and enter it into the database.
The data from the spreadsheet is first converted to String (using POI) and assigned to a TextField.
addSQL = "INSERT INTO MATERIALTBL (MATLINEITEM, MATLINEITEMCOST,
MATLINEITEMQTY, MATLINEITEMXCOST, MATLINEITEMPRICE) VALUES (?,?,?,?,?) ";
pst = conn.prepareStatement(addSQL);
pst.setString(1,jtxtfld_Item.getText());
pst.setString(2,jtxtfld_Cost.getText());
pst.setString(3,jtxtfld_Qty.getText());
pst.setString(4,jtxtfld_XCost.getText());
pst.setString(5,jtxtfld_Price.getText());
pst.executeUpdate();
Any help would be appreciated. Or point me in the right direction as to what I can be using to get this result. Thank you.
If you want to insert multiple INSERT statements, it is possible.
use addBatch() and executeBatch() methods of PrepearedStamement.
You create a PrepearedStatement, and add batches to it, until it is ready to be inserted into the database.
Example
String addSQL = "INSERT INTO MATERIALTBL (MATLINEITEM, MATLINEITEMCOST,
MATLINEITEMQTY, MATLINEITEMXCOST, MATLINEITEMPRICE) VALUES (?,?,?,?,?) ";
PreparedStatemen pst = conn.prepareStatement(addSQL);
//Set auto-commit to false
conn.setAutoCommit(false);
for (int i = 0; i < amountOfIterations; i++){
// Set the variables
pst.setString(1,jtxtfld_Item.getText());
pst.setString(2,jtxtfld_Cost.getText());
pst.setString(3,jtxtfld_Qty.getText());
pst.setString(4,jtxtfld_XCost.getText());
pst.setString(5,jtxtfld_Price.getText());
// Add it to the batch
pst.addBatch();
}
//Create an int[] to hold returned values
int[] count = stmt.executeBatch();
//Explicitly commit statements to apply changes
conn.commit();
Why Use Batches?
This PrepearedStatement will be sent only once to the database, and will insert all your rows at once, saving you unnecessary calls to the database.
I need to insert a couple hundreds of millions of records into the mysql db. I'm batch inserting it 1 million at a time. Please see my code below. It seems to be slow. Is there any way to optimize it?
try {
// Disable auto-commit
connection.setAutoCommit(false);
// Create a prepared statement
String sql = "INSERT INTO mytable (xxx), VALUES(?)";
PreparedStatement pstmt = connection.prepareStatement(sql);
Object[] vals=set.toArray();
for (int i=0; i<vals.length; i++) {
pstmt.setString(1, vals[i].toString());
pstmt.addBatch();
}
// Execute the batch
int [] updateCounts = pstmt.executeBatch();
System.out.append("inserted "+updateCounts.length);
I had a similar performance issue with mysql and solved it by setting the useServerPrepStmts and the rewriteBatchedStatements properties in the connection url.
Connection c = DriverManager.getConnection("jdbc:mysql://host:3306/db?useServerPrepStmts=false&rewriteBatchedStatements=true", "username", "password");
I'd like to expand on Bertil's answer, as I've been experimenting with the connection URL parameters.
rewriteBatchedStatements=true is the important parameter. useServerPrepStmts is already false by default, and even changing it to true doesn't make much difference in terms of batch insert performance.
Now I think is the time to write how rewriteBatchedStatements=true improves the performance so dramatically. It does so by rewriting of prepared statements for INSERT into multi-value inserts when executeBatch() (Source). That means that instead of sending the following n INSERT statements to the mysql server each time executeBatch() is called :
INSERT INTO X VALUES (A1,B1,C1)
INSERT INTO X VALUES (A2,B2,C2)
...
INSERT INTO X VALUES (An,Bn,Cn)
It would send a single INSERT statement :
INSERT INTO X VALUES (A1,B1,C1),(A2,B2,C2),...,(An,Bn,Cn)
You can observe it by toggling on the mysql logging (by SET global general_log = 1) which would log into a file each statement sent to the mysql server.
You can insert multiple rows with one insert statement, doing a few thousands at a time can greatly speed things up, that is, instead of doing e.g. 3 inserts of the form INSERT INTO tbl_name (a,b,c) VALUES(1,2,3); , you do INSERT INTO tbl_name (a,b,c) VALUES(1,2,3),(1,2,3),(1,2,3); (It might be JDBC .addBatch() does similar optimization now - though the mysql addBatch used to be entierly un-optimized and just issuing individual queries anyhow - I don't know if that's still the case with recent drivers)
If you really need speed, load your data from a comma separated file with LOAD DATA INFILE , we get around 7-8 times speedup doing that vs doing tens of millions of inserts.
If:
It's a new table, or the amount to be inserted is greater then the already inserted data
There are indexes on the table
You do not need other access to the table during the insert
Then ALTER TABLE tbl_name DISABLE KEYS can greatly improve the speed of your inserts. When you're done, run ALTER TABLE tbl_name ENABLE KEYS to start building the indexes, which can take a while, but not nearly as long as doing it for every insert.
You may try using DDBulkLoad object.
// Get a DDBulkLoad object
DDBulkLoad bulkLoad = DDBulkLoadFactory.getInstance(connection);
bulkLoad.setTableName(“mytable”);
bulkLoad.load(“data.csv”);
try {
// Disable auto-commit
connection.setAutoCommit(false);
int maxInsertBatch = 10000;
// Create a prepared statement
String sql = "INSERT INTO mytable (xxx), VALUES(?)";
PreparedStatement pstmt = connection.prepareStatement(sql);
Object[] vals=set.toArray();
int count = 1;
for (int i=0; i<vals.length; i++) {
pstmt.setString(1, vals[i].toString());
pstmt.addBatch();
if(count%maxInsertBatch == 0){
pstmt.executeBatch();
}
count++;
}
// Execute the batch
pstmt.executeBatch();
System.out.append("inserted "+count);
The code that I am working with primarily uses Spring and jdbcTemplate as a way to query the database.
As a non-working example, but just to get the idea across of how I get data and display it on my website...
There will be some object called Bike.
List<bikeObject> bikes = new ArrayList<>();
List<Map<String, Object>> rows = jdbcTemplate.queryForList(bikeQuery));
for (Map<String<Object> row : rows){
bikeObject b = new bikeObject();
b.setProperty((String row.get(-property-));
....
bikes.push(bikeObject)
}
However, sometimes the query can be too large and my computer can run out of memory or the database query can timeout.
A solution that was brought to my attention was to just query it into a ResultSet and then iterate through and stream it directly to a file. I can scrap the display on the website and just let the user download an excel table on a click of a button.
I see that I can use something like (copied from the oracle site)
OracleDataSource ods = new OracleDataSource();
ods.setURL(url);
ods.setUser(user);
ods.setPassword(password);
String URL = "jdbc:oracle:thin:scott/tiger#//myhost:1521/orcl");
ods.setURL(URL);
Connection conn = ods.getConnection();
Statement stmt = conn.createStatement();
ResultSet rset = stmt.executeQuery(query);
from here I think I can just iterate through rset and write to a file using BufferedWriter.
The issue I have with this is that my code is pretty consistent so how would I set the URL/User/Password from the Spring properties file that I have? I don't want to type it in the file on a one time occasion.
Also, is this the best way to approach this problem? Can I write to file using jdbcTemplate + ResultSet? I'm stuck on finding a way how.
Slight update:
I assume that the query (passed off from someone else) is optimal and that all the data is necessary. This leaves me with the conclusion of streaming the query results straight to file. Is there a way I can do this with jdbcTemplate or do I have to do it via
Connection conn = ods.getConnection();
Statement stmt = conn.createStatement();
ResultSet rset = stmt.executeQuery(swSb);
And iterating through it on a next() basis?
You don't describe well the problem: Do you really need all data? is database setup with indexes and is the query optimal?
You can use oracle pagination support http://www.oracle.com/technetwork/issue-archive/2007/07-jan/o17asktom-093877.html so the user get first X elements.
If you really need all data and it is a lot I would avoid mapping to an object specially object instantiation inside a loop.
It would help if you could tell how many rows are you expecting
String insert1 = "INSERT INTO Table1(Col1, col2, col3)"
+ "VALUES(?,?,?)";
String insert2 = "INSERT INTO Table2(Colx, coly)"
+ "VALUES(?,?)";
Connection conn = aConn;
PreparedStatement ps = null;
try {
ps = conn.prepareStatement(insert1);
// ps.addBatch(insert2);
I'm trying to Insert Data into multiple tables at a time, and it seems like addBatch(String sql) is not defined for PreparedStatement.
Is there any alternate way?
First of all, a PreparedStatement is used to cache a single SQL statement. This has the advantage that the driver/database might optimize the statement since it is expecting many of them and since it is a parameterized statement. If you want to use it for two different SQL statements you need two PreparedStatements.
In order to add rows to the statement you need to set your parameters using set*(1,...), set*(2,...), set*(3,...), etc. and then call addBatch() (no arguments!). Finally you submit the batch of statements using executeBatch().
I need to insert a couple hundreds of millions of records into the mysql db. I'm batch inserting it 1 million at a time. Please see my code below. It seems to be slow. Is there any way to optimize it?
try {
// Disable auto-commit
connection.setAutoCommit(false);
// Create a prepared statement
String sql = "INSERT INTO mytable (xxx), VALUES(?)";
PreparedStatement pstmt = connection.prepareStatement(sql);
Object[] vals=set.toArray();
for (int i=0; i<vals.length; i++) {
pstmt.setString(1, vals[i].toString());
pstmt.addBatch();
}
// Execute the batch
int [] updateCounts = pstmt.executeBatch();
System.out.append("inserted "+updateCounts.length);
I had a similar performance issue with mysql and solved it by setting the useServerPrepStmts and the rewriteBatchedStatements properties in the connection url.
Connection c = DriverManager.getConnection("jdbc:mysql://host:3306/db?useServerPrepStmts=false&rewriteBatchedStatements=true", "username", "password");
I'd like to expand on Bertil's answer, as I've been experimenting with the connection URL parameters.
rewriteBatchedStatements=true is the important parameter. useServerPrepStmts is already false by default, and even changing it to true doesn't make much difference in terms of batch insert performance.
Now I think is the time to write how rewriteBatchedStatements=true improves the performance so dramatically. It does so by rewriting of prepared statements for INSERT into multi-value inserts when executeBatch() (Source). That means that instead of sending the following n INSERT statements to the mysql server each time executeBatch() is called :
INSERT INTO X VALUES (A1,B1,C1)
INSERT INTO X VALUES (A2,B2,C2)
...
INSERT INTO X VALUES (An,Bn,Cn)
It would send a single INSERT statement :
INSERT INTO X VALUES (A1,B1,C1),(A2,B2,C2),...,(An,Bn,Cn)
You can observe it by toggling on the mysql logging (by SET global general_log = 1) which would log into a file each statement sent to the mysql server.
You can insert multiple rows with one insert statement, doing a few thousands at a time can greatly speed things up, that is, instead of doing e.g. 3 inserts of the form INSERT INTO tbl_name (a,b,c) VALUES(1,2,3); , you do INSERT INTO tbl_name (a,b,c) VALUES(1,2,3),(1,2,3),(1,2,3); (It might be JDBC .addBatch() does similar optimization now - though the mysql addBatch used to be entierly un-optimized and just issuing individual queries anyhow - I don't know if that's still the case with recent drivers)
If you really need speed, load your data from a comma separated file with LOAD DATA INFILE , we get around 7-8 times speedup doing that vs doing tens of millions of inserts.
If:
It's a new table, or the amount to be inserted is greater then the already inserted data
There are indexes on the table
You do not need other access to the table during the insert
Then ALTER TABLE tbl_name DISABLE KEYS can greatly improve the speed of your inserts. When you're done, run ALTER TABLE tbl_name ENABLE KEYS to start building the indexes, which can take a while, but not nearly as long as doing it for every insert.
You may try using DDBulkLoad object.
// Get a DDBulkLoad object
DDBulkLoad bulkLoad = DDBulkLoadFactory.getInstance(connection);
bulkLoad.setTableName(“mytable”);
bulkLoad.load(“data.csv”);
try {
// Disable auto-commit
connection.setAutoCommit(false);
int maxInsertBatch = 10000;
// Create a prepared statement
String sql = "INSERT INTO mytable (xxx), VALUES(?)";
PreparedStatement pstmt = connection.prepareStatement(sql);
Object[] vals=set.toArray();
int count = 1;
for (int i=0; i<vals.length; i++) {
pstmt.setString(1, vals[i].toString());
pstmt.addBatch();
if(count%maxInsertBatch == 0){
pstmt.executeBatch();
}
count++;
}
// Execute the batch
pstmt.executeBatch();
System.out.append("inserted "+count);