I'm parsing CSV file using OpenCSV library. I managed to skip first desired lines, choose only wanted columns and print it to console.
Now I'm struggling with inserting this to MSSQL database.
That's my code for parsing file:
JFileChooser fileopen = new JFileChooser();
FileFilter filter = new FileNameExtensionFilter(
"CSV file", "csv");
fileopen.setFileFilter(filter);
int ret = fileopen.showDialog(null, "Choose file");
if (ret == JFileChooser.APPROVE_OPTION) {
CsvParserSettings settings = new CsvParserSettings();
settings.detectFormatAutomatically();
settings.setHeaderExtractionEnabled(true);
settings.selectIndexes(7, 8, 13, 14);
settings.setNumberOfRowsToSkip(9);
List<String[]> rows = new CsvParser(settings).parseAll((fileopen.getSelectedFile()), "UTF-8");
rows.forEach(arr -> System.out.println(Arrays.toString(arr)));
Now code
INSERT INTO dbo.Glass(Nr_Temp) values(Arrays.toString(rows.get(1)));
Is getting me whole row instead column (which is understandable:)) but is there any other solution to return columns values to insert them to SQL database?
UPDATED
You need to iterate over String[] to get access to each separate value for a column.
PreparedStatement ps = connection.prepareStatement("INSERT INTO dbo.Szyby_temp(nr_zlec_klienta, nr_ref_klienta, szerokosc, wysokosc, ilosc, opis_dodatkowy, data_importu) VALUES(?, ?, ?, ?, ?, ?, getdate())");
int maxBatchSize = 100; //Using batching for performance
int currentBatchSize = 0;
for (String[] row : rows) {
int i = 1;
for (String columnValue : row) {
ps.setString(i++, columnValue); //Parameter indexes start with 1
}
ps.addBatch();
if (++currentBatchSize % maxbatchSize == 0) {
ps.executeUpdate();
}
}
ps.executeUpdate(); //if number of rows in csv file is not divisible by maxbatchSize
Thanks Ivan, I removed optimalization as files are small (less then 100 rows each) and also changed
ps.executeupdate() to `ps.executeBatch()
as it was uploading only last row, now It's working perfect, thank you for your time.
Here is my changed code
try {
PreparedStatement ps = conn.prepareStatement("INSERT INTO dbo.Szyby_temp(nr_zlec_klienta, nr_ref_klienta, szerokosc, wysokosc, ilosc, opis_dodatkowy, data_importu) VALUES(?, ?, ?, ?, ?, ?, getdate())");
for (String[] row : rows) {
int i = 0;
for (String columnValue : row) {
ps.setString(++i, columnValue); //Parameter indexes start with 1
}
ps.addBatch();
}
ps.executeBatch(); //if number of rows in csv file is not divisible by maxbatchSize
} catch (Exception e) {
JOptionPane.showMessageDialog(null, e, "ERROR", JOptionPane.ERROR_MESSAGE);
}
Related
The current format of my List<String[]> is:
60 52 0 0 1512230400
76 52 1 1 1514044800
42 52 4 1 1516464000
Whereby each separated value by space is a row in my database table, for example: 60 52 0 0 1512230400. I want to insert the 5 separate values per loop. I want to insert all these lines into my database but am not sure on exactly how. This is also a working connection to my database as of now.
This is my rough idea:
String query = "INSERT INTO games (team1_id, team2_id, score1, score2, created_at) VALUES (? ,?, ?, ?, ? )";
Connection con = DBConnector.connect();
PreparedStatement stmt = con.prepareStatement(query);//prepare the SQL Query
for (String[] s : fixtures) {
}
Any help is amazing.
Many thanks
In your for-loop, you can do something like this:
stmt.setString(1, s[0]); //team1_id if it's of string type in db
stmt.setInt(2, Integer.parseInt(s[1])); //team2_id if it's of type integer in db
stmt.setInt(3, Integer.parseInt(s[2])); //score1
stmt.setInt(4, Integer.parseInt(s[3])); //score2
stmt.setLong(5, Long.parseLong(s[4])); //created_at
stmt.executeUpdate();
The above code shows you how to deal with String, Long and Integer, you can use other types similarly.
List<String[]> fixtures = new ArrayList<>();
fixtures.add(new String [] {"60","52","0","0","1512230400"});
fixtures.add(new String [] {"76","52","1","1","1514044800"});
fixtures.add(new String [] {"42","52","4","1","1516464000"});
String query =
"INSERT INTO games (team1_id, team2_id, score1, score2, created_at)\n"
+ " VALUES (? ,?, ?, ?, ? )";
try(
Connection con = DBConnector.connect();
PreparedStatement stmt = con.prepareStatement(query);
) {
for (String[] s : fixtures) {
stmt.setString(1,s[0]);
stmt.setString(2,s[1]);
stmt.setString(3,s[2]);
stmt.setString(4,s[3]);
stmt.setString(5,s[4]);
stmt.execute();
}
con.commit();
}
With this approach, we pass the bind variables as strings. If needed, based on the actual type of the columns being inserted to, conversion from string (VARCHAR) to numeric (NUMBER) will happen by the database.
You got basically all of it right, but didn't take the next step of actually setting the bind-variables ...
This can work if the input List is already created:
List<String[]> fixtures = ...; // assuming this data is already created
String query = "INSERT INTO games (team1_id, team2_id, score1, score2, created_at) VALUES (? ,?, ?, ?, ? )";
try (Connection con = DBConnector.connect();
PreparedStatement stmt = con.prepareStatement(query)) {
for (String [] row : fixtures) {
// This gets executed for each row insert
for (int i = 0; i < row.length; i++) {
stmt.setInt(i+1, Integer.parseInt(row[i]);
}
stmt.executeUpdate();
}
}
catch(SQLException ex) {
ex.printStackTrace();
// code that handles exception...
}
I have created a custom function to insert data in my MySQL database. The functions first creates a query based on the input given. The query wil look like INSERT INTO tableName (columnName1, ..., columnNamei) VALUES (?, ..., ?), ..., (?, ...,?). After that, the PreparedStatement needs to made, which contains the real values. These need to be added to a batch, because I want to add multiple rows at once (as showed here: Java: Insert multiple rows into MySQL with PreparedStatement). Here is the code:
insertData() Function
public static void insertData(String table, List<HashMap<String, Object>> list) throws SQLException {
//Create query: make sure all of the rows in the table get the same amount of values passed
//Prepare colnames string
String colNamesParsed = "";
int counter = 1;
//Iterate over only the first hashmap of the list (THATS WHY ALL THE ROWS NEED TO HAVE THE SAME AMOUNT OF VALUES PASSED)
for (String colName : list.get(0).keySet()) {
//Check if it is the last col name
if (counter != list.get(0).keySet().size()) {
colNamesParsed = colNamesParsed + colName+", ";
}
else {
colNamesParsed = colNamesParsed + colName;
}
counter++;
}
//Now create the place holder for the query variables
String queryVariablesPlaceholder = "";
int rowSize = 0;
for (HashMap<String, Object> row : list) {
//This part is to check if all row sizes are equal
if (rowSize == 0) {
rowSize = row.values().size();
}
else {
//Check if the rowsize is equal for all rows
if (row.values().size() != rowSize) {
System.out.println("The rows of the arrays are from a different size");
return;
}
}
String queryVariablesRow = "(?, ";
for (int j = 1; j < (row.values().size()-1); j++) {
queryVariablesRow = queryVariablesRow+"?, ";
}
queryVariablesRow = queryVariablesRow+"?)";
//Make sure the query does not start with a comma
if (queryVariablesPlaceholder.equals("")) {
queryVariablesPlaceholder = queryVariablesRow;
}
else {
queryVariablesPlaceholder = queryVariablesPlaceholder+", "+queryVariablesRow;
}
}
//The MySQL query needs to be built now
String query = "INSERT INTO "+table+" ("+colNamesParsed+") VALUES "+queryVariablesPlaceholder+";";
System.out.println(query);
//Init prepared statement
PreparedStatement statement = con.prepareStatement(query);
for (HashMap<String, Object> map : list) {
int varCounter = 1;
//Iterate over all values that need to be inserted
for (Object object : map.values()) {
if (object instanceof Integer) {
statement.setInt(varCounter, Integer.parseInt(object.toString()));
}
else if (object instanceof String) {
statement.setString(varCounter, object.toString());
}
else if (object instanceof Timestamp) {
statement.setTimestamp(varCounter, parseStringToTimestamp(object.toString()));
}
else if (object instanceof Double) {
statement.setDouble(varCounter, Double.parseDouble(object.toString()));
}
System.out.println(varCounter);
varCounter++;
}
//Add row to the batch
try {
statement.addBatch();
}
catch (SQLException e) {
e.printStackTrace();
}
}
//Execute the query, which is in fact the batch
statement.executeBatch();
}
When I want to insert some data in the database, I execute the following code:
Functional part
List<HashMap<String, Object>> list = new ArrayList<>();
for (Object object : listOfObjects) {
HashMap<String, Object> map = new HashMap<>();
map.put("columnName1", object.getSomeValue());
/....../
map.put("columnName2", object.getSomeOtherValue());
list.add(map);
}
Functions.insertData("tableName", list);
Creating the dynamic query seems to work perfectly. However, I can't get the statement.addBatch() to work. It keeps giving me the following error:
java.sql.SQLException: No value specified for parameter 9
I don't get it, because I only have 8 parameters to pass in every unit of the batch. My target table has 9 columns, so I tried to add a value for that column, but then it says: No value specified for parameter 10, so it seems like it isn't closing the 'batch unit' or something.
What am I missing here?
Any help is greatly appreciated!
This
INSERT INTO tableName (columnName1, ..., columnNamei) VALUES (?, ..., ?), ..., (?, ...,?)
is not standard SQL syntax.
If you use this JDBC will a parameter for each "?" in your query.
Use:
INSERT INTO tableName (columnName1, ..., columnNamei) VALUES (?, ..., ?)
and add every statement to a batch.
I have about 500.000 rows of data to insert into one table.
I am currently inserting them one at a time (I know it's bad) like this :
Dao method :
public static final String SET_DATA = "insert into TABLE (D_ID, N_ID, VALUE, RUN_ID) " + "values (?, ?, ?, ?)";
public void setData(String dId, String nId, BigDecimal value, Run run) throws HibernateException {
if (session == null) {
session = sessionFactory.openSession();
}
SQLQuery select = session.createSQLQuery(SET_DATA);
select.setString(0, dId);
select.setString(1, nId);
select.setBigDecimal(2, value);
select.setLong(3, run.getRunId());
select.executeUpdate();
}
How can I do this more efficiently ?
Why you went for hand written SQL query ?? If you are writing sql in such a way , you are definitely not getting the fruits of hibernate.
Learn Batch Insert Example code for Batch Insert,
Session session = sessionFactory.openSession();
Transaction tx = session.beginTransaction();
for ( int i=0; i<100000; i++ ) {
Customer customer = new Customer(.....);
session.save(customer);
if ( i % 20 == 0 ) { //20, same as the JDBC batch size
//flush a batch of inserts and release memory:
session.flush();
session.clear();
}
}
tx.commit();
session.close();
The fastest way is always using the native bulk import tool of your DBMS.
Do not use hibernate or java for that.
Dump the data into some format your DB understands (most probably in the same file system of your DB) and use your DBMS native import tool.
Ideally you should use batch Insert. Refer to example provided here. This inserts multiple records in DB in a single go.
dbConnection.setAutoCommit(false);//commit trasaction manually
String insertTableSQL = "INSERT INTO DBUSER"
+ "(USER_ID, USERNAME, CREATED_BY, CREATED_DATE) VALUES"
+ "(?,?,?,?)";
PreparedStatement = dbConnection.prepareStatement(insertTableSQL);
for(int i=0;i<500000;i++){
preparedStatement.setInt(1, 101);
preparedStatement.setString(2, "mkyong101");
preparedStatement.setString(3, "system");
preparedStatement.setTimestamp(4, getCurrentTimeStamp());
preparedStatement.addBatch();
}
preparedStatement.executeBatch();
dbConnection.commit();
1.Solution
StringBuilder sb = new StringBuilder();
sb.AppendLine("insert into Table_Name (column1, column1, column1 , column1 ) values ");
foreach (var item in req)
{
sb.AppendFormat("({0},{1},{2},'{3}'),",
item.val1, item.val2, item.val3, item.val4);
}
sb = sb.Remove(sb.Length - 1, 1);
ExecuteNonQuery(sb.ToString());
return true;
if record count great then 1000
StringBuilder sb = new StringBuilder();
foreach (var item in req)
{
sb.AppendLine("insert into Table_Name(column1, column1, column1 , column1) values ");
sb.AppendFormat("({0},{1},{2},'{3}') ;",
item.val1, item.val2, item.val3, item.val4);
}
sb = sb.Remove(sb.Length - 1, 1);
ExecuteNonQuery(sb.ToString());
return true;
i have a method which has to insert into two tables, i have used batch insert for that.
my problem is it takes the second query and inserts in to that table and fails to insert into first table that is fails to take first query.
this is what is my code :
public String saveVillageDetails(VillagesViewModel viewmodel, String functionType) {
int[] saveOrupdateStatus = null;
StringBuffer disctrictQuery = new StringBuffer();
String saveOrupdateStatusMessage = "";
Connection connection = getConnection();
PreparedStatement districtPS = null;
ResultSet rs = null;
if (connection != null) {
try {
connection.setAutoCommit(false);
if(functionType.equals("add")){
// to insert in villages table
disctrictQuery.append(" INSERT INTO m_villages(districtid,village,creation_date,last_created_by) ");
disctrictQuery.append(" VALUES (?, ?, ?,?) ");
districtPS = connection.prepareStatement(disctrictQuery.toString());
districtPS.setInt(1, viewmodel.getDistrictid());
districtPS.setString(2, viewmodel.getVillage());
districtPS.setTimestamp(3, getCurrentDate());
districtPS.setInt(4,viewmodel.getUserid());
districtPS.addBatch();
// to insert in regions table
disctrictQuery=new StringBuffer();
disctrictQuery.append(" INSERT INTO m_regions(villageid,creation_date,last_created_by) ");
disctrictQuery.append(" VALUES (?, ?, ?) ");
districtPS = connection.prepareStatement(disctrictQuery.toString());
districtPS.setInt(1, getLatestVilalgeID());
districtPS.setTimestamp(2, getCurrentDate());
districtPS.setInt(3,viewmodel.getUserid());
districtPS.addBatch();
}else if(functionType.equals("edit")){
// to update villages table
disctrictQuery.append("UPDATE m_villages SET districtid=? ,village=? ,updation_date=? ,last_updated_by=? ");
disctrictQuery.append(" WHERE villageid=? ");
districtPS = connection.prepareStatement(disctrictQuery.toString());
districtPS.setInt(1, viewmodel.getDistrictid());
districtPS.setString(2, viewmodel.getVillage());
districtPS.setTimestamp(3, getCurrentDate());
districtPS.setInt(4,viewmodel.getUserid());
districtPS.setInt(5,viewmodel.getVillageid());
districtPS.addBatch();
disctrictQuery=new StringBuffer();
// to update regiions table
disctrictQuery.append("UPDATE m_regions SET villageid=?,updation_date=? ,last_updated_by=? ");
disctrictQuery.append(" WHERE regionid=? ");
districtPS = connection.prepareStatement(disctrictQuery.toString());
districtPS.setInt(1, viewmodel.getVillageid());
districtPS.setTimestamp(2, getCurrentDate());
districtPS.setInt(3,viewmodel.getUserid());
districtPS.setInt(4,getRegionID(viewmodel.getVillageid()));
districtPS.addBatch();
}
saveOrupdateStatus = districtPS.executeBatch();
if (saveOrupdateStatus.length > 0) {
if(functionType.equals("add")){
saveOrupdateStatusMessage = "Village Details Added successfully";
}else if(functionType.equals("edit")){
saveOrupdateStatusMessage = "Village Details Modified successfully";
}
connection.commit();
connection.setAutoCommit(true);
} else {
if(functionType.equals("add")){
saveOrupdateStatusMessage = "Failed to Add Village Details";
}else if(functionType.equals("edit")){
saveOrupdateStatusMessage = "Failded to Modify Village Details";
}
}
} catch (Exception ex) {
ex.printStackTrace();
//use log to print the exception ex.printStackTrace();
} finally {
try {
closeConnection(connection, rs, districtPS);
} catch (Exception ex) {
ex.printStackTrace();
//use logger here
}
}
}
return saveOrupdateStatusMessage;
}
when i printed the statement i get like this :
com.mysql.jdbc.ServerPreparedStatement[2] - INSERT INTO m_regions(villageid,creation_date,last_created_by) VALUES (18, '2012-11-06 17:41:25', 1)
and the first query is missing in the batch.
what could be the problem.
Please help
Regards
Please refactor your code in the way as below:
if(functionType.equals("add")){
// to insert in villages table
districtPS = connection.prepareStatement("INSERT INTO m_villages(districtid,village,creation_date,last_created_by) VALUES (?, ?, ?,?) ",
Statement.RETURN_GENERATED_KEYS);
districtPS.setInt(1, viewmodel.getDistrictid());
districtPS.setString(2, viewmodel.getVillage());
districtPS.setTimestamp(3, getCurrentDate());
districtPS.setInt(4,viewmodel.getUserid());
districtPS.executeUpdate();
ResultSet keySet = districtPS.getGeneratedKeys();
int villageId = 0;
if (keySet.next()) {
villageId = keySet.getInt(1);
}
// to insert in regions table
districtPS = connection.prepareStatement(" INSERT INTO m_regions(villageid,creation_date,last_created_by) VALUES (?, ?, ?) ",
Statement.RETURN_GENERATED_KEYS);
districtPS.setInt(1, villageId);
districtPS.setTimestamp(2, getCurrentDate());
districtPS.setInt(3,viewmodel.getUserid());
districtPS.executeUpdate();
}
I am inserting 132,000 records in a database table using jdbc batch of prepated statement. The problem i am facing is all records are not inserted in to table only records 1444 are inserted.
try{
StringBuffer insert = new StringBuffer("INSERT INTO mytable (field1,field2,
,field3,field 4 ) VALUES ( ?, ?, ?, ? )");
pstmt = conn.prepareStatement(insert.toString());
Iterator cptIcd9Iterator = cptIcd9List.iterator();
while(cptIcd9Iterator.hasNext()){
cptIcd9VO = (CptIcd9VO)cptIcd9Iterator.next();
count++;
pstmt.setString(1, "field1");
pstmt.setString(2, "field2");
pstmt.setString(3, "field3");
pstmt.setInt(4, 4);
pstmt.addBatch();
}
updateCounts = pstmt.executeBatch();
}
catch (Exception e) {
logger.error(e);
}
Can anyone help me ?
You can't insert all of them in one go, you must do it step by step.
Read this SO post.