Getting inserted (or existing) ids with executeBatch() - java

I am trying to insert some words to database and return newly inserted id or existing id if the word is already in the database.
I found that I can do this using PreparedStatement and including Statement.RETURN_GENERATED_KEYS. But PreparedStatement is terribly slow. I need to insert like 5000 words at once. Another way I could achieve it by running individual query in for loop:
public ArrayList<Integer> addWords(ArrayList<String[]> allTermsForTag) {
ArrayList ids = new ArrayList<Integer>();
ResultSet rs = null;
try{
Statement st = connection.createStatement();
for (String[] articleTerms: allTermsForTag) {
for(String term: articleTerms) {
String query = "WITH a AS (INSERT INTO tag (name) SELECT '"+term+"' WHERE NOT EXISTS (SELECT name FROM tag WHERE name = '"+term+"') " +
"RETURNING id) SELECT id FROM a UNION SELECT id FROM tag WHERE name = '"+term+"'";
rs = st.executeQuery(query);
while (rs.next())
{
int id = rs.getInt(1);
ids.add(id);
System.out.printf("id: "+id);
}
}
}
rs.close();
st.close();
}catch(SQLException e){
System.out.println("SQL exception was raised while performing SELECT: "+e);
}
return ids;
}
This does what I need nicely, but this is too slow as well.
Another method that I wrote uses executeBatch(), however, it does not return ids:
public ArrayList<Integer> addWords(ArrayList<String[]> allTermsForTag){
ResultSet rs = null;
ArrayList ids = new ArrayList<Integer>();
try{
Statement st = connection.createStatement();
for (String[] articleTerms: allTermsForTag) {
for(String term: articleTerms) {
String query = "WITH a AS (INSERT INTO tag (name) SELECT '"+term+"' WHERE NOT EXISTS (SELECT name FROM tag WHERE name = '"+term+"') " +
"RETURNING id) SELECT id FROM a UNION SELECT id FROM tag WHERE name = '"+term+"'";
st.addBatch(query);
}
st.executeBatch();
rs = st.getGeneratedKeys();
while (rs.next()) {
int id = rs.getInt(1);
ids.add(id);
}
}
st.close();
return ids;
}catch (SQLException e){
System.out.println("SQL exception was raised while performing batch INSERT: "+e.getNextException());
System.out.println("dub");
}
return null;
}
So the question is - how to get ids when using executeBatch() or if this is not possible, how to approach this problem? I need it to work as fast as possible, because there will be a lot of INSERT operations with large amount of data.
Thank you!

Set set = new HashSet();
try {
PreparedStatement ps = cn.prepareStatement("delete from myTable where... ",
Statement.RETURN_GENERATED_KEYS);
ps.setInt(1,200);
ps.setInt(2,262);
ps.setString(3, "108gf99");
ps.addBatch();
ps.setInt(1,200);
ps.setInt(2,250);
ps.setString(3, "hgfha");
ps.addBatch();
ps.executeBatch();
ResultSet rs = ps.getGeneratedKeys();
while (rs.next()){
set.addAll(Collections.singleton(rs.getLong(1)));
}
System.out.println(set);
} catch (SQLException e) {
e.printStackTrace();
}

executeBatch can return generated keys in the latest PgJDBC versions. See issue 195 and pull 204. You must use the prepareStatement variant that takes a String[] of returned column names.
However... take a step back here. The solution isn't loops. The solution is almost never loops.
In this case, you should almost certainly use COPY via the PgJDBC CopyManager API to COPY data into a TEMPORARY table. Then do an INSERT INTO ... SELECT ... RETURNING ... to insert the temp table's contents into the final table and return any generated fields. You can also do a SELECT to join on the temp table to return any that already exist. This is basically a bulk upsert or closely related bulk insert-if-not-exists.
If for some reason you can't do that, the next-best option is probably multi-valued INSERTs with large VALUES lists, but this requires some ugly dynamic SQL. Since you need existing values if the row already exists you'll probably need a writeable CTE too. So really, just use COPY and a query to do the table merge.

Related

Get the inserted id of the record before or after execute the insert statement in sql

I have this method in my DAO class to insert record to a table called idea this is my method:
public long addIdea(AddIdeaDto addIdeaDto, int userId) {
Connection connection = null;
PreparedStatement preparedStatement = null;
try {
connection = getConnection();
preparedStatement = connection.prepareStatement(
"INSERT INTO IDEA ( IDEA.I_ID,IDEA.I_NO,IDEA.I_APPROVER_NAME_CODE, IDEA.I_TITLE,IDEA.I_DESCRIPITION, IDEA.I_CREATED_DATE,IDEA.I_STATUS_CODE, "
+ "IDEA.I_IS_CODE, IDEA.I_CONTRIBUTION_CODE, IDEA.I_POSITIVE_IMPACT, IDEA.I_SECOND_MEMBER_ID,IDEA.I_THIRD_MEMBER_ID,IDEA.I_FOURTH_MEMBER_ID,"
+ "IDEA.I_FIFTH_MEMBER_ID, IDEA.I_POINTS,IDEA.I_CREATED_USER_ID)"
+ " VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)");
preparedStatement.executeQuery("SELECT IDEA_SEQ.nextval FROM DUAL");
// Set parameters
preparedStatement.setObject(1, Types.NUMERIC);
preparedStatement.setObject(2, Types.NUMERIC);
preparedStatement.setObject(3, addIdeaDto.getApproverNameCode());
preparedStatement.setString(4, addIdeaDto.getTitle());
preparedStatement.setString(5, addIdeaDto.getDescription());
preparedStatement.setDate(6, addIdeaDto.getCreatedDate() == null ? null
: new java.sql.Date(addIdeaDto.getCreatedDate().getTime()));
preparedStatement.setObject(7, addIdeaDto.getStatusCode());
preparedStatement.setObject(8, addIdeaDto.getIsNewCode());
preparedStatement.setObject(9, addIdeaDto.getContributionCode());
preparedStatement.setString(10, addIdeaDto.getPositiveImpact());
preparedStatement.setObject(11, addIdeaDto.getSecondMemberName());
preparedStatement.setObject(12, addIdeaDto.getThirdMemberName());
preparedStatement.setObject(13, addIdeaDto.getFourthMemberName());
preparedStatement.setObject(14, addIdeaDto.getFifthMemberName());
preparedStatement.setObject(15, addIdeaDto.getPoints());
preparedStatement.setInt(16, userId);
preparedStatement.executeQuery();
return addIdeaDto.getIdeaId();
} catch (Exception e) {
throw new RuntimeException(e);
} finally {
try {
preparedStatement.close();
} catch (SQLException e) {
throw new RuntimeException(e);
}
}
}
actually what I want is after or before the insert statement I want to get the id (IDEA_SEQ.nextval) and save it in a value in order to use it as an input to insert in anther table.
For example, I insert this record : id = 1 , no = 1, approver code = 2, title = 'test'.............
I want this value id = 1 to use it in order to insert in table A, A_id = 33, IDEA.I_ID = 1, A_name ='testing'
how i can achieve it in properer way?
I update the code based on the comments that i receive but I did not achieve it
Usually ID that need to be reuse can be handle using a previous and separate SQL query
previousPreparedStatement = connection.prepareStatement(
"select IDEA_SEQ.nextval as nextval from dual");
Result saved as a int or String parameter according to column (number or varchar) which is passed to the existing insert statement:
(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)");
Notice also an answer from DBA forum
you won't be able to use plain SQL to overcome this limitation: you will need some PL/SQL
A better way to handle this is the RETURNING INTO clause, which uses a single, atomic statement:
INSERT INTO mytable (id, col1, col2)
VALUES ( seq_id.nextval, c1, c2 )
RETURNING id INTO myval;
You can use PreparedStatement.getGeneratedKeys() to obtain the generated value. There is no need to use a separate statement:
You also can't prefix column names with the table name in list of columns of an INSERT statement.
String insert =
"INSERT INTO IDEA ( I_ID,I_NO,I_APPROVER_NAME_CODE, I_TITLE,I_DESCRIPITION, I_CREATED_DATE,I_STATUS_CODE, "
+ "I_IS_CODE, I_CONTRIBUTION_CODE, I_POSITIVE_IMPACT, I_SECOND_MEMBER_ID,I_THIRD_MEMBER_ID,I_FOURTH_MEMBER_ID,"
+ "I_FIFTH_MEMBER_ID, I_POINTS,I_CREATED_USER_ID)"
+ " VALUES (idea_seq.nextval,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)";
preparedStatement = connection.prepareStatement(insertSql, new String[] {"I_ID"});
preparedStatement.setInt(1, ???); // don't know where the value for I_NO comes from
preparedStatement.setString(2, addIdeaDto.getApproverNameCode());
preparedStatement.setString(3, addIdeaDto.getTitle());
... other parameters
preparedStatement.executeUpdate();
ResultSet rs = preparedStatement.getGeneratedKeys();
long newId = -1;
if (rs.next()) {
newId = rs.getLong("I_ID");
}
... use the NewId ...
The parameter new String[] {"I_ID"} for the prepareStatement() call tells the JDBC driver to return the generated value for that column. That value can be retrieved through getGeneratedKeys() which returns a ResultSet that contains one row for each inserted row (so exactly one in this case). The ID value can then be extracted from the ResultSet using the the usual getLong() (or getInt()) methods.

Insert values of arraylist into mysql table

I am trying to insert values from arraylist into mysql table.
List lstUnique=new ArrayList<Object>();
//code to feed the data into list from resultset.
try{
Class.forName("com.mysql.jdbc.Driver");
con=DriverManager.getConnection("jdbc:mysql://localhost:3306/MYdb","root","");
stmt=con.createStatement();
String sql1="select SourceFrom,Updated,Location_Type,Industry,ET_Rank,BT_Rank from mytable";
rs1=stmt.executeQuery(sql1);
rsmd=rs1.getMetaData();
columnNumber=rsmd.getColumnCount();
while(rs1.next()){
lstUnique.add(rs1.getString("SourceFrom")+","+rs1.getString("Updated")+","+rs1.getString("Location_Type")+","+
rs1.getString("Industry")+","+rs1.getString("ET_Rank")+","+rs1.getString("BT_Rank"));
}
String insertsql="";
String SourceFrom=lstUnique.get(0).toString(); //its first column of the table.
insertsql="Insert into java_uniquedata (SourceFrom,field2,field3,field4,field5) values(?,?,?,?,?)";
PreparedStatement ps=con.prepareStatement(insertsql);
//I tried this way also.
for(int i=0;i<lstUnique.size();i++){
SourceFrom=lstUnique.get(i).toString();
}
for(int i=0;i<lstUnique.size();i++){
System.out.println("\n" + lstUnique.get(i).toString());
}
rs1.close();
con.close();
}catch(Exception e){
System.out.println(e);
}
But I am getting error
java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
My list has only one record in it, which has total 5 columns' values. Can you guide me how do I fetch values of first record from arraylist and insert it into mysql table.
You should separate the values for your insert statement. Either use a custom datastructure (class) or use a List<String>.
Something like this might work:
List<List<String>> lstUnique=new ArrayList<>(); // note the List of List
try {
/* ... your database query here ... */
while(rs1.next()){
List<String> values = new ArrayList<>(); // create list for values
values.add(rs1.getString("SourceFrom"));
values.add(rs1.getString("Updated"));
values.add(rs1.getString("Location_Type"));
values.add(rs1.getString("Industry"));
values.add(rs1.getString("ET_Rank"));
values.add(rs1.getString("BT_Rank"));
lstUnique.add(values); // add values for each row
}
String insertsql="Insert into java_uniquedata (SourceFrom,field2,field3,field4,field5) values(?,?,?,?,?)";
PreparedStatement ps=con.prepareStatement(insertsql);
for(List<String> values : lstUnique) { // use for each for rows
for(int i=0;i<values.size();i++) { // set argument values to prepared statement
ps.setString((i+1), values.get(i));
}
ps.execute(); // execute insert statement
}
ps.close();
} catch(Exception e){
System.out.println(e);
} finally { // close recordset and connection in finally-block! (or use 'try-with-resource'!)
rs1.close();
con.close();
}
try this one
for(int i=0; i < lstUnique.size();i++){
ps.setString(1, lstUnique.get(i).toString());
}
ps.execute(insertsql);
I tried this and it inserts data into database
ArrayList<String> lis = new ArrayList<String>();
lis.add("pare1");
lis.add("2");
//code to feed the data into list from resultset.
Class.forName("com.mysql.jdbc.Driver");
Connection connection = DriverManager.getConnection("jdbc:mysql://localhost:3306/test", "root", "paresh");
String insertsql="";
insertsql="Insert into stud (name,age) VALUES (?,?)";
PreparedStatement ps=connection.prepareStatement(insertsql);
//this will set value for your insert statement
System.out.println(lis.get(0).toString());
ps.setString(1, lis.get(0).toString());
ps.setInt(2, Integer.parseInt(lis.get(1)));
System.out.println(ps);
ps.execute(insertsql);
Completely different aproach for a solution to the task, if it is not mandatory to be solved in Java(?):
Since your code sample gives the impression that you're reading from and writing to the same database, you might also consider to copy the data in-database using the SQL INSERT-SELECT Syntax:
INSERT INTO java_uniquedata (SourceFrom, field2, field3, field4, field5)
SELECT DISTINCT SourceFrom, Updated, Location_Type, Industry, ET_Rank, BT_Rank
FROM mytable;
See MySQL INSERT ... SELECT Syntax.

Loading entries from MySQL into Java table based on search field

I have a MySQL table with entries already in it and I have it connected to my Java program so it displays the table values whenever the program is run. I'm basically trying to implement a search field where the user can type any attribute's value and all the entries that match that value will be loaded into the table. Then the user will be able to select the right entry that matches and they can edit, or update that entry's information. This would be useful for me particularly when you have entries that have the same value, for instance first name, last name, or zip code.
try {
String sql = "SELECT * FROM donors WHERE donor_id = ?";
ps = conn.prepareStatement(sql);
ps.setString(1, txtSearch1.getText());
rs = ps.executeQuery();
tblDonors.setModel(DbUtils.resultSetToTableModel(rs));
} catch (Exception e) {
JOptionPane.showMessageDialog(null, e);
}
try {
String sql = "SELECT * FROM donors WHERE first_name = ?";
ps = conn.prepareStatement(sql);
ps.setString(1, txtSearch1.getText());
rs = ps.executeQuery();
tblDonors.setModel(DbUtils.resultSetToTableModel(rs));
} catch (Exception e) {
JOptionPane.showMessageDialog(null, e);
}
The search field only searches for the second query, but not the first, so I can type a name and the matching names will load into the table, but when I try to input an id number, nothing happens. I'm fairly new to this, but I think it has something to do with my resultset object? Not exactly sure though. Any help would be great.
What happens here is that the second result overwrites the first. I think the easiest solution is to use or in the where clause, like this:
String sql = "SELECT * FROM donors WHERE (donor_id = ?) or (first_name = ?)";
ps.setString(1, txtSearch1.getText());
// but of course there are 2 ?'s now, we have to give the value to the second one
// as well
ps.setString(2, txtSearch1.getText());
Due to the way placeholders work in JDBC you'll have to provide a value for each ?.

How to check if resultset has one row or more?

How to check if resultset has one row or more with JDBC?
ResultSet rs = stmt.executeQuery("SELECT a, b, c FROM Table1");
boolean isMoreThanOneRow = rs.first() && rs.next();
You didn't ask this one, but you may need it:
boolean isEmpty = ! rs.first();
Normally, we don't need the row count because we use a WHILE loop to iterate through the result set instead of a FOR loop:
ResultSet rs = stmt.executeQuery("SELECT a, b, c FROM Table1");
while (rs.next()) {
// retrieve and print the values for the current row
int i = rs.getInt("a");
String s = rs.getString("b");
float f = rs.getFloat("c");
System.out.println("ROW = " + i + " " + s + " " + f);
}
However, in some cases, you might want to window the results, and you need the record count ahead of time to display to the user something like Row 1 to 10 of 100. You can do a separate query with SELECT COUNT(*) first, to get the record count, but note that the count is only approximate, since rows can be added or removed between the time it takes to execute the two queries.
Sample from ResultSet Overview
There are many options, and since you don't provide more context the only thing left is to guess. My answers are sorted by complexity and performance ascending order.
Just run select count(1) FROM ... and get the answer. You'd have to run another query that actually selects and returns the data.
Iterate with rs.next() and count until you're happy. Then if you still need the actual data re-run same query.
If your driver supports backwards iteration, go for rs.next() couple of times and then rewind back with rs.previous().
You don't need JDBC for this. The normal idiom is to collect all results in a collection and make use of the collection methods, such as List#size().
List<Item> items = itemDAO.list();
if (items.isEmpty()) {
// It is empty!
if (items.size() == 1) {
// It has only one row!
} else {
// It has more than one row!
}
where the list() method look like something:
public List<Item> list() throws SQLException {
Connection connection = null;
Statement statement = null;
ResultSet resultSet = null;
List<Item> items = new ArrayList<Item>();
try {
connection = database.getConnection();
statement = connection.createStatement();
resultSet = statement.executeQuery(SQL_LIST);
while (resultSet.next()) {
Item item = new Item();
item.setId(resultSet.getLong("id"));
item.setName(resultSet.getString("name"));
// ...
items.add(item);
}
} finally {
if (resultSet != null) try { resultSet.close(); } catch (SQLException logOrIgnore) {}
if (statement != null) try { statement.close(); } catch (SQLException logOrIgnore) {}
if (connection != null) try { connection.close(); } catch (SQLException logOrIgnore) {}
}
return items;
}
If you want to make sure that there is exactly one row, you can ensure that the first row is the last:
ResultSet rs = stmt.executeQuery("SELECT a FROM Table1 WHERE b=10");
if (rs.isBeforeFirst() && rs.next() && rs.isFirst() && rs.isLast()) {
// Logic for where there's exactly 1 row
Long valA = rs.getLong("a");
// ...
}
else {
// More that one row or 0 rows returned.
// ..
}
My no-brainer suggestion: Fetch the first result row, and then try to fetch the next. If the attempt is successful, you have more than one row.
If there is more than one row and you want to process that data, you'll need to either cache the stuff from the first row, or use a scrollable result set so you can seek back to the top before going through the results.
You can also ask SQL directly for this information by doing a SELECT COUNT(*) on the rest of your query; the result will be 0, 1 or more depending on how many rows the rest of the query would return. That's pretty easy to implement but involves two queries to the DB, assuming you're going to want to read and process the actual query next.
This implementation allows you to check for whether result of the query is empty or not at the cost of duplicating some lines.
ResultSet result = stmt.executeQuery("SELECT * FROM Table");
if(result.next()) {
// Duplicate the code which should be pasted inside while
System.out.println(result.getInt(1));
System.out.println(result.getString(2));
while(result.next()){
System.out.println(result.getInt(1));
System.out.println(result.getString(2));
}
}else{
System.out.println("Query result is empty");
}
Drawbacks:
In this implementation a portion of the code will be duplicated.
You cannot know how many lines are present in the result.
Get the Row Count using ResultSetMetaData class.
From your code u can create ResultSetMetaData like :
ResultSetMetaData rsmd = resultSet.getMetaData(); //get ResultSetMetaData
rsmd.getColumnCount(); // get row count from resultsetmetadata

Is there a way to retrieve the autoincrement ID from a prepared statement

Is there a way to retrieve the auto generated key from a DB query when using a java query with prepared statements.
For example, I know AutoGeneratedKeys can work as follows.
stmt = conn.createStatement();
stmt.executeUpdate(sql, Statement.RETURN_GENERATED_KEYS);
if(returnLastInsertId) {
ResultSet rs = stmt.getGeneratedKeys();
rs.next();
auto_id = rs.getInt(1);
}
However. What if I want to do an insert with a prepared Statement.
String sql = "INSERT INTO table (column1, column2) values(?, ?)";
stmt = conn.prepareStatement(sql);
//this is an error
stmt.executeUpdate(Statement.RETURN_GENERATED_KEYS);
if(returnLastInsertId) {
//this is an error since the above is an error
ResultSet rs = stmt.getGeneratedKeys();
rs.next();
auto_id = rs.getInt(1);
}
Is there a way to do this that I don't know about. It seems from the javadoc that PreparedStatements can't return the Auto Generated ID.
Yes. See here. Section 7.1.9. Change your code to:
String sql = "INSERT INTO table (column1, column2) values(?, ?)";
stmt = conn.prepareStatement(sql, Statement.RETURN_GENERATED_KEYS);
stmt.executeUpdate();
if(returnLastInsertId) {
ResultSet rs = stmt.getGeneratedKeys();
rs.next();
auto_id = rs.getInt(1);
}
There's a couple of ways, and it seems different jdbc drivers handles things a bit different, or not at all in some cases(some will only give you autogenerated primary keys, not other columns) but the basic forms are
stmt = conn.prepareStatement(sql, Statement.RETURN_GENERATED_KEYS);
Or use this form:
String autogenColumns[] = {"column1","column2"};
stmt = conn.prepareStatement(sql, autogenColumns)
Yes, There is a way. I just found this hiding in the java doc.
They way is to pass the AutoGeneratedKeys id as follows
String sql = "INSERT INTO table (column1, column2) values(?, ?)";
stmt = conn.prepareStatement(sql, Statement.RETURN_GENERATED_KEYS);
I'm one of those that surfed through a few threads looking for solution of this issue ... and finally get it to work. FOR THOSE USING jdbc:oracle:thin: with ojdbc6.jar PLEASE TAKE NOTE:
You can use either methods:
(Method 1)
Try{
String yourSQL="insert into Table1(Id,Col2,Col3) values(SEQ.nextval,?,?)";
myPrepStatement = <Connection>.prepareStatement(yourSQL, Statement.RETURN_GENERATED_KEYS);
myPrepStatement.setInt(1, 123);
myPrepStatement.setInt(2, 123);
myPrepStatement.executeUpdate();
ResultSet rs = getGeneratedKeys;
if(rs.next()) {
java.sql.RowId rid=rs.getRowId(1);
//what you get is only a RowId ref, try make use of it anyway U could think of
System.out.println(rid);
}
} catch (SQLException e) {
//
}
(Method 2)
Try{
String yourSQL="insert into Table1(Id,Col2,Col3) values(SEQ.nextval,?,?)";
//IMPORTANT: here's where other threads don tell U, you need to list ALL cols
//mentioned in your query in the array
myPrepStatement = <Connection>.prepareStatement(yourSQL, new String[]{"Id","Col2","Col3"});
myPrepStatement.setInt(1, 123);
myPrepStatement.setInt(2, 123);
myPrepStatement.executeUpdate();
ResultSet rs = getGeneratedKeys;
if(rs.next()) {
//In this exp, the autoKey val is in 1st col
int id=rs.getLong(1);
//now this's a real value of col Id
System.out.println(id);
}
} catch (SQLException e) {
//
}
Basically, try not used Method1 if you just want the value of SEQ.Nextval, b'cse it just return the RowID ref that you may cracked your head finding way to make use of it, which also don fit all data type you tried casting it to! This may works fine (return actual val) in MySQL, DB2 but not in Oracle.
AND, turn off your SQL Developer, Toad or any client which use the same login session to do INSERT when you're debugging. It MAY not affect you every time (debugging call) ... until you find your apps freeze without exception for some time. Yes ... halt without exception!
Connection connection=null;
int generatedkey=0;
PreparedStatement pstmt=connection.prepareStatement("Your insert query");
ResultSet rs=pstmt.getGeneratedKeys();
if (rs.next()) {
generatedkey=rs.getInt(1);
System.out.println("Auto Generated Primary Key " + generatedkey);
}

Categories