I have the following code. It simply executes a query then import to the other database, the second part is irrelevant in my question/problem.
As I see by default jdbcTemplate.query() set Cursor before the first element. I don't know where is my mistake, but this code skips the first row since the cursor initial state #first element. So when the while (resultSet.next()) { starts its skip the first element....
I tried to force to move the cursor
resultSet.beforeFirst();
but it throws error.
Currently I check the Cursor position before the loop and I also thinking about to replace with a do {} while().
jdbcTemplate.query(sqlPull, resultSet -> {
List<List<Object>> batch = new ArrayList<>();
ResultSetMetaData metaData = resultSet.getMetaData();
int batchCount = 1;
int columnCount = metaData.getColumnCount();
if (resultSet.getRow() == 1) {
addAllColumnsToArray(resultSet, batch, columnCount);
}
while (resultSet.next()) {
addAllColumnsToArray(resultSet, batch, columnCount);
if (batch.size() >= 10000) {
callRestAPI(batch);
VaadinSqlApplication.logger.info("Commit batch: " + batchCount++);
insertTableToDb(sqlPush, push, batch);
batch.clear();
}
}
callRestAPI(batch);
VaadinSqlApplication.logger.info("Commit final batch. " + batchCount);
insertTableToDb(sqlPush, push, batch);
batch.clear();
});
....
private void addAllColumnsToArray(ResultSet resultSet, List<List<Object>> batch, int columnCount)
throws SQLException {
List<Object> row = new ArrayList<>();
for (int i = 1; i <= columnCount; i++) {
row.add(resultSet.getObject(i));
}
batch.add(row);
}
You can try do{...}while()
// ...
do{
addAllColumnsToArray(resultSet, batch, columnCount);
if (batch.size() >= 10000) {
callRestAPI(batch);
VaadinSqlApplication.logger.info("Commit batch: " + batchCount++);
insertTableToDb(sqlPush, push, batch);
batch.clear();
}
}while(resultSet.next());
//...
Related
Does jdbcTemplate.batchUpdate execute multiple single insert statements OR 1 multi value list insert on the database server?
I know that it sends the complete query payload at once to the server but am not sure how the execution takes place.
Can someone please explain/help?
From question:
Does jdbcTemplate.batchUpdate execute multiple single insert statements OR 1 multi value list insert on the database server?
From comment:
I was curious about int[] org.springframework.jdbc.core.JdbcTemplate.batchUpdate(String sql, List<Object[]> batchArgs, int[] argTypes)
TL;DR: It executes 1 multi-valued list.
Spring Framework is open-source, so it's easy to look at the source code and see that is actually does.
batchUpdate(String sql, List<Object[]> batchArgs, final int[] argTypes)
#Override
public int[] batchUpdate(String sql, List<Object[]> batchArgs, final int[] argTypes) throws DataAccessException {
if (batchArgs.isEmpty()) {
return new int[0];
}
return batchUpdate(
sql,
new BatchPreparedStatementSetter() {
#Override
public void setValues(PreparedStatement ps, int i) throws SQLException {
Object[] values = batchArgs.get(i);
int colIndex = 0;
for (Object value : values) {
colIndex++;
if (value instanceof SqlParameterValue) {
SqlParameterValue paramValue = (SqlParameterValue) value;
StatementCreatorUtils.setParameterValue(ps, colIndex, paramValue, paramValue.getValue());
}
else {
int colType;
if (argTypes.length < colIndex) {
colType = SqlTypeValue.TYPE_UNKNOWN;
}
else {
colType = argTypes[colIndex - 1];
}
StatementCreatorUtils.setParameterValue(ps, colIndex, colType, value);
}
}
}
#Override
public int getBatchSize() {
return batchArgs.size();
}
});
}
As can be seen, it calls the following method.
batchUpdate(String sql, final BatchPreparedStatementSetter pss)
#Override
public int[] batchUpdate(String sql, final BatchPreparedStatementSetter pss) throws DataAccessException {
if (logger.isDebugEnabled()) {
logger.debug("Executing SQL batch update [" + sql + "]");
}
int[] result = execute(sql, (PreparedStatementCallback<int[]>) ps -> {
try {
int batchSize = pss.getBatchSize();
InterruptibleBatchPreparedStatementSetter ipss =
(pss instanceof InterruptibleBatchPreparedStatementSetter ?
(InterruptibleBatchPreparedStatementSetter) pss : null);
if (JdbcUtils.supportsBatchUpdates(ps.getConnection())) {
for (int i = 0; i < batchSize; i++) {
pss.setValues(ps, i);
if (ipss != null && ipss.isBatchExhausted(i)) {
break;
}
ps.addBatch();
}
return ps.executeBatch();
}
else {
List<Integer> rowsAffected = new ArrayList<>();
for (int i = 0; i < batchSize; i++) {
pss.setValues(ps, i);
if (ipss != null && ipss.isBatchExhausted(i)) {
break;
}
rowsAffected.add(ps.executeUpdate());
}
int[] rowsAffectedArray = new int[rowsAffected.size()];
for (int i = 0; i < rowsAffectedArray.length; i++) {
rowsAffectedArray[i] = rowsAffected.get(i);
}
return rowsAffectedArray;
}
}
finally {
if (pss instanceof ParameterDisposer) {
((ParameterDisposer) pss).cleanupParameters();
}
}
});
Assert.state(result != null, "No result array");
return result;
}
As can be seen, it creates a single PreparedStatement, enters a loop calling addBatch(), and finally calls executeBatch().
So, the short answer is: 1 multi-valued list.
The full answer is that it likely sends one SQL statement and a multi-valued list to the database server, however it is entirely up to the JDBC driver how it actually implements batching, mostly limited by what the communication protocol supports, so the only way to know for sure is to trace the communication with the server.
While jdbcTemplate.batchUpdate(...) is running I can see DB row count is increased gradually(by running count(*) in the table), initially 2k then 3k and goes till 10k. 2k and 3k are not exact numbers sometimes I get 235 and then 4567.
I was expecting 10 k rows (batch size) to be committed in one shot. In my understanding, if initially, I get row count 0 then next row count should be 10k.
I don't want one by one insert for performance reason that's why used batch update feature and seems it also doesn't commit all in one shot.
I want to send data(10k rows) to DB server only once for my batch size. for this is there anything I should specify in the configuration?
Below is the way I am writing jdbcTemplate batch update batch size is 10k.
public void insertRows(...) {
...
jdbcTemplate.batchUpdate(query, new BatchPreparedStatementSetter(){
#Override public void
setValues(PreparedStatement ps, int i) throws SQLException {
...
}
#Override public int getBatchSize() {
if(data == null){
return 0;
}
return data.size();
}
});
}
Edit: Added #Transactional to isertRows method still I can see the same behavior.
using Transnational it commits after 10k rows, but when I see count using with UR (select count(*) from mytable with ur) it shows data being updated gradually (2k 4k so on till 10k). That means data goes to a server in chunks (probably one bye one). How can i send everything in one shot. This question suggests it is achieved using rewriteBatchedStatements in mysql, is there anything similar we have in DB2 as well.
I am using DataSource implementation com.ibm.db2.jcc.DB2BaseDataSource
How about the below method? specify nUpdates =10,000 in your case. I have not tried testing this.please ignore my answer if it does not work.
// the batch size is set in the BatchPreparedStatementSetter, the number of rows we want to process is equal to the nbUpdates parameter
public int[] batchUpdate(String sql, final long nbUpdates, final BatchPreparedStatementSetter pss) throws DataAccessException {
if (logger.isDebugEnabled()) {
logger.debug("Executing SQL batch update [" + sql + "]");
}
return (int[]) execute(sql, new PreparedStatementCallback() {
public Object doInPreparedStatement(PreparedStatement ps) throws SQLException {
try {
int batchSize = pss.getBatchSize();
InterruptibleBatchPreparedStatementSetter ipss = (pss instanceof InterruptibleBatchPreparedStatementSetter ? (InterruptibleBatchPreparedStatementSetter) pss
: null);
if (JdbcUtils.supportsBatchUpdates(ps.getConnection())) {
List<Integer> rowsAffected = new ArrayList<Integer>();
for (int i = 1; i <= nbUpdates; i++) {
pss.setValues(ps, i - 1);
if (ipss != null && ipss.isBatchExhausted(i - 1)) {
if (logger.isDebugEnabled()) {
int batchIdx = (i % batchSize == 0) ? i / batchSize : (i / batchSize) + 1;
logger.debug("Batch exhausted - Sending last SQL batch update #" + batchIdx);
}
int[] res = ps.executeBatch();
for (int j = 0; j < res.length; j++) {
rowsAffected.add(res[j]);
}
break;
}
ps.addBatch();
if (i % batchSize == 0 || i == nbUpdates) {
if (logger.isDebugEnabled()) {
int batchIdx = (i % batchSize == 0) ? i / batchSize : (i / batchSize) + 1;
logger.debug("Sending SQL batch update #" + batchIdx);
}
int[] res = ps.executeBatch();
for (int j = 0; j < res.length; j++) {
rowsAffected.add(res[j]);
}
}
}
int[] result = new int[rowsAffected.size()];
for (int i = 0; i < result.length; i++) {
result[i] = rowsAffected.get(i).intValue();
}
return result;
} else {
List<Integer> rowsAffected = new ArrayList<Integer>();
for (int i = 0; i < nbUpdates; i++) {
pss.setValues(ps, i);
if (ipss != null && ipss.isBatchExhausted(i)) {
break;
}
rowsAffected.add(ps.executeUpdate());
}
int[] rowsAffectedArray = new int[rowsAffected.size()];
for (int i = 0; i < rowsAffectedArray.length; i++) {
rowsAffectedArray[i] = rowsAffected.get(i);
}
return rowsAffectedArray;
}
} finally {
if (pss instanceof ParameterDisposer) {
((ParameterDisposer) pss).cleanupParameters();
}
}
}
});
}
So I have a function that populates JTable from my database.
I have here
public static TableModel resultSetToTableModel(ResultSet rs) {
try {
ResultSetMetaData metaData = rs.getMetaData();
int numberOfColumns = metaData.getColumnCount();
Vector<String> columnNames = new Vector<String>();
// Get the column names
for (int column = 0; column < numberOfColumns; column++) {
columnNames.addElement(metaData.getColumnLabel(column + 1));
}
// Get all rows.
Vector<Vector<Object>> rows = new Vector<Vector<Object>>();
while (rs.next()) {
Vector<Object> newRow = new Vector<Object>();
for (int i = 1; i <= numberOfColumns; i++) {
if(isObjectInteger(rs.getObject(i)) && i>1) //checks if the value is Integer else not and is past the first column
{
System.out.println(i+"="+rs.getObject(i));
String label = columnNames.get(i); //THE ERROR IS ON THIS PART
newRow.addElement(getValue((Integer) rs.getObject(i),label)); //gets the value of specific foreign key id from another table
}
else
{
System.out.println(i+"="+rs.getObject(i));
newRow.addElement(rs.getObject(i));
} //inside row (new Rows)
}
rows.addElement(newRow); //outside row
}
return new DefaultTableModel(rows, columnNames)
{
#Override
public boolean isCellEditable(int row, int column) {
return false;
}
};
} catch (Exception e) {
e.printStackTrace();
return null;
}
}
I have total 8 columns in my database the output of that System.out.println is:
The one's that get's inside the else:
1=1
2=test
3=A.
4=test
5=test
6=test
The one's that get's inside the if
7=1
8=23
As you can see the output is right but it always throws Array index out of range: 8 error on the String label = columnNames.get(i);
While ResultSet.getObject() requires an argument based on one, columnNames is a vector, with its indexes based on zero.
Hence valid values for it would be 0..7, not 1..8. In other words, the first part of your if statement should be:
System.out.println(i + "=" + rs.getObject(i));
String label = columnNames.get(i-1); // NOT "i".
I have a strange error which I can not get my head around where the .size() method does not appear to return the correct value.
This first bit of code creates the ArrayList.
public void createResultList(String query) {
ArrayList<ArrayList<String>> data = new ArrayList();
try {
ResultSet rs = st.executeQuery(query);
ResultSetMetaData meta = rs.getMetaData();
for(int i = 0; i < meta.getColumnCount(); i++) {
data.add(i, new ArrayList<String>());
}
int x = 0;
while(rs.next()) {
for(int y = 0; y < meta.getColumnCount(); y++) {
data.get(x).add(rs.getString(y + 1));
}
x++;
}
} catch(Exception e) {
JOptionPane.showMessageDialog(null, e);
}
ResultTable result = new ResultTable(data);
JTable table = new JTable(result);
JScrollPane scrollpane = new JScrollPane(table);
add(scrollpane);
refresh();
}
This is my TableModel class which is used to create the table when it's passed to it.
public class ResultTable extends AbstractTableModel {
private ArrayList<ArrayList<String>> data;
public ResultTable(ArrayList<ArrayList<String>> data) {
this.data = data;
}
public int getColumnCount() {
return data.get(0).size();
}
public int getRowCount() {
return data.size();
}
public Object getValueAt(int row, int col) {
return data.get(row).get(col);
}
}
Now the the problem is in the ResultTable class, now for a select query returning one row with 12 columns, the first data.get(0).size() call correctly returns 12, but the 2nd data.size() call incorrectly returns 12 also instead of 1, this is causing out of bounds errors, can anyone please explain this seemingly paradoxical result?
This is something you should've found easily when you debug your code...
public void createResultList(String query) {
ArrayList<ArrayList<String>> data = new ArrayList();
data is an ArrayList of ArrayLists
try {
ResultSet rs = st.executeQuery(query);
A query returning 1 row of 12 columns
ResultSetMetaData meta = rs.getMetaData();
for(int i = 0; i < meta.getColumnCount(); i++) {
For each column in your recordset, being 12, you add an empty ArrayList in data
data.add(i, new ArrayList<String>());
}
resulting in data being an ArrayList of 12 empty Arraylists
This already explains why data.size() == 12
int x = 0;
while(rs.next()) {
for each record in your recordset, being 1
for(int y = 0; y < meta.getColumnCount(); y++) {
for each column in your recordset, being 12, you add a string to the ArrayList with the same index as the recordset
data.get(x).add(rs.getString(y + 1));
}
The first ArrayList in data (data.get(0)) will have 12 Strings
All other ArrayLists in data (data.get(x) where x > 0) will remain empty
x++;
}
Resulting in data being an ArrayList of 12 ArrayLists
of which only the first ArrayList has 12 Strings and the others are empty
} catch(Exception e) {
JOptionPane.showMessageDialog(null, e);
}
ResultTable result = new ResultTable(data);
JTable table = new JTable(result);
JScrollPane scrollpane = new JScrollPane(table);
add(scrollpane);
refresh();
}
When you create the data list, you create a sublist to hold the data for each column.
If that is what you want to do, you should add data from each column of the result set to the list that corresponds to it:
while(rs.next()) {
for(int y = 0; y < meta.getColumnCount(); y++) {
data.get(y).add(rs.getString(y + 1));
}
}
I built a function in java that searches in the database for 2 values and if they exist return the values;
public List<Conniction> findPath(int one,int two){
List<Conniction> list = new ArrayList<Conniction>();
Connection c = null;
String sql = "SELECT * FROM conniction WHERE oneid="+one+"&& twoid="+two;
try {
c = ConnectionHelper.getConnection();
Statement s = c.createStatement();
ResultSet rs = s.executeQuery(sql);
ResultSetMetaData metaData = rs.getMetaData();
int columncount = metaData.getColumnCount();
//direct result
if (columncount > 0) {
System.out.println("Match one and two found!");
while (rs.next()) {
list.add(processRow(rs));
}
}
else{
String sql2 = "SELECT * FROM conniction WHERE oneid="+one;
s = c.createStatement();
rs = s.executeQuery(sql2);
metaData = rs.getMetaData();
columncount = metaData.getColumnCount();
if (columncount > 0) {
System.out.println("One Match found!");
while (rs.next()) {
list.add(processRow(rs));
}
}
}
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return list;
}
How do I check what the values of the parameters that are returned are, to check if its really the right answer for oneid and twoid. It always returns something different from what I have sent.
I don't know if that is the best solution, but you could:
* give the list object with parameters
* and then return a code
* check the code and use the list
your method gets the list as a parameter:
public int findPath(List<Conniction> list, int one, int two) {
...
if (columncount > 0) {
System.out.println("Match one and two found!");
while (rs.next()) {
list.add(processRow(rs));
}
return 2;
} else {
String sql2 = "SELECT * FROM conniction WHERE oneid="+one;
s = c.createStatement();
rs = s.executeQuery(sql2);
metaData = rs.getMetaData();
columncount = metaData.getColumnCount();
if (columncount > 0) {
System.out.println("One Match found!");
while (rs.next()) {
list.add(processRow(rs));
}
return 1;
}
}
...
}
in your code, where you call the method you check the return:
...
List<Conniction> connlist = new ArrayList<Conniction>();
int foo = findPath(connlist , one, two);
if (foo == 1) {
/*One Match found!*/
// do something with your connlist-Object;
} else if (foo == 2) {
/*Match one and two found!*/
// do something with your connlist-Object;
}
Your code has two serious problems that prevent it from working:
metaData.getColumnCount() (unsurprisingly) returns the number of columns, not the number of rows. The number of columns returned is a constant depending on what is selected in the query. In this case, you select *, so metaData.getColumnCount() will be exactly the number of columns in the conniction table. You don't know how many rows are in the rowset until you process it
You probably want or logic, not and, ie
"SELECT * FROM conniction WHERE oneid="+one+" OR twoid="+two