We Keep Coding

Batch insert using jdbcTemplate.batchUpdate confusion

Does jdbcTemplate.batchUpdate execute multiple single insert statements OR 1 multi value list insert on the database server?
I know that it sends the complete query payload at once to the server but am not sure how the execution takes place.
Can someone please explain/help?

From question:
Does jdbcTemplate.batchUpdate execute multiple single insert statements OR 1 multi value list insert on the database server?
From comment:
I was curious about int[] org.springframework.jdbc.core.JdbcTemplate.batchUpdate(String sql, List<Object[]> batchArgs, int[] argTypes)
TL;DR: It executes 1 multi-valued list.
Spring Framework is open-source, so it's easy to look at the source code and see that is actually does.
batchUpdate(String sql, List<Object[]> batchArgs, final int[] argTypes)
#Override
public int[] batchUpdate(String sql, List<Object[]> batchArgs, final int[] argTypes) throws DataAccessException {
if (batchArgs.isEmpty()) {
return new int[0];
}
return batchUpdate(
sql,
new BatchPreparedStatementSetter() {
#Override
public void setValues(PreparedStatement ps, int i) throws SQLException {
Object[] values = batchArgs.get(i);
int colIndex = 0;
for (Object value : values) {
colIndex++;
if (value instanceof SqlParameterValue) {
SqlParameterValue paramValue = (SqlParameterValue) value;
StatementCreatorUtils.setParameterValue(ps, colIndex, paramValue, paramValue.getValue());
}
else {
int colType;
if (argTypes.length < colIndex) {
colType = SqlTypeValue.TYPE_UNKNOWN;
}
else {
colType = argTypes[colIndex - 1];
}
StatementCreatorUtils.setParameterValue(ps, colIndex, colType, value);
}
}
}
#Override
public int getBatchSize() {
return batchArgs.size();
}
});
}
As can be seen, it calls the following method.
batchUpdate(String sql, final BatchPreparedStatementSetter pss)
#Override
public int[] batchUpdate(String sql, final BatchPreparedStatementSetter pss) throws DataAccessException {
if (logger.isDebugEnabled()) {
logger.debug("Executing SQL batch update [" + sql + "]");
}
int[] result = execute(sql, (PreparedStatementCallback<int[]>) ps -> {
try {
int batchSize = pss.getBatchSize();
InterruptibleBatchPreparedStatementSetter ipss =
(pss instanceof InterruptibleBatchPreparedStatementSetter ?
(InterruptibleBatchPreparedStatementSetter) pss : null);
if (JdbcUtils.supportsBatchUpdates(ps.getConnection())) {
for (int i = 0; i < batchSize; i++) {
pss.setValues(ps, i);
if (ipss != null && ipss.isBatchExhausted(i)) {
break;
}
ps.addBatch();
}
return ps.executeBatch();
}
else {
List<Integer> rowsAffected = new ArrayList<>();
for (int i = 0; i < batchSize; i++) {
pss.setValues(ps, i);
if (ipss != null && ipss.isBatchExhausted(i)) {
break;
}
rowsAffected.add(ps.executeUpdate());
}
int[] rowsAffectedArray = new int[rowsAffected.size()];
for (int i = 0; i < rowsAffectedArray.length; i++) {
rowsAffectedArray[i] = rowsAffected.get(i);
}
return rowsAffectedArray;
}
}
finally {
if (pss instanceof ParameterDisposer) {
((ParameterDisposer) pss).cleanupParameters();
}
}
});
Assert.state(result != null, "No result array");
return result;
}
As can be seen, it creates a single PreparedStatement, enters a loop calling addBatch(), and finally calls executeBatch().
So, the short answer is: 1 multi-valued list.
The full answer is that it likely sends one SQL statement and a multi-valued list to the database server, however it is entirely up to the JDBC driver how it actually implements batching, mostly limited by what the communication protocol supports, so the only way to know for sure is to trace the communication with the server.

Limit the resultset and write into a file based on ram size, rows count and columns count

I am developing a system in Java which reads all the MySQL database tables, performs some operation and finally writes all data into a file (separate file for each table).
Since all the database table have different number of columns and different number of rows, there can be memory issue if the data is higher than our system can handle. Therefore, I need to write code that reads the tables values block by block and writes that data into file; and after some iterating all the data are written into that file.
I believe this approach would run in any system with any RAM size so that this system works without running into memory issues. Currently, for any table I am limiting the query result and writing that result in a one file, and iterating this process over and over until all the results are not processed. Here the value of limit size and number of iteration for all tables are dynamic, i.e. depends upon number of rows, columns and RAM size.
Following is the code written so far.
public static void main(String[] args) throws Exception {
List<String> dbList = MySqlUtils.getAllTableNames("datahouse");
for (String tableName : dbList) {
processTable(tableName);
}
}
public static void processTable(String tableName) throws Exception {
String dbname = "datahouse";
int startIndex = 0;
int limit = getMySqlQueryLimit(dbname, tableName);
int endIndex = limit;
int iteratorLength = getIteratorLength(dbname, tableName);
for (int i = 1; i <= iteratorLength; i++) {
ResultSet resultSet = getResultSet(tableName, startIndex, endIndex);
while (resultSet.next()) {
// Write into file after some operation
}
startIndex = endIndex;
endIndex += limit;
}
}
public static ResultSet getResultSet(String tableName, int startLimit, int endLimit) throws SQLException {
StringBuilder builder = new StringBuilder();
builder.append("SELECT * FROM " + tableName);
builder.append("ORDER BY id ASC limit (");
builder.append(startLimit);
builder.append(",");
builder.append(endLimit);
builder.append(")");
return MySqlUtils.getStatement().executeQuery(builder.toString());
}
public static int getMySqlQueryLimit(String dbName, String tableName) throws SQLException {
long ramSize = SystemUtils.getPhysicalMemorySize();
int columnSize = getColumnCount(dbName, tableName);
int totalRows = getRowsCount(dbName, tableName);
//TODO
return 0;
}
public static int getIteratorLength(String dbName, String tableName) {
try {
long ramSize = SystemUtils.getPhysicalMemorySize();
int columnSize = getColumnCount(dbName, tableName);
int totalRows = getRowsCount(dbName, tableName);
//TODO
return 0;
} catch (SQLException e) {
e.printStackTrace();
return 0;
}
}
In processTable() method, there is a dependency between limit and iteratorLength. Is there any algorithm (or any mathematical formula) that can calculate the values for getMySqlQueryLimit() and getIteratorLength(), so that this code can be executed in any of the system independent of RAM size i.e. without running into memory issue?

SQL Exception When Sending values from jtable to sql

I am stuck in a problem. When I try to send data from jtable to sql database through stored procedure. here what I am doing:
inserting data to jtable
String b= jLabel116.getText(),c=jTextField6.getText(),e=jTextField20.getText(),f=jTextField25.getText(),g=jTextField48.getText();
float x,y,z;
x=Float.parseFloat(jTextField25.getText());
y=Float.parseFloat(jTextField48.getText());
z=x*y;
String total1=String.valueOf(z);
DefaultTableModel df = (DefaultTableModel) jTable5.getModel();
df.addRow(new Object[]{b,c,d,f,g,total1});
int rowsCount = jTable5.getRowCount();
int Price = 0,Qty=0, total=0;
for(int i=0;i<rowsCount;i++){
Price += Integer.parseInt(jTable5.getValueAt(i,3).toString());
Qty += Integer.parseInt(jTable5.getValueAt(i,4).toString());
}
total = Price*Qty;
System.out.println(total);
jTextField26.setText(String.valueOf(total));
jTextField51.setText(String.valueOf(total));
jTextField50.setText(String.valueOf(Qty));
jTable5.setModel(df);
Sending Data to Database
try{
DefaultTableModel df = new DefaultTableModel();
df=(DefaultTableModel) jTable5.getModel();
CallableStatement cs=m.XC.prepareCall("call Prod_SALE (?,?,?,?,?,?,?,?)");
for (int i = 0; i < df.getRowCount(); i++) {
for (int j = 0; j < df.getColumnCount(); j++) {
Object o = df.getValueAt(i, j);
System.out.println("object from table is : " +o);
cs.setString(j+1, (String)o);
cs.addBatch();
}
cs.executeBatch();
cs.clearParameters();
}
}
catch(Exception ex){
ex.printStackTrace();
Error Exception:
java.sql.SQLException: Parameter-Set has missing values.
at sun.jdbc.odbc.JdbcOdbcPreparedStatement.addBatch(JdbcOdbcPreparedStatement.java:1546)
Please help me....I am unable to solve it

You call addBatch in the inner loop (variable j) after you have set one parameter. Obviously this fails since you have 8 parameters. The Javadoc of PreparedStatement.addBatch says:
Adds a set of parameters to this PreparedStatement object's batch of
commands.
You need to move the call to addBatch out of the inner loop.
(And probably the executeBatch should be moved out of the outer loop (variable i) too.
DefaultTableModel df = (DefaultTableModel) jTable5.getModel();
CallableStatement cs=m.XC.prepareCall("call Prod_SALE (?,?,?,?,?,?,?,?)");
for (int i = 0; i < df.getRowCount(); i++)
{
for (int j = 0; j < df.getColumnCount(); j++)
{
Object o = df.getValueAt(i, j);
System.out.println("object from table is : " +o);
cs.setString(j+1, (String)o);
}
cs.addBatch();
}
cs.executeBatch();

As the message says, you have not set all values in your SQL.

JDBC - query does not return data after where statement

Okay. Here's a piece of code that should return an array after searching a database
with JDBC. It's the Apache Derby database that's built into the JDK.
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.util.ArrayList;
public class readnames {
static Object [] name;
static Object [] type;
static Object [] desc;
static ArrayList<String> l = new ArrayList<String>();
static ArrayList<String> l2 = new ArrayList<String>();
static ArrayList<String> l3 = new ArrayList<String>();
public static Object[] lies(String desq) {
final String DRIVER = "org.apache.derby.jdbc.EmbeddedDriver";
final String CONNECTION = "jdbc:derby:AccountDatabase";
try {
Class.forName(DRIVER).newInstance();
} catch (InstantiationException | IllegalAccessException | ClassNotFoundException e) {
e.printStackTrace();
}
try {
Connection connection = DriverManager.getConnection(CONNECTION);
PreparedStatement statement = connection.prepareStatement("select * from WARPS WHERE CATEGORY=?");
statement.setString(1, "Server");
ResultSet resultset = statement.executeQuery();
if (resultset.next()) {
while(resultset.next()) {
l.add(resultset.getString("NAME"));
l2.add(resultset.getString("CATEGORY"));
l3.add(resultset.getString("DESCRIP"));
}
name = l.toArray();
type = l2.toArray();
desc = l3.toArray();
for (int i = 0; i < name.length; i++){
System.out.println(name[i]);
}
for (int i2= 0; i2 < type.length; i2++) {
System.out.println(type[i2]);
}
for (int i3= 0; i3 < desc.length; i3++) {
System.out.println(desc[i3]);
}
System.out.println("The method is called");
} else {System.out.println("No entries exist in this category");
}
} catch (SQLException e) {
e.printStackTrace();
}
return name;
}
}
My problem is, it doesn't return anything at all. It worked fine until I tried filtering the data using a PreparedStatement with a WHERE-clause in it. But I really don't want to get ALL the data, but specific data. The calls to println were added for debug purposes; nothing is printed when the where-clause is in the code.
EDIT: The l2.add and l3.add clauses in the while-loop are also just for debug purposes. Sry.
btw, the data that I want to retrieve definetely is in the database and gets printed when I delete the where-clause.
Does anyone know what causes this and how to fix it?

If your result just has 1 row, you are going to not get anything.
The reason:
if (resultset.next()) {
while(resultset.next()) {
When you do a .next() in the `while, the cursor has already been advanced once by if.
You should just be doing
while(resultset.next()) {
this does not execute if there are no rows.

try this, don't use if and while, try to use if and do while;
if (resultset.next()) {
do {
l.add(resultset.getString("NAME"));
l2.add(resultset.getString("CATEGORY"));
l3.add(resultset.getString("DESCRIP"));
}while(resultset.next());
name = l.toArray();
type = l2.toArray();
desc = l3.toArray();
for (int i = 0; i < name.length; i++){
System.out.println(name[i]);
}
for (int i2= 0; i2 < type.length; i2++) {
System.out.println(type[i2]);
}
for (int i3= 0; i3 < desc.length; i3++) {
System.out.println(desc[i3]);
}
System.out.println("The method is called");
} else {System.out.println("No entries exist in this category");}
and in query write.
Connection connection = DriverManager.getConnection(CONNECTION);
PreparedStatement statement = connection.prepareStatement("select * from WARPS WHERE lower(CATEGORY)=?");
statement.setString(1, "Server".toLowerCase());
don't forget to use string functions which using where clause.

I suspect that your query is returning exactly one row but you're skipping it. This line:
if (resultset.next()) {
advances the ResultSet to the first row. But then your next statement
while(resultset.next()) {
advances it to the next row, which probably does not exist.
You should have only the while loop; remove the if. You can set a boolean variable to true within the while loop so that you can tell if you had records and later print "No entries exist in this category" if there weren't any rows.

Why do you surround your while with an if statement? The condition moves the resultset one row forward so you're missing the first row. If there aren't any rows left the while statement will exit by itself. The if surrounding it is not required.

How to remove duplicate rows from JTable?

In this class, I am trying to only keep one of each value in the JTable. The Data is coming from a Database. I dont need repetative data though.
public JTable getHeute(){
DBconnect verbinden = new DBconnect();
verbinden.erstelleVerbindung();
JTable Habwesend=new JTable();
Habwesend.setBounds(10, 30, 290, 400);
Habwesend.setEnabled(false);
DefaultTableModel dm=new DefaultTableModel();
try {
ResultSet rs= verbinden.sqlStatement.executeQuery("select MA_ID from AB_Spanne where(Date() >= Start and Date() <= Ende)");
ResultSetMetaData rsmd=rs.getMetaData();
//Coding to get columns-
int cols=rsmd.getColumnCount();
String c[]=new String[cols];
for(int i=0;i<cols;i++){
c[i]=rsmd.getColumnName(i+1);
for (int k = 0; k < c.length; k++) {
dm.addColumn(c[k]);
}
}
Object row[]=new Object[cols];
while(rs.next()){
for(int i=0;i<cols;i++){
row[i]=rs.getString(i+1);
}
dm.addRow(row);
}
Habwesend.setModel(dm);
verbinden.schliesseVerbindung();
}

Change your query to:
"SELECT DISTINCT MA_ID from AB_Spanne where(Date() >= Start and Date() <= Ende)"
You can do it in java like (unrecommended and unperformant):
//UNTESTED CODE
List<Object> Objectlist = new List<Object>();
// ...
Object row[]=new Object[cols];
while(rs.next()){
for(int i=0;i<cols;i++){
row[i]=rs.getString(i+1);
}
if !(Objectlist.Contains(row)){
dm.addRow(row);
}
}
// I guess in order to the contains methode to work you may need to create own objects in which you override the equality / comparrision methods.

Instead of the List, use a Set and keep the rest of your code the same. That's your java solution, to whit:
// Set dm = new HashSet();
Object row[]=new Object[cols];
while(rs.next()){
for(int i=0;i<cols;i++){
row[i]=rs.getString(i+1);
}
dm.addRow(row);
}
Habwesend.setModel(dm);
verbinden.schliesseVerbindung();