We Keep Coding

JDBC for array (or list) values determined at run time

I am looking at this post How to insert array values in Mysql database using java with keen interest. But just to get started I find that MySQL Workbench rejects the indicated sql statements for table creation and update(insert).
I ended up loading a single line dataframe from R into my MySQL demo database to create table itemtable. This is not ideal since the item values are intended to be integers, but loading like this from R creates the fields as double.
itemtable fields are "time", "item1", "item2", "duration"
initial line values are 0.0, 1, 1, 0.0
I created an Item class:
public class Item {
String name;
double value;
public Item(String name, double value) {
this.name = name;
this.value = value;
}
In my real application Items are far more complex and value is calculated at different times. So I use a LinkedList to hold these. The quantity of items is variable from model to model as an input of other data tables just as I did from R.
The full test code - That works!!! as follows:
public class Item {
String name;
double value;
public Item(String name, double value) {
this.name = name;
this.value = value;
}
}
import java.sql.*;
import java.util.LinkedList;
public class ItemTest {
public static void main(String[] args) throws SQLException {
Connection conn = null;
PreparedStatement pstmt = null;
Statement stmt = null;
ResultSet rs = null;
String dbUrl = "jdbc:mysql://localhost:3306/demo?useSSL=false";
String user = "student";
String pass = "student";
LinkedList<Item> itemList = new LinkedList<>();
itemList.add(new Item("item1", 0.0));
itemList.add(new Item("item2", 1.0));
double timeNow = 30.0;
double duration = 0.0;
String qMarks = "";
String names = "";
for(int i = 0; i < itemList.size(); i++) {
qMarks = qMarks + "?,";
names = names + itemList.get(i).name + ",";
}
System.out.println(qMarks);
System.out.println(names);
String pquery = "insert into itemtable(time," + names +
"duration) values(?," + qMarks + "?)";
System.out.println(pquery);
try {
conn = DriverManager.getConnection("jdbc:mysql://localhost:3306/demo", "student" , "student");
pstmt = conn.prepareStatement(pquery);
pstmt.setDouble(1, timeNow);
for(int i = 0; i<itemList.size(); i++) {
pstmt.setDouble(i+2, itemList.get(i).value);
}
pstmt.setDouble(itemList.size()+2, duration);
int count = pstmt.executeUpdate();
System.out.println(count + " inserted");
stmt = conn.createStatement();
rs = stmt.executeQuery("select * from itemtable");
while (rs.next()) {
System.out.println(rs.getDouble("time") + ", " + rs.getDouble("item1"));
}
}
catch (SQLException exc){
}
finally {
if (rs != null) rs.close();
if (stmt != null) stmt.close();
if (conn != null) conn.close();
}
}
}
Full verification of the table insert is confirmed in MySQL Workbench.
It would still be nice if I could create the itemtable from within java code.

As the linked SO post tells you: SQL's 'ARRAY' concept (which is mirrored by JDBC's .setArray and .getArray and java.sql.Array) just doesn't work in mysql. At all.
I suggest you use a better DB engine, such as postgres. Otherwise, well, you can't have arrays; what you can try to do is make a table to represent it. The usual many-to-one stuff in DBs: Instead of 'an array', you have a second table containing 1 row for each 'slot' in all arrays across the entire dataset: Each row in this 'array table' contains the data as well as a reference.
So, for example:
-- This works in PSQL:
CREATE TABLE example (id integer, whatever text, names text[], PRIMARY KEY(id));
-- but the above does not work in MySQL, so there do e.g.:
CREATE TABLE example (id integer, whatever text, PRIMARY KEY(id));
CREATE TABLE example_names (id integer, example_id integer, name text, PRIMARY KEY(id));
CREATE INDEX example_names_lnk ON example_names(example_id);

It turns out that what I was trying to do with the list item with quantity defined at runtime was indeed different than trying to enter an array in a MySQL table, which I now understand cannot be done. I do appreciate the earlier discussion, which sparked my interest in solution I needed.
The create table sql is generated in a straight forward way, the key to handling the unknown content at runtime was to build the query string using a loop for the item columns. A single loop built internal string items for the create statement as well as the prepared statements.
So the sql for creating the table in java code is straight forward. Item values can now be integers as intended.
String qMarks = "";
String names = "";
String items = "";
for(int i = 0; i < itemList.size(); i++) {
qMarks = qMarks + "?,";
names = names + itemList.get(i).name + ",";
items = items + itemList.get(i).name +" INTEGER,";
}
String createsql = "CREATE TABLE ITEMTABLE (" +
"time DOUBLE, " + items +
"duration DOUBLE)";

batch processing with dynamic insert query using spring "jdbcTemplate.batchUpdate"

Well I have been updating a legacy code since last few days.
Explanation:
There is a table CUSTOMER with columns blah1, blah2, blah3, blah4, blah.....
As per our architecture I need to create an insert statement dynamically which can insert data into any table with any number of column.
Currently we have following code.
public void save(Table table, Connection conn) throws Exception {
PreparedStatement pstmt = null;
try {
List<Row> rows = table.getRows();
String sql = "";
if(!rows.isEmpty() && rows != null)
{
for(Row row: rows) //READ EACH ROW
{
String columnName = ""; String columnValue = "";
List<String> params = new ArrayList<String>();
List<Column> columns = row.getColumns();
if(!columns.isEmpty() && columns != null)
{
for(Column column: columns) //GET EACH COLUMN DATA
{
columnName += ", "+column.getName();
columnValue += ", ?";
String value = column.getValue();
params.add(value); //ADD VALUE TO PARAMS
}
//INSERT QUERY
sql = "INSERT INTO "+table.getTableName()+" ("+columnName+") VALUES ("+columnValue+")";
if(pstmt == null) pstmt = conn.prepareStatement(sql);
//POPULATE PREPARED STATEMENT
for (int i =0; i<params.size(); i++) {
pstmt.setString(i+1, (String)params.get(i));
}
pstmt.addBatch();
}
}
pstmt.executeBatch();//BATCH COMMIT
conn.commit();
}
} catch (Exception e) {
if (conn != null) {
conn.rollback();
}
throw e;
}
}
Now instead of using the typical pstmt.executeBatch(). I want to use spring batch update as follows:
public void save(Table table, Connection conn) throws Exception{
String sql = createSaveQuery(table);//CREATES the INSERT Query
getJdbcTemplate().batchUpdate(sql.toString(), new BatchPreparedStatementSetter() {
#Override
public void setValues(PreparedStatement ps, int j) throws SQLException {
//PROBLEM AREA: How to map this for each insert statement?
for(int i =0; i < params.size(); i++){
ps.setString(i+1, (String)params.get(i));
}
}
#Override
public int getBatchSize() {
return 0;
}
});
}
But I cannot figure out how to set the params for the each insert Query. As in the we can set the pstmt.setString(i, params.get(i)); for each row. How to achieve the same in 'new BatchPreparedStatementSetter()'.
any suggestions will be appreciated. If you need to further imporve the explanation. Please let me know.

I guess it's against Spring Batch nature to create dynamic SQL queries. That's because of PreparedStatement nature (read more here: What does it mean when I say Prepared statement is pre-compiled?). Long story short - PreparedStatements are compiled on DB side and they are really fast. If you modify SQL query, it can't be reused. It is hard to give advise without knowing of actual DB schema, but in general you should have one POJO for CUSTOMER table (I hope you don't have BLOBS) and you should read-write all fields. It will be faster than "optimized" dynamic query.

How to check if two values exists?

I'm trying to implement a program (in Java) that check if two values exists in a database (PostgreSQL). This is my problem:
I have an external document that Java reads this values:
# letter_id, child_id, toy_id
18,1,2
18,1,3
18,2,1
The first row put into a database this values for this reason: letter doesn't exists in my database and then, insert letter_id and child_id into LETTER (it's a table) and insert WISHED TOY (it's a table).
The second row is inserted into the database for this reason: letter exists and child_id is the same, but toy_id is different. Then, update values to LETTER and insert letter_id and toy_id into WISHED TOY.
The third row will fail, for this reasons: letter exists and child_id is different and this is not possible, only one child can write 1 letter. It is not possible to add this case because two children cannot write the same letter.
I have this functions that I want use:
private Integer getLetterId(List<String> row)
{
String integer;
String[] rowArray = (String[]) row.toArray(new String[0]);
integer = rowArray[0]; //Letter_Id
return (null != integer) ? Integer.valueOf(integer) : null;
}
private Integer getChildId(List<String> row)
{
String integer;
String[] rowArray = (String[]) row.toArray(new String[0]);
integer = rowArray[1]; //Child_Id
return (null != integer) ? Integer.valueOf(integer) : null;
}
My idea is this:
If letter exists {
If letter exists for this child {
UPDATE LETTER
INSERT WISHED
}
ELSE {
//this case will be return an error because letter not coincide with child
}
Else {
Insert Letter
Insert Wished
}
}
Wished is an another table that puts letter and toy_id in case that exists or not letter.
I understand how to implement the first 'if': (if (rs.next()))
But I think I have problems with this 'if': if (childId==row.getChildId(1))
So, how can I do it? Any help please?
EDIT: Sorry, I don't know what I'm doing bad:
if (conn != null) {
PreparedStatement selectPS = null;
ResultSet rs = null;
Integer childId;
Integer letterId;
try {
String selectSql = "SELECT child_id FROM letter WHERE letter_id = ?";
selectPS = conn.prepareStatement(selectSql);
for (List<String> row : fileContents) {
// TODO Update or insert record from PLAYER for every row in file
selectPS.clearParameters();
setSelectPreparedStatement (selectPS,row);
rs = selectPS.executeQuery();
childId = rs.getInt("child_id");
letterId = rs.getInt("letter_id");
if (rs.next())
{
if (childId == getChildId (row))
{
System.out.println("Updating");
}
if (childId != getChildId (row) && letterId == getLetterId(row))
{
System.out.println("This letter are used by other child");
}
}
else
{
System.out.println("Inserting values to letter and wished");
}
}
}
}
I obtain this error:
ERROR: Executing
SQL ResultSet not positioned properly, perhaps you need to call next.

After rs = selectPS.executeQuery(); you have to check if there is any row at all, it means rs.next() should be run. I mean you have to move
childId = rs.getInt("child_id");
letterId = rs.getInt("letter_id");
after the if(rs.next()){

Query in a loop which is executed only once Java

I am currently working on a Java project (on NetBeans) and I am struggling with a problem.
In fact, I have a jTable which contains several elements, which element has a jCheckBox in the second column and I would like to make a query to add the selected element (selected by the jCheckBox of course) in a table.
I can get the data that I want to add, but my query works only once. I have already check my loop but I don't where the problem comes from.
I let you see the code :
try {
// Getting id of the selected value in the jComboBox
String idParcours = oParcoursDAO.findIdParcours(jComboBoxParcours.getSelectedItem().toString());
int id = Integer.parseInt(idParcours);
// for each value in the jTable
for(int i=0; i <jTable2.getRowCount(); i++){
boolean isChecked = (Boolean)jTable2.getValueAt(i, 1);
String nomPoi = (String)jTable2.getValueAt(i, 0);
// if the value is selected
if(isChecked){
String IDPoi = oParcoursDAO.findIdPoi(nomPoi);
int idpoi = Integer.parseInt(IDPoi);
System.out.println("idpoi "+idpoi); // It works I saw as idpoi as I have choose
System.out.println("id "+id) // It works too
oParcoursDAO.addPoi(idpoi,id); // it works only once
}
}
}catch (SQLException ex) {
Logger.getLogger(ModificationParcoursJInternalFrame.class.getName()).log(Level.SEVERE, null, ex);
}
Thank you in advance for your help.
This is my statement
public void addPoi(int idPoi,int idParcours) throws SQLException{
String query = "INSERT INTO TB_POI_PARCOURS (id_poi,id_parcours) VALUES (?,?) ";
PreparedStatement preparedStatement = conn.prepareStatement(query);
preparedStatement.setInt(1,idPoi);
preparedStatement.setInt(2,idParcours);
preparedStatement.executeUpdate();
preparedStatement.close();
}

Why are you running one query per line? You can execute all of them in a single SQL using batch queries. It will require you to change the code but it will make it more efficient:
public void addPoi(Map<integer,Integer> poiMap) throws SQLException{
String query = "INSERT INTO TB_POI_PARCOURS (id_poi,id_parcours) VALUES (?,?) ";
PreparedStatement preparedStatement = conn.prepareStatement(query);
for(Integer idPoi:poiMap.keySet()) {
preparedStatement.setInt(1,idPoi);
preparedStatement.setInt(2,poiMap.get(idPoi));
preparedStatement.addBatch();
}
preparedStatement.executeBatch();
preparedStatement.close();
}
Of course the original method has to be changed accordingly.

Faster batch loading of nested data

I am populating two tables, which have a 1-many relationship.
So I insert a line in outer, get the (autoincrement primary key) id for that line, and then insert 100 lines into inner (all with a foreign key pointing to outer.id).
Then I repeat, 50 times. For every entry in outer I have to insert, read id, and then insert into inner.
This is slow. Most of the time is spent in loading the 100 lines into inner. I suspect it would be much faster if I could insert all 50*100 lines into inner in one batch operation. But I cannot see how to do that - how can I make the foreign keys work?
How do other people make this efficient?
I am using Java / Spring. The 100 lines are inserted with a JdbcTemplate.batchUpdate().
public final void insert(final JdbcTemplate db,
final Iterable<DataBlock> data) {
String insertSql = getInsertSql();
String idQuery = getIdQuery();
ItemRowMapper.IdRowMapper mapper = new ItemRowMapper.IdRowMapper();
for (DataBlock block: data) {
Object[] outer = block.getOuter();
LOG.trace("Loading outer");
db.update(insertSql, outer);
LOG.trace("Getting index");
// currently retrieve index based on natural key, but could use last index
int id = db.query(idQuery, mapper, uniqueData(outer)).get(0);
LOG.trace("Getting inner");
List<Object[]> inner = block.getInner(id);
// most time spent here
LOG.trace(format("Loading inner (%d)", inner.size()));
innerTable.insert(db, inner);
}
}
And pseudo-SQL:
create table outer (
integer id primary key autoincrement,
...
);
create table inner (
integer outer references outer(id),
...
);
Update - The following appears to work with Spring 3.1.1 and Postgres 9.2-1003.jdbc4.
/**
* An alternative implementation that should be faster, since it inserts
* in just two batches (one for inner and one fo router).
*
* #param db A connection to the database.
* #param data The data to insert.
*/
public final void insertBatchier(final JdbcTemplate db,
final AllDataBlocks data) {
final List<Object[]> outers = data.getOuter();
List<Integer> ids = db.execute(
new PreparedStatementCreator() {
#Override
public PreparedStatement createPreparedStatement(
final Connection con) throws SQLException {
return con.prepareStatement(getInsertSql(),
Statement.RETURN_GENERATED_KEYS);
}
},
new PreparedStatementCallback<List<Integer>>() {
#Override
public List<Integer> doInPreparedStatement(final PreparedStatement ps)
throws SQLException {
for (Object[] outer: outers) {
for (int i = 0; i < outer.length; ++i) {
setParameterValue(ps, i + 1,
SqlTypeValue.TYPE_UNKNOWN, outer[i]);
}
ps.addBatch();
}
ps.executeBatch();
RowMapperResultSetExtractor<Integer> ids =
new RowMapperResultSetExtractor<Integer>(
new ItemRowMapper.IdRowMapper());
try (ResultSet keys = ps.getGeneratedKeys()) {
return ids.extractData(keys);
}
}
});
innerTable.insert(db, data.getInner(ids));
}

I'm not as familiar with JdbcTemplate, but assuming it is similar to JDBC I would do it with something similar (I would probably break this into multiple methods) to the following code:
private static final int BATCH_SIZE = 50;
public void addBatch(Connection connection, List<Outer> outers) {
PreparedStatement outerInsertStatement = connection.prepareStatement("...", Statement.RETURN_GENERATED_KEYS);
PreparedStatement innerInsertStatement = connection.prepareStatement("...", Statement.RETURN_GENERATED_KEYS);
List<Integer> outerIds = new ArrayList<Integer>();
for(Outer outer : outers) {
outerInsertStatement.setParameter(...);
...
outerInsertStatement.setParameter(...);
outerInsertStatement.addBatch();
}
outerInsertStatement.executeBatch();
//Note, this line requires JDBC3
ResultSet primaryKeys = outerInsertStatement.getGeneratedKeys();
while(!primaryKeys.isAfterLast()) {
outerIds.add(primaryKeys.getInt(0));
}
for(int i = 0; i < outers.size(); i++) {
Outer outer = outers.get(i);
Integer outerId = outerIds.get(i);
for(Inner inner : outer.getInners()) {
//One of these setParameter calls would use outerId
innerInsertStatement.setParameter(...);
...
innerInsertStatement.setParameter(...);
innerInsertStatement.addBatch();
if( (i+1) % BATCH_SIZE == 0) {
innerInsertStatement.executeBatch();
}
}
innerInsertStatement.executeBatch();
}
}