We Keep Coding

NamedParameterJdbcTemplate not updating when using batchUpdate

I have a list of objects provided by another service which I use to update my own data. When I try to use NamedParameterJdbcTemplate.batchUpdate, all returned values are zero.
public void updateWeather(List<Weather> weatherList) {
String query = "UPDATE weather \n" +
"SET rain_probability = ROUND(:rainProbability, 4), \n" +
"wind_speed = :windSpeed \n" +
"WHERE city_id = :cityId AND date = :date;";
List<MapSqlParameterSource> batchList = new ArrayList<>();
for(Weather weather : weatherList) {
MapSqlParameterSource params = new MapSqlParameterSource();
params.addValue("rainProbability", weather.getRainProbability());
params.addValue("windSpeed", weather.getWindSpeed());
params.addValue("cityId", weather.getCityId());
params.addValue("date", weather.getDate());
batchList.add(params);
}
this.namedParameterJdbcParameter
.batchUpdate(query, batchList.toArray(new MapSqlParameterSource[] {});
}
If I run this UPDATE directly in the database, it works fine. Futhermore, if I run it one by one, that is, replacing values (instead of adding the parameter source to batchList) it also works.
For example:
for (Weather weather : weatherList) {
String query = String.format("UPDATE weather \n" +
"SET rain_probability = ROUND('%d', 4), \n" +
" wind_speed = %d \n" +
" WHERE city_id = :cityId AND date = :date;",
weather.getRainProbability(),
weather.getWindSpeed(),
weather.getCityId(),
weather.getDate()
);
this.namedParameterJdbcTemplate.update(query, Collections.emptyMap());
}
Any suggestions of what I'm doing wrong?

Is it the use of "\n" or the ";" at the end of the statement within the String? (I'm surprised you don't get a SQL Syntax exception with the ; inside the actual query string)
Also dates are always a bit tricky and if that isn't converting properly then your WHERE clause isn't going to match and is possibly why 0 rows are returned. Could you temporarily try converting dates to Strings and see if the count is correct (e.g. for Oracle: AND date = TO_DATE(:dateStr, 'DD/MM/YYYY') )

Improve speed of insertion of big amount of data

I have a rest service that take xml with 400_000 records, each record contain the following fields: code,type,price.
In DB (MySql )I have table named PriceData with 2_000_000 rows. The purpose of this rest is: select all PriceDatas from DB according to code,type from XML, replace price of each PriceData with price from XML, if there is no PriceData with this code,type create new with provided price.
Now it work as : select one PriceData from DB accroding to first record from XML, set new price or create new PriceData, save PriceData and these steps repeats 400_000 times.(It takes about 5 minutes)
I want to speed up this process.
First try:
Select 1000 elements step by step from PriceData, and when all elements will be selected update them:
Code:
private void updateAll(final List<XmlData> prices/*data from xml*/) {
int end= 1000;
int begin= 0;
final List<PriceData> models = new ArrayList<>();
while(end != prices.size() || begin !=end){
models.addAll(dao.findByPrices(prices.subList(begin,end)));
begin = end;
end +=1000;
}
final Map<String,XmlData> xmlData= prices.stream()
.collect(Collectors.toMap(this::keyForPriceDate,e->e));
final Map<String,PriceData> modelMap = models.stream()
.collect(Collectors.toMap(this::keyForRowModel,e->e));
final List<PriceData> modelsToSave = new ArrayList<>();
for(final String key : xmlData.keySet()){
final XmlData price = xmlData.get(key);
PriceData model = modelMap.get(key);
if(model == null){
model = onEmptyPriceData(price);
}
model.setPrice(price.getPrice());
modelsToSave.add(model);
}
modelService.saveAll(modelsToSave);
}
I convert two lists to maps to know does PriceData exist (keys for xmlData and modelMap created as (code+type))
findByPrices method create query in following format
select * from PriceData where (code =123 and type ='qwe') or (...)//and this `Or` repeats 1000 times
Now it takes 2 minutes.
Second try:
Select all PriceData from db (2 millions)
and use the algorithm above
It takes 3 minutes. First try is better but in future my rest can take 500_000 and I want to know which try will be better in this scenario or maybe there is the better way to do this task.
My select method
public List<PriceData> findBy(final List<XmlData> selectData) {
final StringBuilder query = new StringBuilder("SELECT * from PriceData ");
query.append("WHERE \n");
final Iterator<PriceRowSelectData> selectDataIterator = selectData.iterator();
while(selectDataIterator.hasNext()){
final PriceRowSelectData data = selectDataIterator.next();
query.append("( \n")
.append("productCode = "+ data.getProductId()+" \n")
.append(" AND type = "+ data.getPriceind()+" \n")
.append(" ) \n");
if(selectDataIterator.hasNext()){
query.append("OR \n");
}
}
final SearchResult<PriceRowModel> searchRes = search(query.toString());
/*
Here i use custom mapper that map list of result to my object
*/
return searchRes.getResult();
}

You should use the MySQL INSERT ... ON DUPLICATE KEY UPDATE statement, combined with JDBC batch processing. This of course assumes that code,type is the primary key, or at least a unique index.
private void updateAll(final List<XmlData> prices) throws SQLException {
String sql = "INSERT INTO PriceData (code, type, price)" +
" VALUES (?,?,?)" +
" ON DUPLICATE KEY" +
" UPDATE price = ?";
try (PreparedStatement stmt = this.conn.prepareStatement(sql)) {
int batchSize = 0;
for (XmlData price : prices) {
if (batchSize == 1000) { // flush batch every 1000
stmt.executeBatch();
batchSize = 0;
}
stmt.setInt (1, price.getCode());
stmt.setString (2, price.getType());
stmt.setBigDecimal(3, price.getPrice());
stmt.setBigDecimal(4, price.getPrice());
stmt.addBatch();
batchSize++;
}
if (batchSize != 0)
stmt.executeBatch();
}
}
You can twiddle the batch size, but not flushing will use a lot of memory. I think 1000 statements per batch is good, but I have no numbers backing that.

confusion over using values in Java within SQL statements

I need to retrieve a list of vehicle registrations from an SQL database and put them into a dropdown list in the java GUI. Once a Vehicle Registration is selected (as part of a login sequence) from the list I want to use this in the "where" statement of subsequent SQL queries made to the database to get things like inventory statuses etc that are only for that vehicle.
I have done the first part i.e. retrieved the list from the database and displayed the veh regs in the dropdown for the user to select, I have managed to display this in an optional panel as well as display it in another class in a Jlabel box but I cannot seem to figure out how to use this selected veh reg in a seperate query class i.e as part of a select/where statement?
I have been trying to find a solution to this but am getting confused on whether to use an array list and put the registration number into this and then how to make this available to other classes so I can retrieve the value to use in the SQL statement. I am quite lost now so any advice in the right direction will be very helpful.
I am very new to Java and programming altogether so if you feel the need to make sarcastic (you are such a noob) type comments then don't bother posting!
This is the code that grabs the registrations from the database and adds them to the combo box in one GUI:
private void populateRegistration() {
try {
// create a connection to database
pst = conn.prepareStatement("SELECT VehicleRegistrationNumber FROM vehicle;");
// create a query to get vehicle regs
rs = pst.executeQuery();
// add vehicle regs to combobox
while (rs.next()) {
jComboBox1.addItem(rs.getString("VehicleRegistrationNumber"));
}
} catch (SQLException ex) {
Logger.getLogger(Login_GUI.class.getName()).log(Level.SEVERE, null, ex);
}
This is the code In another GUI Class that shows this vehicle registration in a dialogue box and then shows it in a text field in the GUI:
public ArrayList<String> getTableContent()
{
DatabaseConnection db = new DatabaseConnection();
try {
//Sql Return Statement
String newQuery= "SELECT P.PatientFirstName, P.PatientLastName, P.PatientHouseNumber, P.PatientStreetName, P.PatientPostcode, P.PatientBreathing, P.ProblemInformation, C.AMPDSCategory, \n" +
"I.NumberHurt,T.TaskClosed, H.HospitalSpaceAvailable, H.HospitalName, H.HospitalPostcode, V.VehicleRegistrationNumber, EM.DateTimeReported\n" +
"FROM Patient AS P\n" +
"JOIN Category AS C\n" +
"ON C.Category_ID = P.CategoryID\n" +
"--AND P.Patient_ID = 2\n" +
"JOIN Incident AS I\n" +
"ON I.Incident_ID = P.IncidentID\n" +
"JOIN TASK AS T\n" +
"ON T.IncidentID = I.Incident_ID\n" +
"JOIN Hospital AS H\n" +
"ON H.Hospital_ID = T.HospitalID\n" +
"JOIN Vehicle AS V\n" +
"ON T.Task_ID = V.TaskID\n" +
"JOIN ECCPersonnel AS EC\n" +
"ON I.ECCPersonnelID = EC.ECCPersonnel_ID\n" +
"JOIN EmergencyCall AS EM\n" +
"ON EC.CallID = EM.Call_ID\n" +
"WHERE T.Task_ID=1" +
"--WHERE V.VehicleRegistrationNumber = '?'";
I know the SQL is ugly but it works so I will refine it later if I get time. I believe I need to define the ? but have no idea how to reference the ? to the vehReg value!!

First you should allways close what you have opened. So in populateRegistration() do not forget rs.close() and pst.close().
Next to reference the ? to the vehReg, you create a PreparedStatement and call its setString method (supposing vehReg is a String).
pst = conn.prepareStatement(newQuery);
pst.setString(1, vehReg);
rs = pst.executeQuery();
(try, catch, declarations, and actual read ommitted for brevity)
Remember that columns are numbered starting by 1.

ResultSet to ArrayList<String[]>

I want to create an array for each row.
while (result.next()) {
String[] result1 = {
"Dog ID: " + result.getLong("dogs.id"),
", Dog name: " + result.getString("dogs.first_name"),
", Owner name: "
+ result.getString("owners.first_name"),
", Owner phone: " + result.getString("owners.phone") };
resultList.add(result1);
My code write every one row in one array.
Can i get numbers of columns and put a limit?
while (resultset.next()) {
int i = 1;
while(i <= numberOfColumns) {
It's because i can't send entire table as a result from server to client.

You can query by column number result.getLong(columnIndex) but it doesn't make sense in your case withing a loop because you have columns of different types (unless complicating the code).
If you want to optimize the traffic from server to client the way to go is querying for just the columns you need.

If you want to limit the rows returned, it might be better to put the limiting criteria into the SQL query and only return the rows you want to include.

In order to get number of columns in your ResultSet you can use the following piece of code :
Statement stat = conn.createStatement();
ResultSet rs = stat.executeQuery(myQuery);
ResultSetMetaData metaData = rs.getMetaData();
int numOfColumns = metaData .getColumnCount();

Error: Before start of result set in Java

I know this would be a foolish question to ask but still i need to do this.
This is a basic program in java application where I want to use 3 queries simultaneously to print the table.
(I'm not using any Primary key in this case so please help me to resolve this without making my attributes as primary keys - I know this is not a good practice but for now i need to complete it.)
my code:
Connection con = null;
Statement stat1 = null, stat2 = null, stat3 = null;
ResultSet rs1, rs2, rs3;
stat1 = con.createStatement();
stat2 = con.createStatement();
stat3 = con.createStatement();
String str = "\nProduct\tC.P\tS.P.\tStock\tExpenditure\tSales";
info.setText(str);
String s1 = "SELECT type, cp, sp, stock FROM ts_items GROUP BY type ORDER BY type";
String s2 = "SELECT expenditure FROM ts_expenditure GROUP BY type ORDER BY type";
String s3 = "SELECT sales FROM ts_sales GROUP BY type ORDER BY type";
rs1 = stat1.executeQuery(s1);
rs2 = stat2.executeQuery(s2);
rs3 = stat3.executeQuery(s3);
String type;
int cp, sp, stock, expenditure, sales;
while( rs1.next() || rs2.next() || rs3.next() )
{
type = rs1.getString("type");
cp = rs1.getInt("cp");
sp = rs1.getInt("sp");
stock = rs1.getInt("stock");
expenditure = rs2.getInt("expenditure");
sales = rs3.getInt("sales");
info.append("\n" + type + "\t" + cp + "\t" + sp + "\t" + stock + "\t" + expenditure + "\t" + sales);
}
Output:
Runtime Exception: Before start of result set

This is the problem:
while( rs1.next() || rs2.next() || rs3.next() )
If rs1.next() returns true, rs2.next() and rs3.next() won't be called due to short-circuiting. So rs2 and rs3 will both be before the first row. And if rs1.next() returns false, then you couldn't read from that anyway...
I suspect you actually want:
while (rs1.next() && rs2.next() && rs3.next())
After all, you only want to keep going while all three result sets have more information, right?
It's not clear why you're not doing an appropriate join, to be honest. That would make a lot more sense to me... Then you wouldn't be trying to use multiple result sets on a single connection, and you wouldn't be relying on there being the exact same type values in all the different tables.

You do an OR so imagine only one ResultSet has a result.
What you end up with is trying to read from empty result sets.

Suppose rs1 has one result and rs3 has 3 results. Now as per your code it will fail for rs1.getString("type"); during second iteration.
Better to loop over each resultSet separately.

This is going to go badly wrong, in the event that there is a type value that's missing from one of your three tables. Your code just assumes you'll get all of the types from all of the tables. It may be the case for your current data set, but it means that your code is not at all robust.
I would seriously recommend having just one SQL statement, that has each of your three selects as subselects, then joins them all together. Your java can just iterate over the result from this one SQL statement.