We Keep Coding

NumberFormatException because NULL not recognised on prepared statement Java

I am using a prepared statement to insert to my database. Some of the values im inserting are NULL as the matches are yet to be played, and so the score is NULL, NULL.
void insertFixtures(List<String[]> fixtures) throws SQLException {
String query = "REPLACE INTO games (team1_id, team2_id, score1, score2, created_at, winner) VALUES (? ,?, ?, ?, ?, ?)";
Connection con = DBConnector.connect();
PreparedStatement stmt = con.prepareStatement(query);
for (String[] s : fixtures) {
int winner;
stmt.setString(1, s[0]);
stmt.setString(2, s[1]);
String nullTest1 = s[2];
if (nullTest1 != null) {
stmt.setString(3, s[2]);
stmt.setString(4, s[3]);
stmt.setString(5, s[4]);
int score1 = Integer.parseInt(s[2]);
int score2 = Integer.parseInt(s[3]);
System.out.println(score1);
System.out.println(score2);
if (score1 > score2) {
winner = 1;
} else if (score2 > score1) {
winner = 2;
} else {
winner = 0;
}
String gameWinner = Integer.toString(winner);
stmt.setString(6, gameWinner);
} else {
System.out.println("empty");
stmt.setString(3, null);
stmt.setString(4, null);
stmt.setString(5, s[4]);
stmt.setString(6, null);
}
}
stmt.execute();
stmt.close();
con.close();
}
InsertFixtures takes the list string array and inserts these into my database using a for loop.
The problem i have is with:
if(nullTest1 != null ){
When i run this code in debug mode and set nullTest1 to equal null it skips over this and goes into the else statement. However, when i run it real time it goes into this if statement and has an issue with parseInt on a null value.
This is an example of the strings im trying to insert into my database:
Fixture 45 42 1 0 1554642300
Fixture 49 48 null null 0
Any help is useful.
Thanks

You should check null before parsing a String into Integer:
int score2 = s[3] != null ? Integer.parseInt(s[3]) : 0;
You need to decide what should be the value if s[3] is null. I have put 0 just for example purpose.

Insert statement contains more than 1000 elements

I'm trying to add 100,000 names to a database using JDBC. I'm aware that MS SQL doesnt allow mass inserts of more than 1000, elmts, so I accomplish this by breaking the major set down into sets containing 1000 or fewer. The following is my code:
StringJoiner joiner = new StringJoiner("\'), (\'", "INSERT INTO Names (Name) VALUES (\'", "\');");
ExecutorService threadpool = Executors.newFixedThreadPool(100);
while(names.size() > 0) {
int count = Math.min(1000, names.size());
HashSet<String> set = new HashSet();
Iterator iterator = names.iterator();
for(int i = 0; i < count; i++) {
set.add((String) iterator.next());
}
names.removeAll(set);
for(String s: set)
{
joiner.add(s);
}
// System.out.println(joiner.toString());
threadpool.submit(() -> {
PreparedStatement query = null;
try {
query = connect.prepareStatement(joiner.toString());
} catch (SQLException e) {
e.printStackTrace();
}
try {
query.executeUpdate();
} catch (SQLException e) {
e.printStackTrace();
}
});
}
}
This throws the following exception: com.microsoft.sqlserver.jdbc.SQLServerException: The number of row value expressions in the INSERT statement exceeds the maximum allowed number of 1000 row values.
Why are there more than 1000 lines being added?

Why not use batch inserts? Define batch size 1000 and insert queries in the batches of 1000 queries at a time.
String sql = "insert into names(name) values (?)";
Connection connection = new getConnection();
PreparedStatement ps = connection.prepareStatement(sql);
final int batchSize = 1000;
int count = 0;
for (Employee employee: employees) {
ps.setString(1, employee.getName());
ps.addBatch();
if(++count % batchSize == 0) {
ps.executeBatch();
}
}
ps.executeBatch(); // insert remaining records
ps.close();
connection.close();

Java Heap space with jpa native Query

I am having an OOME problem when trying to run a native SQL query through the jpa / hibernate EM.
The treatment serves to make millions of insertion per pack of 50.
Here is code of my algorithm:
private void createNewJetonsForIndividus(boolean isGlobal, List<String> entreprises, List<String> services,
String user, Timestamp dateDB) {
LocalDateTime timer = LocalDateTime.now();
List<Object[]> MinMaxId = getMinMaxIdDroitsIndividusActifsForCreation(
isGlobal, entreprises, services);
if (null != MinMaxId.get(0)[0]) {
int idStart = ((BigInteger) MinMaxId.get(0)[0]).intValue();
int idEnd = idStart + PAS;
int idMax = ((BigInteger) MinMaxId.get(0)[1]).intValue();
int nbRowsTotal = 0;
Logger.debug("Droits Individus : ID Min {} - ID Max {}", idStart, idMax);
do {
int finalIdStart = idStart;
int finalIdEnd = idEnd;
callTransaction(() -> create(false, true,isGlobal, entreprises, services, finalIdStart,
finalIdEnd, user, dateDB));
idStart = idEnd + 1;
idEnd = idEnd + PAS;
}
while (idMax > idEnd);
}
}
the method is used to calculate the id min and max of the records that interest my treatment. Subsequently, I use the create method whose code is below :
int nbRowsFind;
List<Object[]> listeDroitsIndividusActifsForCreation = getDroitsIndividusActifsForCreation(
isGlobal, entreprises, services, idStart, idEnd);
if (ValidationUtils.isNotEmpty(listeDroitsIndividusActifsForCreation)) {
nbRowsFind = listeDroitsIndividusActifsForCreation.size();
StringBuilder sbJeton = new StringBuilder();
sbJeton.append("INSERT INTO sigs_nv_jeton VALUES ");
StringBuilder sbDroitHasJeton = new StringBuilder();
if (isCreateForIndiv) {
sbDroitHasJeton.append("INSERT INTO sigs_droits_individu_has_nv_jeton VALUES ");
}
listeDroitsIndividusActifsForCreation.stream().forEach(object -> {
sbJeton.append("(");
sbDroitHasJeton.append("(");
BigInteger idDroit = (BigInteger) object[0];
String jetonGenerated = IdJetonGenerator.codeGenerator(idDroit.toString(), DateUtils.now());
sbJeton.append("'").append(jetonGenerated).append("', ");
appendDate(sbJeton, object[1]);
appendDate(sbJeton, object[2]);
sbJeton.append(0).append(", ");
sbJeton.append(0).append(", ");
sbJeton.append("'").append(dateDB).append("', ");
sbJeton.append("'").append(user).append("'");
sbDroitHasJeton.append(idDroit).append(",'").append(jetonGenerated).append("'");
sbJeton.append("),");
sbDroitHasJeton.append("),");
});
String requestJeton = sbJeton.toString();
sbJeton.delete(33, sbJeton.length());
requestJeton = requestJeton.substring(0, requestJeton.length() - 1);
jpaApi.em().createNativeQuery(requestJeton).executeUpdate();
String requestDroitHasJeton = sbDroitHasJeton.toString();
sbDroitHasJeton.delete(54, sbDroitHasJeton.length());
requestDroitHasJeton = requestDroitHasJeton.substring(0, requestDroitHasJeton.length() - 1);
**jpaApi.em().createNativeQuery(requestDroitHasJeton).executeUpdate();**
**jpaApi.em().flush();
jpaApi.em().clear();**
}
When I analyze the Heap Dump, I notice that despite the flush and clear, queries are still referenced in the SessionFactory, is this normal?
enter image description here

flush is executing changes made in the unit of work and clear removes entities from the persistence context.
Both have nothing to do with native SQL queries. I assume that the native queries are cached anywhere else (Hibernate, JDBC...)
I would suggest that you use a prepared statement instead of your dynamic insert statement.

This is my new implementation :
Statement statement = null;
ResultSet resultSet = null;
int nbRows = 0;
try {
statement = connection.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE,
ResultSet.CONCUR_READ_ONLY);
try {
resultSet = statement.executeQuery(queryGetDroitsEntreprisesActifsForCreation(isGlobal, entreprises, services));
resultSet.beforeFirst();
while (resultSet.next()) {
nbRows += 1;
queryInsertJetonsAndLinkDroitsJeton(isCreateForEntreprise, isCreateForIndiv, connection, resultSet,
user, dateDB);
}
System.out.println(nbRows);
} finally {
if (resultSet != null) {
resultSet.close();
}
}
} finally {
if (statement != null) {
statement.close();
}
}
The method "queryGetDroitsEntreprisesActifsForCreation" returns a SQL query in String.
The method "queryInsertJetonsAndLinkDroitsJeton" use the famous PreparedStatement :
PreparedStatement psJeton = null;
PreparedStatement psDroitJeton = null;
try {
String sbJeton = "INSERT INTO TABLE1 VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) ";
psJeton = connection.prepareStatement(sbJeton);
String jetonGenerated;
if (isCreateForEntreprise) {
jetonGenerated = IdJetonGenerator.codeGenerator(String.valueOf(resultSet.getInt("id_e")), DateUtils.now());
} else {
jetonGenerated = IdJetonGenerator.codeGenerator(String.valueOf(resultSet.getInt("id_i")), DateUtils.now());
}
psJeton.setString(1, jetonGenerated);
psJeton.setString(2, ContextType.GD.name());
psJeton.setString(3, JetonType.ANONYME.name());
psJeton.setInt(4, 1);
psJeton.setTimestamp(5, resultSet.getTimestamp("dt_debut"));
psJeton.setTimestamp(6, resultSet.getTimestamp("dt_fin"));
psJeton.setInt(7, 0);
psJeton.setInt(8, 0);
psJeton.setInt(9, 0);
psJeton.setInt(10, 0);
psJeton.setTimestamp(11, dateDB);
psJeton.setString(12, user);
psJeton.setTimestamp(13, dateDB);
psJeton.setString(14, user);
psJeton.executeUpdate();
String sbDroitHasJeton = null;
if (isCreateForEntreprise) {
sbDroitHasJeton = "INSERT INTO sigs_dej VALUES (?, ?)";
}
if (isCreateForIndiv) {
sbDroitHasJeton = "INSERT INTO sigs_dij VALUES (?, ?)";
}
psDroitJeton = connection.prepareStatement(sbDroitHasJeton);
if(isCreateForEntreprise) {
psDroitJeton.setInt(1, resultSet.getInt("id_e"));
} else {
psDroitJeton.setInt(1, resultSet.getInt("id_i"));
}
psDroitJeton.setString(2, jetonGenerated);
psDroitJeton.executeUpdate();
} catch (SQLException e) {
e.printStackTrace();
} finally {
if (psJeton != null) {
psJeton.close();
}
if (psDroitJeton != null) {
psDroitJeton.close();
}
}
I hope this is the best implementation of PreparedStatement & Scrollable ResultSet

how to auto generate id in java textfield from mysql

What I've tried:
try
{
PreparedStatement pstat = con.prepareStatement("insert into student_info values (?,?,?,?)", PreparedStatement.RETURN_GENERATED_KEYS);
pstat.setInt(1, Integer.parseInt(txtId.getText()));
pstat.setString(2, txtFName.getText());
pstat.setString(3, txtLName.getText());
pstat.setString(4, gender);
int result = pstat.executeUpdate();
if(result > 0)
{
JOptionPane.showMessageDialog(null,"Data Inserted Succesfully !!!");
txtId.setText("");
txtFName.setText("");
txtLName.setText("");
bg.clearSelection();
ResultSet rs = pstat.getGeneratedKeys();
int newId= -1;
if(rs != null && rs.next())
{
newId = rs.getInt(1);
txtId.setText(String.valueOf(newId));
}
}
}
What I want:
The new id to be displayed in the textbox
What doesn't happen:
The new id doesnt get displayed in textfield..

Make this changes to your code:
PreparedStatement pstat = con
.prepareStatement("insert into student_info (firstname,lastname,gender)
values (?,?,?)", PreparedStatement.RETURN_GENERATED_KEYS); // omit id column
pstat.setString(1, txtFName.getText());
pstat.setString(2, txtLName.getText());
pstat.setString(3, gender);
int result = pstat.executeUpdate();
// now got fetch the generated id

Updating a large Resultset with JDBC and MySQL

I am trying update a large set of rows (around 5M). I first came across the heap overflow issue of having so many rows fetched in a resultset. Since I don't want to raise my heap size on this machine I was wondering if there is an effective way of doing this.
I tried setting the setFetchSize(Integer.MIN_VALUE) but then when I call the update function I get this error:
Streaming result set com.mysql.jdbc.RowDataDynamic#2087c268 is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.
If I call close() on the result set I cannot update it of course. Here is my code
public void getRows()
{
Statement stmt = null;
ResultSet rs = null;
int countSpecialChars = 0;
int upper = 0, lower = 0, digits =0;
String pass = null;
int id = 0;
char thisChar;
String query = "select id,pass from datatable";
try {
this.conn.setAutoCommit(false);
stmt = this.conn.createStatement();
stmt.setFetchSize(Integer.MIN_VALUE);
rs = stmt.executeQuery(query);
while (rs.next()) {
pass = rs.getString(2).trim();
id = rs.getInt(1);
for (int i=0; i<=pass.length()-1; i++)
{
thisChar= pass.charAt(i);
if (thisChar >= 65 && thisChar <= 90) {
upper++;
} else if (thisChar >= 97 && thisChar <= 122) {
lower++;
} else if( thisChar >= 48 && thisChar <= 57) {
digits++;
}
else
{countSpecialChars++;}
}
Entropy entropy = new Entropy();
double naiveEntropy = entropy.naiveEntropy(pass);
NumberFormat formatter = new DecimalFormat("#0.00");
this.updateRow(id, pass.length(), digits, upper, lower, countSpecialChars, Double.parseDouble(formatter.format(naiveEntropy)));
countSpecialChars = 0;
upper=digits=0;
lower = 0;
}
rs.close();
}
catch (SQLException e)
...
}
public void updateRow(int id, int length, int numbers, int upper,
int lower, int specialChars, double naiveEntropy )
{
PreparedStatement updatePassEntry = null;
String updateString = "update cwlCompatiblePassUnique " +
"set length = ?, numbers = ?, upper = ?, lower = ?, specialChars = ?, ShannonEntropy = ? where id = ?";
try {
this.conn.setAutoCommit(false);
updatePassEntry = this.conn.prepareStatement(updateString);
updatePassEntry.setInt(1, length);
...
updatePassEntry.setInt(7, id);
updatePassEntry.executeUpdate();
this.conn.commit();
}
catch (SQLException e)
...
}
Any ideas on what can be done?
Thanks

you call updateRow() method inside the rs.next() loop; which tries to make a SQL update on a SQL field (id) that is currently being processed inside your while (rs.next()) loop. this will raise the error you get. i suggest you write a method for pulling rs and storing them in java objects vector as a first step. this method will close the rs after exiting. then write another method to do both processing and update data on your cached vector objects .
something like this:
private void Vector<DataSet> getDataSet(){
Vector<DataSet> data=new Vector<DataSet>();
String query = "select id,pass from datatable";
try {
this.conn.setAutoCommit(false);
stmt = this.conn.createStatement();
stmt.setFetchSize(Integer.MIN_VALUE);
rs = stmt.executeQuery(query);
while (rs.next()) {
pass = rs.getString(2).trim();
id = rs.getInt(1);
data.addElement(new DataSet(id,pass));
}
}catch(Exception e){
// here close connection and rs
}
}
private void udpateData(Vector<dataSet> data){
//process data and update her
}
static class DataSet{
int id;
String pass;
//constructor here
}

Connection object should not hold multiple resultset object at a time.
After creation of ResultSet and Statement Objects, each has to close explicitly like,
resultSet.close()
statement.close()