I am having an OOME problem when trying to run a native SQL query through the jpa / hibernate EM.
The treatment serves to make millions of insertion per pack of 50.
Here is code of my algorithm:
private void createNewJetonsForIndividus(boolean isGlobal, List<String> entreprises, List<String> services,
String user, Timestamp dateDB) {
LocalDateTime timer = LocalDateTime.now();
List<Object[]> MinMaxId = getMinMaxIdDroitsIndividusActifsForCreation(
isGlobal, entreprises, services);
if (null != MinMaxId.get(0)[0]) {
int idStart = ((BigInteger) MinMaxId.get(0)[0]).intValue();
int idEnd = idStart + PAS;
int idMax = ((BigInteger) MinMaxId.get(0)[1]).intValue();
int nbRowsTotal = 0;
Logger.debug("Droits Individus : ID Min {} - ID Max {}", idStart, idMax);
do {
int finalIdStart = idStart;
int finalIdEnd = idEnd;
callTransaction(() -> create(false, true,isGlobal, entreprises, services, finalIdStart,
finalIdEnd, user, dateDB));
idStart = idEnd + 1;
idEnd = idEnd + PAS;
}
while (idMax > idEnd);
}
}
the method is used to calculate the id min and max of the records that interest my treatment. Subsequently, I use the create method whose code is below :
int nbRowsFind;
List<Object[]> listeDroitsIndividusActifsForCreation = getDroitsIndividusActifsForCreation(
isGlobal, entreprises, services, idStart, idEnd);
if (ValidationUtils.isNotEmpty(listeDroitsIndividusActifsForCreation)) {
nbRowsFind = listeDroitsIndividusActifsForCreation.size();
StringBuilder sbJeton = new StringBuilder();
sbJeton.append("INSERT INTO sigs_nv_jeton VALUES ");
StringBuilder sbDroitHasJeton = new StringBuilder();
if (isCreateForIndiv) {
sbDroitHasJeton.append("INSERT INTO sigs_droits_individu_has_nv_jeton VALUES ");
}
listeDroitsIndividusActifsForCreation.stream().forEach(object -> {
sbJeton.append("(");
sbDroitHasJeton.append("(");
BigInteger idDroit = (BigInteger) object[0];
String jetonGenerated = IdJetonGenerator.codeGenerator(idDroit.toString(), DateUtils.now());
sbJeton.append("'").append(jetonGenerated).append("', ");
appendDate(sbJeton, object[1]);
appendDate(sbJeton, object[2]);
sbJeton.append(0).append(", ");
sbJeton.append(0).append(", ");
sbJeton.append("'").append(dateDB).append("', ");
sbJeton.append("'").append(user).append("'");
sbDroitHasJeton.append(idDroit).append(",'").append(jetonGenerated).append("'");
sbJeton.append("),");
sbDroitHasJeton.append("),");
});
String requestJeton = sbJeton.toString();
sbJeton.delete(33, sbJeton.length());
requestJeton = requestJeton.substring(0, requestJeton.length() - 1);
jpaApi.em().createNativeQuery(requestJeton).executeUpdate();
String requestDroitHasJeton = sbDroitHasJeton.toString();
sbDroitHasJeton.delete(54, sbDroitHasJeton.length());
requestDroitHasJeton = requestDroitHasJeton.substring(0, requestDroitHasJeton.length() - 1);
**jpaApi.em().createNativeQuery(requestDroitHasJeton).executeUpdate();**
**jpaApi.em().flush();
jpaApi.em().clear();**
}
When I analyze the Heap Dump, I notice that despite the flush and clear, queries are still referenced in the SessionFactory, is this normal?
enter image description here
flush is executing changes made in the unit of work and clear removes entities from the persistence context.
Both have nothing to do with native SQL queries. I assume that the native queries are cached anywhere else (Hibernate, JDBC...)
I would suggest that you use a prepared statement instead of your dynamic insert statement.
This is my new implementation :
Statement statement = null;
ResultSet resultSet = null;
int nbRows = 0;
try {
statement = connection.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE,
ResultSet.CONCUR_READ_ONLY);
try {
resultSet = statement.executeQuery(queryGetDroitsEntreprisesActifsForCreation(isGlobal, entreprises, services));
resultSet.beforeFirst();
while (resultSet.next()) {
nbRows += 1;
queryInsertJetonsAndLinkDroitsJeton(isCreateForEntreprise, isCreateForIndiv, connection, resultSet,
user, dateDB);
}
System.out.println(nbRows);
} finally {
if (resultSet != null) {
resultSet.close();
}
}
} finally {
if (statement != null) {
statement.close();
}
}
The method "queryGetDroitsEntreprisesActifsForCreation" returns a SQL query in String.
The method "queryInsertJetonsAndLinkDroitsJeton" use the famous PreparedStatement :
PreparedStatement psJeton = null;
PreparedStatement psDroitJeton = null;
try {
String sbJeton = "INSERT INTO TABLE1 VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) ";
psJeton = connection.prepareStatement(sbJeton);
String jetonGenerated;
if (isCreateForEntreprise) {
jetonGenerated = IdJetonGenerator.codeGenerator(String.valueOf(resultSet.getInt("id_e")), DateUtils.now());
} else {
jetonGenerated = IdJetonGenerator.codeGenerator(String.valueOf(resultSet.getInt("id_i")), DateUtils.now());
}
psJeton.setString(1, jetonGenerated);
psJeton.setString(2, ContextType.GD.name());
psJeton.setString(3, JetonType.ANONYME.name());
psJeton.setInt(4, 1);
psJeton.setTimestamp(5, resultSet.getTimestamp("dt_debut"));
psJeton.setTimestamp(6, resultSet.getTimestamp("dt_fin"));
psJeton.setInt(7, 0);
psJeton.setInt(8, 0);
psJeton.setInt(9, 0);
psJeton.setInt(10, 0);
psJeton.setTimestamp(11, dateDB);
psJeton.setString(12, user);
psJeton.setTimestamp(13, dateDB);
psJeton.setString(14, user);
psJeton.executeUpdate();
String sbDroitHasJeton = null;
if (isCreateForEntreprise) {
sbDroitHasJeton = "INSERT INTO sigs_dej VALUES (?, ?)";
}
if (isCreateForIndiv) {
sbDroitHasJeton = "INSERT INTO sigs_dij VALUES (?, ?)";
}
psDroitJeton = connection.prepareStatement(sbDroitHasJeton);
if(isCreateForEntreprise) {
psDroitJeton.setInt(1, resultSet.getInt("id_e"));
} else {
psDroitJeton.setInt(1, resultSet.getInt("id_i"));
}
psDroitJeton.setString(2, jetonGenerated);
psDroitJeton.executeUpdate();
} catch (SQLException e) {
e.printStackTrace();
} finally {
if (psJeton != null) {
psJeton.close();
}
if (psDroitJeton != null) {
psDroitJeton.close();
}
}
I hope this is the best implementation of PreparedStatement & Scrollable ResultSet
Related
I know this question has been asked many times here on stackoverflow, however I still can't pinpoint the exact reason on why my code is not working.
This is the query:
String QUERY = "INSERT INTO orders (user, product_id, final_price, key_id) VALUES (?, ?, ?, ?)";
Manually running the query using the same parameters that I use on my failing function works perfectly.
This is the preparedStatement:
// ...more code up here...
PreparedStatement preparedStatement = connectionManager.databaseConnection.prepareStatement(QUERY);
preparedStatement.setInt(1, user_id);
preparedStatement.setInt(2, product_id);
preparedStatement.setFloat(3, _finalprice);
// ...more code down there...
preparedStatement.setInt(4, _keyid);
The error I'm getting is the following:
java.sql.SQLException: Parameter index out of range (0 < 1 ).
I have several other INSERTs done with the same technique and they all work flawlessly, I'm starting to think that there is some problem with the mysqlconnector I'm using (mysql-connector-java-8.0.19).
Full code for context:
private static boolean assignProduct(int product_id, int user_id)
{
String QUERY = "INSERT INTO orders (user, product_id, final_price, key_id) VALUES (?, ?, ?, ?)";
float _finalprice = 0;
int _discount = 0;
int _keyid = 0;
if(product_id <= 0 || user_id <= 0) {return false;}
Product _p;
try {
_p = Product_utils.productByID(product_id);
if(_p != null)
{
_finalprice = _p.getPrice();
_discount = _p.getDiscount();
if(_discount > 0)
{
_finalprice = _finalprice - (_finalprice * _discount / 100);
}
}else { return false;}
PreparedStatement preparedStatement = connectionManager.databaseConnection.prepareStatement(QUERY);
preparedStatement.setInt(1, user_id);
preparedStatement.setInt(2, product_id);
preparedStatement.setFloat(3, _finalprice);
_keyid = retriveKey(product_id);
if(_keyid == 0) { return false; }
preparedStatement.setInt(4, _keyid);
if(preparedStatement.executeUpdate() == 1) {return true;} else {return false;}
} catch (SQLException e) { System.out.println(e.toString()); return false;}
}
Resolved by upgrading my mysql connector to 8.0.20
try {
Class.forName("com.mysql.jdbc.Driver");
Connection dbaTo = DriverManager.getConnection("jdbc:mysql://localhost:3306/ourDB1", "root", "");
PreparedStatement stat2 = dbaTo.prepareStatement("SELECT * FROM tblEntry");
journEnt = journCombo.getSelectedItem().toString();
String sqlbebe = "INSERT INTO tblJournEnt(strEntJournCode, strEntJournType) VALUES (?, ?)";
PreparedStatement stat3 = dbaTo.prepareStatement(sqlbebe);
ResultSet resultaNgSet = stat2.executeQuery();
if(resultaNgSet.next()) {
do{
kuhaEntCode = resultaNgSet.getString(1);
substring2 = kuhaEntCode.substring(Math.max(kuhaEntCode.length() - 3, 0));
}while(resultaNgSet.next());
} //IF////////////////////////////////////////////////////////////
else{
stringsaEnt = "Ent000";
}
int convertToInt2 = Integer.parseInt(substring2);
int addition2 = convertToInt2 + 1;
if (addition2 >= 10) {
String prd = "ent0";
stringsaEnt = prd + addition2;
}
else {
String prd2 = "ent00";
stringsaEnt = prd2 + addition2;
//pasaEnt(stringsaEnt);
}
stat3.setString(1, stringsaEnt);
stat3.setString(2, journEnt);
stat3.addBatch();
stat3.executeBatch();
stat3.close();
}catch(Exception saiko) {
}
This is my code and i don't know but it won't insert into my database
You're not providing the two parameters for stat3 / sqlbebe.
Also, having it all wrapped in a try block with empty catch doesn't help, this leads to any exceptions being silently ignored. Print / log / rethrow the error and you'll see why your SQL statement failed.
I have a JFrame that has 3 JTextfields and 2 JDatechooser, what I am trying to do is if only one JTextfield has something typed in it and I press the search button, then I will be able to retrieve the data to JTable, but the problem is I have to fill out all JTextFileds and JDatechooser in order to retrieve data. My idea is to ignore null JTextfields and JTdatechooser if only one JTextfield has the keyword I want ?? Any suggestions ?? Thanks in advance,
public ArrayList<BillsRecord> getBillRecordByID(int EmpCode, String Fname, String Lname, String sDate, String eDate) throws SQLException {
String sql = "SELECT B.DATE AS DT, B.EMP_ID, E.FNAME, E.LNAME, MONEY_SENT, RENT, PHONE, GAS, ELECTRICITY, INTERNET, OTHER"
+ " FROM EMPLOYEE E INNER JOIN BILLS B ON E.EMP_ID = B.EMP_ID"
+ " WHERE B.EMP_ID = ? "
+ " OR E.FNAME = ? "
+ " OR E.LNAME = ? "
+ " OR DATE BETWEEN ? AND ? "
+ " ORDER BY B.DATE";
DBConnection con = new DBConnection();
Connection connect = con.getConnection();
PreparedStatement ps = null;
ArrayList<BillsRecord> records = new ArrayList<>();
try {
ps = connect.prepareStatement(sql);
ps.setInt(1, EmpCode);
ps.setString(2, Fname);
ps.setString(3, Lname);
ps.setString(4, sDate);
ps.setString(5, eDate);
ResultSet rs = ps.executeQuery();
if (rs.next()) {
BillsRecord billrec = new BillsRecord();
billrec.setDATE(rs.getString("DT"));
billrec.setEMP_ID(rs.getInt("EMP_ID"));
billrec.setFNAME(rs.getString("FNAME"));
billrec.setLNAME(rs.getString("LNAME"));
billrec.setMONEY_SENT(rs.getDouble("MONEY_SENT"));
billrec.setRENT(rs.getDouble("RENT"));
billrec.setPHONE(rs.getDouble("PHONE"));
billrec.setGAS(rs.getDouble("GAS"));
billrec.setELECTRICITY(rs.getDouble("ELECTRICITY"));
billrec.setINTERNET(rs.getDouble("INTERNET"));
billrec.setOTHER(rs.getDouble("OTHER"));
records.add(billrec);
return records;
}
} catch (SQLException e) {
System.out.println(e.toString());
} finally {
if (ps != null) {
ps.close();
}
if (connect != null) {
connect.close();
}
}
return null;
}
private void search() {
try {
JTextField stxt = ((JTextField) startdatetxt.getDateEditor().getUiComponent());
String sDATE = stxt.getText().trim();
JTextField etxt = ((JTextField) enddatetxt.getDateEditor().getUiComponent());
String eDATE = etxt.getText().trim();
int EMP_ID = Integer.parseInt(this.empidtxt.getText().trim());
String FNAME = this.firstnametxt.getText().trim();
String LNAME = this.lastnametxt.getText().trim();
BillRecordDao billrecdao = new BillRecordDao();
ArrayList<BillsRecord> records = billrecdao.getBillRecordByID(EMP_ID, FNAME, LNAME, sDATE, eDATE);
Object[] tableColumnName = new Object[11];
tableColumnName[0] = "Date";
tableColumnName[1] = "H.License";
tableColumnName[2] = "First Name";
tableColumnName[3] = "Last Name";
tableColumnName[4] = "MONEY SENT";
tableColumnName[5] = "RENT";
tableColumnName[6] = "PHONE";
tableColumnName[7] = "GASE";
tableColumnName[8] = "ELECTRICITY";
tableColumnName[9] = "INTERNET";
tableColumnName[10] = "OTHER";
DefaultTableModel tbd = new DefaultTableModel();
tbd.setColumnIdentifiers(tableColumnName);
this.BillsSummaryTable.setModel(tbd);
Object[] RowRec = new Object[11];
for (int i = 0; i < records.size(); i++) {
RowRec[0] = records.get(i).getDATE();
RowRec[1] = records.get(i).getEMP_ID();
RowRec[2] = records.get(i).getFNAME().toUpperCase();
RowRec[3] = records.get(i).getLNAME().toUpperCase();
RowRec[4] = records.get(i).getMONEY_SENT();
RowRec[5] = records.get(i).getRENT();
RowRec[6] = records.get(i).getPHONE();
RowRec[7] = records.get(i).getGAS();
RowRec[8] = records.get(i).getELECTRICITY();
RowRec[9] = records.get(i).getINTERNET();
RowRec[10] = records.get(i).getOTHER();
tbd.addRow(RowRec);
}
} catch (SQLException e) {
System.out.println(e.toString());
}
}
Basically, you need to create a variable/dynamic query based on the available values
Now, you can do this using something like StringBuilder or even storing each query element in a List or array, but you always end up with the "trailing OR" problem (you need to know when you've got to the last element and not append the "OR" to the String or remove the trailing "OR" from the resulting String). While not difficult, it's just a pain.
However, if you're using Java 8, you can use StringJoiner!
StringJoiner sj = new StringJoiner(" OR ");
String sql = "SELECT B.DATE AS DT, B.EMP_ID, E.FNAME, E.LNAME, MONEY_SENT, RENT, PHONE, GAS, ELECTRICITY, INTERNET, OTHER"
+ " FROM EMPLOYEE E INNER JOIN BILLS B ON E.EMP_ID = B.EMP_ID"
+ " WHERE ";
List values = new ArrayList();
// EmpCode MUST be a Integer, so it can be null
if (EmpCode != null) {
sj.add("B.EMP_ID = ?");
values.add(EmpCode);
}
if (FName != null) {
sj.add("E.FNAME = ?");
values.add(FName);
}
if (LName != null) {
sj.add("E.LNAME = ?");
values.add(LName);
}
if (sDate != null && eDate != null) {
sj.add("DATE BETWEEN ? AND ?");
values.add(sDate);
values.add(eDate);
}
sql += sj.toString();
Connection connect = null;
try (PreparedStatement ps = connect.prepareStatement(sql)) {
for (int index = 0; index < values.size(); index++) {
ps.setObject(index + 1, values.get(index));
}
try (ResultSet rs = ps.executeQuery()) {
if (rs.next()) {
//...
}
}
} catch (SQLException exp) {
exp.printStackTrace();
}
You might also like to have a look at The try-with-resources Statement and have a read through Code Conventions for the Java TM Programming Language, it will make it easier for people to read your code and for you to read others
I am trying update a large set of rows (around 5M). I first came across the heap overflow issue of having so many rows fetched in a resultset. Since I don't want to raise my heap size on this machine I was wondering if there is an effective way of doing this.
I tried setting the setFetchSize(Integer.MIN_VALUE) but then when I call the update function I get this error:
Streaming result set com.mysql.jdbc.RowDataDynamic#2087c268 is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.
If I call close() on the result set I cannot update it of course. Here is my code
public void getRows()
{
Statement stmt = null;
ResultSet rs = null;
int countSpecialChars = 0;
int upper = 0, lower = 0, digits =0;
String pass = null;
int id = 0;
char thisChar;
String query = "select id,pass from datatable";
try {
this.conn.setAutoCommit(false);
stmt = this.conn.createStatement();
stmt.setFetchSize(Integer.MIN_VALUE);
rs = stmt.executeQuery(query);
while (rs.next()) {
pass = rs.getString(2).trim();
id = rs.getInt(1);
for (int i=0; i<=pass.length()-1; i++)
{
thisChar= pass.charAt(i);
if (thisChar >= 65 && thisChar <= 90) {
upper++;
} else if (thisChar >= 97 && thisChar <= 122) {
lower++;
} else if( thisChar >= 48 && thisChar <= 57) {
digits++;
}
else
{countSpecialChars++;}
}
Entropy entropy = new Entropy();
double naiveEntropy = entropy.naiveEntropy(pass);
NumberFormat formatter = new DecimalFormat("#0.00");
this.updateRow(id, pass.length(), digits, upper, lower, countSpecialChars, Double.parseDouble(formatter.format(naiveEntropy)));
countSpecialChars = 0;
upper=digits=0;
lower = 0;
}
rs.close();
}
catch (SQLException e)
...
}
public void updateRow(int id, int length, int numbers, int upper,
int lower, int specialChars, double naiveEntropy )
{
PreparedStatement updatePassEntry = null;
String updateString = "update cwlCompatiblePassUnique " +
"set length = ?, numbers = ?, upper = ?, lower = ?, specialChars = ?, ShannonEntropy = ? where id = ?";
try {
this.conn.setAutoCommit(false);
updatePassEntry = this.conn.prepareStatement(updateString);
updatePassEntry.setInt(1, length);
...
updatePassEntry.setInt(7, id);
updatePassEntry.executeUpdate();
this.conn.commit();
}
catch (SQLException e)
...
}
Any ideas on what can be done?
Thanks
you call updateRow() method inside the rs.next() loop; which tries to make a SQL update on a SQL field (id) that is currently being processed inside your while (rs.next()) loop. this will raise the error you get. i suggest you write a method for pulling rs and storing them in java objects vector as a first step. this method will close the rs after exiting. then write another method to do both processing and update data on your cached vector objects .
something like this:
private void Vector<DataSet> getDataSet(){
Vector<DataSet> data=new Vector<DataSet>();
String query = "select id,pass from datatable";
try {
this.conn.setAutoCommit(false);
stmt = this.conn.createStatement();
stmt.setFetchSize(Integer.MIN_VALUE);
rs = stmt.executeQuery(query);
while (rs.next()) {
pass = rs.getString(2).trim();
id = rs.getInt(1);
data.addElement(new DataSet(id,pass));
}
}catch(Exception e){
// here close connection and rs
}
}
private void udpateData(Vector<dataSet> data){
//process data and update her
}
static class DataSet{
int id;
String pass;
//constructor here
}
Connection object should not hold multiple resultset object at a time.
After creation of ResultSet and Statement Objects, each has to close explicitly like,
resultSet.close()
statement.close()
I used the following codes to update column Clob in oracle, it seems to be okay and work properly, after performance testing, it reported that need consumed more than 200ms while the length of string is more than 130000. Is it any good way to improve it?
private void updateClobDetailsField(Map<Integer, String> idToDetails){
long s1 = System.currentTimeMillis();
Connection conn = null;
PreparedStatement pStmt = null;
ResultSet rset = null;
Map<Integer, Clob> idToDetailsClob = new HashMap<Integer, Clob>();
int BATCH_SIZE = CMType.BATCH_UPDATE_MAXSIZE;
try
{
conn = getConnection();
ServerAdapter adapter = ServerAdapter.getServerAdapter();
List<Integer> IDList = new ArrayList<Integer>();
for(Integer id : idToDetails.keySet()){
IDList.add(id);
}
List<Integer> tempIDList = new ArrayList<Integer>(IDList);
while(!tempIDList.isEmpty()){
int size = tempIDList.size() < BATCH_SIZE ? tempIDList.size() : BATCH_SIZE;
List<Integer> currentBatch = tempIDList.subList(0, size);
String inClause = SQLHelper.prepareInClause("ID",currentBatch.size());
pStmt = conn.prepareStatement("SELECT ID, DETAILS FROM PROGRAM_HISTORY WHERE " + inClause);
for(int i = 0; i < currentBatch.size(); i++){
pStmt.setInt(i+1, (currentBatch.get(i)));
}
rset = pStmt.executeQuery();
while(rset.next()){
int id = rset.getInt(1);
Clob detailsClob = rset.getClob(2);
Writer writer = adapter.getCharacterOutputStream(detailsClob);
String details = idToDetails.get(id);
if (details != null) {
writer.write(details);
}
writer.flush();
writer.close();
idToDetailsClob.put(id, detailsClob);
}
currentBatch.clear();
BaseSQLHelper.close(pStmt, rset);
}
int counter = 0;
pStmt = conn.prepareStatement("UPDATE PROGRAM_HISTORY SET DETAILS = ? WHERE ID = ?");
for(int i=0; i<IDList.size(); i++){
int index = 1;
Clob detailsClob = (Clob) idToDetailsClob.get(IDList.get(i));
pStmt.setClob(index++, detailsClob);
pStmt.setInt(index++, IDList.get(i));
pStmt.addBatch();
counter++;
if(counter % BATCH_SIZE == 0) {
pStmt.executeBatch();
pStmt.clearBatch();
counter = 0;
}
}
if(IDList.size() % BATCH_SIZE > 0) {
pStmt.executeBatch();
}
}
catch (SQLException se)
{
se.printStackTrace();
}
catch (IOException se)
{
se.printStackTrace();
}
finally
{
cleanup(conn, pStmt, null);
}
System.out.println(System.currentTimeMillis()-s1);
}
If I understand your code correctly, you are appending text to your details clob column.
Doing it in PL/SQL would be faster since you wouldn't have to fetch the clob across the network. For example you could prepare this statement:
DECLARE
l_details CLOB;
BEGIN
SELECT details INTO l_details FROM program_history WHERE ID = ?;
dbms_lob.append(l_details, ?);
END;
and bind currentBatch.get(i) and idToDetails.get(id).
Notice that you don't need an additional update with PL/SQL.
Execute your query with an updatable ResultSet so that you can update the data as you scroll through without separate update statements being executed.
You need to create your prepared statement with the resultSetConcurrency set to ResultSet.CONCUR_UPDATABLE. Check out the oracle documentation on dealing with streams for the various ways you can handle the clob data.