I'm trying to add 100,000 names to a database using JDBC. I'm aware that MS SQL doesnt allow mass inserts of more than 1000, elmts, so I accomplish this by breaking the major set down into sets containing 1000 or fewer. The following is my code:
StringJoiner joiner = new StringJoiner("\'), (\'", "INSERT INTO Names (Name) VALUES (\'", "\');");
ExecutorService threadpool = Executors.newFixedThreadPool(100);
while(names.size() > 0) {
int count = Math.min(1000, names.size());
HashSet<String> set = new HashSet();
Iterator iterator = names.iterator();
for(int i = 0; i < count; i++) {
set.add((String) iterator.next());
}
names.removeAll(set);
for(String s: set)
{
joiner.add(s);
}
// System.out.println(joiner.toString());
threadpool.submit(() -> {
PreparedStatement query = null;
try {
query = connect.prepareStatement(joiner.toString());
} catch (SQLException e) {
e.printStackTrace();
}
try {
query.executeUpdate();
} catch (SQLException e) {
e.printStackTrace();
}
});
}
}
This throws the following exception: com.microsoft.sqlserver.jdbc.SQLServerException: The number of row value expressions in the INSERT statement exceeds the maximum allowed number of 1000 row values.
Why are there more than 1000 lines being added?
Why not use batch inserts? Define batch size 1000 and insert queries in the batches of 1000 queries at a time.
String sql = "insert into names(name) values (?)";
Connection connection = new getConnection();
PreparedStatement ps = connection.prepareStatement(sql);
final int batchSize = 1000;
int count = 0;
for (Employee employee: employees) {
ps.setString(1, employee.getName());
ps.addBatch();
if(++count % batchSize == 0) {
ps.executeBatch();
}
}
ps.executeBatch(); // insert remaining records
ps.close();
connection.close();
Related
I'm trying to take two random rowid from my database. Everything works but I have a scenario when there is only one rowid. I want to make a loop on my try/catch until there is second number in my database.
What I'm doing wrong? Thank you
public void Kaslaimejo() {
String sql = "SELECT rowid FROM Zaidejai WHERE Pirmas < 4 ORDER BY random() LIMIT 2";
Integer value1 = null, value2 = null;
Integer judesiukas1 = null, judesiukas2 = null;
int a = 0;
int k = 15; // kiek kartu? Reikia infinity padaryti
for (a = 0; a < 3; a++) {
try {
Connection conn = Serveris.connect();
Statement stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery(sql)) {
if (rs.next()) {
value1 = rs.getInt("rowid");
if (rs.next()) {
value2 = rs.getInt("rowid");
PreparedStatement buvo = conn.prepareStatement("UPDATE Zaidejai SET Numeriukas = ? WHERE rowid = ?");
buvo.setInt(1, i);
buvo.setInt(2, value1);
int buvolala = buvo.executeUpdate ();
PreparedStatement buvo2 = conn.prepareStatement("UPDATE Zaidejai SET Numeriukas = ? WHERE rowid = ?");
buvo2.setInt(1, i);
buvo2.setInt(2, value2);
int buvolala2 = buvo2.executeUpdate ();//
i++;
}
System.out.println("Pirmas zaidejas" + value1); // atspausdina 1 random zaideja is duomenu bazes
System.out.println("Antras zaidejas" + value2); // atspausdina 2 random zaideja is duomenu bazes
}
} catch (SQLException e) {
a--;
//System.out.println(e.getMessage());
}
}
}
Right now my program loops two times and then gives me SQLException. How I can loop my program until there is no SQLException?
OK, I've tried to write what I think you're trying to do.
You wait for ever until someone puts at least two entries in the database.
You extract two values, process them, then wait some more.
Some points to watch out:
1. Object comparisons need to be made with .equals() not with ==
2. You might want to provide some way to break out of the infinite loop I've written (while(true)).
3. Careful with null values. They might produce NullPointerException.
4. Try to break up your code into methods. Each large block of code could go into each own method.
public void Kaslaimejo(){
String sql = "SELECT rowid FROM Zaidejai WHERE Pirmas < 4 ORDER BY random() LIMIT 2";
Integer judesiukas1 = null, judesiukas2 = null;
while(true) {
List<Integer> values = new ArrayList<>();
while (values.size() < 2) {
try (Connection conn = Serveris.connect();
Statement stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery(sql)) {
if( rs.next() ){
Integer value = rs.getInt("rowid");
values.add(value);
}
} catch (SQLException e) {
e.printStackTrace();
}
}
try( Connection conn = Serveris.connect()) {
PreparedStatement buvo = conn.prepareStatement("UPDATE Zaidejai SET Numeriukas = ? WHERE rowid = ?");
buvo.setInt(1, i);
buvo.setInt(2, values.get(0));
int buvolala = buvo.executeUpdate ();
PreparedStatement buvo2 = conn.prepareStatement("UPDATE Zaidejai SET Numeriukas = ? WHERE rowid = ?");
buvo2.setInt(1, i);
buvo2.setInt(2, values.get(1));
int buvolala2 = buvo2.executeUpdate ();//
i++;
}catch (SQLException e) {
e.printStackTrace();
}
Connection conn = Serveris.connect();
try {
PreparedStatement pstmt = conn.prepareStatement("SELECT Pirmas FROM Zaidejai WHERE rowid = ?");
PreparedStatement pstmt2 = conn.prepareStatement("SELECT Pirmas FROM Zaidejai WHERE rowid = ?");
pstmt.setInt(1, values.get(0));
pstmt2.setInt(1, values.get(1));
ResultSet myrsv = pstmt.executeQuery();
ResultSet myrsv2 = pstmt2.executeQuery();
{
if (myrsv.next()) {
judesiukas1 = myrsv.getInt("Pirmas");
if (myrsv2.next()) {
judesiukas2 = myrsv2.getInt("Pirmas");
}
}
//System.out.println("Pirmo zaidejo veiksmas" + myrsv.getInt("Pirmas"));
//System.out.println("Antro zaidejo veiksmas" + myrsv2.getInt("Pirmas"));
}
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
if (judesiukas1.equals(judesiukas2)) // careful here. NullPointerException may happen.
{
try {
PreparedStatement laim = conn.prepareStatement("UPDATE Zaidejai SET Rezultatas = ? WHERE rowid = ?"); // ble ble update reikia naudoti , o ne insert into. Insert kai sukuriame nauja kazka tik
PreparedStatement laim2 = conn.prepareStatement("UPDATE Zaidejai SET Rezultatas = ? WHERE rowid = ?");
laim.setString(1, "Lygiosios");
laim.setInt(2, values.get(0));
laim2.setString(1, "Lygiosios");
laim2.setInt(2, values.get(1));
int irasyk = laim.executeUpdate (); // kodel executeupdate, o ne executequery????
int irasyk2 = laim2.executeUpdate (); // kodel executeupdate, o ne executequery????
{
}
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
System.out.print("Lygiosios");
} else {
// (1) - Rock
// (2) Scissors
// (3) - Paper
switch (values.get(0)){
case 1:
if (judesiukas2 == 2)
System.out.print("Zaidejas 1 wins!");
else
System.out.print("Zaidejas 2 wins!");
break;
case 2:
if (judesiukas2 == 3)
System.out.print("Zaidejas 1 wins!");
else
System.out.print("Zaidejas 2 wins!");
break;
case 3:
if (judesiukas2 == 1)
System.out.print("Zaidejas 1 wins!");
else
System.out.print("Zaidejas 2 wins!");
break;
}
}
try {
conn.close();
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
The logic becomes easier if you add the values to a list
var values = new ArrayList<Integer>();
while (values.Count < 2) {
try (Connection conn = Serveris.connect();
Statement stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery(sql))
{
while (values.Count < 2 && rs.next()) {
Integer v = rs.getInt("rowid");
values.Add(v);
}
} catch (SQLException e) {
}
}
//TODO: process the values here
The advantage is, that you can retrieve one value at the first database query and the second at a later one or both in the same round and you don't have to keep track of which one of two variables to use.
(Bear with me with the syntax details, I'm not a Java programmer.)
How i can loop my program until there is no SQLException?
Change this (because, it will only allow to loop two times)
for (a = 0; a < 2; a++) {
to
while(true)
Put everything inside while(true), if exception occurred, then it will come out from the while loop. Something similar :
try
{
while(true)
{
...
...
}
...
}
catch(SQLException e)
{
// do somthing
}
I have an hashmap with a key and value (Array of integers) HashMap<key, array> and I would like to retrieve data from a sql table with a column that matches the values in array stored in a hashmap
I can do this in two ways one getting all the data from table and iterating through hashmap and other being iterating through hashmap and applying the where clause in each iteration which makes n calls to database.
Is there an efficient way to do this other than the above ?
Use a PreparedStatement to set up a query having an IN condition in the WHERE clause. This snippet should give you the general idea:
values = new int[]{1,2,3,4,5};
String query =
"SELECT film_id, title\n"
+ "FROM film\n"
+ "WHERE film_Id IN (";
for (int i = 0; i < values.length; ++i) {
if (i > 0) {
query += ",";
}
query += "?";
}
query += ")";
try {
Connection conn = DriverManager.getConnection("db-url", "user", "pwd");
PreparedStatement stmt = conn.prepareStatement(query);
for (int i = 0; i < values.length; ++i)
stmt.setInt(i+1, values[i]);
ResultSet rs = stmt.executeQuery();
while (rs.next()){
// process results
}
} catch (Exception ex) {
ex.printStackTrace();
}
In my application, I am getting(SELECT) around 1000 records from the DB and saving it in a map.
Then I need to do a check and then UPDATE every 100 records(using batchupdate).
So there will be (10*100=1000) 10 set of the batchUpdates performed.
If any set fails, we need to retry(re-loop) the set(of 100). The total records can be any number.
I tried the following. I just want to know if there are any better logic.
x=1000; //total records
y=100; //set of records that need to be updated
for(i=1 to y) {
try{
batchUpdate();
} catch() {
retry();
}
if(i%y==0) { break;}
}
Can anyone please help me in getting any better logic?
If you use java jdbc batch update:
int x = 1000;//your total records
int y = 100;//set of records that need to be updated
String sql = "update xxx set xx = ?,yy=? ";
Connection dbConnection = null;
PreparedStatement preparedStatement = null;
try{
dbConnection = getDBConnection(); //get your db connetion
preparedStatement = dbConnection.prepareStatement(sql);
dbConnection.setAutoCommit(false);
for(int i = 1;i < x;i++){
preparedStatement.setInt(1, 101);
preparedStatement.setString(2, "zzzz");
preparedStatement.addBatch();
if(i % y == 0){
try{
preparedStatement.executeBatch();// update y records
}catch(Exception e){
x -= 100; //if failed ,roll back x
}
}
}
preparedStatement.executeBatch(); // update rest of the reords
dbConnection.commit();
}catch(Exception e){
}
wish can help you!
I am trying update a large set of rows (around 5M). I first came across the heap overflow issue of having so many rows fetched in a resultset. Since I don't want to raise my heap size on this machine I was wondering if there is an effective way of doing this.
I tried setting the setFetchSize(Integer.MIN_VALUE) but then when I call the update function I get this error:
Streaming result set com.mysql.jdbc.RowDataDynamic#2087c268 is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.
If I call close() on the result set I cannot update it of course. Here is my code
public void getRows()
{
Statement stmt = null;
ResultSet rs = null;
int countSpecialChars = 0;
int upper = 0, lower = 0, digits =0;
String pass = null;
int id = 0;
char thisChar;
String query = "select id,pass from datatable";
try {
this.conn.setAutoCommit(false);
stmt = this.conn.createStatement();
stmt.setFetchSize(Integer.MIN_VALUE);
rs = stmt.executeQuery(query);
while (rs.next()) {
pass = rs.getString(2).trim();
id = rs.getInt(1);
for (int i=0; i<=pass.length()-1; i++)
{
thisChar= pass.charAt(i);
if (thisChar >= 65 && thisChar <= 90) {
upper++;
} else if (thisChar >= 97 && thisChar <= 122) {
lower++;
} else if( thisChar >= 48 && thisChar <= 57) {
digits++;
}
else
{countSpecialChars++;}
}
Entropy entropy = new Entropy();
double naiveEntropy = entropy.naiveEntropy(pass);
NumberFormat formatter = new DecimalFormat("#0.00");
this.updateRow(id, pass.length(), digits, upper, lower, countSpecialChars, Double.parseDouble(formatter.format(naiveEntropy)));
countSpecialChars = 0;
upper=digits=0;
lower = 0;
}
rs.close();
}
catch (SQLException e)
...
}
public void updateRow(int id, int length, int numbers, int upper,
int lower, int specialChars, double naiveEntropy )
{
PreparedStatement updatePassEntry = null;
String updateString = "update cwlCompatiblePassUnique " +
"set length = ?, numbers = ?, upper = ?, lower = ?, specialChars = ?, ShannonEntropy = ? where id = ?";
try {
this.conn.setAutoCommit(false);
updatePassEntry = this.conn.prepareStatement(updateString);
updatePassEntry.setInt(1, length);
...
updatePassEntry.setInt(7, id);
updatePassEntry.executeUpdate();
this.conn.commit();
}
catch (SQLException e)
...
}
Any ideas on what can be done?
Thanks
you call updateRow() method inside the rs.next() loop; which tries to make a SQL update on a SQL field (id) that is currently being processed inside your while (rs.next()) loop. this will raise the error you get. i suggest you write a method for pulling rs and storing them in java objects vector as a first step. this method will close the rs after exiting. then write another method to do both processing and update data on your cached vector objects .
something like this:
private void Vector<DataSet> getDataSet(){
Vector<DataSet> data=new Vector<DataSet>();
String query = "select id,pass from datatable";
try {
this.conn.setAutoCommit(false);
stmt = this.conn.createStatement();
stmt.setFetchSize(Integer.MIN_VALUE);
rs = stmt.executeQuery(query);
while (rs.next()) {
pass = rs.getString(2).trim();
id = rs.getInt(1);
data.addElement(new DataSet(id,pass));
}
}catch(Exception e){
// here close connection and rs
}
}
private void udpateData(Vector<dataSet> data){
//process data and update her
}
static class DataSet{
int id;
String pass;
//constructor here
}
Connection object should not hold multiple resultset object at a time.
After creation of ResultSet and Statement Objects, each has to close explicitly like,
resultSet.close()
statement.close()
I used the following codes to update column Clob in oracle, it seems to be okay and work properly, after performance testing, it reported that need consumed more than 200ms while the length of string is more than 130000. Is it any good way to improve it?
private void updateClobDetailsField(Map<Integer, String> idToDetails){
long s1 = System.currentTimeMillis();
Connection conn = null;
PreparedStatement pStmt = null;
ResultSet rset = null;
Map<Integer, Clob> idToDetailsClob = new HashMap<Integer, Clob>();
int BATCH_SIZE = CMType.BATCH_UPDATE_MAXSIZE;
try
{
conn = getConnection();
ServerAdapter adapter = ServerAdapter.getServerAdapter();
List<Integer> IDList = new ArrayList<Integer>();
for(Integer id : idToDetails.keySet()){
IDList.add(id);
}
List<Integer> tempIDList = new ArrayList<Integer>(IDList);
while(!tempIDList.isEmpty()){
int size = tempIDList.size() < BATCH_SIZE ? tempIDList.size() : BATCH_SIZE;
List<Integer> currentBatch = tempIDList.subList(0, size);
String inClause = SQLHelper.prepareInClause("ID",currentBatch.size());
pStmt = conn.prepareStatement("SELECT ID, DETAILS FROM PROGRAM_HISTORY WHERE " + inClause);
for(int i = 0; i < currentBatch.size(); i++){
pStmt.setInt(i+1, (currentBatch.get(i)));
}
rset = pStmt.executeQuery();
while(rset.next()){
int id = rset.getInt(1);
Clob detailsClob = rset.getClob(2);
Writer writer = adapter.getCharacterOutputStream(detailsClob);
String details = idToDetails.get(id);
if (details != null) {
writer.write(details);
}
writer.flush();
writer.close();
idToDetailsClob.put(id, detailsClob);
}
currentBatch.clear();
BaseSQLHelper.close(pStmt, rset);
}
int counter = 0;
pStmt = conn.prepareStatement("UPDATE PROGRAM_HISTORY SET DETAILS = ? WHERE ID = ?");
for(int i=0; i<IDList.size(); i++){
int index = 1;
Clob detailsClob = (Clob) idToDetailsClob.get(IDList.get(i));
pStmt.setClob(index++, detailsClob);
pStmt.setInt(index++, IDList.get(i));
pStmt.addBatch();
counter++;
if(counter % BATCH_SIZE == 0) {
pStmt.executeBatch();
pStmt.clearBatch();
counter = 0;
}
}
if(IDList.size() % BATCH_SIZE > 0) {
pStmt.executeBatch();
}
}
catch (SQLException se)
{
se.printStackTrace();
}
catch (IOException se)
{
se.printStackTrace();
}
finally
{
cleanup(conn, pStmt, null);
}
System.out.println(System.currentTimeMillis()-s1);
}
If I understand your code correctly, you are appending text to your details clob column.
Doing it in PL/SQL would be faster since you wouldn't have to fetch the clob across the network. For example you could prepare this statement:
DECLARE
l_details CLOB;
BEGIN
SELECT details INTO l_details FROM program_history WHERE ID = ?;
dbms_lob.append(l_details, ?);
END;
and bind currentBatch.get(i) and idToDetails.get(id).
Notice that you don't need an additional update with PL/SQL.
Execute your query with an updatable ResultSet so that you can update the data as you scroll through without separate update statements being executed.
You need to create your prepared statement with the resultSetConcurrency set to ResultSet.CONCUR_UPDATABLE. Check out the oracle documentation on dealing with streams for the various ways you can handle the clob data.