why to close result set when handling large dataset - java

preparedbatch method takes resultset of size 1000.
I am inserting data using prepared statement and batch.
Using LIMIT I am pulling 1000 rows from mysql database, java.. and calculating some value from column values of each row..
public class Method {
void preparedBatch (ResultSet rs, PreparedStatement insert) throws Exception{
int y;
int arr[]= new int[40];
while (rs.next()){
for(y=2;y<=41;y++){
arr[y-2]=rs.getInt(y);
}
insert.setString(1, rs.getString(1));
insert.setLong(2, calculate(arr));
insert.addBatch();
}
{
insert.executeBatch();
insert.clearBatch(); // I did this to avoid oOM
}
rs.close(); //if I do not close, then I get OOM error
}
long calculate (int[] arr){
long sum =0;
for(int k=0;k<40;k++){
sum+= arr[k]*pow(2,k);
}
return sum;
}
long pow(long base, int exponent)
{
if (exponent == 0)
return 1; // base case;
double temp = pow(base, exponent/2);
if (exponent % 2 == 0)
return (long) (temp * temp);
else
return (long) (base * temp * temp);
}
void method(){
Method k = Insert.m1;
System.out.println("in metho");
Database d1= new Database(); // for reading from tables
d1.setPassword("abc");
d1.setUrl("jdbc:mysql://localhost/DB");
d1.setUser("root");
// object to calculate 2 raise to 39 stuff
ResultSet rs =null;// result set for 2500 lines selected
PreparedStatement ps=null,insert=null; //ps for selecting 1000 lines //insert for inserting into binary table
final Connection con;
try {
con = DriverManager.getConnection(d1.getUrl(), d1.getUser(), d1.getPassword());
con.setAutoCommit(false);
insert = con.prepareStatement("insert into binarydata values(?,?)");
int z;
for(z=0;z<=850000;z=z+1000)
{
ps = con.prepareStatement("select * from table1 LIMIT "+z +", 1000");
rs= ps.executeQuery();
try {
k.preparedBatch(rs,insert);//k is global object of Methodclass
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
rs.close();
if(z%200000==0){
System.out.println(z +" inserted this time");
}
if(z==850000){
System.out.println(z +"mouse");
}
}
{
ps = con.prepareStatement("select * from table1 LIMIT 851000 , 364");
insert = con.prepareStatement("insert into binarydata values(?,?)");
rs= ps.executeQuery();
try {
k.preparedBatch(rs,insert);
System.out.println("finally inserting");
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
System.out.println(rs.getString(1));
}
rs.close();
}
try {
con.commit();
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
finally{
con.close();
ps.close();
insert.close();
}
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
My main class creates object of Method and invokes method from it,
my array is 0,1,0,1,1,1,1,1,,0,0,0,0,0,0,0,1,0 ( it is an example, all 851364 rows contain such data)
if my arr for example is 0,1,0, then I am calculating sum+= arr[index]*2 (power index).. like this..
I have no knowledge of Springs, otherwise I would have used that to handle data of my size.
Please help me understand, why in method preparedBatch, I have to close resultset?
Also, please suggest optimized code of this.
It took me some 9 minutes, to read 85136 lines and insert respective sum..

Related

how to sort a table from database

I want to be able to sort a table from the database, according to either the quatity or the name, but how do i decided what happens in what case?
Below is the code for the table.
public void tableupdate(JTable jTable1, String fill) {
try {
try {
Class.forName("org.h2.Driver");
Connection con = DriverManager.getConnection("jdbc:h2:file:D:/Inventory.db", "sa", "");
Statement stat = con.createStatement();
fill = "SELECT * FROM BOOKDESC ";
ResultSet rs = stat.executeQuery(fill);
while (jTable1.getRowCount() > 0) {
((DefaultTableModel) jTable1.getModel()).removeRow(0);
}
int columns = rs.getMetaData().getColumnCount();
while (rs.next()) {
Object[] row = new Object[columns];
for (int i = 1; i <= columns; i++) {
row[i - 1] = rs.getObject(i);
}
((DefaultTableModel) jTable1.getModel()).insertRow(rs.getRow() - 1, row);
}
rs.close();
stat.close();
con.close();
} catch (ClassNotFoundException e) {
JOptionPane.showMessageDialog(null, e);
}
} catch (SQLException e) {
JOptionPane.showMessageDialog(null, e);
}
}
MySQL is offering a method for sorting data in your SELECT statement, it's called ORDER BY.
Usage is found here.
This way, your code doesn't have to do the work, as your ResultSet already gets sorted data.

Looping try/catch statement

I'm trying to take two random rowid from my database. Everything works but I have a scenario when there is only one rowid. I want to make a loop on my try/catch until there is second number in my database.
What I'm doing wrong? Thank you
public void Kaslaimejo() {
String sql = "SELECT rowid FROM Zaidejai WHERE Pirmas < 4 ORDER BY random() LIMIT 2";
Integer value1 = null, value2 = null;
Integer judesiukas1 = null, judesiukas2 = null;
int a = 0;
int k = 15; // kiek kartu? Reikia infinity padaryti
for (a = 0; a < 3; a++) {
try {
Connection conn = Serveris.connect();
Statement stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery(sql)) {
if (rs.next()) {
value1 = rs.getInt("rowid");
if (rs.next()) {
value2 = rs.getInt("rowid");
PreparedStatement buvo = conn.prepareStatement("UPDATE Zaidejai SET Numeriukas = ? WHERE rowid = ?");
buvo.setInt(1, i);
buvo.setInt(2, value1);
int buvolala = buvo.executeUpdate ();
PreparedStatement buvo2 = conn.prepareStatement("UPDATE Zaidejai SET Numeriukas = ? WHERE rowid = ?");
buvo2.setInt(1, i);
buvo2.setInt(2, value2);
int buvolala2 = buvo2.executeUpdate ();//
i++;
}
System.out.println("Pirmas zaidejas" + value1); // atspausdina 1 random zaideja is duomenu bazes
System.out.println("Antras zaidejas" + value2); // atspausdina 2 random zaideja is duomenu bazes
}
} catch (SQLException e) {
a--;
//System.out.println(e.getMessage());
}
}
}
Right now my program loops two times and then gives me SQLException. How I can loop my program until there is no SQLException?
OK, I've tried to write what I think you're trying to do.
You wait for ever until someone puts at least two entries in the database.
You extract two values, process them, then wait some more.
Some points to watch out:
1. Object comparisons need to be made with .equals() not with ==
2. You might want to provide some way to break out of the infinite loop I've written (while(true)).
3. Careful with null values. They might produce NullPointerException.
4. Try to break up your code into methods. Each large block of code could go into each own method.
public void Kaslaimejo(){
String sql = "SELECT rowid FROM Zaidejai WHERE Pirmas < 4 ORDER BY random() LIMIT 2";
Integer judesiukas1 = null, judesiukas2 = null;
while(true) {
List<Integer> values = new ArrayList<>();
while (values.size() < 2) {
try (Connection conn = Serveris.connect();
Statement stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery(sql)) {
if( rs.next() ){
Integer value = rs.getInt("rowid");
values.add(value);
}
} catch (SQLException e) {
e.printStackTrace();
}
}
try( Connection conn = Serveris.connect()) {
PreparedStatement buvo = conn.prepareStatement("UPDATE Zaidejai SET Numeriukas = ? WHERE rowid = ?");
buvo.setInt(1, i);
buvo.setInt(2, values.get(0));
int buvolala = buvo.executeUpdate ();
PreparedStatement buvo2 = conn.prepareStatement("UPDATE Zaidejai SET Numeriukas = ? WHERE rowid = ?");
buvo2.setInt(1, i);
buvo2.setInt(2, values.get(1));
int buvolala2 = buvo2.executeUpdate ();//
i++;
}catch (SQLException e) {
e.printStackTrace();
}
Connection conn = Serveris.connect();
try {
PreparedStatement pstmt = conn.prepareStatement("SELECT Pirmas FROM Zaidejai WHERE rowid = ?");
PreparedStatement pstmt2 = conn.prepareStatement("SELECT Pirmas FROM Zaidejai WHERE rowid = ?");
pstmt.setInt(1, values.get(0));
pstmt2.setInt(1, values.get(1));
ResultSet myrsv = pstmt.executeQuery();
ResultSet myrsv2 = pstmt2.executeQuery();
{
if (myrsv.next()) {
judesiukas1 = myrsv.getInt("Pirmas");
if (myrsv2.next()) {
judesiukas2 = myrsv2.getInt("Pirmas");
}
}
//System.out.println("Pirmo zaidejo veiksmas" + myrsv.getInt("Pirmas"));
//System.out.println("Antro zaidejo veiksmas" + myrsv2.getInt("Pirmas"));
}
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
if (judesiukas1.equals(judesiukas2)) // careful here. NullPointerException may happen.
{
try {
PreparedStatement laim = conn.prepareStatement("UPDATE Zaidejai SET Rezultatas = ? WHERE rowid = ?"); // ble ble update reikia naudoti , o ne insert into. Insert kai sukuriame nauja kazka tik
PreparedStatement laim2 = conn.prepareStatement("UPDATE Zaidejai SET Rezultatas = ? WHERE rowid = ?");
laim.setString(1, "Lygiosios");
laim.setInt(2, values.get(0));
laim2.setString(1, "Lygiosios");
laim2.setInt(2, values.get(1));
int irasyk = laim.executeUpdate (); // kodel executeupdate, o ne executequery????
int irasyk2 = laim2.executeUpdate (); // kodel executeupdate, o ne executequery????
{
}
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
System.out.print("Lygiosios");
} else {
// (1) - Rock
// (2) Scissors
// (3) - Paper
switch (values.get(0)){
case 1:
if (judesiukas2 == 2)
System.out.print("Zaidejas 1 wins!");
else
System.out.print("Zaidejas 2 wins!");
break;
case 2:
if (judesiukas2 == 3)
System.out.print("Zaidejas 1 wins!");
else
System.out.print("Zaidejas 2 wins!");
break;
case 3:
if (judesiukas2 == 1)
System.out.print("Zaidejas 1 wins!");
else
System.out.print("Zaidejas 2 wins!");
break;
}
}
try {
conn.close();
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
The logic becomes easier if you add the values to a list
var values = new ArrayList<Integer>();
while (values.Count < 2) {
try (Connection conn = Serveris.connect();
Statement stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery(sql))
{
while (values.Count < 2 && rs.next()) {
Integer v = rs.getInt("rowid");
values.Add(v);
}
} catch (SQLException e) {
}
}
//TODO: process the values here
The advantage is, that you can retrieve one value at the first database query and the second at a later one or both in the same round and you don't have to keep track of which one of two variables to use.
(Bear with me with the syntax details, I'm not a Java programmer.)
How i can loop my program until there is no SQLException?
Change this (because, it will only allow to loop two times)
for (a = 0; a < 2; a++) {
to
while(true)
Put everything inside while(true), if exception occurred, then it will come out from the while loop. Something similar :
try
{
while(true)
{
...
...
}
...
}
catch(SQLException e)
{
// do somthing
}

Java - NULL ResultSet

I have a function to fetch data from MySQL table
public ResultSet getAddressID(String city) throws SQLException{
String q = "SELECT PK_ADDRESS_ID FROM tbl_addresses WHERE city =" + "\""+ city+ "\";";
ResultSet rs = executeSearch(q);
return rs;
}
When I try System.out.println(n.getAddressID("Sheffield")); it returns null. Why this happened even though there are data in my table (see picture).
public ResultSet executeSearch(String q){
openConnection();
try{
Statement statement = connection.createStatement();
ResultSet resultSet = statement.executeQuery(q);
closeConnection();
return resultSet;
}
catch (Exception e){
JOptionPane.showMessageDialog(null, e.getMessage());
}
finally {
closeConnection();
return null;
}
}
The problem appears to be in your executeSearch method; the finally block will always execute, so by returning null in the finally block, you essentially override what you returned in the try block!
This could be an alternative solution; note that I'm returning at the end of the method instead of within any parts of the try-catch-finally block.
/**
* Converts a provided ResultSet into a generic List so that the
* ResultSet can be closed while the data persists.
* Source: http://stackoverflow.com/a/7507225/899126
*/
public List convertResultSetToList(ResultSet rs) throws SQLException
{
ResultSetMetaData md = rs.getMetaData();
int columns = md.getColumnCount();
List list = new ArrayList(50);
while (rs.next())
{
HashMap row = new HashMap(columns);
for(int i = 1; i <= columns; ++i)
{
row.put(md.getColumnName(i), rs.getObject(i));
}
list.add(row);
}
return list;
}
public List executeSearch(String q)
{
List toReturn;
openConnection();
try {
Statement statement = connection.createStatement();
toReturn = this.convertResultSetToList(statement.executeQuery(q));
}
catch (Exception e) {
JOptionPane.showMessageDialog(null, e.getMessage());
toReturn = new ArrayList();
}
finally {
closeConnection();
}
return toReturn;
}

jdbc consume too much memory

I wrote program in that I need to update 10000. The query is:
for (Ticket ticket : existInBoth) {
if (!ticket.isUpdatable() && !ticket.isDeleted()) continue;
String query = String.format("update tickets SET vehicle_id = %d , price = %d , free = %d,time = '%s',date='%s',url='%s',deleted_at=NULL WHERE unique_key = '%s'",
ticket.getVehicle().getId(), ticket.getPrice(), ticket.getFree(), ticket.getTime(), ticket.getDate(), ticket.getUrl(), ticket.getUniqueKey());
if (db.update(query) > 0)
updatedDates.add(ticket.getDate());
}
and db.update
#Override
public int update(String sql) {
synchronized (Main.THREAD_GUARD) {
int result = -1;
Connection connection = open();
try {
Statement statement = connection.createStatement();
result = statement.executeUpdate(sql);
statement.close();
} catch (SQLException e) {
e.printStackTrace();
} finally {
close(connection);
}
return result;
}
}
It seems realy easy. But when I ran it I see this updating take about 300MB of memory.
I carefully close all statements and connections.

Creating table with values in Java Derby

I am trying to create a table with values that are in a String array. I am looking for the best way to loop through it without having to send a bunch of queries. This is what I have but obviously the loop wont work. Anyone have a better way of performing this?
public static void createTable(String table, String[] values) throws SQLException{
try {
Class.forName(driver);
conn = DriverManager.getConnection(connectionURL);
Statement state = conn.createStatement();
for(int x = 0; x < values.length; x++){
state.execute("CREATE TABLE" + table + " ( " + values[x] + " );");
}
conn.commit();
} catch (ClassNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} finally {
try {
conn.close();
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
Concatenate the elements of Values array.
StringBuilder sb=new StringBuilder();
sb.append("CREATE TABLE " + table + " ( ");
for (int i = 0; i < values.length; i++) {
sb.append(values[i]);
if (i >= values.length-1) {break;}
sb.append(",");
}
sb.append(" )");
state.execute(sb.toString());

Categories