I used the following codes to update column Clob in oracle, it seems to be okay and work properly, after performance testing, it reported that need consumed more than 200ms while the length of string is more than 130000. Is it any good way to improve it?
private void updateClobDetailsField(Map<Integer, String> idToDetails){
long s1 = System.currentTimeMillis();
Connection conn = null;
PreparedStatement pStmt = null;
ResultSet rset = null;
Map<Integer, Clob> idToDetailsClob = new HashMap<Integer, Clob>();
int BATCH_SIZE = CMType.BATCH_UPDATE_MAXSIZE;
try
{
conn = getConnection();
ServerAdapter adapter = ServerAdapter.getServerAdapter();
List<Integer> IDList = new ArrayList<Integer>();
for(Integer id : idToDetails.keySet()){
IDList.add(id);
}
List<Integer> tempIDList = new ArrayList<Integer>(IDList);
while(!tempIDList.isEmpty()){
int size = tempIDList.size() < BATCH_SIZE ? tempIDList.size() : BATCH_SIZE;
List<Integer> currentBatch = tempIDList.subList(0, size);
String inClause = SQLHelper.prepareInClause("ID",currentBatch.size());
pStmt = conn.prepareStatement("SELECT ID, DETAILS FROM PROGRAM_HISTORY WHERE " + inClause);
for(int i = 0; i < currentBatch.size(); i++){
pStmt.setInt(i+1, (currentBatch.get(i)));
}
rset = pStmt.executeQuery();
while(rset.next()){
int id = rset.getInt(1);
Clob detailsClob = rset.getClob(2);
Writer writer = adapter.getCharacterOutputStream(detailsClob);
String details = idToDetails.get(id);
if (details != null) {
writer.write(details);
}
writer.flush();
writer.close();
idToDetailsClob.put(id, detailsClob);
}
currentBatch.clear();
BaseSQLHelper.close(pStmt, rset);
}
int counter = 0;
pStmt = conn.prepareStatement("UPDATE PROGRAM_HISTORY SET DETAILS = ? WHERE ID = ?");
for(int i=0; i<IDList.size(); i++){
int index = 1;
Clob detailsClob = (Clob) idToDetailsClob.get(IDList.get(i));
pStmt.setClob(index++, detailsClob);
pStmt.setInt(index++, IDList.get(i));
pStmt.addBatch();
counter++;
if(counter % BATCH_SIZE == 0) {
pStmt.executeBatch();
pStmt.clearBatch();
counter = 0;
}
}
if(IDList.size() % BATCH_SIZE > 0) {
pStmt.executeBatch();
}
}
catch (SQLException se)
{
se.printStackTrace();
}
catch (IOException se)
{
se.printStackTrace();
}
finally
{
cleanup(conn, pStmt, null);
}
System.out.println(System.currentTimeMillis()-s1);
}
If I understand your code correctly, you are appending text to your details clob column.
Doing it in PL/SQL would be faster since you wouldn't have to fetch the clob across the network. For example you could prepare this statement:
DECLARE
l_details CLOB;
BEGIN
SELECT details INTO l_details FROM program_history WHERE ID = ?;
dbms_lob.append(l_details, ?);
END;
and bind currentBatch.get(i) and idToDetails.get(id).
Notice that you don't need an additional update with PL/SQL.
Execute your query with an updatable ResultSet so that you can update the data as you scroll through without separate update statements being executed.
You need to create your prepared statement with the resultSetConcurrency set to ResultSet.CONCUR_UPDATABLE. Check out the oracle documentation on dealing with streams for the various ways you can handle the clob data.
Related
In Java, how do I pass values of type set to a procedure. This seems too basic, but I can't solve it, I spent days searching sample Java code on how to pass set values to Informix procedure.
Tools
IBM Informix Dynamic Server Version 12.10.FC13
JDBC 4.10.14, 4.50.7
Java version "1.8.0_172"
Informix procedure
create procedure sp_demo_set_arg(
arg1 set(integer not null)
)
...
end procedure
Java code
#Override
public Integer callProcedure(List<Integer> listOfId) {
String sql = "{ call sp_demo_set_arg(?) }";
#SuppressWarnings("rawtypes")
java.util.HashSet arg1 = new HashSet();
Integer intObject;
int i;
for (i=1; i <= 3; i++)
{
intObject = new Integer(i);
arg1.add(intObject);
}
Connection conn = null;
try {
conn = dataSource.getConnection();
CallableStatement stmt = conn.prepareCall(sql);
stmt.setObject(1, arg1);
stmt.executeUpdate();
return 0;
} catch (SQLException e) {
e.printStackTrace();
}
return 1;
}
Stacktrace
...
java.sql.SQLException: Routine (sp_demo_set_arg) can not be resolved.
at com.informix.jdbc.IfxSqli.addException(IfxSqli.java:3133)
at com.informix.jdbc.IfxSqli.receiveError(IfxSqli.java:3417)
at com.informix.jdbc.IfxSqli.dispatchMsg(IfxSqli.java:2324)
at com.informix.jdbc.IfxSqli.receiveMessage(IfxSqli.java:2249)
at com.informix.jdbc.IfxSqli.executeCommand(IfxSqli.java:850)
at com.informix.jdbc.IfxResultSet.executeUpdate(IfxResultSet.java:230)
at com.informix.jdbc.IfxStatement.executeUpdateImpl(IfxStatement.java:1054)
at com.informix.jdbc.IfxPreparedStatement.executeUpdate(IfxPreparedStatement.java:396)
at
...
Your java code may fail with an -674 "Routine can not be resolved" error because the server may not know the parameter type at execution.
Try giving it some 'hints' changing '?' for a '?::SET(integer not null)'
Something like:
D:\Infx\work\Java>cat t2.java
import java.sql.*;
import java.util.*;
public class t2 {
public static void main( String [] args ) {
Connection conn = null;
ResultSet dbRes = null;
Statement is = null;
try {
Class.forName("com.informix.jdbc.IfxDriver");
conn = DriverManager.getConnection("jdbc:informix-sqli://420ito:9091/stores7:INFORMIXSERVER=ids1410;user=informix;password=passw;SQLIDEBUG=pp;");
is = conn.createStatement();
is.executeUpdate("drop table t2; create table t2 (c1 SET(integer not null) );");
java.util.HashSet arg1 = new HashSet();
Integer intObject;
int i;
for (i=1; i <= 3; i++)
{
intObject = new Integer(i);
arg1.add(intObject);
}
try {
//CallableStatement stmt = conn.prepareCall("{ CALL p2(?)}");
CallableStatement stmt = conn.prepareCall("{ CALL p2(?::SET(integer not null))}");
stmt.setObject(1, arg1);
stmt.executeUpdate();
} catch (SQLException e) {
e.printStackTrace();
}
Statement st = conn.createStatement();
ResultSet rs = st.executeQuery("SELECT * FROM t2");
while (rs.next()) {
java.util.HashSet set = (HashSet) rs.getObject(1);
Iterator it = set.iterator();
Object obj;
i = 0;
while (it.hasNext())
{
obj = it.next();
System.out.println(" element[" + i + "] = " + obj.toString());
i++;
}
}
rs.close();
conn.close();
}
catch ( Exception e ) {
System.err.println(e);
e.printStackTrace();
}
}
}
D:\Infx\work\Java>javac t2.java
Note: t2.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
D:\Infx\work\Java>java t2
element[0] = 1
element[1] = 2
element[2] = 3
D:\Infx\work\Java>
I'm trying to take two random rowid from my database. Everything works but I have a scenario when there is only one rowid. I want to make a loop on my try/catch until there is second number in my database.
What I'm doing wrong? Thank you
public void Kaslaimejo() {
String sql = "SELECT rowid FROM Zaidejai WHERE Pirmas < 4 ORDER BY random() LIMIT 2";
Integer value1 = null, value2 = null;
Integer judesiukas1 = null, judesiukas2 = null;
int a = 0;
int k = 15; // kiek kartu? Reikia infinity padaryti
for (a = 0; a < 3; a++) {
try {
Connection conn = Serveris.connect();
Statement stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery(sql)) {
if (rs.next()) {
value1 = rs.getInt("rowid");
if (rs.next()) {
value2 = rs.getInt("rowid");
PreparedStatement buvo = conn.prepareStatement("UPDATE Zaidejai SET Numeriukas = ? WHERE rowid = ?");
buvo.setInt(1, i);
buvo.setInt(2, value1);
int buvolala = buvo.executeUpdate ();
PreparedStatement buvo2 = conn.prepareStatement("UPDATE Zaidejai SET Numeriukas = ? WHERE rowid = ?");
buvo2.setInt(1, i);
buvo2.setInt(2, value2);
int buvolala2 = buvo2.executeUpdate ();//
i++;
}
System.out.println("Pirmas zaidejas" + value1); // atspausdina 1 random zaideja is duomenu bazes
System.out.println("Antras zaidejas" + value2); // atspausdina 2 random zaideja is duomenu bazes
}
} catch (SQLException e) {
a--;
//System.out.println(e.getMessage());
}
}
}
Right now my program loops two times and then gives me SQLException. How I can loop my program until there is no SQLException?
OK, I've tried to write what I think you're trying to do.
You wait for ever until someone puts at least two entries in the database.
You extract two values, process them, then wait some more.
Some points to watch out:
1. Object comparisons need to be made with .equals() not with ==
2. You might want to provide some way to break out of the infinite loop I've written (while(true)).
3. Careful with null values. They might produce NullPointerException.
4. Try to break up your code into methods. Each large block of code could go into each own method.
public void Kaslaimejo(){
String sql = "SELECT rowid FROM Zaidejai WHERE Pirmas < 4 ORDER BY random() LIMIT 2";
Integer judesiukas1 = null, judesiukas2 = null;
while(true) {
List<Integer> values = new ArrayList<>();
while (values.size() < 2) {
try (Connection conn = Serveris.connect();
Statement stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery(sql)) {
if( rs.next() ){
Integer value = rs.getInt("rowid");
values.add(value);
}
} catch (SQLException e) {
e.printStackTrace();
}
}
try( Connection conn = Serveris.connect()) {
PreparedStatement buvo = conn.prepareStatement("UPDATE Zaidejai SET Numeriukas = ? WHERE rowid = ?");
buvo.setInt(1, i);
buvo.setInt(2, values.get(0));
int buvolala = buvo.executeUpdate ();
PreparedStatement buvo2 = conn.prepareStatement("UPDATE Zaidejai SET Numeriukas = ? WHERE rowid = ?");
buvo2.setInt(1, i);
buvo2.setInt(2, values.get(1));
int buvolala2 = buvo2.executeUpdate ();//
i++;
}catch (SQLException e) {
e.printStackTrace();
}
Connection conn = Serveris.connect();
try {
PreparedStatement pstmt = conn.prepareStatement("SELECT Pirmas FROM Zaidejai WHERE rowid = ?");
PreparedStatement pstmt2 = conn.prepareStatement("SELECT Pirmas FROM Zaidejai WHERE rowid = ?");
pstmt.setInt(1, values.get(0));
pstmt2.setInt(1, values.get(1));
ResultSet myrsv = pstmt.executeQuery();
ResultSet myrsv2 = pstmt2.executeQuery();
{
if (myrsv.next()) {
judesiukas1 = myrsv.getInt("Pirmas");
if (myrsv2.next()) {
judesiukas2 = myrsv2.getInt("Pirmas");
}
}
//System.out.println("Pirmo zaidejo veiksmas" + myrsv.getInt("Pirmas"));
//System.out.println("Antro zaidejo veiksmas" + myrsv2.getInt("Pirmas"));
}
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
if (judesiukas1.equals(judesiukas2)) // careful here. NullPointerException may happen.
{
try {
PreparedStatement laim = conn.prepareStatement("UPDATE Zaidejai SET Rezultatas = ? WHERE rowid = ?"); // ble ble update reikia naudoti , o ne insert into. Insert kai sukuriame nauja kazka tik
PreparedStatement laim2 = conn.prepareStatement("UPDATE Zaidejai SET Rezultatas = ? WHERE rowid = ?");
laim.setString(1, "Lygiosios");
laim.setInt(2, values.get(0));
laim2.setString(1, "Lygiosios");
laim2.setInt(2, values.get(1));
int irasyk = laim.executeUpdate (); // kodel executeupdate, o ne executequery????
int irasyk2 = laim2.executeUpdate (); // kodel executeupdate, o ne executequery????
{
}
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
System.out.print("Lygiosios");
} else {
// (1) - Rock
// (2) Scissors
// (3) - Paper
switch (values.get(0)){
case 1:
if (judesiukas2 == 2)
System.out.print("Zaidejas 1 wins!");
else
System.out.print("Zaidejas 2 wins!");
break;
case 2:
if (judesiukas2 == 3)
System.out.print("Zaidejas 1 wins!");
else
System.out.print("Zaidejas 2 wins!");
break;
case 3:
if (judesiukas2 == 1)
System.out.print("Zaidejas 1 wins!");
else
System.out.print("Zaidejas 2 wins!");
break;
}
}
try {
conn.close();
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
The logic becomes easier if you add the values to a list
var values = new ArrayList<Integer>();
while (values.Count < 2) {
try (Connection conn = Serveris.connect();
Statement stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery(sql))
{
while (values.Count < 2 && rs.next()) {
Integer v = rs.getInt("rowid");
values.Add(v);
}
} catch (SQLException e) {
}
}
//TODO: process the values here
The advantage is, that you can retrieve one value at the first database query and the second at a later one or both in the same round and you don't have to keep track of which one of two variables to use.
(Bear with me with the syntax details, I'm not a Java programmer.)
How i can loop my program until there is no SQLException?
Change this (because, it will only allow to loop two times)
for (a = 0; a < 2; a++) {
to
while(true)
Put everything inside while(true), if exception occurred, then it will come out from the while loop. Something similar :
try
{
while(true)
{
...
...
}
...
}
catch(SQLException e)
{
// do somthing
}
I'm trying to add 100,000 names to a database using JDBC. I'm aware that MS SQL doesnt allow mass inserts of more than 1000, elmts, so I accomplish this by breaking the major set down into sets containing 1000 or fewer. The following is my code:
StringJoiner joiner = new StringJoiner("\'), (\'", "INSERT INTO Names (Name) VALUES (\'", "\');");
ExecutorService threadpool = Executors.newFixedThreadPool(100);
while(names.size() > 0) {
int count = Math.min(1000, names.size());
HashSet<String> set = new HashSet();
Iterator iterator = names.iterator();
for(int i = 0; i < count; i++) {
set.add((String) iterator.next());
}
names.removeAll(set);
for(String s: set)
{
joiner.add(s);
}
// System.out.println(joiner.toString());
threadpool.submit(() -> {
PreparedStatement query = null;
try {
query = connect.prepareStatement(joiner.toString());
} catch (SQLException e) {
e.printStackTrace();
}
try {
query.executeUpdate();
} catch (SQLException e) {
e.printStackTrace();
}
});
}
}
This throws the following exception: com.microsoft.sqlserver.jdbc.SQLServerException: The number of row value expressions in the INSERT statement exceeds the maximum allowed number of 1000 row values.
Why are there more than 1000 lines being added?
Why not use batch inserts? Define batch size 1000 and insert queries in the batches of 1000 queries at a time.
String sql = "insert into names(name) values (?)";
Connection connection = new getConnection();
PreparedStatement ps = connection.prepareStatement(sql);
final int batchSize = 1000;
int count = 0;
for (Employee employee: employees) {
ps.setString(1, employee.getName());
ps.addBatch();
if(++count % batchSize == 0) {
ps.executeBatch();
}
}
ps.executeBatch(); // insert remaining records
ps.close();
connection.close();
I have code, where I have single quote or APOSTROPHE in my search
I have database which is having test table and in name column of value is "my'test"
When running
SELECT * from test WHERE name = 'my''test';
this works fine
If I use the same in a Java program I am not getting any error or any result
But If I give the name with only single quote then it works
SELECT * from test WHERE name = 'my'test';
Could you please help me out to understand.
Java code is
Connection con = null;
PreparedStatement prSt = null;
try {
Class.forName("oracle.jdbc.driver.OracleDriver");
con = DriverManager.
getConnection("jdbc:oracle:thin:#localhost:1521:orcl"
,"user","pwd");
String query = "SELECT * from "
+ "WHERE name = ? ";
prSt = con.prepareStatement(query);
String value = "my'mobile";
char content[] = new char[value.length()];
value.getChars(0, value.length(), content, 0);
StringBuffer result = new StringBuffer(content.length + 50);
for (int i = 0; i < content.length; i++) {
if (content[i] == '\'')
{
result.append("\'");
result.append("\'");
}
else
{
result.append(content[i]);
}
}
prSt.setObject(1, result.toString());
int count = prSt.executeUpdate();
System.out.println("===============> "+count);
} catch (ClassNotFoundException e) {
e.printStackTrace();
} catch (SQLException e) {
e.printStackTrace();
} finally{
try{
if(prSt != null) prSt.close();
if(con != null) con.close();
} catch(Exception ex){}
}
You don't have to escape anything for the parameter of a PreparedStatement
Just use:
prSt = con.prepareStatement(query);
prSt.setString("my'mobile");
Additionally: if you are using a SELECT statement to retrieve data, you need to use executeQuery() not executeUpdate()
ResultSet rs = prst.executeQuery();
while (rs.next())
{
// process the result here
}
You might want to go through the JDBC tutorial before you continue with your project: http://docs.oracle.com/javase/tutorial/jdbc/index.html
I am trying update a large set of rows (around 5M). I first came across the heap overflow issue of having so many rows fetched in a resultset. Since I don't want to raise my heap size on this machine I was wondering if there is an effective way of doing this.
I tried setting the setFetchSize(Integer.MIN_VALUE) but then when I call the update function I get this error:
Streaming result set com.mysql.jdbc.RowDataDynamic#2087c268 is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.
If I call close() on the result set I cannot update it of course. Here is my code
public void getRows()
{
Statement stmt = null;
ResultSet rs = null;
int countSpecialChars = 0;
int upper = 0, lower = 0, digits =0;
String pass = null;
int id = 0;
char thisChar;
String query = "select id,pass from datatable";
try {
this.conn.setAutoCommit(false);
stmt = this.conn.createStatement();
stmt.setFetchSize(Integer.MIN_VALUE);
rs = stmt.executeQuery(query);
while (rs.next()) {
pass = rs.getString(2).trim();
id = rs.getInt(1);
for (int i=0; i<=pass.length()-1; i++)
{
thisChar= pass.charAt(i);
if (thisChar >= 65 && thisChar <= 90) {
upper++;
} else if (thisChar >= 97 && thisChar <= 122) {
lower++;
} else if( thisChar >= 48 && thisChar <= 57) {
digits++;
}
else
{countSpecialChars++;}
}
Entropy entropy = new Entropy();
double naiveEntropy = entropy.naiveEntropy(pass);
NumberFormat formatter = new DecimalFormat("#0.00");
this.updateRow(id, pass.length(), digits, upper, lower, countSpecialChars, Double.parseDouble(formatter.format(naiveEntropy)));
countSpecialChars = 0;
upper=digits=0;
lower = 0;
}
rs.close();
}
catch (SQLException e)
...
}
public void updateRow(int id, int length, int numbers, int upper,
int lower, int specialChars, double naiveEntropy )
{
PreparedStatement updatePassEntry = null;
String updateString = "update cwlCompatiblePassUnique " +
"set length = ?, numbers = ?, upper = ?, lower = ?, specialChars = ?, ShannonEntropy = ? where id = ?";
try {
this.conn.setAutoCommit(false);
updatePassEntry = this.conn.prepareStatement(updateString);
updatePassEntry.setInt(1, length);
...
updatePassEntry.setInt(7, id);
updatePassEntry.executeUpdate();
this.conn.commit();
}
catch (SQLException e)
...
}
Any ideas on what can be done?
Thanks
you call updateRow() method inside the rs.next() loop; which tries to make a SQL update on a SQL field (id) that is currently being processed inside your while (rs.next()) loop. this will raise the error you get. i suggest you write a method for pulling rs and storing them in java objects vector as a first step. this method will close the rs after exiting. then write another method to do both processing and update data on your cached vector objects .
something like this:
private void Vector<DataSet> getDataSet(){
Vector<DataSet> data=new Vector<DataSet>();
String query = "select id,pass from datatable";
try {
this.conn.setAutoCommit(false);
stmt = this.conn.createStatement();
stmt.setFetchSize(Integer.MIN_VALUE);
rs = stmt.executeQuery(query);
while (rs.next()) {
pass = rs.getString(2).trim();
id = rs.getInt(1);
data.addElement(new DataSet(id,pass));
}
}catch(Exception e){
// here close connection and rs
}
}
private void udpateData(Vector<dataSet> data){
//process data and update her
}
static class DataSet{
int id;
String pass;
//constructor here
}
Connection object should not hold multiple resultset object at a time.
After creation of ResultSet and Statement Objects, each has to close explicitly like,
resultSet.close()
statement.close()