I am optimizing a code written a long time ago by a developer that I never met. I came across a method that requires modification. The first thing that came to mind is to use stored procedure. Maybe there are better ways of achieving this hence this question.
The code goes this way:
public void execute()
{
String query = "select a, b, e from table1";
....
ResultSet rs = stmt.executeQuery(query);
String query2 = null;
List<Integer> list1 = ....
List<Integer> list2 = ....
while(rs.next)
{
query2 = "select count(*) as rowcount from vw_view1 where f='" + rs.getString("a") + "' and d='" + rs.getString("b") + "'";
.....
ResultSet rs2 = stmt2.executeQuery(query2);
list1.add(rs2.getInt(rowcount));
query3 = "select count(*) as rowcount from vw_view1 where c='" + rs.getString("a") + "' and e='" + rs.getString("e") + "'";
.....
ResultSet rs3 = stmt3.executeQuery(query3);
list2.add(rs3.getInt(rowcount));
}
}
Apart from using a stored procedure, is there a better way of avoiding unnecessary trips to the database in this method.
Try this query, check if it gives you the same result. With this you should have in one query directly all the value you wanted
SELECT c,
SUM(CASE WHEN vw1.d = tb1.b THEN 1 ELSE 0 END) as rowcountListOne,
SUM(CASE WHEN vw1.e = tb1.e THEN 1 ELSE 0 END) as rowcountListTwo
from vw_view1 vw1
left join table1 tb1 on vw1.c=tb1.a
GROUP BY c
In the new case you posted you should actually join by c and f:
SELECT c, f,
SUM(CASE WHEN (vw1.d = tb1.b) AND (vw1.f=tb1.a) THEN 1 ELSE 0 END) as rowcountListOne,
SUM(CASE WHEN (vw1.e = tb1.e) AND (vw1.c=tb1.a) THEN 1 ELSE 0 END) as rowcountListTwo
FROM vw_view1 vw1
LEFT JOIN table1 tb1 on (vw1.c=tb1.a OR vw1.f=tb1.a)
GROUP BY c, f
You should check also what Lennart said about COUNT, I think I should add on those case some NULL management, but I can't test it right now.
Maybe is enough if you check vw1.f not null in the case (and vw1.c in the other):
CASE WHEN (vw1.f IS NOT NULL) AND (vw1.d = tb1.b) AND (vw1.f=tb1.a) THEN 1 ELSE 0 END
You could just perform the counts at the same time as your initial select, removing the need to run two further queries for every row in rs:
SELECT t.a, t.b, t.e, t1.RowCount1, t2.RowCount2
FROM table1 AS t
LEFT JOIN
( SELECT c, d, COUNT(*) AS RowCount1
FROM vw_view1
GROUP BY c, d
) AS t2
ON t2.c = t.a
AND t2.d = t.b
LEFT JOIN
( SELECT c, e, COUNT(*) AS RowCount2
FROM vw_view1
GROUP BY c, e
) AS t3
ON t3.c = t.a
AND t3.e = t.e;
The initial select will be slower, but the subsequent loop will be much more efficient.
Related
i have a simple stored procedure. i use this to show data from my db. i want to optimize this query because i think this is slow. 31 seconds for 220.000 rows. so this is my query
BEGIN
sql_base :=
'SELECT
p.id
,p.cis
,p.account_no
,p.name
,p.address
,p.village_name
,p.postal_code
,p.subdistrict_name
,p.regency_name
,p.province_name
,p.country_name
,m.group_id
,i.name
,m.id
,m.name
,m.address
,m.village_name
,m.postal_code
,m.subdistrict_name
,m.regency_name
,m.province_name
,m.country_name
,listagg (mMcc.name, '','') within group (order by mMcc.name) merchantMccName
,o.mid
,o.name
,o.mcc_id outletMccId
,REPLACE(REPLACE(o.jenis_outlet,''ecomm'',''e-commerce''),''retail'',''toko'') jenisOutlet
,o.MODIFIEDDATE
,o.CREATEDDATE
,o.CC_testing_ACCOUNTNO outletCcAccountNo
,o.DEBIT_testing_ACCNO outletDebitAccountNo
,MCC.name outletMccName
,o.MCC_CODE outletMccCode
,m.code merchantCode
,p.jenisn abah jenisn abah
,p.segmenn abah segmenn abah
,p.IDENTITY_NO identityNo
,p.IDENTITY_TYPE identityType
,o.status status
,NVL(o.ISOUTLET, 0) isOutlet
,NVL(p.IS_ACCOUNT_testing, 0) isAccounttesting
,p.ACCOUNT_OTHERBANK_NO accountOtherbankNo
,p.ACCOUNT_OTHERBANK_BANKNAME accountOtherbankBankname
,p.ACCOUNT_OTHERBANK_NO pemilikOtherBankAccountNo
,o.ADDRESS1 outletAddress1
,o.ADDRESS2 outletAddress2
,o.VILLAGE_NAME outletVillageName
,o.POSTAL_CODE outletPostalCode
,o.SUBDISTRICT_NAME outletSubdistrictName
,o.REGENCY_NAME outletRegencyName
,o.PROVINCE_NAME outletProvinceName
,C E WHEN NVL(o.IS_CC_TESTING,0)=1 THEN ''TESTING'' ELSE TO_CHAR(o.CC_OTHERBANK_NAME) END outletCcBankName
,C E WHEN NVL(o.IS_DEBIT_TESTING,0)=1 THEN ''TESTING'' ELSE TO_CHAR(o.CC_OTHERBANK_NAME) END outletDebitBankName
,NVL(o.IS_CC_OTHERBANK,0) isCcOtherBank
,NVL(o.IS_CC_TESTING,0) is_cc_testing
,NVL(o.IS_DEBIT_testing,0) is_debit_testing
,o.CC_OTHERBANK_ACCNO ccOtherBankAccNo
,o.QRIS_STATIS_NMID nmid
,o.AGENT_BANK_CODE outletAgentBank
FROM PEMILIK p
LEFT JOIN mytable m ON p.ID = m.PEM_ID
LEFT JOIN mytable2 i ON m.GROUP_ID = i.ID
INNER JOIN mytable3 o ON m.ID = o.MERCH_ID
LEFT JOIN mytable4 mMcc ON m.ID = mMcc.MERCH_ID
LEFT JOIN mytable5 mcc ON MCC.ID = o.MCC_ID
GROUP BY
*SAME LIKE SELECT*;
final_result:='SELECT rec.*, count (*) over ()CountData FROM ('
|| sql_base || ') rec '
||WHERE_Q
||ORDERBY_Q||' '||PAGING_Q ;
OPEN O_RESULT_REC FOR
final_result;
i already tried to change the query but still i think my query still not good. any tips to improve my query? should i change the connection in java? or this is just query problem?
I am developing a web application (java-maven project) and there is a problem with the query I use. The UI of my web app has 2 search fields: PTT and ID.
The user needs to fill at least one of the fields to make a search. So both fields are nullable but not at the same time.
Before, I had only one field: PTR and it was showing a result array of size 52. (also getting the same number if
I execute select * from users where ptr='smthing' ). After that I added ID field and updated my query as below:
I execute this query in my webservice:
String query= "SELECT t.ptr, t.id ";
query+= "FROM users t ";
query+= "WHERE t.ptr = COALESCE(?, t.ptr) AND " ;
query+= "t.id = COALESCE(?, t.id) ";
and set the fields with the help of Prepared Statement.
Now if the ptr field is filled, but id field is left blank (this can be null or empty string) on the UI and user makes a search, result array size becomes 30. I compared with database
and it does not fetch the rows where ID is null. So coalesce is not what I need when both of its parameters (?, t.ptr) is null.
How can I fix this problem, any suggestions?
I think the logic you want is:
WHERE (t.ptr = ? OR ? IS NULL) AND
(t.id = ? OR ? IS NULL)
I would recommend using named parameters, so you don't have to pass them in twice.
Check this statement:
String query= "SELECT t.ptr, t.id ";
query+= "FROM users t ";
query+= "WHERE (t.ptr = ? OR 1 = ?)"
query+= " AND " ;
query+= "(t.id = ? OR 1 = ?)";
You see that for each of t.id and t.ptr there is a counterpart parameter. In total there will be 4 parameters.
You say that at least 1 of t.id or t.ptr has a valid value, so there are 2 cases:
[1] t.id and t.ptr both have valid values.
For both the counterpart parameters you pass 0 and the query becomes:
"SELECT t.ptr, t.id FROM users t WHERE (t.ptr = valueptr OR 1 = 0) AND (t.id = valueid OR 1 = 0)"
In the WHERE part:
t.ptr = valueptr OR 1 = 0 is equivalent to t.ptr = valueptr, and
t.id = valueid OR 1 = 0 is equivalent to t.id = valueid,
and the query finally becomes:
"SELECT t.ptr, t.id FROM users t WHERE t.ptr = valueptr AND t.id = valueid"
[2] from t.id or t.ptr only one has a valid value, let's say this is t.ptr.
For the counterpart of t.ptr you pass 0, for t.id you pass -1 (or any other non existing value) and for the counterpart of t.id you pass 1 and the query becomes:
"SELECT t.ptr, t.id FROM users t WHERE (t.ptr = valueptr OR 1 = 0) AND (t.id = -1 OR 1 = 1)"
In the WHERE part:
t.ptr = valueptr OR 1 = 0 is equivalent to t.ptr = valueptr, and
t.id = -1 OR 1 = 1 is equivalent to true because 1 = 1 is always true,
and the query finally becomes:
"SELECT t.ptr, t.id FROM users t WHERE (t.ptr = valueptr OR 1 = 0)"
equivalent to:
"SELECT t.ptr, t.id FROM users t WHERE (t.ptr = valueptr)"
(In the case where only t.id has a valid value then you pass an invalid value for t.ptr and 1 for its counterpart and for the counterpart if t.id you pass 0.)
Maybe it seems complicated but it's working and it can be extended for more than 2 columns.
Oracle has build function for this. But it's hard to understand how to use its.
lnnvl(a = b) = true if (a != b ) or ( a = null ) or (b = null)
in your case
WHERE lnnvl(t.ptr != ? ) AND lnnvl( t.id != ?)
LNNVL
I have sql query which has a conditions(in where clause, having clause, or in join). I need to find condition which contains value with brackets like this: {some name} and replace that condition with 1=1(in some cases) with java.
E.g.
Select *
From customer c inner join address a on c.id = a.customer_id
where c.id > {var1} AND (c.name LIKE {var2} OR a.city = {var3}) AND ... (there could be written all posible operaters and conditions which are allowed by sql)
if var1 = 2, var2 = *ALL, var3 = 'aa' then query should looks like
Select *
From customer c inner join address a on c.id = a.customer_id
where c.id > 2 AND (1=1 OR a.city = 'aa')
I thought to split where condition with AND|OR then check if it contains {var2} then replace with 1=1, but this will not work. (in above described example after the split it will be (c.name LIKE {var2} so ( also will be replaced).
Does somebody came across with this situation and how solved.
Is there any open source libraray which will find and replace or how can I do that with regex ?
First I assume you are not able to dynamically build your sql. For this you would go the trivial way and could build your sql using string concats, e.g.
"select ... where c.id > " + myvalue + " ... "
The use of regular expressions for sqls will work, if you can guarantee that your expression will only find the specific parts of your sql and not more. This sounds a bit trivial but is hard to achieve for specific sqls, e.g.:
select '{var2}' from mytable
Should {var2} replaced here or not? I think not, because it is part of a string literal and is not part of a where statement.
Therefore you need a more structured look at your sql which is provided by a sql parser.
Using e.g. JSqlParser (https://github.com/JSQLParser/JSqlParser) you would be able to identify the parts of your where statement and replace the parts of your SQL in a controlled manner.
But to get this parsing going, you have to replace your {val} with something that is SQL conform, e.g. __val__.
So your replacement could be achieved with something like the following code. It assumes, that the replacement is only done within the right expression, but you could easily change that.
String sqlTxt = "Select * from customer c inner join address a on c.id = a.customer_id where c.id > {var1} AND (c.name LIKE {var2} OR a.city = {var3})";
//replace macro constructs
String sql = sqlTxt.replace("{", "__").replace("}", "__");
//build replacement data
Map<String, String> data = new HashMap<>();
data.put("__var1__", "2");
data.put("__var2__", "*ALL");
data.put("__var3__", "'aa'");
//parse sql
Select select = (Select) CCJSqlParserUtil.parse(sql);
StringBuilder b = new StringBuilder(sql);
//rewrite sql to fit your needs
((PlainSelect) select.getSelectBody()).getWhere().accept(new ExpressionVisitorAdapter() {
int delta = 0; //to correct the position due to former replacements
#Override
protected void visitBinaryExpression(BinaryExpression expr) {
if (expr instanceof ASTNodeAccess) {
if (expr.getRightExpression() instanceof Column) {
Column c = ((Column) expr.getRightExpression());
if (data.containsKey(c.getColumnName())) {
if ("__var2__".equals(c.getColumnName())) {
delta = replaceASTNodeWith(b, delta, (ASTNodeAccess) expr, "1=1");
} else {
delta = replaceASTNodeWith(b, delta, (ASTNodeAccess) expr,
expr.getLeftExpression() + expr.getStringExpression() + data.get(c.getColumnName()));
}
}
}
}
super.visitBinaryExpression(expr);
}
});
System.out.println("parsed sql = " + select.toString());
System.out.println("changed sql = " + b.toString());
The replacement in your sql is somewhat tricky due to the position change of former replacements.
//do the text replacement within the sql
private static int replaceASTNodeWith(StringBuilder sql, int delta, ASTNodeAccess node, String expr) {
sql.replace(
node.getASTNode().jjtGetFirstToken().absoluteBegin + delta - 1,
node.getASTNode().jjtGetLastToken().absoluteEnd + delta - 1,
expr);
return delta + expr.length()
- (node.getASTNode().jjtGetLastToken().absoluteEnd - node.getASTNode().jjtGetFirstToken().absoluteBegin);
}
This will not answer your question, but just more a suggestion on how I used the 1=1 approach.
I think you might want to write something like this
String query = "select * from customer c inner join address a on c.id = a.customer_id where 1=1"
if(var1 != null)
query += "and c.id > " + var1
if(var2 != null)
query += "and c.name LIKE " + var2
if(var3 != null)
query += "and a.city = " + var3
for example you want to had that logic, it will be a little tricky that it will look like this.
//you should realize that 1=1 here to make sure that we add "AND " command
String query = "select * from customer c inner join address a on c.id = a.customer_id where 1=1"
if(var1 != null)
query += "and c.id > " + var1
if(var2 != null && var3 != null) { //because if var2 or var3 is all, then return all.
query += "and ( c.name LIKE " + var2;
query += "or a.city = " + var3 + ")";
}
Hope this help
use the following:-
In case of single word variable between {singleWord}
queryString = queryString.replaceAll("\\{[A-Za-z0-9]+\\}", "1=1");
In case of multi word variable between {multi word}
queryString = queryString.replaceAll("\\{[A-Za-z0-9\\s]+\\}", "1=1");
Refer for more details:- https://docs.oracle.com/javase/1.5.0/docs/api/java/util/regex/Pattern.html#sum
I have written a function that I would like to call in Java. But I don't think it is able to do anything with the query that I passed. Following is my code from java:
String QUERY_LOCATION = "select (license_plate) as test from carInst( (select category_name from reservation where rid = ?) , (select lname from reservation where rid = ?))";
//PreparedStatement check_location = null;
PreparedStatement check_location = connection.prepareStatement(QUERY_LOCATION);
check_location.setInt(1, rid);
check_location.setInt(2, rid);
rs = check_location.executeQuery();
if (rs.next()) {
System.out.print("Car found: "+rs.getString("test")+"\n");
license_plate = rs.getString("test");
update_reservation.setString(5, license_plate);
bool = false;
} else {
System.out
.print("There is no car available\n");
}
And following is my stored procedure written in PL/pgSQL (PostgreSQL):
CREATE OR REPLACE FUNCTION carInst(cname varchar(20), loc varchar(20) )
RETURNS TABLE (license_plate varchar(6) ) AS $$
BEGIN
DECLARE cur CURSOR
FOR SELECT carinstance.license_plate, carmodel.category_name, carinstance.lname FROM carinstance,carmodel
WHERE carinstance.mid = carmodel.mid ;
BEGIN
FOR rec IN cur LOOP
RETURN QUERY SELECT distinct carinstance.license_plate FROM Carinstance
WHERE rec.category_name = cname
AND rec.lname = loc
AND rec.license_plate=carinstance.license_plate;
END LOOP;
END;
END;
$$ LANGUAGE plpgsql;
When I run the code in Java, the print statement prints a null value for Car found. I would really appreciate some help here.
Problems
Most importantly, the query in the LOOP is nonsense. You select rows from carinstance, but all conditions are on rec. This select all rows multiple times.
One END too many. FOR has no END, only LOOP has.
Whenever you feel the temptation to work with an explicit cursor in plpgsql, stop right there. Chances are, you are doing it wrong. A FOR loop has an implicit cursor anyway.
Don't mess with mixed case identifiers without double quotes. I converted all identifiers to lower case.
You use one simple query, spread out over a cursor and another query. This can all be much simpler.
Solution
Try this simple SQL function instead:
CREATE OR REPLACE FUNCTION car_inst(_cname text, _loc text)
RETURNS TABLE (license_plate text)
LANGUAGE sql AS
$func$
SELECT DISTINCT ci.license_plate
FROM carmodel cm
JOIN carinstance ci USING (mid)
WHERE cm.category_name = $1
AND ci.lname = $2
$func$;
Call:
SELECT license_plate AS test FROM car_inst(
(SELECT category_name FROM reservation WHERE rid = ?)
, (SELECT lname FROM reservation WHERE rid = ?)
);
Or build it all into your function:
CREATE OR REPLACE FUNCTION car_inst(_cname text, _loc text)
RETURNS TABLE (license_plate text)
LANGUAGE sql AS
$func$
SELECT DISTINCT ci.license_plate
FROM carmodel cm
JOIN carinstance ci USING (mid)
JOIN reservation r1 ON r1.category_name = cm.category_name
JOIN reservation r2 ON r2.lname = ci.lname
WHERE r1.rid = $1
AND r2.rid = $2;
$func$;
Call:
"SELECT license_plate AS test FROM car_inst(? , ?)";
Remember: The OUT parameter license_plate is visible anywhere in the body of the function. Therefore you must table-qualify the column of the same name at all times to prevent a naming collision.
DISTINCT may or may not be redundant.
I'm using posgresql as a database and java as programming language with hibernate. My problem is this query:
select cast(sum(CASE WHEN p.nropack > 0 THEN p.nropack ELSE 0 END) as integer),
cast(sum(CASE WHEN p.nropack < 0 THEN p.nropack ELSE 0 END) as integer),
cast(p.fechareg as date)
from pronostico p
inner join aeropuerto a on (a.idaeropuerto=p.idaeropuerto)
inner join ciudad c on (a.idciudad=c.idciudad)
inner join pais ps on (ps.idpais=c.idpais)
inner join continente ct on (ct.idcontinente=ps.idcontinente)
where c.idciudad=105
group by cast (p.fechareg as date);
As a result I get:
sum;sum;fechareg
30;-15;"2012-11-15"
But when I use it in my program:
public ArrayList<RepKardex> listarKardex(int ciud){
ciud=105;
ArrayList<RepKardex> listaKardex = new ArrayList<>();
Session session = HibernateUtil.getSessionFactory().openSession();
Transaction tx = session.beginTransaction();
String q = "select cast(sum( case when p.nropack > 0 then p.nropack ELSE 0 end ) as integer), "
+ "cast(sum( case when p.nropack < 0 then p.nropack ELSE 0 end ) as integer), "
+ "cast(p.fechareg as date) "
+ "from Pronostico p "
+ "inner join Aeropuerto a on (p.idaeropuerto = a.idaeropuerto) "
+ "inner join Ciudad c on (a.idciudad = c.idciudad) "
+ "inner join Pais ps on (c.idpais = ps.idpais) "
+ "inner join Continente ct on (ct.idcontinente = ps.idcontinente) "
+ "where c.idciudad = :ciud "
+ "group by cast(p.fechareg as date) ";
Query query = session.createSQLQuery(q);
query.setInteger("ciud", ciud);
List lista = query.list();
Iterator iter = lista.iterator();
while (iter.hasNext()) {
Object[] row = (Object[]) iter.next();
if (row!=null){
System.out.println("entrantes : "+(Integer)row[0]);
System.out.println("salientes : "+(Integer)row[1]);
RepKardex rep = new RepKardex((int)row[0],(int)row[1],(Date)row[2]);
listaKardex.add(rep);
}
}
tx.commit();
session.close();
return listaKardex;
}
It prints
entrantes: 30
salida: 30
Can someone help me figure out why it repeats the positive numbers even when I use the case statement inside the query? Thanks in advance.
You can simplify your query with LEAST and GREATEST:
SELECT sum(GREATEST(p.nropack, 0))::int AS entrantes
,sum(LEAST(p.nropack, 0))::int AS salida
,p.fechareg::date AS fechareg
from pronostico p
JOIN aeropuerto a ON a.idaeropuerto = p.idaeropuerto
JOIN ciudad c ON c.idciudad = a.idciudad
JOIN pais ps ON ps.idpais = c.idpais
JOIN continente ct ON ct.idcontinente = ps.idcontinente
where c.idciudad = 105
GROUP BY p.fechareg::date;
Other than that it looks just fine. I see no way the second column could return a positive number. Something else must go wrong here.
Edit:
Comment by #hvd helped to find that the client code seems to get confused by identical column names (both sum columns defaulted to "sum"). Explicit column aliases seem to fix this.