We Keep Coding

Postgres Query returning empty result set from Java

String query = "SELECT distinct on (l->>'test_id') l->>'test_id' test_id,
l->>'result' result,l->>'indication' indication
FROM Ltable,
jsonb_array_elements(lines) with ordinality a(l) WHERE pat_id = ?
AND result_received <= ? AND (l->>'test_id' = ? OR l->>'test_id' = ? )
and archived = false ORDER BY l->>'test_id', result_received DESC LIMIT 2";
jdbcTemplate.query(query, rs -> {
while(rs.next()){
}
}val1, val2, val3, val4);
The above query when executed from pgAdmin returns some rows but when executed from a java app returns an empty resultSet, the line while(rs.next()) never gets entered.
I have logged the query parameters to make sure the correct data is being passed into the query.
Please help look at at and let me know if there's a mistake I'm making somewhere.

How to prevent SQL injection when the statement has a dynamic table name?

I am having code something like this.
final PreparedStatement stmt = connection
.prepareStatement("delete from " + fullTableName
+ " where name= ?");
stmt.setString(1, addressName);
Calculation of fullTableName is something like:
public String getFullTableName(final String table) {
if (this.schemaDB != null) {
return this.schemaDB + "." + table;
}
return table;
}
Here schemaDB is the name of the environment(which can be changed over time) and table is the table name(which will be fixed).
Value for schemaDB is coming from an XML file which makes the query vulnerable to SQL injection.
Query: I am not sure how the table name can be used as a prepared statement(like the name used in this example), which is the 100% security measure against SQL injection.
Could anyone please suggest me, what could be the possible approach to deal with this?
Note: We can be migrated to DB2 in future so the solution should compatible with both Oracle and DB2(and if possible database independent).

JDBC, sort of unfortunately, does not allow you to make the table name a bound variable inside statements. (It has its reasons for this).
So you can not write, or achieve this kind of functionnality :
connection.prepareStatement("SELECT * FROM ? where id=?", "TUSERS", 123);
And have TUSER be bound to the table name of the statement.
Therefore, your only safe way forward is to validate the user input. The safest way, though, is not to validate it and allow user-input go through the DB, because from a security point of view, you can always count on a user being smarter than your validation.
Never trust a dynamic, user generated String, concatenated inside your statement.
So what is a safe validation pattern ?
Pattern 1 : prebuild safe queries
1) Create all your valid statements once and for all, in code.
Map<String, String> statementByTableName = new HashMap<>();
statementByTableName.put("table_1", "DELETE FROM table_1 where name= ?");
statementByTableName.put("table_2", "DELETE FROM table_2 where name= ?");
If need be, this creation itself can be made dynamic, with a select * from ALL_TABLES; statement. ALL_TABLES will return all the tables your SQL user has access to, and you can also get the table name, and schema name from this.
2) Select the statement inside the map
String unsafeUserContent = ...
String safeStatement = statementByTableName.get(usafeUserContent);
conn.prepareStatement(safeStatement, name);
See how the unsafeUserContent variable never reaches the DB.
3) Make some kind of policy, or unit test, that checks that all you statementByTableName are valid against your schemas for future evolutions of it, and that no table is missing.
Pattern 2 : double check
You can 1) validate that the user input is indeed a table name, using an injection free query (I'm typing pseudo sql code here, you'd have to adapt it to make it work cause I have no Oracle instance to actually check it works) :
select * FROM
(select schema_name || '.' || table_name as fullName FROM all_tables)
WHERE fullName = ?
And bind your fullName as a prepared statement variable here. If you have a result, then it is a valid table name. Then you can use this result to build a safe query.
Pattern 3
It's sort of a mix between 1 and 2.
You create a table that is named, e.g., "TABLES_ALLOWED_FOR_DELETION", and you statically populate it with all tables that are fit for deletion.
Then you make your validation step be
conn.prepareStatement(SELECT safe_table_name FROM TABLES_ALLOWED_FOR_DELETION WHERE table_name = ?", unsafeDynamicString);
If this has a result, then you execute the safe_table_name. For extra safety, this table should not be writable by the standard application user.
I somehow feel the first pattern is better.

You can avoid attack by checking your table name using regular expression:
if (fullTableName.matches("[_a-zA-Z0-9\\.]+")) {
final PreparedStatement stmt = connection
.prepareStatement("delete from " + fullTableName
+ " where name= ?");
stmt.setString(1, addressName);
}
It's impossible to inject SQL using such a restricted set of characters.
Also, we can escape any quotes from table name, and safely add it to our query:
fullTableName = StringEscapeUtils.escapeSql(fullTableName);
final PreparedStatement stmt = connection
.prepareStatement("delete from " + fullTableName
+ " where name= ?");
stmt.setString(1, addressName);
StringEscapeUtils comes with Apache's commons-lang library.

I think that the best approach is to create a set of possible table names and check for existance in this set before creating query.
Set<String> validTables=.... // prepare this set yourself
if(validTables.contains(fullTableName))
{
final PreparedStatement stmt = connection
.prepareStatement("delete from " + fullTableName
+ " where name= ?");
//and so on
}else{
// ooooh you nasty haker!
}

create table MYTAB(n number);
insert into MYTAB values(10);
commit;
select * from mytab;
N
10
create table TABS2DEL(tname varchar2(32));
insert into TABS2DEL values('MYTAB');
commit;
select * from TABS2DEL;
TNAME
MYTAB
create or replace procedure deltab(v in varchar2)
is
LvSQL varchar2(32767);
LvChk number;
begin
LvChk := 0;
begin
select count(1)
into LvChk
from TABS2DEL
where tname = v;
if LvChk = 0 then
raise_application_error(-20001, 'Input table name '||v||' is not a valid table name');
end if;
exception when others
then raise;
end;
LvSQL := 'delete from '||v||' where n = 10';
execute immediate LvSQL;
commit;
end deltab;
begin
deltab('MYTAB');
end;
select * from mytab;
no rows found
begin
deltab('InvalidTableName');
end;
ORA-20001: Input table name InvalidTableName is not a valid table name ORA-06512: at "SQL_PHOYNSAMOMWLFRCCFWUMTBQWC.DELTAB", line 21
ORA-06512: at "SQL_PHOYNSAMOMWLFRCCFWUMTBQWC.DELTAB", line 16
ORA-06512: at line 2
ORA-06512: at "SYS.DBMS_SQL", line 1721

Getting ResultSet from stored procedure within another stored procedure

I have a stored procedure that calls another stored procedure. The inner stored procedure returns a result set. After using a CallableStatement to execute the calling stored procedure I am unable to get the result set returned by called stored procedure.
I tried both execute and executeQuery for execution of callable statement. When I execute the calling stored procedure from SQL Server I am getting proper results.
Calling procedure:-
ALTER PROC [User].[Get_Data]
(#UserID NVARCHAR(20))
AS
BEGIN
Select 'User Data'
Exec [Order].[Get_Order] #UserID
END
Called procedure:-
ALTER PROC [Order].[Get_Order]
(#UserID NVARCHAR(20))
AS
BEGIN
Select * from orders where userId=#UserID
END

Your outer stored procedure is returning two result sets:
The results from Select 'User Data'
The results from Exec [Order].[Get_Order] #UserID
You need to call .getMoreResults() in order to retrieve the second result set, e.g.,
CallableStatement cs = connection.prepareCall("{CALL Get_Data (?)}");
cs.setString(1, "gord");
ResultSet rs = cs.executeQuery();
System.out.println("[First result set]");
while (rs.next()) {
System.out.printf("(No column name): %s%n", rs.getString(1));
}
cs.getMoreResults();
rs = cs.getResultSet();
System.out.println();
System.out.println("[Second result set]");
while (rs.next()) {
System.out.printf("userId: %s, orderId: %s%n",
rs.getString("userId"), rs.getString("orderId"));
}
producing
[First result set]
(No column name): User Data
[Second result set]
userId: gord, orderId: order1
userId: gord, orderId: order2
(Tested using mssql-jdbc-6.2.1.jre8.jar connecting to SQL Server 2014.)
For more details, see
How to get *everything* back from a stored procedure using JDBC

You cannot select the results of a stored procedure directly within SQL Server itself. You need to first insert the result into a temp table as per example below.
Example use:
-- Create a tempoary table to store the results.
CREATE TABLE #UserOrderDetail
(
UserData NVARCHAR(50) -- Your columns here
)
-- Insert result into temp table.
-- Note that the columns returned from procedure has to match columns in your temp table.
INSERT INTO #UserOrderDetail
EXEC [Order].[Get_Order] #UserID
-- Select the results out of the temp table.
SELECT *
FROM #UserOrderDetail
If the intent is to simply return one or more result sets to a client application, you should ensure that the SET NOCOUNT ON statement is added to the top of your stored procedures, this will prevent SQL Server from sending the DONE_IN_PROC messages to the client for each statement in the stored procedure. Database libraries like ODBC, JDBC and OLEDB can get confused by the row counts returned by the various insert and update statements executed within a SQL Server stored procedures. Your original procedure will look as follow:
ALTER PROC [User].[Get_Data]
(
#UserID NVARCHAR(20)
)
AS
BEGIN
SET NOCOUNT ON
SELECT 'User Data'
EXEC [Order].[Get_Order] #UserID
END

The correct way to do this with JDBC
Getting this right with JDBC is quite hard. The accepted answer by Gord Thompson might work, but it doesn't follow the JDBC spec to the word, so there might be edge cases where it fails, e.g. when there are interleaved update counts (known or accidental), or exceptions / messages.
I've blogged about the correct approach in detail here. The Oracle version is even more tricky, in case you need it. So here it goes:
// If you're daring, use an infinite loop. But you never know...
fetchLoop:
for (int i = 0, updateCount = 0; i < 256; i++) {
// Use execute(), instead of executeQuery() to handle
// leading update counts or exceptions
boolean result = (i == 0)
? s.execute()
: s.getMoreResults();
// Warnings here
SQLWarning w = s.getWarnings();
for (int j = 0; j < 255 && w != null; j++) {
System.out.println("Warning : " + w.getMessage());
w = w.getNextWarning();
}
// Don't forget this
s.clearWarnings();
if (result)
try (ResultSet rs = s.getResultSet()) {
System.out.println("Result :");
while (rs.next())
System.out.println(" " + rs.getString(1));
}
else if ((updateCount = s.getUpdateCount()) != -1)
System.out.println("Update Count: " + updateCount);
else
break fetchLoop;
}
Using jOOQ
Note that in case you're using jOOQ, you could leverage code generation for your stored procedures and call the simplified API to do this in a few lines only:
GetDatap = new GetData();
p.setUserId("gord");
p.execute(configuration);
Results results = p.getResults();
for (Result<?> result : results)
for (Record record : result)
System.out.println(record);
Disclaimer: I work for the company behind jOOQ

Using a database API cursor with JDBC and SQLServer to select batch results

SOLVED (See answer below.)
I did not understand my problem within the proper context. The real issue was that my query was returning multiple ResultSet objects, and I had never come across that before. I have posted code below that solves the problem.
PROBLEM
I have an SQL Server database table with many thousand rows. My goal is to pull the data back from the source database and write it to a second database. Because of application memory constraints, I will not be able to pull the data back all at once. Also, because of this particular table's schema (over which I have no control) there is no good way for me to tick off the rows using some sort of ID column.
A gentleman over at the Database Administrators StackExchange helped me out by putting together something called a database API cursor, and basically wrote this complicated query that I only need to drop my statement into. When I run the query in SQL Management Studio (SSMS) it works great. I get all the data back, a thousand rows at a time.
Unfortunately, when I try to translate this into JDBC code, I get back the first thousand rows only.
QUESTION
Is it possible using JDBC to retrieve a database API cursor, pull the first set of rows from it, allow the cursor to advance, and then pull the subsequent sets one at a time? (In this case, a thousand rows at a time.)
SQL CODE
This gets complicated, so I'm going to break it up.
The actual query can be simple or complicated. It doesn't matter. I've tried several different queries during my experimentation and they all work. You just basically drop it into the the SQL code in the appropriate place. So, let's take this simple statement as our query:
SELECT MyColumn FROM MyTable;
The actual SQL database API cursor is far more complicated. I will print it out below. You can see the above query buried in it:
-- http://dba.stackexchange.com/a/82806
DECLARE #cur INTEGER
,
-- FAST_FORWARD | AUTO_FETCH | AUTO_CLOSE
#scrollopt INTEGER = 16 | 8192 | 16384
,
-- READ_ONLY, CHECK_ACCEPTED_OPTS, READ_ONLY_ACCEPTABLE
#ccopt INTEGER = 1 | 32768 | 65536
,#rowcount INTEGER = 1000
,#rc INTEGER;
-- Open the cursor and return the first 1,000 rows
EXECUTE #rc = sys.sp_cursoropen #cur OUTPUT
,'SELECT MyColumn FROM MyTable'
,#scrollopt OUTPUT
,#ccopt OUTPUT
,#rowcount OUTPUT;
IF #rc <> 16 -- FastForward cursor automatically closed
BEGIN
-- Name the cursor so we can use CURSOR_STATUS
EXECUTE sys.sp_cursoroption #cur
,2
,'MyCursorName';
-- Until the cursor auto-closes
WHILE CURSOR_STATUS('global', 'MyCursorName') = 1
BEGIN
EXECUTE sys.sp_cursorfetch #cur
,2
,0
,1000;
END;
END;
As I've said, the above creates a cursor in the database and asks the database to execute the statement, keep track (internally) of the data it's returning, and return the data a thousand rows at a time. It works great.
JDBC CODE
Here's where I'm having the problem. I have no compilation problems or run-time problems with my Java code. The problem I am having is that it returns only the first thousand rows. I don't understand how to utilize the database cursor properly. I have tried variations on the Java basics:
// Hoping to get all of the data, but I only get the first thousand.
ResultSet rs = stmt.executeQuery(fq.getQuery());
while (rs.next()) {
System.out.println(rs.getString("MyColumn"));
}
I'm not surprised by the results, but all of the variations I've tried produce the same results.
From my research it seems like the JDBC does something with database cursors when the database is Oracle, but you have to set the data type returned in the result set as an Oracle cursor object. I'm guessing there is something similar with SQL Server, but I have been unable to find anything yet.
Does anyone know of a way?
I'm including example Java code in full (as ugly as that gets).
// FancyQuery.java
import java.sql.*;
public class FancyQuery {
// Adapted from http://dba.stackexchange.com/a/82806
String query = "DECLARE #cur INTEGER\n"
+ " ,\n"
+ " -- FAST_FORWARD | AUTO_FETCH | AUTO_CLOSE\n"
+ " #scrollopt INTEGER = 16 | 8192 | 16384\n"
+ " ,\n"
+ " -- READ_ONLY, CHECK_ACCEPTED_OPTS, READ_ONLY_ACCEPTABLE\n"
+ " #ccopt INTEGER = 1 | 32768 | 65536\n"
+ " ,#rowcount INTEGER = 1000\n"
+ " ,#rc INTEGER;\n"
+ "\n"
+ "-- Open the cursor and return the first 1,000 rows\n"
+ "EXECUTE #rc = sys.sp_cursoropen #cur OUTPUT\n"
+ " ,'SELECT MyColumn FROM MyTable;'\n"
+ " ,#scrollopt OUTPUT\n"
+ " ,#ccopt OUTPUT\n"
+ " ,#rowcount OUTPUT;\n"
+ " \n"
+ "IF #rc <> 16 -- FastForward cursor automatically closed\n"
+ "BEGIN\n"
+ " -- Name the cursor so we can use CURSOR_STATUS\n"
+ " EXECUTE sys.sp_cursoroption #cur\n"
+ " ,2\n"
+ " ,'MyCursorName';\n"
+ "\n"
+ " -- Until the cursor auto-closes\n"
+ " WHILE CURSOR_STATUS('global', 'MyCursorName') = 1\n"
+ " BEGIN\n"
+ " EXECUTE sys.sp_cursorfetch #cur\n"
+ " ,2\n"
+ " ,0\n"
+ " ,1000;\n"
+ " END;\n"
+ "END;\n";
public String getQuery() {
return this.query;
}
public static void main(String[ ] args) throws Exception {
String dbUrl = "jdbc:sqlserver://tc-sqlserver:1433;database=MyBigDatabase";
String user = "mario";
String password = "p#ssw0rd";
String driver = "com.microsoft.sqlserver.jdbc.SQLServerDriver";
FancyQuery fq = new FancyQuery();
Class.forName(driver);
Connection conn = DriverManager.getConnection(dbUrl, user, password);
Statement stmt = conn.createStatement();
// We expect to get 1,000 rows at a time.
ResultSet rs = stmt.executeQuery(fq.getQuery());
while (rs.next()) {
System.out.println(rs.getString("MyColumn"));
}
// Alas, we've only gotten 1,000 rows, total.
rs.close();
stmt.close();
conn.close();
}
}

I figured it out.
stmt.execute(fq.getQuery());
ResultSet rs = null;
for (;;) {
rs = stmt.getResultSet();
while (rs.next()) {
System.out.println(rs.getString("MyColumn"));
}
if ((stmt.getMoreResults() == false) && (stmt.getUpdateCount() == -1)) {
break;
}
}
if (rs != null) {
rs.close();
}
After some additional googling, I found a bit of code posted back in 2004:
http://www.coderanch.com/t/300865/JDBC/databases/SQL-Server-JDBC-Registering-cursor
The gentleman who posted the snippet that I found helpful (Julian Kennedy) suggested: "Read the Javadoc for getUpdateCount() and getMoreResults() for a clear understanding." I was able to piece it together from that.
Basically, I don't think I understood my problem well enough at the outset in order to phrase it correctly. What it comes down to is that my query will be returning the data in multiple ResultSet instances. What I needed was a way to not merely iterate through each row in a ResultSet but, rather, iterate through the entire set of ResultSets. That's what the code above does.

If you want all records from the table, just do "Select * from table".
The only reason to retrieve in chunks is if there is some intermediate place for the data: e.g. if you are showing it on the screen, or storing it in memory.
If you are simply reading from one and inserting to another, just read everything from the first.You will not get any better performance by trying to retrieve in batches. If there is a difference, it will be negative. Frame your query in a way that brings back everything. The JDBC software will handle all the other breaking-up and reconstituting that you need.
However, you should batch the update/insert side of things.
The set-up would create two statements on the two connections:
Statement stmt = null;
ResultSet rs = null;
PreparedStatement insStmt = null;
stmt = conDb1.createStatement();
insStmt = conDb2.prepareStament("insert into tgt_db2_table (?,?,?,?,?......etc. ?,?) ");
rs = stmt.executeQuery("select * from src_db1_table");
Then, loop over the select as normal, but use batching on the target.
int batchedRecordCount = 0;
while (rs.next()) {
System.out.println(rs.getString("MyColumn"));
//Here you read values from the cursor and set them to the insStmt ...
String field1 = rs.getString(1);
String field2 = rs.getString(2);
int field3 = rs.getInt(3);
//--- etc.
insStmt.setString(1, field1);
insStmt.setString(2, field2);
insStmt.setInt(3, field3);
//----- etc. for all the fields
batchedRecordCount++;
insStmt.addBatch();
if (batchRecordCount > 1000) {
insStmt.executeBatch();
}
}
if (batchRecordCount > 0) {
//Finish of the final (partial) set of records
insStmt.executeBatch();
}
//Close resources...

Is it possible to use "WHERE" clause to select all records in SQL Statement?

Good Evening, I am curious if it is possible to make a WHERE-clause in a SQL statement which can show all records?
Below some explanation:
Random SQL Statement (Java)-(JSP example), Normal Situation
String SqlStatement = "SELECT * FROM table_example WHERE First_Col = '<%=passVar%>' ";
db.query(SqlStatement );
//........
//........
What if the passVar is 'ALL', and we need to prompt all the records out when passVar = All? I know I can do it with if-else and check if the passVar is "ALL" then query the without-WHERE statement to make it work..
**without-WHERE statement (Java)-(JSP example)**
if(<%=passVar%> == "ALL") {
SqlStatement = "SELECT * FROM table_example";
} else {
SqlStatement = "SELECT * FROM table_example WHERE First_Col = '<%=passVar%>' ";
}
but can I just code one SQL statement to make all the records prompt? Something like below:
(Java)-(JSP example)
String ShowAll = "";
if(<%=passVar%> == "ALL") {
ShowAll = *;
} else {
ShowAll = <%=passVar%>;
}
SqlStatement = "SELECT * FROM table_example WHERE First_Col = ShowAll ";

Try with WHERE 1=1::
Select * from myTable WHERE 1=1

This also works:
WHERE columnname LIKE '%'
Except for NULL values.

where 1=1 worked for me, Although where clause was being used all records were selected.
You can also try
SELECT * FROM Customers
WHERE CustomerID=CustomerID; /* query */
or
[any_column_name]=[column_name_in_LHL]
(LHL=left hand side.)
copy the query and
click here to try code

It would be better to differ the 2 situations and make 2 queries out of it.
If there is no where condition then the DB does not need to evaluate it (potencially faster)
The source code/debugging output is clearer.

Consider moving the special case inside the query itself, e.g.
SELECT * FROM table_example WHERE '<%=passVar%>' IN ('ALL', First_Col)

Try with wildcard value '%' but I would recommend to use a Factory here to create the SQL statement, what you are trying to do smells a bit.

Something else you could do, is making that combination of code and SQL a single query. Which means the IF..ELSE will be in SQL language.
Check these links for some more info:
MySQL
Using If else in SQL Select statement

On sqlserver you can make proc:
create proc select_all_on_null
#a int
as
begin
select * from Records where (#a is null or Record_id=#a )
end
When you select be your program:
make #a in null will select all
if i is numder there will select row with this id