I want to export huge data from oracle to csv file. so i used simple JDBC select statement to get data in memory but and then write it to file, But data is very large of i am getting Out of memory exception. So i thought of using CallableStatement to call Stored Procedure which will return CURSOR with ResultSet as below :-
String getDBTableCursorSql = "{call getDBTableCursor(?,?)}";
callableStatement = dbConnection.prepareCall(getDBTableCursorSql);
callableStatement.setString(1, "test");
callableStatement.registerOutParameter(2, OracleTypes.CURSOR);
// execute getDBTableCursorSqlstore procedure
callableStatement.executeUpdate();
// get cursor and cast it to ResultSet
rs = (ResultSet) callableStatement.getObject(2);
// loop it like normal
while (rs.next()) {
String userid = rs.getString("ID");
String userName = rs.getString("NAME");
..
..
}
Oracle Proc :-
CREATE OR REPLACE PROCEDURE getDBTableCursor(
p_username IN DBUSER.USERNAME%TYPE,
c_dbuser OUT SYS_REFCURSOR)
IS
BEGIN
OPEN c_dbuser FOR
SELECT * FROM CUSTOMER WHERE USERNAME LIKE p_username || '%';
END;
Question 1 :-
does above ResultSet will fetch all the data in single shot ? or it will go to database for each rs.next(),
Question 2:-
is there any other approach which can deal with large data export to file in java using chunks so it wont get Out of memory issue?
I can't use pagination in this condition because of requirement.
Regarding your first question: the Oracle jdbc driver by default fetches 10 rows at a time. This can be verified or set to other value via standard jdbc:
https://docs.oracle.com/cd/E11882_01/java.112/e16548/resltset.htm#JJDBC28621
Related
I am using snowflake-jdbc-3.13.8.jar and jdk1.8.0_151 to implement snowflake stored procedure which has an OUT parameter. I am getting
net.snowflake.client.jdbc SnowflakeLoggedFeatureNotSupportedException
on the line where registerOutParameter is called.
‛‛‛
CallableStatement st = connection.prepareCall("call sp_poc(?)"); //Call proc
st.registerOutParameter(1,java.sql.Types.VARCHAR); //exception here
ResultSet rs= st.executeQuery(); //execute st
‛‛‛
Below is the procedure created on snowflake db
CREATE or Replace PROCEDURE sp_poc()
RETURNS VARCHAR
LANGUAGE javaScript
AS
$$
var rs="test";
return rs;
$$;
You cannot use the output of a Snowflake stored procedure directly. If you want to get its return output, you need to run another query. This is true whether it's running from the web UI or through a driver or connector:
call sp_poc();
select * from table(result_scan(last_query_id()));
If you don't want to depend on the last_query_id() function, you can capture the query ID in the JDBC call. You can refer to the getQueryId() method of the Snowflake JDBC driver here:
https://docs.snowflake.com/en/user-guide/jdbc-api.html#interface-snowflakecallablestatement
If you capture the query ID that way, you can then get the output results of the SP for up to 24 hours after its completion using:
select * from table(result_scan(?));
Where the bind variable is the query ID returned from the getQueryId() JDBC method.
I want to get access to the source code of some stored procedures, which are stored in a Sybase ASE 15.7 database.
At the moment I copy the text manually in a textfile and read it via executed java-code from the disk.
Now it is my goal to improve this process: I will get the sourcecode directly from the database within my executed java-code.
Is that possible?
Sorry, I have no idea.
I have Rapid SQL to work on the database.
you could run an ISQL script that calls the stored procedure sp_helptext to get the code of the stored procedure. Hope this helps.
Vince's hint was to use the stored procedure sp_helptext. This stored procedure produces more than one resultset and caused me difficulties.
RobV's answer was to use the stored procedure sp_showtext. This seems the better solution, cause it returns only one resultset. But I tried it with jdbc and it ever throws in a sql exception ("must return a resultset").
After my fails I find another solution, which need against the other two answers only a execution of a select statement. This works for me! Here is the code:
StringBuffer sbSP = new StringBuffer();
String query = "select c.text "
+"from syscomments c, sysobjects o "
+"where o.id=c.id and o.name='"+spName+"' order by c.colid";
PreparedStatement stmt = connection.prepareStatement(query);
ResultSet rs = stmt.executeQuery();
while (rs.next()) {
//loop over every row of the stored procedure
sbSP.append(rs.getString("text") + "\n");
}
Best use sp_showtext; it gives better formatted results (no broken SQL lines) than sp_helptext
I have one stored proc written in MS Sql Server which has cursor and temporal table in it and returns result of one select query at the end.
I am trying to call it from java with following piece of code
final Connection conn = getConnection();
final CallableStatement statement = conn.prepareCall("{call dbo.storedProcName (?) }");
statement.setString(1, "value1,value2");
final ResultSet rs = statement.executeQuery();
while (rs.next()) {
//some code
}
When I execute this code in java I get the resultset as null.
The stored proc returns values when I run it on sql server console with the same parameters I am passing from java code.
Any idea what causing this issue here?
I am using sqljdbc4.jar, Java 7 and SQL Server 2008 R2.
Error stack
com.microsoft.sqlserver.jdbc.SQLServerException: The statement did not return a result set.
at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(SQLServerException.java:170)
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(SQLServerPreparedStatement.java:392)
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(SQLServerPreparedStatement.java:338)
at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:4026)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:1416)
at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:185)
at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:160)
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeQuery(SQLServerPreparedStatement.java:281)
Stored proc rough format
create procedure [dbo].[storedProcName](
#inpurparam varchar(4000)
)
as
begin
select some_values into #temp_table where value in (#inputparam)
//declare some_variables
declare #table2 table(col1 varchar(10), col2 varchar(10))
declare cursor for select * from #temp_table
open cursor
fetch next from cursor into some_params
while ##fetch_status = 0
begin
//some processing
//insert into table2 based on some logic
fetch next ..
end
close cursor
deallocate cursor
drop table #temp_table
select col1, col2 from #table2
order by col1
end
go
I was able to fix it with slight change to the stored proc.
create procedure [dbo].[storedProcName](
#inpurparam varchar(4000)
)
as
begin
**SET NOCOUNT ON**
..
..
I don't know how this fixed the issue but it is working like a charm.
I'd appreciate if anybody could throw some light on it.
I have a stored procedure which creates a temp table and populates the temp table. After populating it, it returns the rows inside the table with (SELECT).
When running it from MySQL workbench works tremendously but in JAVA with JDBC, the ResultSet is empty. I have gone through some posts which say that we should not use PreparedStatment but even Statement doesn't work.
For test, I divided the job in stored procedure. Now I call:
CallableStatement ps = connection.prepareCall("{CALL myProcedure(?)}");
ps.setString(1, "value");
ps.execute();
which creates the temp table and populates it. Thereafter with same connection I try to do
Statement st = connection.createStatement();
ResultSet rs = st.executeQuery("SELECT * FROM tmp_table");
Still ResultSet is empty. What can I do? Any help is appreciated.
Thank you folks for help!
I found the error. The error was in code.
I send a parameter when calling procedure. For testing purposes I did
SET #param = "value";
Later on, I removed it but I forgot to remove it when inserting data into temporary table which used that #param. MySQL somehow could see that #param and IN param is same because of name but JDBC could not see that and Therefore, it was not returning any data.
I have to improve some code where an Oracle stored procedure is called from a Java program. Currently the code is really really slow: up to about 8 seconds on my development machine. On the same machine, if I directly call an SQL query that does about the same treatment and returns the same data, it takes under 100 ms...
The code creates a CallableStatement, registers one of the output parameters to be an Oracle cursor, and then retrieves the cursor using the getObject method of the statement and parse it to ResultSet:
cstmt = conn.prepareCall("{ call PKG_ESPECEW.P_ListEspece( ?, ?, ?, ?, ?, ? ) }");
cstmt.registerOutParameter(4, oracle.jdbc.OracleTypes.CURSOR);
[...]
cstmt.executeQuery();
rs = (ResultSet)cstmt.getObject(4);
rs.setFetchSize(1000); //supposed to help ?
options = new HashMap<String, String>(1000);
rs.next() //added that to measure exactly the length of the first call
while(rs.next()) {
[...]
}
I put some timestamps in the code to know which part is taking so long. The result: The first call to rs.next() is taking up to various seconds. The result sets are average, from 10 to a couple thousands rows. As I said before, handling similar result sets coming from a regular PreparedStatement takes 10-100 ms depending the size.
Is anything wrong with the code? How do I improve it? I'll do direct SQL where critical if I haven't any other solution, but I'd prefer a solution that allows me to not rewrite all the procedures!
Here is the definition of the stored procedure:
PROCEDURE P_ListEspece(P_CLT_ID IN ESPECE.ESP_CLT_ID%TYPE, -- Langue de l'utilisateur
P_ESP_GROUP_CODE IN ESPECE.ESP_CODE%TYPE,-- Code du groupe ou NULL
P_Filter IN VARCHAR2, -- Filtre de la requête
P_Cursor OUT L_CURSOR_TYPE, -- Curseur
P_RecordCount OUT NUMBER, -- Nombre d'enregistrement retourne
P_ReturnStatus OUT NUMBER); -- Code d'erreur
"I thought the procedure was executed, then it's result stored in oracle server's memory, and finally transmitted back to the client (the java app) through the cursor and result set and JDBC"
That's incorrect.
What oracle returns as a cursor is basically a pointer to a query (all ready with any bind variables). It has not materialized the result set in memory. It could be a massive result set of millions/billions of rows.
So it could well be a slow query that takes a long time to deliver results.
Apparently the stored procedure is doing some data conversion/massaging forth and back (e.g. int <--> varchar). This is known to take a lot of time in case of large tables. Ensure that you've declared the right datatypes in the SP arguments and are setting the right datatypes in CallableStatement.
How long does it take to execute the procedure outside of Java? Check with a script like this in SQL*Plus:
var ref refcursor
var cnt number
var status number
exec p_listespece (xx, yy, zz, :ref, :cnt, :status);--replace with actual values
print :ref
If it takes more than 10-100 ms, your problem may come from the stored procedure.
I had the same problem, we solved (me and the oracle dedicated guy) by changing the returned parameter from a cursor to a varchar, that was the plain query the stored was executing internally.
this was an huge implementation, I don't know if this is applicable for your scenario.
here's the snippet :
`
String sql = "call MyStored(?,?,?,?)";
CallableStatement st = Conn.prepareCall(sql);
st.setInt(1, 10);
st.setInt(2, 20);
st.setInt(3, 30);
st.registerOutParameter(4, OracleTypes.VARCHAR);
st.execute();
String query = (String) st.getObject(4);
Statement stmt = Conn.createStatement();
rs = stmt.executeQuery(query);
[...]
//work with resultset
[...]
stmt.close();
stmt = null;
`