I have this table in oracle and i need to retrieve two columns from the table desc_data
eg:
select ticket_id, date_logged from desc_data;
I would have around 10,000 records in this table, so if I do this operation from java and perform some operations in java by putting these values in a list and then based on some conditions filter data and insert back into some other table, would it be possible and if it's possible would it be an overhead?
I think better to use a stored procedure in database and just call it from java. But what you consider is a possible solution too.
It depends on what type of filtering you wish to do on your 10000 records. If the filtering is simple, such as filtering the records in a date range, then you can achieve that just using SQL. If your processing is more complex, then you could also use an stored procedure. As you are running on Oracle these can be written in Java. See here for an example.
Related
I am currently trying to push some data from Java into BigQuery, in order to be able to make use of it only for the next query and then to get rid of it.
The data consists of 3 columns which exists in the table I want to query. Therefore, by creating a temporary table containing this data, I could make a left join and get the query results I am in need of.
This process will happen on a scheduled basis, having different sets of data.
Can you please tell me if that can be done ?
Many thanks !
Using the jobs.query API, you can specify a destinationTable as part of configuration.query. Does that help? You can control the table expiration time using the tables.update API and setting expirationTime.
Alternatively, you can "inline" the table as part of the query that you want to run using a WITH clause in standard SQL rather than writing to a temporary table.
I am executing a custom built DML statement using the
namedParameterJdbcTemplate.update(sql, valueMap);
call, where the sql is built based on the values in the map. Here my map could get very large and thus the sql might also get very lengthy. I understand that in Oracle, there is no fixed number for how long a query can be and there are many factors including the database configuration that may affect this value, but I would like to limit the query length to a fixed number.
What is the best way to limit the query length? Would the spring-batch API be any useful here?
Thanks in advance for any pointers.
I would choose one of these approaches:
Temporary table - Insert data in batch to the temporary table and then use MERGE INTO statement with that table.
Create SQL type for your rows and bind just that one parameter. (google for OraData - it is a bit tricky but it works)
Both will enable you to have a small static query and therefore avoid potential problems with too large query (and its parsing, polluting library cache etc.).
I have 5000 records as search result and based on product number have to pull the related data associated with product number.that means seperate the 5000 product number and sending to database to pull the data.Creating one query and hiting the database for each product number is not efficient.
I'm looking for some idea to handle this situation.
Note:using hibernate and oracle and java
You got that search result with some query, it might be simpler to reuse that query with a join to retrieve the related data.
Instead of 5000 queries to get the result, you may use the IN clause.
You should probably split it in chunks, however, since such long SQL queries can throw errors, or use a temporary table and do a JOIN. Take a look at this.
maybe you could use a Materialized View and some basic paging? http://docs.oracle.com/cd/A97630_01/server.920/a96567/repmview.htm
I want to use Bulk API of Salesforce to run queries of this format.
Select Id from Object where field='<value>'.
I have thousands of such field values and want to retrieve Id of those objects. AFAIK, Bulk query of Salesforce supports only one SOQL statement as input.
One option could be to form a query like
Select Id,field where field in (<all field values>)
but problem is SOQL has 10000 characters limitation.
Any suggestions here?
Thanks
It seems like you are attempting to perform some kind of search query. If so you might look into using a SOSL query as opposed to SOQL as long as the fields you are searching are indexed by SFDC.
Otherwise, I agree with Born2BeMild. Your second approach is better and breaking up your list of values into batches would help get around the limits.
It would also help if you described a bit of your use case in more detail. Typically queries on a dynamic set of fields and values doesn't always yield the best performance even with the bulk api. You are almost better off downloading the data to a local database and exploring the data that way.
You could break those down into batches of 200 or so values and iteratively query Salesforce to build up a result set in memory or process subsets of the data.
You would have to check the governor limits for the maximum number of SOQL queries though. You should be able to track your usage via the API at runtime to avoid going over the maximum.
The problem is that you are hitting the governor limits. Saleforce can only process 200 records at a time if its coming from a database. Therefore to be able to work with all this records first you need to add all records to a list for example:
List<Account> accounts= [SELECT id, name, FROM Account];
Then you can work with the list accounts do everything you need to do with it then when you done you can update the database using:
Update accounts;
this link might be helpful:
https://help.salesforce.com/apex/HTViewSolution?id=000004410&language=en_US
I have a producer thread in Java pulling items from an Oracle table every n milliseconds.
The current implementation relies on a Java timestamp in order to retrieve data and never re-retrieve them again.
My objective is to get rid of the timestamp pattern and directly update the same items I'm pulling from the database.
Is there a way to SELECT a set of items and UPDATE them at the same time to mark them as "Being processed"?
If not, would a separate UPDATE query relying on the IN clause be a major performance hit?
I tried using a temporary table for that purpose, but I've seen that performance was severely affected.
Don't know if it helps, but the application is using iBatis.
If you are using oracle 10g or higher, you can use the RETURNING clause of the update statement. If you wish the retrieve more than one row you can use the BULK COLLECT statement.
Here is a link to some examples;
http://psoug.org/snippet/UPDATE-with-RETURNING-clause_604.htm