I have two tables table1 and table2 and joining them using inner join on one column.
There is a possibility that child table can have more than 50 million recorrds.
It took 30 mins to delete 17 million records using spring jdbc update().
is there a optimized way to reduce deletion time.
Use batchUpdate with some copeable batch size, eg. 5000.
EDIT: The problem is probably not in Spring jdbc but in your query.
Would this work for you?
DELETE
res
FROM
RESULT res
INNER JOIN
POSITION pos
ON res.POSITION_ID = pos.POSITION_ID
WHERE
pos.AS_OF_DATE = '2012-11-29 11:11:11'
This removes entries from RESULT table. Simplified SQL fiddle demo: http://www.sqlfiddle.com/#!3/4a71e/15
Related
I am reading data from a table using Spring JPA.
This Entity object has one-to-many relationship to other six tables.
All tables together has 20,000 records in them.
I am using below query to fetch data from DB.
SELECT * FROM A WHER ID IN (SELECT ID FROM B WHERE COL1 = '?')
A table has relationship to other 6 tables.
Spring JPA is taking around 30 seconds of time to read this data from DB.
Any idea to improve the data fetch time here.
I am using native Queries here and i am looking for query rewriting that will optimize the data fetch time.
Please suggest thanks.
You might need consider below to identify the root cause:
Check if you are ending up with n+1 query issue. Your query might end up calling n queries for each join table, where n is no. of associations with the join table. You can check this by setting spring.jpa.show-sql=true
If you see the issue as n+1 then you need set appropriate FetchMode, refer https://www.baeldung.com/hibernate-fetchmode for detailed explanation of using different FetchModes.
If it is not n+1 query issue you might need to check the performance of the genarated queries using EXPLAIN command. Usually IN clause on a non indexed columns have performance impact.
So set spring.jpa.show-sql=true and check queries generated and run to debug and optimize your code or query.
I have SQL query joining multiple tables.
SELECT a.fourbnumber,a.fourbdate,a.taxcollector,b.cashcheque,c.propertycode
from tbl_rphead a
inner join tbl_rpdetail b on a.rpid = b.rpid
inner join tbl_assessmregister c on b.assessmid = c.assessmid
I can execute that query in Sql Editor with fast manner (3 secs). When I execute that query using JAVA(JDBC), it doesn’t returns any results and no exceptions
I don’t know how to fix that problem.
Each table has 200k records
Your Sql Editor might limiting the result to some count to show the records in view. See the editor you may find the hint showing 500 of XXXXXX
When you calling it from JDBC it may get the results faster from DB, but it need to fill up the result set objects for those lacs of records. It will more time and memory.
If you are working with oracle DB try limiting records in your query with help of rownum < 100 , so you could get the results in java/jdbc. If it works go with SQL pagination technique with rownum < x and rownum > y
I want only update first N row, in SQL:
UPDATE Table1 SET c1 = 'XXX' WHERE Id IN (SELECT TOP 10 Id FROM Table1 ORDER BY c2)
Can Hibernate do that in ONE update?
With Hibernate, you can always issue a native query as such, but the current running Persistence Context won't be aware of the deleted entries.
As long as you only deleted a relatively small amount of entries, you can simply select N entities and then use the remove operation so that you can benefit from optimistic locking checks and prevent lost updates.
If you want to deletes lots of entries, then a bulk delete query is much more appropriate. You can even run the SQL DELETE query that you mentioned. That's exactly the reason why JPA and Hibernate allow you to use native SQL queries anyway.
The following hibernate query is being used to fetch a list of ProductCatalogue records, by passing in catId and inventoryId
select prodcat from ProductCatalogue prodcat where prodcat.prodSec.prodId=:catId and prodcat.prodPlacedOrder.inventoryId=:inventoryId
The tables ProductCatalogue and ProdPlacedOrder are tables with 3 lakh + records. inventoryId is a column in prodOrder table, and prodPlacedOrder extends prodOrder table.
This query on execution takes a lot of time, and the single hibernate query shoots off many complex sql queries.
Any suggestions on what might be the issue and how to modify it such that the query is executed faster?
Difficult to say without more info but try making ProdPlacedOrder as LAZY fetch if you dont need any data from that table.
Also as phatmanace, mentioned - check your indices.
This is more like a design question but related to SQL optimization as well.
My project has to import a large number of records into the database (more than 100k records). In the meantime, the project has logic to check each record to make sure it meets the criteria which are configurable. It then will mark the record as no warning or has warning in the database. The inserting and warning checking are done within one importing process.
For each criteria it has to query the database. The query needs to join two other tables and sometimes add additional nested query inside the conditions, such as
select * from TableA a
join TableB on ...
join TableC on ...
where
(select count(*) from TableA
where TableA.Field = Bla) > 100
Although the queries take unnoticeable time, to query the entire record set takes a considerable amount of time which may be 4 - 5 hours on a server. Especially if there are many criteria, at the end the project will stop running the import and rollback.
I've tried changing "SELECT * FROM" to "SELECT TableA.ID FROM" but it seems it has no effect at all. Is there a better design to improve the performance of this process?
How about making a temp table (or more than one) that stores the aggregated results of the sub-queries, then indexing that/those with covering indexes.
From your code above, we'd make a temp table grouping on TableA.Field1 and including a count, then index on Field1, theCount. On SQL server the fastest approach would then be:
select * from TableA a
join TableB on ...
join TableC on ...
join (select Field1 from #temp1 where theCount > 100) t on...
The reason this works is that we are doing the same trick twice.
First, we pre-aggregate into the temp table, which is a simple operation and very easy for SQL Server to optimize. So we have taken a piece of the problem and solved in an optimizable way.
Then we repeat this trick by joining to a subquery, putting the filter inside the subquery, so that the join acts as a filter.
I would suggest you batch your records together (500 or so at a time) and send it to a stored proc which can do the calculation.
Use simple statements instead of joins in there. That saves as well. This link might help as well.
Good choice is using indexed view.
http://msdn.microsoft.com/en-us/library/dd171921(SQL.100).aspx