In java code I generate thousands of ids, next I need to get those ids in the postgresql database, so far I have this (used createNativeQuery as pseudocode):
Query q = em.createNativeQuery(
"select * from mytable where id in (:ids)"
).setParameter("ids", listofIds);
I'm afraid about the IN clause, I fear that there might be a huge number of values, is there another better way to perform this?
I think you can scale to quite large lists by doing
select * from mytable where id = any(cast(:ids as bigint[]))
but you have to format the list of ids as a string in Java first, like
ids = "{" + String.join(",", listofIds) + "}";
Still, there will always be some threshold where the list is just too large for Java or Postgresql to handle, so you'll need to break it up into batches if you want to support arbitrarily large lists.
Related
I'm writing a sql query to count objects. I have about 50 pre-written queries, and each of them is essentially written like so:
"select count(1) from SOME_TABLE where identifier = :ID"
I have it in Java. SOME_TABLE is already written into the query (different for each query). At runtime, a switch/case determines which query to execute and which ID to send it.
Essentially, there are various queries which can be called, but ID is the only thing that changes inside the queries.
So, if I want to count all the pinetrees in a forest, I'd send pineTree as an ID to a query that counts Tree. Simple enough. As I mentioned, there's a giant switch/case which determines TABLE_NAME, ID, and the exact query.
The problem is, I'm trying to count all of the objects of a table, and I'm trying to count them with various IDs. For example:
query1: "select count(1) from LIST_OF_TOYS where animalId = theListOfIDsfromQuery2"
query2: "select allThePetIDs from LIST_OF_PETS where ownerId = :myID
This is essentially what I'm trying to do. I want to count all the toys owned by my pets, but there is no direct link between my ID and the LIST_OF_TOYS table.
I know how to do this with two separate queries, but I would strongly prefer to write one query that just takes ownerId as a parameter, so that I can add it to my giant switch/case (and keep my code neat). I can write the query however I want, but I was wondering if it's possible to do this as a single query and a single argument. Thanks for the help!
select count(1) from LIST_OF_TOYS where animalId in (select allThePetIDs from LIST_OF_PETS where ownerId = :myID)
Suppose I have a Table X that has a billion records.
Table X
ProductID AccountID ContractID
ProductID and AccountID make a composite key for Table X.
Now, in memory, I have a map (let's say Java HashMap) that contains a million (ProductID, AccountID) pairs.
I want to create a file that will contain all the (ProductID, AccountID) as well as the corresponding ContractID for that pair.
Now I can use a for loop and for each (ProductID, AccountID) I can query the table, but then I would have to do this a million times and it would be really inefficient.
The question is, how to write a query that will do this efficiently? Or can such a query be written at all? Is there another way out?
If speed and efficiency are of importance, then a query with a million "unions" or a million items in an IN clause is not going to be acceptable.
A more performant solution would be to perform a bulk insert of your ProductID/AccountID hashmap into a temp table, let's call it #temp. I'm not going to describe the bulk insert because that is database dependent. Then you can perform a simple join query:
SELECT ProductID, AccountID, ContractID
FROM X
INNER JOIN #temp t ON t.ProductID = X.ProductID AND t.AccountID = X.AccountID
Without knowing the exact SQL dialect, I'd perform an INNER JOIN:
SELECT ProductID, AccountID, ContractID
FROM X
INNER JOIN MemTable m ON m.ProductID = X.ProductID AND m.AccountID = X.AccountID
You now added Java as a tag, so am I right in thinking that the map is within your Java application? If so, it will get tough - you may actually need to query the database a million times.
On the other hand you could construct a string containing one single, large SQL statement like that:
SELECT * FROM X WHERE ProductID IN (...) AND AccountID IN (...)
where your loop just fills in a list of product IDs and account IDs comma separated. Then you issue that command once. The command should for example look like this, assuming both IDs are numeric:
SELECT * FROM X WHERE ProductID IN (1,2,3,4) AND AccountID IN (99,88,77)
EDIT
Please note that my last suggestion may have the following flaw (you'll have to decide whether this is actually a problem for you):
Assume your map contains (1, 99) and (3, 77), but in table X there are additional records (1, 77) and (3, 99). The result of my query will be (1,99), (3, 77), (1, 77) and (3, 99) as both IDs are not treated as an "entity", but individually.
So as long as there are rows that contain any combination of the given ProductID and AccountID, they will be returned.
Assuming the DB system you're using allows for this, you could expand the SELECT statement into something like this:
SELECT ProductID, AccountID, ContractID FROM X WHERE ProductID = <ValueFromMap> AND AccountID = <ValueFromMap>
UNION ALL
SELECT ProductID, AccountID, ContractID FROM X WHERE ...
UNION ALL
...
I guess your memory map is in your Java program? If so I think there is no efficient solution that will be database independent. Best I can think of is to try and find continous id-ranges in your memory map so that you can write SELECT FROM X where ID >= xx AND id <= yy and avoid selecting duplicate ids.
I need to get an equivalent to this SQL that can be run using Hibernate. It doesn't work as is due to special characters like #.
SELECT place from (select #curRow := #curRow + 1 AS place, time, id FROM `testing`.`competitor` JOIN (SELECT #curRow := 0) r order by time) competitorList where competitorList.id=4;
My application is managing results of running competitions. The above query is selecting for a specific competitor, it's place based on his/her overall time.
For simplicity I'll only list the COMPETITOR table structure (only the relevant fields). My actual query involves a few joins, but they are not relevant for the question:
CREATE TABLE competitor {
id INT,
name VARCHAR,
time INT
}
Note that competitors are not already ordered by time, thus, the ID cannot be used as rank. As well, it is possible to have two competitors with the same overall time.
Any idea how I could make this work with Hibernate?
Hard to tell without a schema, but you may be able to use something like
SELECT COUNT(*) FROM testing ts
WHERE ts.score < $obj.score
where I am using the $ to stand for whatever Hibernate notation you need to refer to the live object.
I couldn't find any way to do this, so I had to change the way I'm calculating the position. I'm now taking the top results and am creating the ladder in Java, rather than in the SQL query.
How can I retrieve the last entered value of the column in the database (MS ACCESS 2007)
I used the following code
String sql = "SELECT Last(RegNumber) FROM Death ";
but it does not work in MS ACCESS and when I run the program Error generates as
java.sql.SQLException: Column not found
but I have created a column in database as RegNumber
I am using Java for programming in which I used this query
EDIT:
RegNumber is in String form not in integer form so I cant use DESC or ASC
Please help me
Sort your table by whatever criteria you'd like and use SELECT TOP 1 * FROM myTable ORDER BY RegNumber ASC.
Or ORDER BY incrementingId DESC
Basically there must be some logical order to the sorting for what you refer to as the "last entered column" (which I assume means row, not column)
EDIT: Your function is correct in Access, and should return the correct value. However Java may not interpret it correctly. Try your query in an Access native query, then try debugging your Java. If it's simply that Java does not support this function, consider using the built in ResultSet() functions in Java.sql
ResultSet rs = ....;
rs.last();
int RegNumber = rs.getRow();
I do not know about the last() function in MS ACCESS, but I have another idea:
Usually there is an automatically generated id for each table, so you can sort on it and get the first record from the result set like this:
SELECT RegNumber
FROM Death
ORDER BY id DESC
That depend of your database structure.
Typically with table come some unique identifier, if you are sure that it comes always in order to database you could use function MAX to retrieve the identifier and then whole row.
Another scenario is just to a timestamp columns that describe the time when column was created , this approach satisfying if the sequence is really crucial if not the id should be enough.
Following will return the last and lastest RegNumber :
SELECT RegNumber FROM Death ORDER BY RegNumber DESC
How do I implement paging in Hibernate? The Query objects has methods called setMaxResults and setFirstResult which are certainly helpful. But where can I get the total number of results, so that I can show link to last page of results, and print things such as results 200 to 250 of xxx?
You can use Query.setMaxResults(int results) and Query.setFirstResult(int offset).
Editing too: There's no way to know how many results you'll get. So, first you must query with "select count(*)...". A little ugly, IMHO.
You must do a separate query to get the max results...and in the case where between time A of the first time the client issues a paging request to time B when another request is issued, if new records are added or some records now fit the criteria then you have to query the max again to reflect such. I usually do this in HQL like this
Integer count = (Integer) session.createQuery("select count(*) from ....").uniqueResult();
for Criteria queries I usually push my data into a DTO like this
ScrollableResults scrollable = criteria.scroll(ScrollMode.SCROLL_INSENSITIVE);
if(scrollable.last()){//returns true if there is a resultset
genericDTO.setTotalCount(scrollable.getRowNumber() + 1);
criteria.setFirstResult(command.getStart())
.setMaxResults(command.getLimit());
genericDTO.setLineItems(Collections.unmodifiableList(criteria.list()));
}
scrollable.close();
return genericDTO;
you could perform two queries - a count(*) type query, which should be cheap if you are not joining too many tables together, and a second query that has the limits set. Then you know how many items exists but only grab the ones being viewed.
You can do one thing. just prepare Criteria query as per your busness requirement with all Predicates , sorting , searching etc.
and then do as below :-
CriteriaBuilder criteriaBuilder = em.getCriteriaBuilder();
CriteriaQuery<Feedback> criteriaQuery = criteriaBuilder.createQuery(Feedback.class);
//Just Prepare your all Predicates as per your business need.
//eg :-
yourPredicateAsPerYourBusnessNeed = criteriaBuilder.equal(Root.get("applicationName"), applicationName);
criteriaQuery.where(yourPredicateAsPerYourBusnessNeed).distinct(true);
TypedQuery<Feedback> criteriaQueryWithPredicate = em.createQuery(criteriaQuery);
//Getting total Count Here
Long totalCount = criteriaQueryWithPredicate.getResultStream().distinct().count();
Now we have our actual data with us as above with total count , right.
So now we can apply pagination on the data we have in our hand above , as below :-
List<Feedback> feedbackList = criteriaQueryWithPredicate.setFirstResult(offset).setMaxResults(pageSize).getResultList();
Now You can prepare a wrapper with your List return by DB along with the totalCount , startingPageNo that is offset here in this case, page Size etc and can return to your service / controller class.
I am 101 % sure , this will solve your problem, Because I was facing same problem and sorted it out same way.
Thanks- Sunil Kumar Mali
You can just setMaxResults to the maximum number of rows you want returned. There is no harm in setting this value greater than the number of actual rows available. The problem the other solutions is they assume the ordering of records remains the same each repeat of the query, and there are no changes going on between commands.
To avoid that if you really want to scroll through results, it is best to use the ScrollableResults. Don't throw this object away between paging, but use it to keep the records in the same order. To find out the number of records from the ScrollableResults, you can simply move to the last() position, and then get the row number. Remember to add 1 to this value, since row numbers start counting at 0.
I personally think you should handle the paging in the front-end. I know this isn't that efficiënt but at least it would be less error prone.
If you would use the count(*) thing what would happen if records get deleted from the table in between requests for a certain page? Lots of things could go wrong this way.