Is there a workaround for
'ORA-01795: maximum number of expressions in a list is 1000 error'
I have a query and it is selecting fields based on the value of one field. I am using the in clause and there are 10000+ values
example:
select field1, field2, field3
from table1
where name in
(
'value1',
'value2',
...
'value10000+'
);
Every time I execute the query I get the ORA-01795: maximum number of expressions in a list is 1000 error. I am trying to execute the query in TOAD, no difference, the same error. How would I modify the query to get it to work?
Just use multiple in-clauses to get around this:
select field1, field2, field3 from table1
where name in ('value1', 'value2', ..., 'value999')
or name in ('value1000', ..., 'value1999')
or ...;
Some workaround solutions are:
1. Split up IN clause
Split IN clause to multiple IN clauses where literals are less than 1000 and combine them using OR clauses:
Split the original "WHERE" clause from one "IN" condition to several "IN" condition:
Select id from x where id in (1, 2, ..., 1000,…,1500);
To:
Select id from x where id in (1, 2, ..., 999) OR id in (1000,...,1500);
2. Use tuples
The limit of 1000 applies to sets of single items: (x) IN ((1), (2), (3), ...).
There is no limit if the sets contain two or more items: (x, 0) IN ((1,0), (2,0), (3,0), ...):
Select id from x where (x.id, 0) IN ((1, 0), (2, 0), (3, 0),.....(n, 0));
3. Use temporary table
Select id from x where id in (select id from <temporary-table>);
I ran into this issue recently and figured out a cheeky way of doing it without stringing together additional IN clauses
You could make use of Tuples
SELECT field1, field2, field3
FROM table1
WHERE (1, name) IN ((1, value1), (1, value2), (1, value3),.....(1, value5000));
Oracle does allow >1000 Tuples but not simple values. More on this here,
https://community.oracle.com/message/3515498#3515498
and
https://community.oracle.com/thread/958612
This is of course if you don't have the option of using a subquery inside IN to get the values you need from a temp table.
One more way:
CREATE OR REPLACE TYPE TYPE_TABLE_OF_VARCHAR2 AS TABLE OF VARCHAR(100);
-- ...
SELECT field1, field2, field3
FROM table1
WHERE name IN (
SELECT * FROM table (SELECT CAST(? AS TYPE_TABLE_OF_VARCHAR2) FROM dual)
);
I don't consider it's optimal, but it works. The hint /*+ CARDINALITY(...) */ would be very useful because Oracle does not understand cardinality of the array passed and can't estimate optimal execution plan.
As another alternative - batch insert into temporary table and using the last in subquery for IN predicate.
Please use an inner query inside of the in-clause:
select col1, col2, col3... from table1
where id in (select id from table2 where conditions...)
There is another option: with syntax. To use the OPs example, this would look like:
with data as (
select 'value1' name from dual
union all
select 'value2' name from dual
union all
...
select 'value10000+' name from dual)
select field1, field2, field3
from table1 t1
inner join data on t1.name = data.name;
I ran into this problem. In my case I had a list of data in Java where each item had an item_id and a customer_id. I have two tables in the DB with subscriptions to items respective customers. I want to get a list of all subscriptions to the items or to the customer for that item, together with the item id.
I tried three variants:
Multiple selects from Java (using tuples to get around the limit)
With-syntax
Temporary table
Option 1: Multiple Selects from Java
Basically, I first
select item_id, token
from item_subs
where (item_id, 0) in ((:item_id_0, 0)...(:item_id_n, 0))
Then
select cus_id, token
from cus_subs
where (cus_id, 0) in ((:cus_id_0, 0)...(:cus_id_n, 0))
Then I build a Map in Java with the cus_id as the key and a list of items as value, and for each found customer subscription I add (to the list returned from the first select) an entry for all relevant items with that item_id. It's much messier code
Option 2: With-syntax
Get everything at once with an SQL like
with data as (
select :item_id_0 item_id, :cus_id_0 cus_id
union all
...
select :item_id_n item_id, :cus_id_n cus_id )
select I.item_id item_id, I.token token
from item_subs I
inner join data D on I.item_id = D.item_id
union all
select D.item_id item_id, C.token token
from cus_subs C
inner join data D on C.cus_id = D.cus_id
Option 3: Temporary table
Create a global temporary table with three fields: rownr (primary key), item_id and cus_id. Insert all the data there then run a very similar select to option 2, but linking in the temporary table instead of the with data
Performance
This is not a fully-scientific performance analysis.
I'm running against a development database, with slightly over 1000 rows in my data set that I want to find subscriptions for.
I've only tried one data set.
I'm not in the same physical location as my DB server. It's not that far away, but I do notice if I try from home over the VPN then it's all much slower, even though it's the same distance (and it's not my home internet that's the problem).
I was testing the full call, so my API calls another (also running in the same instance in dev) which also connects to to the DB to get the initial data set. But that is the same in all three cases.
YMMV.
That said, the temporary table option was much slower. As in double so slow. I was getting 14-15 seconds for option 1, 15-16 for option 2 and 30 for option 3.
I'll try them again from the same network as the DB server and check if that changes things when I get the chance.
I realize this is an old question and referring to TOAD but if you need to code around this using c# you can split up the list through a for loop. You can essentially do the same with Java using subList();
List<Address> allAddresses = GetAllAddresses();
List<Employee> employees = GetAllEmployees(); // count > 1000
List<Address> addresses = new List<Address>();
for (int i = 0; i < employees.Count; i += 1000)
{
int count = ((employees.Count - i) < 1000) ? (employees.Count - i) - 1 : 1000;
var query = (from address in allAddresses
where employees.GetRange(i, count).Contains(address.EmployeeId)
&& address.State == "UT"
select address).ToList();
addresses.AddRange(query);
}
Hope this helps someone.
there is also another way to resolve this issue. lets say you have two tables Table1 and Table2. and it is required to fetch all entries of Table1 not referred/present in Table2 using Criteria query. So go ahead like this...
List list=new ArrayList();
Criteria cr=session.createCriteria(Table1.class);
cr.add(Restrictions.sqlRestriction("this_.id not in (select t2.t1_id from Table2 t2 )"));
.
.
. . . It will perform all the subquery function directly in SQL without including 1000 or more parameters in SQL converted by Hibernate framework. It worked for me. Note: You may need to change SQL portion as per your requirement.
Operato union
select * from tableA where tableA.Field1 in (1,2,...999)
union
select * from tableA where tableA.Field1 in (1000,1001,...1999)
union
select * from tableA where tableA.Field1 in (2000,2001,...2999)
**Divide a list to lists of n size**
import java.util.AbstractList;
import java.util.ArrayList;
import java.util.List;
public final class PartitionUtil<T> extends AbstractList<List<T>> {
private final List<T> list;
private final int chunkSize;
private PartitionUtil(List<T> list, int chunkSize) {
this.list = new ArrayList<>(list);
this.chunkSize = chunkSize;
}
public static <T> PartitionUtil<T> ofSize(List<T> list, int chunkSize) {
return new PartitionUtil<>(list, chunkSize);
}
#Override
public List<T> get(int index) {
int start = index * chunkSize;
int end = Math.min(start + chunkSize, list.size());
if (start > end) {
throw new IndexOutOfBoundsException("Index " + index + " is out of the list range <0," + (size() - 1) + ">");
}
return new ArrayList<>(list.subList(start, end));
}
#Override
public int size() {
return (int) Math.ceil((double) list.size() / (double) chunkSize);
}
}
Function call :
List<List<String>> containerNumChunks = PartitionUtil.ofSize(list, 999)
more details: https://e.printstacktrace.blog/divide-a-list-to-lists-of-n-size-in-Java-8/
There's also workaround doing disjunction of your array, worked for me as other solutions were hard to implement using some old framework.
select * from tableA where id = 1 or id = 2 or id = 3 ...
But for better perfo, I would use Nikolai Nechai's solution with unions, if possible.
Pass the list and the number of records needs to return in the loop most cases = 999.
List<List<Long>> getSubLists = batchList(inputList, 999);
List<Long> newList = new ArrayList<>();
for (List<Long> subSet : getSubLists) { newList.addALL(daoCall) // add in the required list in loop }
public static <T> List<List<T>> batchList(List<T> inputList, final int maxSize) {
List<List<T>> sublists = new ArrayList<>();
final int size = inputList.size();
for (int i = 0; i < size; i += maxSize) {
sublists.add(new ArrayList<>(inputList.subList(i, Math.min(size, i + maxSize))));
}
return sublists;
}
Use Tuple :
Let's suppose input is :
List<Long> userIdList = Arrays.asList(100L,200L,300L);
StringBuilder tuple = new StringBuilder();
for(Long userId : userIdList) {
tuple.append("(1,").append(userId).append("),");
}
tuple.deleteCharAt(tuple.length()-1);
Output will be :
(1,100),(1,200),(1,300)
And we can pass this to below query like this (And YES, we can pass more than 1000 elements):
SELECT * FROM MyTable WHERE (1, USR_ID) IN ((1,100),(1,200),(1,300));
ORA-01795: maximum number of expressions in a list is 1000.
Issue:
When user selects long list of values (greate or qual to 1000 values/expression) for IN/OR list of where clause, system throws error: “ORA-01795: maximum number of expressions in a list is 1000”
Root Cause:
Oracle IN / OR list has limit of 1000 (actually its 999) number of expression/value list.
Proposed Solution:
You need to split the list of expressions into multiple sets (using OR) and each should be less than 1000 list/expressoin combine using IN / Or list.
Example:
Suppose you have a table ABC with column ZIP of CLOB type and table contains more than 1000 rows.
You need to break them in multiple list, like shown below:
( ZIP IN (1,2,3,.........N999)
OR ZIP IN (1000,1001,......N999)
.....
.....
)
If you are using Hibernate Query Language(HQL) in your repository class and using IN clause, Oracle database is not going to be let you run your query. Because you are giving more then 1000 value. Oracle in clause limit is 1000. So you cannot give more then 1000 value to your IN clause.
In java, you can use this codes. It's going to be very help full. You can call your services like this, and in this example limit is 750.
Don't write more than 1000. Happy coding.
import java.util.List;
public class ListUtils {
public static int getPeriodCount(List<?> list, int limit) {
int size = list.size();
int leap = size % limit;
int periodCount = leap != 0 ? (size / limit) + 1 : (size / limit);
return periodCount;
}
}
public class MainClass {
private static final int LIMIT = 750;
private List<GivenObject> getList(List<Integer> numbers) {
if (CollectionUtils.isEmpty(numbers)) {
return Collections.emptyList();
}
List<Integer> resultList= new ArrayList<>();
int startIndex = 0;
int periodCount = ListUtils.getPeriodCount(numbers, LIMIT);
for (int i = 1; i <= periodCount; i++) {
List<Integer> collectedNumbers = numbers.stream()
.skip(startIndex)
.limit(LIMIT)
.collect(Collectors.toList());
resultList.addAll(collectedNumbers);
startIndex += LIMIT;
}
return resultList;
}
}
I need to send a query to retrieve values that has a specific group of characters as following:
Lets say I am interested in 'XX' so it should search for any field that its value starts with 'XX' or has ' XX' (space XX). For example XXCDEF, PD XXRF and CMKJIEK XX are valid results.
I have following query that
returns the correct results but I need to sort them
in a way that it first return those with XX at the beginning then other results. As following:
XXABCD
XXPLER
XXRFKF
AB XXAB
CD XXCD
ZZ XXOI
POLO XX
Code
Criteria criteria = session.createCriteria(Name.class, "name")
.add(Restrictions.disjunction()
.add(Restrictions.ilike("name.fname", fname + "%"))
.add(Restrictions.ilike("name.fname", "%" + " " + fname + "%"))
)
.setProjection(Projections.property("name.fname").as("fname"));
List<String> names = (List<String>) criteria.list();
With JPQL (HQL):
select fname from Name
where upper(fname) like :fnameStart or upper(fname) like :fnameMiddle
order by (case when upper(fname) like :fnameStart then 1 else 2 end), fname
query.setParameter("fnameStart", "XX%");
query.setParameter("fnameMiddle", "% XX%");
With Criteria
With Criteria it's much trickier. Firstly, you have to resort to native SQL in the order clause. Secondly, you have to bind the variable.
public class FirstNameOrder extends Order {
public FirstNameOrder() {
super("", true);
}
#Override
public String toSqlString(Criteria criteria, CriteriaQuery criteriaQuery) throws HibernateException {
return "case when upper(FIRST_NAME) like ? then 1 else 2 end";
}
}
The case expression syntax and the upper function name should be changed in accordance with your database (and the column name if it's different, of course).
It is easy to add this to the Criteria, but there is no API to bind the parameter.
I tried to trick Hibernate by passing in an unused variable to the custom sql restriction so that it is effectively used for the variable in the order by clause:
Criteria criteria = session.createCriteria(Name.class, "name")
.add(Restrictions.disjunction()
.add(Restrictions.ilike("name.fname", fname + "%"))
.add(Restrictions.ilike("name.fname", "%" + " " + fname + "%")))
.setProjection(Projections.property("name.fname").as("fname"))
.add(Restrictions.sqlRestriction("1 = 1", fname + "%", StringType.INSTANCE))
.addOrder(new FirstNameOrder())
.addOrder(Order.asc("fname"));
and it works fine.
Obviously, this solution is not recommended and I suggest using JPQL for this query.
Hibernate supports Order: http://docs.jboss.org/hibernate/orm/4.2/devguide/en-US/html/ch11.html#ql-ordering
Because of the special criteria, I think you have to custom the Order in Hibernate. This link may help:
http://blog.tremend.ro/2008/06/10/how-to-order-by-a-custom-sql-formulaexpression-when-using-hibernate-criteria-api/
Run two selects, one filtered for all the strings starting with 'XX', the second filtered for the others.
You can use Predicates in criteria... something like this:
public List<Name> findByParameter(String key, String value, String orderKey)
CriteriaBuilder builder = this.entityManager.getCriteriaBuilder();
CriteriaQuery<Name> criteria = builder.createQuery(this.getClazz());
Root<Name> root = criteria.from(Name.getClass());
criteria.select(root);
List<Predicate> predicates = new ArrayList<Predicate>();
predicates.add(builder.equal(root.get(key), value));
criteria.where(predicates.toArray(new Predicate[predicates.size()]));
if (orderKey!=null && !orderKey.isEmpty()) {
criteria.orderBy(builder.asc(root.get(orderKey)));
}
result = this.entityManager.createQuery(criteria).getResultList();
return result;
}
Stupid but it may work for your case.
Since you got your correct result you can just reconstruct your results as follows:
pick up all results starting with XX you put them into a list L1 and do the normal sort like Collections.sort(L1);
for all other results, do the same like Collections.sort(L2)as list of L2
At last , put them together
List newList = new ArrayList(L1);
newList.addAll(L2);
Please note. Collections.sort are following the natural ordering of its elements.
If you don't want to sort the result in memory,you can modify your criteria,I'll show you the SQL
select * from table where fname like 'XX%' order by fname
union all
select * from table where fname like '% XX%' order by fname
union all
select * from table where fname like '% XX' order by fname
the result will be your order and alphabetical and then apply your filter.
I'm trying to do the following with Jooq and can't for the life of me figure out how to do it properly:
select name, id in (
select capability_id
from a.capabilities_users
where user_id = ?)
from a.capabilities;
Basically I want to get all items (capabilities) and know whether each one applies to a particular user. It seems that all the condition type operators (like greater than or in) can only be used in the where and not the select. And I can't think of how else to express this.
Worst case, I can do a select count and then do the boolean logic in Java, but I was hoping to use fetchMap.
Depending on your database and schema meta data, a LEFT JOIN might be a better choice than a predicate in the projection. You should of course verify this in the execution plan.
Solving this with a LEFT JOIN:
-- NVL2 is Oracle syntax.
-- jOOQ will emulate NVL2 using CASE, if it's not available in your database
SELECT c.name, NVL2(cu.capability_id, 1, 0)
FROM a.capabilities c
LEFT OUTER JOIN a.capabilities_users cu
ON (c.id = cu.capability_id
AND cu.user_id = ?)
The above assumes, of course, that there is a unqiue constraint on cu(user_id, capability_id). This would then translate into jOOQ as such:
Capabilities c = CAPABILITIES.as("c");
CapabilitiesUsers cu = CAPABILITIES_USERS.as("cu");
Field<String> key = c.NAME.as("key");
Field<Boolean> value = nvl2(
CAPABILITIES_USER.CAPABILITY_ID, true, false
).as("value");
Map<String, Boolean> map =
DSL.using(configuration)
.select(key, value)
.from(c)
.leftOuterJoin(cu)
.on(c.ID.eq(cu.CAPABILITY_ID))
.and(cu.USER_ID.eq(...))
.fetchMap(key, value);
Solving this with a predicate in the projection:
If you really prefer a predicate in the projection, you might try DSL.field(Condition), which allows for precisely this:
Field<String> key = CAPABILITIES.NAME.as("key");
Field<Boolean> value = field(
CAPABILITIES.ID.in(
select(CAPABILITY_ID)
.from(CAPABILITIES_USERS)
.where(CAPABILITIES_USERS.USER_ID.eq(...))
)
).as("value");
Map<String, Boolean> map =
DSL.using(configuration)
.select(key, value)
.from(CAPABILITIES)
.fetchMap(key, value);
Note that if you're using a standards-compliant database, which doesn't allow for predicates to be treated as columns, DSL.field(Condition) will render an equivalent CASE statement for you.
I'm using,
JPA 2.0
Mojarra 2.1.9
JSF component library, Primefaces 3.5.
MySQL 5.6.11
I have a table in MySQL database named state_table with three columns as an example.
state_id (BigInt)
state_name (Varchar)
country_id (BigInt)
state_id is a auto-generated primary key and country_id is a foreign key that references a primary key of the country table.
This table is mapped by its corresponding entity class named StateTable and the data held by this table are displayed in a Primefaces DataTable, <p:dataTable>...</p:dataTable>.
The DataTable column header contains a clickable sort area, <div> for each column with a sort direction for sorting, when this area is clicked, a String, either ASCENDING or DESCENDING representing the sort order is rendered and a text box for filtering (searching) in which a user enters a search item for each column.
So ultimately, what I get in JSF managed bean is a List of type java.util.List<org.primefaces.model.SortMeta> representing sort orders of the columns of the DataTable that a user wishes.
And a Map of type java.util.Map<java.lang.String, java.lang.String> representing the search column names as keys and search items of the corresponding columns as values (a search item is entered by a user in a text box on the column header of each column of DataTable).
In short, I use List<SortMeta> for sorting and Map<String, String> for filtering/searching.
My code in one of the DAOs to get a list of rows after sorting and filtering is as follows.
#Override
#SuppressWarnings("unchecked")
public List<StateTable> getList(int first, int pageSize, List<SortMeta> multiSortMeta, Map<String, String>filters)
{
CriteriaBuilder criteriaBuilder = entityManager.getCriteriaBuilder();
CriteriaQuery<StateTable> criteriaQuery = criteriaBuilder.createQuery(StateTable.class);
Metamodel metamodel=entityManager.getMetamodel();
EntityType<StateTable> entityType = metamodel.entity(StateTable.class);
Root<StateTable>root=criteriaQuery.from(entityType);
Join<StateTable, Country> join = null;
//Sorting
List<Order> orders=new ArrayList<Order>();
if(multiSortMeta!=null&&!multiSortMeta.isEmpty())
{
for(SortMeta sortMeta:multiSortMeta)
{
if(sortMeta.getSortField().equalsIgnoreCase("stateId"))
{
orders.add(sortMeta.getSortOrder().equals(SortOrder.ASCENDING)?criteriaBuilder.asc(root.get(StateTable_.stateId)):criteriaBuilder.desc(root.get(StateTable_.stateId)));
}
else if(sortMeta.getSortField().equalsIgnoreCase("stateName"))
{
orders.add(sortMeta.getSortOrder().equals(SortOrder.ASCENDING)?criteriaBuilder.asc(root.get(StateTable_.stateName)):criteriaBuilder.desc(root.get(StateTable_.stateName)));
}
else if(sortMeta.getSortField().equalsIgnoreCase("country.countryName")) // Yes, Primefaces DataTable renders this ugly name in case of a nested property representing a foreign key relationship.
{
join = root.join(StateTable_.countryId, JoinType.INNER);
orders.add(sortMeta.getSortOrder().equals(SortOrder.ASCENDING)?criteriaBuilder.asc(join.get(Country_.countryName)):criteriaBuilder.desc(join.get(Country_.countryName)));
}
}
}
//Filtering/searching
List<Predicate>predicates=new ArrayList<Predicate>();
if(filters!=null&&!filters.isEmpty())
{
for(Entry<String, String>entry:filters.entrySet())
{
if(entry.getKey().equalsIgnoreCase("stateId"))
{
predicates.add(criteriaBuilder.equal(root.get(StateTable_.stateId), Long.parseLong(entry.getValue())));
}
else if(entry.getKey().equalsIgnoreCase("stateName"))
{
predicates.add(criteriaBuilder.like(root.get(StateTable_.stateName), "%"+entry.getValue()+"%"));
}
else if(entry.getKey().equalsIgnoreCase("country.countryName"))// Yes, Primefaces DataTable renders this ugly name in case of a nested property representing a foreign key relationship.
{
if(join==null)
{
join = root.join(StateTable_.countryId, JoinType.INNER);
}
predicates.add(criteriaBuilder.like(join.get(Country_.countryName), "%"+entry.getValue()+"%"));
}
}
}
if(predicates!=null&&!predicates.isEmpty())
{
criteriaQuery.where(predicates.toArray(new Predicate[0]));
}
if(orders!=null&&!orders.isEmpty())
{
criteriaQuery.orderBy(orders);
}
else
{
criteriaQuery.orderBy(criteriaBuilder.desc(root.get(StateTable_.stateId)));
}
TypedQuery<StateTable> typedQuery = entityManager.createQuery(criteriaQuery).setFirstResult(first).setMaxResults(pageSize);
return typedQuery.getResultList();
}
This works as expected but as it can be noticed, the if-else if ladder inside the foreach loop can contain many conditional checks as the number of columns in a database table are increased.
Each column requires a conditional check for both sorting and searching. Is there an efficient way to get rid of these conditional checks that can ultimately remove or at least minimize this if-else if ladder?
P.S. In case of country, I'm doing sorting and searching on countryName (which is available in the parent table country) rather than countryId. Hence, I'm using Join, in this case.
If you drop the usage of SingularAttribute values and you make sure that the caller calls the method with exactly the desired column names in sort/filter fields, then you could simplify it a lot more by just reusing the iterated sort/filter field as column name without the need for an if/else check on the field in order to specify the right column name (which is after all actually identical to the sort/filter field name).
Essentially, you don't need those equalsIgnoreCase() checks in if-else ladder at all. As to case sensitivity, if the caller is doing it wrong, just fix it over there instead of being too forgiving on caller's mistakes.
Here's how you could refactor it then:
/**
* #throws NullPointerException When <code>multiSortMeta</code> or <code>filters</code> argument is null.
*/
#SuppressWarnings({ "unchecked", "rawtypes" })
public List<?> getList(int first, int pageSize, List<SortMeta> multiSortMeta, Map<String, String> filters) {
// ...
Root<StateTable> root = criteriaQuery.from(entityType);
Join<StateTable, Country> join = root.join(StateTable_.countryId, JoinType.INNER);
List<Order> orders = new ArrayList<Order>();
for (SortMeta sortMeta : multiSortMeta) {
String[] sortField = sortMeta.getSortField().split("\\.", 2);
Path<Object> path = sortField.length == 1 ? root.get(sortField[0]) : join.get(sortField[1]);
orders.add(sortMeta.getSortOrder() == SortOrder.ASCENDING
? criteriaBuilder.asc(path)
: criteriaBuilder.desc(path));
}
List<Predicate>predicates = new ArrayList<Predicate>();
for (Entry<String, String> filter : filters.entrySet()) {
String[] filterField = filter.getKey().split("\\.", 2);
Path path = filterField.length == 1 ? root.get(filterField[0]): join.get(filterField[1]);
predicates.add(filter.getValue().matches("[0-9]+")
? criteriaBuilder.equal(path, Long.valueOf(filter.getValue()))
: criteriaBuilder.like(path, "%" + filter.getValue() + "%"));
}
// ...
}
Note that I also modified the method to not accept null as sort and filter meta, so that you can safely trim out all those null checks. Those empty checks are unnecessary as the for loop won't iterate anyway if it's empty. Also note that the filtering uses CriteriaBuilder#equal() if a numeric input is given, otherwise it uses like(). I'm not sure if that covers all your cases, you may want to finetune that more.
You can if necessary refactor the obtaining of Path even more with the following helper method:
#SuppressWarnings("rawtypes")
private static Path<?> getPath(String field, Root root, Join join) {
String[] fields = field.split("\\.", 2);
return fields.length == 1 ? root.get(fields[0]): join.get(fields[1]);
}