We Keep Coding

Java EE + Oracle: how to use GTT(global temporary table) to avoid possible long(1000+) IN clause?

I have a query for Oracle database, built with CriteriaBuilder of Hibernate. Now, it has a IN clause which already takes about 800+ params.
Team says this may surpass 1000 and hitting the hard upper limit of Oracle itself, which only allows 1000 param for an IN clause. We need to optimize that.
select ih from ItemHistory as ih
where ih.number=:param0
and
ih.companyId in (
select c.id from Company as c
where (
( c.organizationId in (:param1) )
or
( c.organizationId like :param2 )
) and (
c.organizationId in (:param3, :param4, :param5, :param6, :param7, :param8, :param9, :param10, ..... :param818)
)
)
order by ih.eventDate desc
So, two solutions I can think of:
The easy one, as now the list from :param3 to :param818 are below 1000, and in the future, we may hit 1000, we can separate the list if size > 1000, into another IN clause, so it becomes:
c.organizationId in (:param3, :param4, :param5, :param6, :param7, :param8, :param9, :param10, ..... :param1002) or c.organizationId in (:param1003, ...)
Both the original code and solution 1 are not very efficient already. Although it can fetch 40K records in 25 seconds, we should use a GTT(Global Temporary Table), as per what I can find on AskTom or other sites, by professional DBAs. But I can only find SQL examples, not Java code.
What I can imagine is:
createNativeQuery("create global temporary table GTT_COMPANIES if not exist (companyId varchar(32)) ON COMMIT DELETE ROWS;"); and execute(Do we need index here?)
createNativeQuery("insert into GTT_COMPANIES (list)"); query.bind("1", query.getCompanyIds()); and execute(can we bind a list and insert it?)
use CriteriaQuery to select from this table(but I doubt, as CriteriaQueryBuilder will require type safe meta model class to be generated beforehand, and here we don't have the entity; this is an ad-hoc table and no model entity is mapped to it)
and, do we need to create GTT even the list size is < 1000? As often it is big, 700~800.
So, any suggestion? Someone got a working example of Hibernate CriteriaQuery + Oracle GTT?
The whole method is like this:
public List<ItemHistory> findByIdTypePermissionAndOrganizationIds(final Query<String> query, final ItemIdType idType) throws DataLookupException {
String id = query.getObjectId();
String type = idType.name();
Set<String> companyIds = query.getCompanyIds();
Set<String> allowedOrgIds = query.getAllowedOrganizationIds();
Set<String> excludedOrgIds = query.getExcludedOrganizationIds();
// if no orgs are allowed, we should return empty list
if (CollectionUtils.isEmpty(allowedOrgIds)) {
return Collections.emptyList();
}
try {
CriteriaBuilder builder = entityManager.getCriteriaBuilder();
CriteriaQuery<ItemHistory> criteriaQuery = builder.createQuery(ItemHistory.class);
Subquery<String> subQueryCompanyIds = filterByPermissionAndOrgIdsInSubquery(query, builder, criteriaQuery);
Subquery<String> subQueryCompanyIds = criteriaQuery.subquery(String.class);
Root<Company> companies = subQueryCompanyIds.from(Company.class);
companies.alias(COMPANY_ALIAS);
Path<String> orgIdColumn = companies.get(Company_.organizationId);
/* 1. get permission based restrictions */
// select COMPANY_ID where (ORG_ID in ... or like ...) and (ORG_ID not in ... and not like ...)
// actually query.getExcludedOrganizationIds() can also be very long list(1000+), but let's do it later
Predicate permissionPredicate = getCompanyIdRangeByPermission(
builder, query.getAllowedOrganizationIds(), query.getExcludedOrganizationIds(), orgIdColumn
);
/* 2. get org id based restrictions, which was done on top of permission restrictions */
// ... and where (ORG_ID in ... or like ...)
// process companyIds with and without "*" by adding different predicates, like (xxx%, yyy%) vs in (xxx, yyy)
// here, query.getCompanyIds() could be very long, may be 1000+
Predicate orgIdPredicate = groupByWildcardsAndCombine(builder, query.getCompanyIds(), orgIdColumn, false);
/* 3. Join two predicates with AND, because originally filtering is done twice, 2nd is done on basis of 1st */
Predicate subqueryWhere = CriteriaQueryUtils.joinWith(builder, true, permissionPredicate, orgIdPredicate); // join predicates with AND
subQueryCompanyIds.select(companies.get(Company_.id)); // id -> COMPANY_ID
if (subqueryWhere != null) {
subQueryCompanyIds.where(subqueryWhere);
} else {
LOGGER.warn("Cannot build subquery of org id and permission. " +
"Org ids: {}, allowed companies: {}, excluded companies: {}",
query.getCompanyIds(), query.getAllowedOrganizationIds(), query.getExcludedOrganizationIds());
}
Root<ItemHistory> itemHistory = criteriaQuery.from(ItemHistory.class);
itemHistory.alias(ITEM_HISTORY_ALIAS);
criteriaQuery.select(itemHistory)
.where(builder.and(
builder.equal(getColumnByIdType(itemHistory, idType), id),
builder.in(itemHistory.get(ItemHistory_.companyId)).value(subQueryCompanyIds)
))
.orderBy(builder.desc(itemHistory.get(ItemHistory_.eventDate)));
TypedQuery<ItemHistory> finalQuery = entityManager.createQuery(criteriaQuery);
LOGGER.trace(LOG_MESSAGE_FINAL_QUERY, finalQuery.unwrap(org.hibernate.Query.class).getQueryString());
return finalQuery.setMaxResults(MAX_LIST_FETCH_SIZE).getResultList();
} catch (NoResultException e) {
LOGGER.info("No item history events found by permission and org ids with {}={}", type, id);
throw new DataLookupException(ErrorCode.DATA_LOOKUP_NO_RESULT);
} catch (Exception e) {
LOGGER.error("Error when fetching item history events by permission and org ids with {}={}", type, id, e);
throw new DataLookupException(ErrorCode.DATA_LOOKUP_ERROR,
"Error when fetching item history events by permission and org ids with " + type + "=" + id);
}
}

Script in Groovy to update and insert Initial_Range and Final_Range

Im new on Groovy and need some help.
I have that table called RANGE with :
ID_RANGE - Integer
Year - Integer
six_Month - Integer
Initial_Range - String
Final_Range - String
Last_Update - TimeStamp
I have to do a script to update/insert Initial_Range and Final_Range.
I will receive the new data from a map called "df_map" with the following data:
df_map.Update_date -> String
df_map.six_Month -> Integer
df_map.Initial_range -> String (Format "AA-123456678")
df_map.Final_range -> String (Format "AA-123456678")
That script have to validate some requirements,
if it doesn't, it can´t continues:
the date have to be a valid timeStamp (yyyy-MM-dd HH:mm:ss.SSS)
there must be 1000 values ​​between Initial_Range and Final_Range
Only can update or insert a future date
When check all this, have to search if the register to modify exits, if not exits, have to create it.
that script has helpers to do 2 actions:
sqlQuery -
Make a query against the bbdd and show and array with the data. It get parameters like bbdd,query and parameters maps.
EXAMPLE:
query = "SELECT * FROM RANGE WHERE ID_RANGE = :RANGE"
params = [:]
params.RANGE = 1
outcome = sqlQuery(bbdd,query,params)
-sqlUpdate -
It make an insert or an update against the database and returns an array with the result. It receives as parameters the database, query, parameter map.
EXAMPLE:
query = "UPDATE RANGE SET Initial_Range = :Initial_Range WHERE ID_RANGE = :RANGE"
params = [:]
params.RANGE = 1
outcome = sqlUpdate(bbdd,query,params)
I work with sql but never before with groovy.....
Thanks in advance ;)

How to implement proper pagination in Google App Engine (Java)?

I tried to implement pagination in google app engine (Java), but I am not able to achieve. It is working only forward pagination and reverse pagination is not able to achieved.
I tried storing the previous cursor value through HTTP request as below:
JSP file:
<a href='/myServlet?previousCursor=${previousCursor}'>Previous page</a>
<a href='/myServlet?nextCursor=${nextCursor}'>Next page</a>
Servlet file:
String previousCursor = req.getParameter("previousCursor");
String nextCursor = req.getParameter("nextCursor");
String startCursor = null;
if(previousCursor != null){
startCursor = previousCursor;
}
if(nextCursor != null){
startCursor = nextCursor;
}
int pageSize = 3;
FetchOptions fetchOptions = FetchOptions.Builder.withLimit(pageSize);
if (startCursor != null) {
fetchOptions.startCursor(Cursor.fromWebSafeString(startCursor));
}
Query q = new Query("MyQuery");
PreparedQuery pq = datastore.prepare(q);
QueryResultList<Entity> results = pq.asQueryResultList(fetchOptions);
for (Entity entity : results) {
//Get the properties from the entity
}
String endCursor = results.getCursor().toWebSafeString();
req.setAttribute("previousCursor", startCursor);
req.setAttribute("nextCursor", endCursor);
With this I am able to retain the previous cursor value, but unfortunately the previous cursor seems to be invalid.
I also tried using reverse() method, but it is of no use. It work same as forward.
So is the any way to implement proper pagination (forward and backword) in google app engine (Java)?
I found similar one that was posted in 2010. Here also the answer was to use Cursor. But as I shown above it is not working.
Pagination in Google App Engine with Java

If you are familiar with JPA you can give it a try.
Have tested it and pagination works in GAE.
I think they support JPA 1.0 as of now.
What I tried was, created an Employee entity.
Created DAO layer and persisted few employee entities.
To have a paginated fetch, did this:
Query query = em.createQuery("select e from Employee e");
query.setFirstResult(0);
query.setMaxResults(2);
List<Employee> resultList = query.getResultList();
(In this example we get first page which has 2 entities. Argument to
setFirstResult would be start index and argument to setMaxResult would be your page size)
You can easily change the arguments to query.setFirstResult and setMaxResults
and have a pagination logic around it.
Hope this helps,
Regards,

How to fetch WorkItem links with the TFS Java API

We use the TFS Java API to fetch WorkItems from a TFS server:
TFSTeamProjectCollection collection = TFSTeamProjectCollectionUtils
.openTeamProjectCollection(serverUrl, credentials,
new DefaultConnectionAdvisor(Locale.getDefault(),
TimeZone.getDefault()));
WorkItemClient client = collection.getWorkItemClient();
List<WorkItem> result = new ArrayList<>();
try {
WorkItemCollection workItems = client.query(wiqlQuery, null, false);
for (int i = 0; i < workItems.size(); i++) {
WorkItem item = workItems.getWorkItem(i);
result.add(item);
}
return result;
} catch (TECoreException e) {
throw new ConQATException("Failed to fetch work items from TFS", e);
}
If I run the query select * from workitems I get all workitems on the server with all fields and all links. Since I'm only interested in some of the fields, I would like to restrict the query to only those and save some bandwidth/time: select ID, Title from workitems
This works fine, but now the links of the items are missing (i.e. item.getLinks() always returns an empty collection).
Is there a way to select the links other than select * from workitems?

After some more digging around, I found that you can create a link query and run it like this:
WorkItemLinkInfo[] infos = client.createQuery("select * from workitemlinks").runLinkQuery()
With this, you can get the links as WorkItemLinkInfo objects that contain the IDs of the target and source node and the link type.

The solution using WorkItemLinkInfo is correct.
Just as remark: Using a WIQL Query you only receive the attributes you were querying - which cannot be the set of links of a work item (therefore always empty). If you query a single workitem using
WorkItemClient client = TFSConnection.getClient();
WorkItem firstWorkItem = client.getWorkItemByID(id);
then you also get the LinkCollection using (containing RelatedLinks, ExternalLinks or HyperLinks)
LinkCollection linkcoll = firstWorkItem.getLinks()

Grouping Solr results in Solr 3.6.1 API causes NullPointerException when parsing result

As long as I limit my query to:
SolrQuery solrQuery = new SolrQuery();
solrQuery.set("q", query); //where query is solr query string (e.g. *:*)
solrQuery.set("start", 0);
solrQuery.set("rows", 10);
everything works fine - results are returned and so on.
Things are getting worse when I try to group results by my field "Token_group" to avoid duplicates:
SolrQuery solrQuery = new SolrQuery();
solrQuery.set("q", query); //where query is solr query string (e.g. *:*)
solrQuery.set("start", 0);
solrQuery.set("rows", 10);
solrQuery.set("group", true);
solrQuery.set("group.field", "token_group");
solrQuery.set("group.ngroups", true);
solrQuery.set("group.limit", 20);
Using this results in HttpSolrServer no exceptions are being thrown, but trying to access results ends up in NPE.
My querying Solr method:
public SolrDocumentList query(SolrQuery query) throws SolrServerException {
QueryResponse response = this.solr.query(query); //(this.solr is handle to HttpSolrSelver)
SolrDocumentList list = response.getResults();
return list;
}
note that similar grouping (using the very same field) is made in our other apps (PHP) and works fine, so this is not a schema issue.

I solved my issue. In case someone needs this in future:
When you perform a group query, you should use different methods to get and parse results.
While in ungrouped queries
QueryResponse response = this.solr.query(query); //(this.solr is handle to HttpSolrSelver)
SolrDocumentList list = response.getResults();
will work, when you want to query for groups, it won't.
So, how do I make and parse query?
Below code for building query is perfectly fine:
SolrQuery solrQuery = new SolrQuery();
solrQuery.set("q", query); //where query is solr query string (e.g. *:*)
solrQuery.set("start", 0);
solrQuery.set("rows", 10);
solrQuery.set("group", true);
solrQuery.set("group.field", "token_group");
solrQuery.set("group.ngroups", true);
solrQuery.set("group.limit", 20);
where last four lines define that Solr should group results and parameters of grouping. In this case group.limit will define how many maximum results within a group you want, and rows will tell how many max results should be there.
Making grouped query looks like this:
List<GroupCommand> groupCommands = this.solr.query(query).getGroupResponse().getValues();
referring to documentation, GroupCommand contains info about grouping as well as list of results, divided by groups.
Okay, I want to get to the results. How to do it?
Well, in my example there's only one position in List<GroupCommand> groupCommands, so to get list of found groups within it:
GroupCommand groupCommand = groupCommands.get(0);
List<Group> groups = groupCommand.getValues();
This will result in list of groups. Each group contains its own SolrDocumentList. To get it:
for(Group g : groups){
SolrDocumentList groupList = g.getResult();
(...)
}
Having this, well just proceed with SolrDocumentList for each group.
I used grouping query to get list of distinct results. How to do it?
This was exacly my case. It seems easy but there's a tricky part that can catch you if you're refactoring already running code that uses getNumFound() from SolrDocumentList.
Just analyze my code:
/**
* Gets distinct resultlist from grouped query
*
* #param query
* #return results list
* #throws SolrServerException
*/
public SolrDocumentList queryGrouped(SolrQuery query) throws SolrServerException {
List<GroupCommand> groupCommands = this.solr.query(query).getGroupResponse().getValues();
GroupCommand groupCommand = groupCommands.get(0);
List<Group> groups = groupCommand.getValues();
SolrDocumentList list = new SolrDocumentList();
if(groups.size() > 0){
long totalNumFound = groupCommand.getNGroups();
int iteratorLimit = 1;
for(Group g : groups){
SolrDocumentList groupList = g.getResult();
list.add(groupList.get(0));
//I wanted to limit list to 10 records
if(iteratorLimit++ > 10){
break;
}
}
list.setNumFound(totalNumFound);
}
return list;
}