We Keep Coding

Java - CMIS query - WHERE IN

I have my connection to SP through open CMIS.
I am trying to get cmis:contentStreamLength based on many cmis:objectId.
This select, for all documents works fine.
SELECT cmis:contentStreamLength FROM cmis:document
But this query with where cond. doesn't.
SELECT cmis:contentStreamLength FROM cmis:document d WHERE ANY d.cmis:objectId IN ('id',...)
OR
SELECT * FROM cmis:document d WHERE d.cmis:objectId = '"+contentId+"'" (just one id in form "id")
Here is the code in Java (1.4).
public int getSizeInBytes(String contentIds) {
String query = "SELECT * FROM cmis:document d WHERE ANY d.cmis:objectId IN ("+contentIds+")";
ItemIterable results = session.query(query,false);
int size=0;
Iterator iterator = results.iterator();
while (iterator.hasNext()) {
QueryResult res = (QueryResult) iterator.next();
size += Integer.parseInt(res.getPropertyById("cmis:contentStreamLength").getFirstValue().toString());
}
return size;
}
contetntIds are in the form of string like so - "'id','id',..." - id as xxxx:xxxx
It breaks on while (iterator.hasNext())
exception :
CmisInvalidArgumentException: One or more of the input parameters to the service method is missing or invalid.
When I did
where d.cmis:createdBy = "nonsens/non existent"
It just worked, like it didn't show a thing, so empty result is ok.
What seems to be the problem ?
Everything I tried is in details of my problem.
I expect to get size of all documents with inputed ids.

Java EE + Oracle: how to use GTT(global temporary table) to avoid possible long(1000+) IN clause?

I have a query for Oracle database, built with CriteriaBuilder of Hibernate. Now, it has a IN clause which already takes about 800+ params.
Team says this may surpass 1000 and hitting the hard upper limit of Oracle itself, which only allows 1000 param for an IN clause. We need to optimize that.
select ih from ItemHistory as ih
where ih.number=:param0
and
ih.companyId in (
select c.id from Company as c
where (
( c.organizationId in (:param1) )
or
( c.organizationId like :param2 )
) and (
c.organizationId in (:param3, :param4, :param5, :param6, :param7, :param8, :param9, :param10, ..... :param818)
)
)
order by ih.eventDate desc
So, two solutions I can think of:
The easy one, as now the list from :param3 to :param818 are below 1000, and in the future, we may hit 1000, we can separate the list if size > 1000, into another IN clause, so it becomes:
c.organizationId in (:param3, :param4, :param5, :param6, :param7, :param8, :param9, :param10, ..... :param1002) or c.organizationId in (:param1003, ...)
Both the original code and solution 1 are not very efficient already. Although it can fetch 40K records in 25 seconds, we should use a GTT(Global Temporary Table), as per what I can find on AskTom or other sites, by professional DBAs. But I can only find SQL examples, not Java code.
What I can imagine is:
createNativeQuery("create global temporary table GTT_COMPANIES if not exist (companyId varchar(32)) ON COMMIT DELETE ROWS;"); and execute(Do we need index here?)
createNativeQuery("insert into GTT_COMPANIES (list)"); query.bind("1", query.getCompanyIds()); and execute(can we bind a list and insert it?)
use CriteriaQuery to select from this table(but I doubt, as CriteriaQueryBuilder will require type safe meta model class to be generated beforehand, and here we don't have the entity; this is an ad-hoc table and no model entity is mapped to it)
and, do we need to create GTT even the list size is < 1000? As often it is big, 700~800.
So, any suggestion? Someone got a working example of Hibernate CriteriaQuery + Oracle GTT?
The whole method is like this:
public List<ItemHistory> findByIdTypePermissionAndOrganizationIds(final Query<String> query, final ItemIdType idType) throws DataLookupException {
String id = query.getObjectId();
String type = idType.name();
Set<String> companyIds = query.getCompanyIds();
Set<String> allowedOrgIds = query.getAllowedOrganizationIds();
Set<String> excludedOrgIds = query.getExcludedOrganizationIds();
// if no orgs are allowed, we should return empty list
if (CollectionUtils.isEmpty(allowedOrgIds)) {
return Collections.emptyList();
}
try {
CriteriaBuilder builder = entityManager.getCriteriaBuilder();
CriteriaQuery<ItemHistory> criteriaQuery = builder.createQuery(ItemHistory.class);
Subquery<String> subQueryCompanyIds = filterByPermissionAndOrgIdsInSubquery(query, builder, criteriaQuery);
Subquery<String> subQueryCompanyIds = criteriaQuery.subquery(String.class);
Root<Company> companies = subQueryCompanyIds.from(Company.class);
companies.alias(COMPANY_ALIAS);
Path<String> orgIdColumn = companies.get(Company_.organizationId);
/* 1. get permission based restrictions */
// select COMPANY_ID where (ORG_ID in ... or like ...) and (ORG_ID not in ... and not like ...)
// actually query.getExcludedOrganizationIds() can also be very long list(1000+), but let's do it later
Predicate permissionPredicate = getCompanyIdRangeByPermission(
builder, query.getAllowedOrganizationIds(), query.getExcludedOrganizationIds(), orgIdColumn
);
/* 2. get org id based restrictions, which was done on top of permission restrictions */
// ... and where (ORG_ID in ... or like ...)
// process companyIds with and without "*" by adding different predicates, like (xxx%, yyy%) vs in (xxx, yyy)
// here, query.getCompanyIds() could be very long, may be 1000+
Predicate orgIdPredicate = groupByWildcardsAndCombine(builder, query.getCompanyIds(), orgIdColumn, false);
/* 3. Join two predicates with AND, because originally filtering is done twice, 2nd is done on basis of 1st */
Predicate subqueryWhere = CriteriaQueryUtils.joinWith(builder, true, permissionPredicate, orgIdPredicate); // join predicates with AND
subQueryCompanyIds.select(companies.get(Company_.id)); // id -> COMPANY_ID
if (subqueryWhere != null) {
subQueryCompanyIds.where(subqueryWhere);
} else {
LOGGER.warn("Cannot build subquery of org id and permission. " +
"Org ids: {}, allowed companies: {}, excluded companies: {}",
query.getCompanyIds(), query.getAllowedOrganizationIds(), query.getExcludedOrganizationIds());
}
Root<ItemHistory> itemHistory = criteriaQuery.from(ItemHistory.class);
itemHistory.alias(ITEM_HISTORY_ALIAS);
criteriaQuery.select(itemHistory)
.where(builder.and(
builder.equal(getColumnByIdType(itemHistory, idType), id),
builder.in(itemHistory.get(ItemHistory_.companyId)).value(subQueryCompanyIds)
))
.orderBy(builder.desc(itemHistory.get(ItemHistory_.eventDate)));
TypedQuery<ItemHistory> finalQuery = entityManager.createQuery(criteriaQuery);
LOGGER.trace(LOG_MESSAGE_FINAL_QUERY, finalQuery.unwrap(org.hibernate.Query.class).getQueryString());
return finalQuery.setMaxResults(MAX_LIST_FETCH_SIZE).getResultList();
} catch (NoResultException e) {
LOGGER.info("No item history events found by permission and org ids with {}={}", type, id);
throw new DataLookupException(ErrorCode.DATA_LOOKUP_NO_RESULT);
} catch (Exception e) {
LOGGER.error("Error when fetching item history events by permission and org ids with {}={}", type, id, e);
throw new DataLookupException(ErrorCode.DATA_LOOKUP_ERROR,
"Error when fetching item history events by permission and org ids with " + type + "=" + id);
}
}

Script in Groovy to update and insert Initial_Range and Final_Range

Im new on Groovy and need some help.
I have that table called RANGE with :
ID_RANGE - Integer
Year - Integer
six_Month - Integer
Initial_Range - String
Final_Range - String
Last_Update - TimeStamp
I have to do a script to update/insert Initial_Range and Final_Range.
I will receive the new data from a map called "df_map" with the following data:
df_map.Update_date -> String
df_map.six_Month -> Integer
df_map.Initial_range -> String (Format "AA-123456678")
df_map.Final_range -> String (Format "AA-123456678")
That script have to validate some requirements,
if it doesn't, it can´t continues:
the date have to be a valid timeStamp (yyyy-MM-dd HH:mm:ss.SSS)
there must be 1000 values ​​between Initial_Range and Final_Range
Only can update or insert a future date
When check all this, have to search if the register to modify exits, if not exits, have to create it.
that script has helpers to do 2 actions:
sqlQuery -
Make a query against the bbdd and show and array with the data. It get parameters like bbdd,query and parameters maps.
EXAMPLE:
query = "SELECT * FROM RANGE WHERE ID_RANGE = :RANGE"
params = [:]
params.RANGE = 1
outcome = sqlQuery(bbdd,query,params)
-sqlUpdate -
It make an insert or an update against the database and returns an array with the result. It receives as parameters the database, query, parameter map.
EXAMPLE:
query = "UPDATE RANGE SET Initial_Range = :Initial_Range WHERE ID_RANGE = :RANGE"
params = [:]
params.RANGE = 1
outcome = sqlUpdate(bbdd,query,params)
I work with sql but never before with groovy.....
Thanks in advance ;)

How to fetch complex model with a result of select statment using fetchgoups and map

I'm using Jooq with Kotlin and i want to write a statement that fetches data from a query that uses couple of tables using join statement(example attached)
The problem I'm facing is that I want to map the result to my complex model which consist of one to many relationships and also many to many.
According to my knowledge I know i can use fetchgroups operation in Jooq to some how map the records but i still can't figure out how to get the result into my model.
my model:
data class MicroserviceDto(
val microservice_id: Long = 1,
val microservice_name: String? = "",
val endpoint: String? = "",
val mappings: String? = "",
val solutionDefinitionMinimalDtoList: List<SolutionDefinitionDto> = emptyList(),
val projectFileDtoList: List<ProjectFileDto> = emptyList()
)
data class SolutionDefinitionDto(
val solution_definition_id: Long = 0L,
val solution_definition_name: String = "",
val solutionId: String = "",
val solutionVersion: String = ""
)
data class ProjectFileDto(
val project_file_id: Long = 1,
val model: String = "",
val relativePath: String = "",
val fileContentDtoList: List<FileContentDto> = emptyList()
)
data class FileContentDto(
val file_content_id: Long = 1,
val content: ByteArray = ByteArray(0)
)
Link to my schema diagram
Database Diagram visualization
Explanation of the diagram:
Microservice has many to many relationship with SolutionDefinistion
ProjectFile has one to many relationship with Microservice
ProjectFile has one to many relationship with SolutionDefinition
FileContent has one to many with ProjectFile
I've created a view to represent my desired query with all tables and the join statements between them.
Here is the View:
CREATE OR REPLACE VIEW Microservice_Metadata_by_Microservice_Id AS
select
# microservice
M.id as `microservice_id`,
M.name as `microservice_name`,
M.mappings,
M.endpoint,
# solution definition
SD.id as `solution_definition_id`,
SD.name as `solution_definition_name`,
SD.solutionId,
SD.solutionVersion,
# project file of microservice
PF.id as `project_file_id`,
PF.relativePath,
PF.model,
# file content data of project file
FC.id as `file_content_id`,
FC.content
from Microservice M
# get project file
left join Microservice_SolutionDefinition MSD
on MSD.microserviceId = M.id
left join ProjectFile PF
on PF.microserviceId = M.id
# get data content
left JOIN FileContent FC
on PF.id = FC.projectFileId
# get solutions of microservice
left join SolutionDefinition SD
on SD.id = MSD.solutionDefinitionId;
How can I implement such a Jooq dsl query that map the ResultSet to my data model

SPARK SQL not in or NOT EXISTS

I have a scenario where there are two sheets(csv). Created two tables for it.
When there s a good data I could map it with a value( id statistic value) from the second table. If I have bad data I should map it with id statistic value again (but a different value). however, I'm unable to use not exists in spark SQL. And I get the error below:
mismatched input 'from' expecting {, 'WHERE', 'GROUP', 'ORDER', 'HAVING', 'LIMIT', 'LATERAL', 'WINDOW', 'UNION', 'EXCEPT', 'INTERSECT', 'SORT', 'CLUSTER', 'DISTRIBUTE'}(line 1, pos 386)
at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:197)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:99)
at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:45)
CODE:
select
a.ptf_id,a.ptf_code,a.share_id,a.share_code,a.bench_id,a.bench_code
, a.l1_calculation_date,a.l1_begin_date,a.l1_end_date,a.l1_running_date
, a.l1_frequency,a.l1_calculation_step,a.l1_performance_currency
, a.l1_configuration,a.l1_valuation_source,a.l1_nav_valuation_type
, a.l1_setting_reference_type, a.l1_setting_valuation_type
, a.l1_sharpe_ratio_annualized as value,b.id_statistic
from
parquetFile a,
pairRDD b,
stats c
where
a.l1_nav_valuation_type= b.l1_nav_valuation_type
and a.l1_valuation_source = b.l1_valuation_source
and b.l1_Perf = 'l1_sharpe_ratio_annualized'
OR (a.ptf_id not EXISTS (
select e.ptf_id from pairRDD d, parquetFile e
where d.l1_valuation_source = e.l1_valuation_source
AND d.l1_nav_valuation_type = e.l1_nav_valuation_type)
and b.l1_valuation_source ='')
this query works in SQL if I use "NOT in"
please help me on what other options to use in this scenario other than not exists.

The query is a bit hard to follow as written. It's not clear (to me anyway) what your intentions are for the stats table since you don't select anything from it so I removed it. Obviously I haven't tried this myself but at a glance, you might try something like this instead:
select
a.PTF_ID,
a.PTF_CODE,
a.SHARE_ID,
a.SHARE_CODE,
a.BENCH_ID,
a.BENCH_CODE,
a.L1_CALCULATION_DATE,
a.L1_BEGIN_DATE,
a.L1_END_DATE,
a.L1_RUNNING_DATE,
a.L1_FREQUENCY,
a.L1_CALCULATION_STEP,
a.L1_PERFORMANCE_CURRENCY,
a.L1_CONFIGURATION,
a.L1_VALUATION_SOURCE,
a.L1_NAV_VALUATION_TYPE,
a.L1_SETTING_REFERENCE_TYPE,
a.L1_SETTING_VALUATION_TYPE,
a.L1_SHARPE_RATIO_ANNUALIZED as VALUE,
b.ID_STATISTIC
from
PARQUETFILE a
inner join
PAIRRDD b
on
a.L1_NAV_VALUATION_TYPE = b.L1_NAV_VALUATION_TYPE and
a.L1_VALUATION_SOURCE = b.L1_VALUATION_SOURCE
left outer join (
select
e.PTF_ID
from
PAIRRDD d
inner join
PARQUETFILE e
on
d.L1_VALUATION_SOURCE = e.L1_VALUATION_SOURCE AND
d.L1_NAV_VALUATION_TYPE = e.L1_NAV_VALUATION_TYPE
where
b.L1_VALUATION_SOURCE = ''
) x
on
a.PTF_ID = x.PTF_ID
where
b.L1_PERF = 'l1_sharpe_ratio_annualized' or
x.PTF_ID is null