How to avoid automatic truncation of GROUP_CONCAT result - java

I am trying to retrieve a long set of result and in doing so I find that the result data is being truncated automatically. I also noticed that the limit of result(String) is 1024 characters. How can I avoid this to get whole data?
I am accessing tables through JDBC, if that matters.
Code:-
SELECT OLL.EMAIL_ID,COUNT(1) AS TOTAL_ORDERS,OLL.SHIPPING_NAME ,SUM(OFF.UNIT_PRICE*OFF.QUANTITY) AS TOTAL_GMV,
MIN(ODD.CREATION_DATE) AS FIRST_PURCHASE,MAX(ODD.CREATION_DATE) AS LAST_PURCHASE,GROUP_CONCAT(OFF.PAYMENT_MODE) AS PAY_MODES,
GROUP_CONCAT(OFLL.PRODUCT_ID)AS PRODUCT_IDS,LEFT(GROUP_CONCAT(OFLL.PRODUCT_NAME),2048) AS PRODUCTS,OLL.BRAND_NAME,GROUP_CONCAT(OFF.GCTYPE) AS GCTYPE,
OLL.SHIPPING_CITY,OLL.SHIPPING_STATE,GROUP_CONCAT(OFF.UNIT_PRICE-OFF.UNIT_SHIPPING_PRICE) AS PRODUCT_PRICE,GROUP_CONCAT(CEILING(OFF.QUANTITY*OFF.UNIT_PRICE)) AS PRODUCT_GMV,GROUP_CONCAT(OFLL.CHANNEL) AS SALES_CHANNEL
FROM (
SELECT COUNT(1), OL.ORDER_ID, OL.EMAIL_ID, OFL.CREATED_BY,OL.SELLER_CITY,OL.SHIPPING_STATE
FROM (SELECT CREATION_DATE,ORDER_ID,ORDER_ITEM_SEQ_ID FROM ORDER_DATES WHERE CREATION_DATE
BETWEEN '2015-01-30 23:55:00' AND '2015-01-30 23:59:59') OD
INNER JOIN ORDER_LOGISTICS OL ON OD.ORDER_ID = OL.ORDER_ID AND OD.ORDER_ITEM_SEQ_ID = OL.ORDER_ITEM_SEQ_ID
INNER JOIN ORDER_FINANCE OF ON OL.ORDER_ID = OF.ORDER_ID AND OL.ORDER_ITEM_SEQ_ID = OF.ORDER_ITEM_SEQ_ID
INNER JOIN ORDER_FILTERS OFL ON OL.ORDER_ID = OFL.ORDER_ID AND OL.ORDER_ITEM_SEQ_ID = OFL.ORDER_ITEM_SEQ_ID
INNER JOIN ORDER_STATUS OS ON OL.ORDER_ID = OS.ORDER_ID AND OL.ORDER_ITEM_SEQ_ID = OS.ORDER_ITEM_SEQ_ID
WHERE (OF.PAYMENT_MODE='Cash On Delivery' OR OS.PAYMENT_STATUS='Received')AND (OFL.PRODUCT_ID IN ('B4333897','B5163012','B5654542') OR OF.UNIT_PRICE-OF.UNIT_SHIPPING_PRICE='1.00') AND
OFL.CHANNEL IN ('Web Channel', 'Mobile Web Channel') AND
OL.EMAIL_ID IS NOT NULL GROUP BY OL.EMAIL_ID) AAA, ORDER_LOGISTICS OLL, ORDER_FILTERS OFLL,
ORDER_FINANCE OFF,ORDER_DATES ODD WHERE AAA.EMAIL_ID=OLL.EMAIL_ID AND
OLL.ORDER_ID=OFLL.ORDER_ID AND OLL.ORDER_ITEM_SEQ_ID=OFLL.ORDER_ITEM_SEQ_ID AND (OFLL.PRODUCT_ID IN ('B4333897','B5163012','B5654542') OR OFF.UNIT_PRICE-OFF.UNIT_SHIPPING_PRICE='1.00') AND
OFLL.ORDER_ID=ODD.ORDER_ID AND OFF.ORDER_ID=OLL.ORDER_ID AND OFF.ORDER_ITEM_SEQ_ID=OLL.ORDER_ITEM_SEQ_ID
GROUP BY OLL.EMAIL_ID
Since I am using GROUP_CONCAT(), the result comes out to be lengthy and that's where it gets truncated.

Related

How resolve "unknown field 'TableField'

Hi guys I have a problem with SPEL - Flexible Search, this is my error log when into my entity I click on button search:
ERROR [hybrisHTTP6] [PagingDelegateController] cannot search unknown field 'TableField(name='inStockStatus',langPK='null',type=Product)' within type Product unless you disable checking
and this is the query that I must create :
select {p.code},{p.description},{bs.uid}, (CASE WHEN ({p.onlineDate} is not null AND {p.onlineDate} > current_timestamp ) THEN 1 else 0 END)
from {Product as p join StockLevel as s on {s.productCode} = {p.code} join BaseStore2WarehouseRel as b on {b.target} = {s.warehouse} join BaseStore as bs on {bs.pk} = {b.source}}
where {bs.uid} in ('baseStorePk')
and {p.code} = '?productCode'
and {p.description} = '?description'
and {p.descriptionCics} = '?descriptionCics'
and {p.onlinedate} <= '?onlineDateFrom'
and {p.onlinedate} >= '?onlineDateTo'
and {s.inStockStatus} = '?inStockStatus'
and {p.doneBy} = '?doneBy'
and {s.outOfStockCause} = '?oosCause'
and {p.department} = '?department'
and {p.grm} = '?grm'
and in the image the report that I have create into myExtension-items-core.xml
based on Exception it's clear you are referring wrong attribute of product is "inStockStatus".
please replace with {s.inStockStatus} instead of {p.inStockStatus}

CASE WHEN on a passed in variable

I'm using Java 11 and Postgres 9.6. I am writing a SQL query that will be included in my Java application and I will pass in values to the variables designated with a:
SQL
CASE
WHEN (:designatorInclude IS NOT NULL)
THEN totdl.designator_id IN (:designatorInclude)
ELSE ttl.product_code IS NOT NULL
END
Java
String filterProductCodes = IOUtil.readResourceText(MyService.class, "sql_files/FilterProductCodes.sql");
Query filterProductCodesQuery = this.em.createNativeQuery(filterProductCodes)
if(inDto.getIdList() != null)
{
filterProductCodesQuery.setParameter("designatorInclude", inDto.getIdList);
}else{
filterProductCodesQuery.setParameter("designatorInclude", null);
}
Passing variables in like this works in many other places in the application. The difference here is that I'm using a CASE/WHERE statement.
Everything compiles fine, but when I try to hit this part of the code, I get the following error:
PSQLException: ERROR: argument of CASE/WHEN must be type boolean, not type record
I found 1 post on the internet about this and it did not apply to my situation.
Can anyone shed some light on this? Any suggestions on how I can get this to work?
Full SQL - case statement is towards the end.
SELECT DISTINCT ttl.product_code,
si.condition,
totdl.designator_id,
ttl.active as ttl_active,
l.active as l_active,
ii.mls_order_id,
ii.damaged,
ttl.id as ttl_id,
ll.location_id as ll_locationId,
cl.location_id as cl_locationId
FROM serial_item si
JOIN inventory_item ii ON ii.id = si.id
AND ii.mls_order_id IS NULL
AND ii.damaged = false
JOIN title ttl ON ttl.id = si.title_id
AND ttl.active = true
JOIN public.title_owner_title_designator_lookup totdl ON ttl.id = totdl.title_id
JOIN locator l ON l.id = ii.locator_id
AND l.active = true
LEFT JOIN location_locator ll ON ll.id = l.id
AND ll.status IN ('Shelf', 'Intake', 'Donation', 'Wholesaler')
LEFT JOIN course_locator cl ON cl.id = l.id
WHERE COALESCE(ll.location_id, cl.location_id) = :locationId
AND ll.do_not_list_on_markets = false
AND CASE
WHEN (:designatorInclude IS NOT NULL)
THEN totdl.designator_id IN (:designatorInclude)
ELSE ttl.product_code IS NOT NULL
END
I got this to work by changing my IS NOT NULL check to a COALESCE
CASE
WHEN **(COALESCE(:include, 0) != 0)**
THEN totdl.designator_id IN (:include)
ELSE ttl.product_code IS NOT NULL
If I would you, I would select just row column such
SELECT ttl.product_code, totdl.designator_id
FROM ...
I mapped this query normal to an Bean or Entity.
And an transient getter could solve your problem.

How can I look at the SQL query really used by myBatis?

The query, created by myBatis, falls with the cause ORA-00933, "SQL command not properly ended".
All advice that I found on the net, says that there is some error in the SQL syntax. On the other hand, if I make the SQL by hand, Oracle SQL developer takes it as a correct one.
Obviously, I am making it by hand differently that myBatis does. And I need to check it. But how can I check the SQL query really sent by myBatis to the DB?
The MyBatis version used: 3.0
I am interested much more in the possibility to look for problems myself than to look for help with every problem. But some people here think that the question must contain the code, so here it is. (mapper.xml)
<sql id="pracovisteSql">
/* pozor, pracuje pouze do max 3 vrstev v cis_pracovist */
cis_pracoviste A
join cis_pracoviste B
on (A.stupen_rizeni in (0,1) and B.kod_nadrizeneho = A.kod_pracoviste) or
(A.stupen_rizeni = 2 and A.kod_pracoviste = B.kod_pracoviste)
join cis_pracoviste C
on (B.stupen_rizeni = 1 and C.kod_nadrizeneho = B.kod_pracoviste) or
(B.stupen_rizeni = 2 and C.kod_pracoviste = B.kod_pracoviste)
</sql>
<sql id="organizaceSql">
WITH organizace
AS (
SELECT a.kod_pracoviste as AKP, a.nazev as ANZ, a.stupen_rizeni as AST, a.kod_nadrizeneho as ANR,
b.kod_pracoviste as BKP, b.nazev as BNZ, b.stupen_rizeni as BST, b.kod_nadrizeneho as BNR,
c.kod_pracoviste as CKP, c.nazev as CNZ, c.stupen_rizeni as CST, c.kod_nadrizeneho as CNR
from
<include refid="pracovisteSql"/>
)
</sql>
<sql id="zahajeniOdDo">
(r01.dat_zahajeni between to_date(#{mesicRokOd}, 'MMYYYY') and to_date(#{mesicRokDo, 'MMYYYY'))
</sql>
<select id="getReportSR02Sql1"
parameterType="amcssz.spr.srv.main.dto.reports.ReportSR02QueryDTO"
resultType="amcssz.spr.srv.main.dto.reports.ReportSR02Sql1DTO">
<include refid="organizaceSql"/>
SELECT Count(Distinct (r01.id_r01_rizeni)) as pocetVRSP
From organizace
Left join r01_rizeni r01
on organizace.ckp = r01.kod_pracoviste and
r01.je_stornovano = 0 and
<include refid="zahajeniOdDo"/> and
r01.kod_skup_rizeni = 'VRSP' /* r01.kod_rizeni in ('VRSPUC', 'VRSPSR', 'VRSPPE', 'VRSPJI') */
Join r02_stavrizeni r02
on R01.ID_R01_RIZENI = R02.ID_R01_RIZENI and /* Změna 17.1.2020 */
R02.JE_AKTUALNI = '1' and
R02.KOD_STAV_RIZENI != 'STR'
Join r08_ukon r08
on R01.ID_R01_RIZENI = R08.ID_R01_RIZENI and /* Změna 17.1.2020 */
R08.KOD_UKON IN ('1','14','23','32') and
R08.JE_STORNOVAN = '0'
Join d02_obalka d02
on R08.ID_R08_UKON = D02.ID_R08_UKON and
d02.dat_doruceni IS NOT NULL
Where organizace.AKP = #{kodPracoviste} and
r01.kod_skup_rizeni is Not Null
Group by r01.kod_skup_rizeni
Order by 1;
</select>
You can configure your log4j level to show the mybatis logging like log4j.logger.org.mybatis.example=DEBUG
Full documentation will be found here
If you use intellij idea, try to add a plugin mybatis-log-plugin.The executed SQL will be output in the console

Cassandra Exception

For my current project i'm using Cassandra Db for fetching data frequently. Within every second at least 30 Db requests will hit. For each request at least 40000 rows needed to fetch from Db. Following is my current code and this method will return Hash Map.
public Map<String,String> loadObject(ArrayList<Integer> tradigAccountList){
com.datastax.driver.core.Session session;
Map<String,String> orderListMap = new HashMap<>();
List<ResultSetFuture> futures = new ArrayList<>();
List<ListenableFuture<ResultSet>> Future;
try {
session =jdbcUtils.getCassandraSession();
PreparedStatement statement = jdbcUtils.getCassandraPS(CassandraPS.LOAD_ORDER_LIST);
for (Integer tradingAccount:tradigAccountList){
futures.add(session.executeAsync(statement.bind(tradingAccount).setFetchSize(3000)));
}
Future = Futures.inCompletionOrder(futures);
for (ListenableFuture<ResultSet> future : Future){
for (Row row: future.get()){
orderListMap.put(row.getString("cliordid"), row.getString("ordermsg"));
}
}
}catch (Exception e){
}finally {
}
return orderListMap;
}
My data request query is something like this,
"SELECT cliordid,ordermsg FROM omsks_v1.ordersStringV1 WHERE tradacntid = ?".
My Cassandra cluster has 2 nodes with 32 concurrent read and write thread for each and my Db schema as follow
CREATE TABLE omsks_v1.ordersstringv1_copy1 (
tradacntid int,
cliordid text,
ordermsg text,
PRIMARY KEY (tradacntid, cliordid)
) WITH bloom_filter_fp_chance = 0.01
AND comment = ''
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99.0PERCENTILE'
AND caching = {
'keys' : 'ALL',
'rows_per_partition' : 'NONE'
}
AND compression = {
'sstable_compression' : 'LZ4Compressor'
}
AND compaction = {
'class' : 'SizeTieredCompactionStrategy'
};
My problem is getting Cassandra timeout exception, how to optimize my code to handle all these requests
It would be better if you would attach the snnipet of that Exception (Read/write exception). I assume you are getting read time out. You are trying to fetch a large data set on a single request.
For each request at least 40000 rows needed to fetch from Db
If you have a large record and resultset is too big, it throws exception if results cannot be returned within a time limit mentioned in Cassandra.yaml.
read_request_timeout_in_ms
You can increase the timeout but this is not a good option. It may resolve the issue (may not throw exception but will take more time to return result).
Solution: For big data set you can get the result using manual pagination (range query) with limit.
SELECT cliordid,ordermsg FROM omsks_v1.ordersStringV1
WHERE tradacntid > = ? and cliordid > ? limit ?;
Or use range query
SELECT cliordid,ordermsg FROM omsks_v1.ordersStringV1 WHERE tradacntid
= ? and cliordid >= ? and cliordid <= ?;
This will be much more faster than fetching the whole resultset.
You can also try by reducing the fetch size. Although it will return the whole resultset.
public Statement setFetchSize(int fetchSize) to check if exception is thrown.
setFetchSize controls the page size, but it doesn't control the
maximum rows returned in a ResultSet.
Another point to be noted:
What's the size of tradigAccountList?
Too many requests at a time also may lead to timeout. Large size of tradigAccountList and a lot of read requests are done at a time (load balancing of requests are handled by Cassandra and how many requests can be handled depends on cluster size and some other factors) may cause this exception .
Some related Links:
Cassandra read timeout
NoHostAvailableException With Cassandra & DataStax Java Driver If Large ResultSet
Cassandra .setFetchSize() on statement is not honoured

Update and return list from hibernate (using sql query)

I have a query which updates (table1) and inserts into (table2) and returns a list of ids at the end which have been updated by the query.
When I run it from sql it runs properly.
But when I run from my code it returns the affected ids but it does not update & the rows.
Is it because I am doing it as: namedQuery.list() ??
SQL Query:
step 1: create table variable
step 2: update table1
step 3: store "Output Into" table variable; insert table variable content into table2
step 4: retrun affected rows (table variable column)
Here's the sql:
Declare #tempHistoryTable Table(
AG_TASK_ID int,Fulfillment_Request_ID int,Task_Type_ID int,Task_Status varchar(25)
,CF_System_User_ID int,Modified_By_System_User_ID int,AG_Task_Date datetime,row_version int
)
Declare #infoNeeded int,
#reviewResult int,
#researcherClass varchar(20),
#infoNeededReview int;
Update ag_task Set task_status = case when task_status = 'awaitingHitEntry' or task_status = 'Uploaded'
then task_status when (data_source = 'New_Jersey' or data_source = 'Illinois') and :action = 'Assign'
then 'sentForProcessing' else 'New' end
,db_version = case when db_version is null then 0 else db_version + 1 end
,modified_by_system_user_id = :assignedBy
,system_user_id = :assigningTo
,bpm_version = db_version
,task_type_id = case when task_type_id = #infoNeeded and :action = 'Assign'
then #reviewResult
when task_type_id = #infoNeeded and #researcherClass != 'External'
then #reviewResult
when (task_type_id = #infoNeededReview or task_type_id = #reviewResult) and #researcherClass = 'External'
then #infoNeeded
else task_type_id end
,creation_date = case when (task_type_id = #infoNeededReview or task_type_id = #reviewResult) and #researcherClass = 'External'
then getDate() else creation_date end
,task_assigned_date = case when :assigningTo is null then null else GETDATE() end
Output inserted.ag_task_id,inserted.Fulfillment_Request_ID,inserted.Task_Type_ID
,inserted.Task_Status,inserted.system_user_id
,inserted.Modified_By_System_User_ID,getdate(),inserted.row_version
Into #tempHistoryTable
Where fulfillment_request_id in (:fulfillmentRequestIds) and completion_date is null and
1 = case when data_source in ('New_Jersey', 'Illinois') and processing_date is null then 2 else 1 end
and 1 = case when :action = 'Claim' and system_user_id is not null then 2 else 1 end
and 1 = case when system_user_id is null and :assigningTo is null then 2 else 1 end
insert into AG_TASK_HISTORY (
AG_TASK_ID
,Fulfillment_Request_ID
,Task_Type_ID
,Task_Status
,CF_System_User_ID
,Modified_By_System_User_ID
,AG_Task_Date
,row_version
)
SELECT AG_TASK_ID
,Fulfillment_Request_ID
,Task_Type_ID
,Task_Status
,CF_System_User_ID
,Modified_By_System_User_ID
,AG_Task_Date
,row_version
from #tempHistoryTable
select Fulfillment_Request_ID from #tempHistoryTable
and this is how i'm calling the code:
String queryName = AgTask.class.getName () + ".claimAssignTasks";
final Query namedQuery = getNamedQuery ( queryName );
logger.debug("assigningTo: "+assigningTo);
logger.debug("assignedBy: "+assignedBy);
logger.debug("action: "+action);
logger.debug("fulfillmentRequestIdList: "+fulfillmentRequestIdList);
namedQuery.setParameterList("fulfillmentRequestIds", fulfillmentRequestIdList);
namedQuery.setParameter("assigningTo", assigningTo);
namedQuery.setParameter("assignedBy",assignedBy);
namedQuery.setParameter("action",action);
logger.debug("Query: "+namedQuery.getQueryString());
List<Integer> frIdList = new ArrayList<Integer>();
frIdList = namedQuery.list();
This is how i am running my test (which properly updates and inserts into table in my DB):
#Test
public void testclaimAssignTasks(){
transactionTemplate.execute(new TransactionCallbackWithoutResult ()
{
public void doInTransactionWithoutResult (TransactionStatus arg0)
{
Set<Integer> frIds = new HashSet<Integer>();
frIds.add(190195);
frIds.add(190257);
frIds.add(190243);
frIds.add(190205);
//java.util.List<Integer> frIdList =
java.util.List<Integer> frIdList = agTaskSearchDao.claimAssignTasks(frIds,846,846,"Reassign");
log.info("collection size: "+frIdList.size());
for(Integer fulfillmentRequestId : frIdList){
log.debug("fulfillmentRequestId: "+fulfillmentRequestId);
}
}
});
}
First off, I think putting such a complex query into the context of Hibernate/JPA is a really bad idea. If I were you I would try to analyze the sql and break it down, in order to see if this is something that could be solved much cleaner by lending from JPA/Hibernate functionality. Or maybe you'll end up scrapping the whole thing, solving the problem at hand with different methods.
That being said, if you want to execute an SQL (or HQL/JPQL) that is supposed to change the state of the database, you will need to use the method executeUpdate() of the Query interface, as opposed to list() or uniqueResult() that is used for retrieval of data only.

Categories