Related
I've developed a java application that uses an Atlas MongoDB Serverless DB.
This application performs an Aggregation query with the following steps:
$match
$project
$addFields
$sort
$facet
$project
When I perform a query thats returns a lot of results I'm obtaining this exception: QueryExceededMemoryLimitNoDiskUseAllowed.
I've tried to modify my code adding allowDiskUse: true in the aggregation, but didn't resolve the exception.
I've tried to replicate my aggregation pipeline in Atlas console and found that every think works fine until $facet step that returns
Reason: PlanExecutor error during aggregation :: caused by :: Sort exceeded memory limit of 33554432 bytes, but did not opt in to external sorting.
This is my $facet step:
{$facet: {
paginatedResults: [{ $skip: 0 }, { $limit: 50 }],
totalCount: [
{
$count: 'count'
}
]
}
}
As you can see I'm using it to paginate my query results.
Any suggestion to avoid this problem?
I was thinking about making two different query one for the results and one for total count, but I'm not sure this is the best solution.
EDIT: added query
db.vendor_search.aggregate(
{$match: {
$or: [
{'searchKeys.value': {$regex: "vendor"}},
{'searchKeys.value': {$regex: "test"}},
{'searchKeys.valueClean': {$regex: "vendor"}},
{'searchKeys.valueClean': {$regex: "test"}},
],
buyerId: 7
}},
{$project: {
companyId: 1,
buyerId: 1,
companyName: 1,
legalForm: 1,
country: 1,
supplhiCompanyCode: 1,
vat: 1,
erpCode: 1,
visibility: 1,
businessStatus: 1,
city: 1,
logo: 1,
location: {$concat : ["$country.value",'$city']},
searchKeys: {
"$filter": {
"input": "$searchKeys",
"cond": {
"$or": [
{$regexMatch: {input: "$$this.value",regex: "vendor"}},
{$regexMatch: {input: "$$this.value",regex: "test"}}
{$regexMatch: {input: "$$this.valueClean",regex: "vendor"}},
{$regexMatch: {input: "$$this.valueClean",regex: "test"}}
]
}
}
}
}},
{$addFields: {
searchMatching: {
$reduce: {
input: "$searchKeys.type",
initialValue: [],
in: {
$concatArrays: [
"$$value",
{$cond: [{$in: ["$$this", "$$value"]},[],["$$this"]]}
]
}
}
},
'sort.supplhiId': { $toLower: "$supplhiCompanyCode" },
'sort.companyName': { $toLower: "$companyName" },
'sort.location': { $toLower: {$concat : ["$country.value"," ","$city"]}},
'sort.vat': { $toLower: "$vat" },
'sort.companyStatus': { $toLower: "$businessStatus" },
'sort.erpCode': { $toLower: "$erpCode" }
}},
{$sort: {"sort.companyName": 1}},
{$facet: {
paginatedResults: [{ $skip: 0 }, { $limit: 50 }],
totalCount: [
{
$count: 'count'
}
]
}
},
{$project: {paginatedResults:1, 'totalCount': {$first : '$totalCount.count'}}}
)
EDIT: Added model
{
"buyerId": 1,
"companyId": 869048,
"address": "FP8R+52H",
"businessStatus": "AC",
"city": "Chiffa",
"companyName": "Test Algeria 25 agosto",
"country": {
"lookupId": 78,
"code": "DZA",
"value": "Algeria"
},
"erpCode": null,
"legalForm": "Ltd.",
"logo": "fc4d821a-e814-49e4-96d1-f32421fdaa6d_1.jpg",
"searchKeys": [
{
"type": "contact",
"value": "pebiw81522#xitudy.com",
"valueClean": "pebiw81522xitudycom"
},
{
"type": "company_registration_number",
"value": "112211331144",
"valueClean": "112211331144"
},
{
"type": "vendor_name",
"value": "test algeria 25 agosto ltd.",
"valueClean": "test algeria 25 agosto ltd"
},
{
"type": "contact",
"value": "tredicisf2#ottobre2022.com",
"valueClean": "tredicisf2ottobre2022com"
},
{
"type": "contact",
"value": "ty#s.com",
"valueClean": "tyscom"
},
{
"type": "contact",
"value": "info#x.com",
"valueClean": "infoxcom"
},
{
"type": "tin",
"value": "00112341675",
"valueClean": "00112341675"
},
{
"type": "contact",
"value": "hatikog381#rxcay.com",
"valueClean": "hatikog381rxcaycom"
},
{
"type": "supplhi_id",
"value": "100059410",
"valueClean": "100059410"
},
{
"type": "contact",
"value": "tredici#ottobre2022.com",
"valueClean": "trediciottobre2022com"
},
{
"type": "country_key",
"value": "00112341675",
"valueClean": "00112341675"
},
{
"type": "vat",
"value": "00112341675",
"valueClean": "00112341675"
},
{
"type": "address",
"value": "fp8r+52h",
"valueClean": "fp8r52h"
},
{
"type": "city",
"value": "chiffa",
"valueClean": "chiffa"
},
{
"type": "contact",
"value": "prova#supplhi.com",
"valueClean": "provasupplhicom"
},
{
"type": "contact",
"value": "saraxo2669#dmonies.com",
"valueClean": "saraxo2669dmoniescom"
}
],
"supplhiCompanyCode": "100059410",
"vat": "00112341675",
"visibility": true
}
in ATLAS M0 free clusters and M2/M5 shared clusters sort in memory limit is 32 MB. ( ref ) , this limit seems to apply also to serverless
For not limited mongod you may usually increase this limit from 32MB for example to 320MB as follow:
db.adminCommand({setParameter: 1, internalQueryExecMaxBlockingSortBytes: 335544320})
You can check the current value with:
db.runCommand( { getParameter : 1, "internalQueryExecMaxBlockingSortBytes" : 1 } )
But it is best to optimize your queries to not hit this limit , if you post your full query and indexes ( db.collection.getIndexes() perhaps there is a better way ...
I using JOLT to transform and extract a URL of an object, but it's only the URL slug. Is it possible to concatenate a static string to that slug to have a full URL?
The current output is:
{
"title" : "Where is deleted user data visible, and for how long is it there?",
"externalId" : "where_is_deleted_user_data_visible__and_for_how_lo",
"summary" : "Deleted user data is stored in LP and visible for 13 months following deletion.",
"detail" : "<p>Deleted user data is stored in LP and visible for <strong>13 months following deletion.</strong></p>\n<p>Deleted agents are displayed with a '(deleted)' indicator in the Open Conversations, All Conversations, Web History lists, and the Conversation History widget.</p>\n<object type=\"application/kenticocloud\" data-type=\"item\" data-rel=\"component\" data-codename=\"n6fdd091f_fcb5_012a_8745_c1671c268e6b\"></object>\n<p>Filter by deleted agents is available in the Open Conversations, All Conversations, and web history lists.</p>\n<p>In the visitor conversation window, the name of the deleted agent will be displayed in the window header (with no indication that they have been deleted).</p>\n<p>The LP user limit '<code>collections.maxElements</code>' will not count the number of deleted users in the total.</p>\n<object type=\"application/kenticocloud\" data-type=\"item\" data-rel=\"component\" data-codename=\"n3f1dea42_ac2e_01c6_cbd6_3f802d4e8fb3\"></object>",
"url" : "faqs-data-reporting1.html"
}
And the expected output is:
{
"title" : "Where is deleted user data visible, and for how long is it there?",
"externalId" : "where_is_deleted_user_data_visible__and_for_how_lo",
"summary" : "Deleted user data is stored in LP and visible for 13 months following deletion.",
"detail" : "<p>Deleted user data is stored in LP and visible for <strong>13 months following deletion.</strong></p>\n<p>Deleted agents are displayed with a '(deleted)' indicator in the Open Conversations, All Conversations, Web History lists, and the Conversation History widget.</p>\n<object type=\"application/kenticocloud\" data-type=\"item\" data-rel=\"component\" data-codename=\"n6fdd091f_fcb5_012a_8745_c1671c268e6b\"></object>\n<p>Filter by deleted agents is available in the Open Conversations, All Conversations, and web history lists.</p>\n<p>In the visitor conversation window, the name of the deleted agent will be displayed in the window header (with no indication that they have been deleted).</p>\n<p>The LP user limit '<code>collections.maxElements</code>' will not count the number of deleted users in the total.</p>\n<object type=\"application/kenticocloud\" data-type=\"item\" data-rel=\"component\" data-codename=\"n3f1dea42_ac2e_01c6_cbd6_3f802d4e8fb3\"></object>",
"url" : "https://www.static-url.com/faqs-data-reporting1.html"
}
The JOLT Spec is:
[
{
"operation": "shift",
"spec": {
"item": {
"system": {
"name": "title",
"codename": "externalId"
},
"elements": {
"short_answer": {
"value": ["summary"]
},
"long_answer": {
"value": "detail"
}
}
},
"modular_content": {
"*": {
"elements": {
"permalink": {
"value": "url"
}
}
}
}
}
}
]
The JSON input is:
{
"item": {
"system": {
"id": "6e55g261aee-34d9-47be-8830-145ec71b564f",
"name": "Where is deleted user data visible, and for how long is it there?",
"codename": "where_is_deleted_user_data_visible__and_for_how_lo",
"language": "en-US",
"type": "faq",
"collection": "default",
"sitemap_locations": [],
"last_modified": "2022-04-11T15:32:08.0844256Z",
"workflow_step": "published"
},
"elements": {
"question": {
"type": "text",
"name": "Question",
"value": "Where is deleted user data visible, and for how long is it there?"
},
"short_answer": {
"type": "text",
"name": "Bot Teaser Answer",
"value": "Deleted user data is stored in LP and visible for 13 months following deletion."
},
"long_answer": {
"type": "rich_text",
"name": "Full Answer",
"images": {},
"links": {},
"modular_content": [
"n6fdd091f_fcb5_012a_87545_c1671c268e6b",
"n3f1dea42_ac2e_01cdf6_cbd6_3f802d4e8fb3"
],
"value": "<p>Deleted user data is stored in LP and visible for <strong>13 months following deletion.</strong></p>\n<p>Deleted agents are displayed with a '(deleted)' indicator in the Open Conversations, All Conversations, Web History lists, and the Conversation History widget.</p>\n<object type=\"application/kenticocloud\" data-type=\"item\" data-rel=\"component\" data-codename=\"n6fdd091f_fcb5_012a_8745_c1671c268e6b\"></object>\n<p>Filter by deleted agents is available in the Open Conversations, All Conversations, and web history lists.</p>\n<p>In the visitor conversation window, the name of the deleted agent will be displayed in the window header (with no indication that they have been deleted).</p>\n<p>The LP user limit '<code>collections.maxElements</code>' will not count the number of deleted users in the total.</p>\n<object type=\"application/kenticocloud\" data-type=\"item\" data-rel=\"component\" data-codename=\"n3f1dea42_ac2e_01c6_cbd6_3f802d4e8fb3\"></object>"
},
"faq_page": {
"type": "modular_content",
"name": "This FAQ item's FAQ page(s)",
"value": [
"data___reporting_9658b36"
]
},
"related_article": {
"type": "modular_content",
"name": "Related article",
"value": []
}
}
},
"modular_content": {
"data___reporting_9658b36": {
"system": {
"id": "9658bddf362-735b-47fd3c-9232-26de14835d8f",
"name": "Data & Reporting",
"codename": "data___reporting_9658b36",
"language": "en-US",
"type": "kc_faqs",
"collection": "default",
"sitemap_locations": [],
"last_modified": "2022-04-13T16:18:12.2538592Z",
"workflow_step": "published"
},
"elements": {
"pagename": {
"type": "text",
"name": "Title",
"value": "Data & Reporting"
},
"faq_items": {
"type": "modular_content",
"name": "FAQ Items",
"value": [
"how_do_i_find_chats_that_only_contain_private_mess",
"how_do_i_understand_why_a_chat_ended_",
"how_do_i_change_the_email_address_for_scheduled_re",
"how_do_i_retrieve_data_that_has_been_masked_",
"where_is_deleted_user_data_visible__and_for_how_lo"
]
},
"related_articles": {
"type": "modular_content",
"name": "Related Articles",
"value": []
},
"permalink": {
"type": "url_slug",
"name": "URL slug",
"value": "faqs-data-reporting1.html"
},
"redirects": {
"type": "text",
"name": "redirects",
"value": "faqs-data-reporting1.html"
}
}
},
"n3f1dea42_ac2e_01c6_cbd6_3f802d4e8fb3": {
"system": {
"id": "3f1dea42-ac2e-01c6-cbd6-3f802d4e8fb3",
"name": "3f1dea42-ac2e-01c6-cbd6-3f802d4e8fb3",
"codename": "n3f1dea42_ac2e_01c6_cbd6_3f802d4e8fb3",
"language": "en-US",
"type": "contentbox",
"collection": "default",
"sitemap_locations": [],
"last_modified": "2022-04-11T15:32:08.0844256Z"
},
"elements": {
"type": {
"type": "multiple_choice",
"name": "Type",
"value": [
{
"name": "Warning (orange)",
"codename": "warning"
}
]
},
"notice_text": {
"type": "rich_text",
"name": "Text",
"images": {},
"links": {},
"modular_content": [],
"value": "<p>A user who joined a chat and then was deleted won’t be reflected as deleted in the history.</p>"
}
}
},
"n6fdd091f_fcb5_012a_8745_c1671c268e6b": {
"system": {
"id": "6fdd091f-fcb5-012a-8745-c1671c268e6b",
"name": "6fdd091f-fcb5-012a-8745-c1671c268e6b",
"codename": "n6fdd091f_fcb5_012a_8745_c1671c268e6b",
"language": "en-US",
"type": "contentbox",
"collection": "default",
"sitemap_locations": [],
"last_modified": "2022-04-11T15:32:08.0844256Z"
},
"elements": {
"type": {
"type": "multiple_choice",
"name": "Type",
"value": [
{
"name": "Notice (blue)",
"codename": "notice"
}
]
},
"notice_text": {
"type": "rich_text",
"name": "Text",
"images": {},
"links": {},
"modular_content": [],
"value": "<p>Before deleting a user, ensure that the agent doesn’t own any open conversations (if so, these conversations should be transferred to the queue) and that the user is logged out of LP.</p>"
}
}
}
}
}
Any option to add the static URL link to the output?
You can add a modify spec with concat function after the current shift transformation spec such as
{
"operation": "modify-overwrite-beta",
"spec": {
"url": "=concat('https://www.static-url.com/',#(1,&))" // first argument is static, and the second one is dynamic(directly inherited from the current value of "url") content
}
}
I have a JSON
{
"Id": "xxx",
"Type": "Transaction.Create",
"Payload": {
"result": 2,
"description": "Pending",
"body": {
"redirect": {
"url": "xxx",
"fields": {
"MD": "8a829449620619e80162252adeb66a39"
}
},
"card": {
"expiryMonth": "1",
"expiryYear": "2033"
},
"order": {
"amount": 1
}
}
}
}
And I want to remove the card info of it like this:
{
"Id": "xxx",
"Type": "Transaction.Create",
"Payload": {
"result": 2,
"description": "Pending",
"body": {
"redirect": {
"url": "xxx",
"fields": {
"MD": "8a829449620619e80162252adeb66a39"
}
},
"order": {
"amount": 1
}
}
}
}
How can I do this with Apache velocity?
What works is:
#set($content = $util.urlEncode($input.json('$')))
#set($new = $content.replaceAll("2033","2055"))
Action=SendMessage&MessageBody={"body": "$new","Event-Signature": "$util.urlEncode($input.params('Event-Signature'))"}
This gives me
{
"Id": "xxx",
"Type": "Transaction.Create",
"Payload": {
"result": 2,
"description": "Pending",
"body": {
"redirect": {
"url": "xxx",
"fields": {
"MD": "8a829449620619e80162252adeb66a39"
}
},
"card": {
"expiryMonth": "1",
"expiryYear": "2050"
},
"order": {
"amount": 1
}
}
}
}
But now I want to remove the card part but it does not work:
#set($content = $util.urlEncode($input.json('$')))
#set($new = $content.delete("$.Payload.body.card"))
Action=SendMessage&MessageBody={"body": "$new","Event-Signature": "$util.urlEncode($input.params('Event-Signature'))"}
what am I doing wrong?
Main goal is transform a mapping template in API Gateway for a webhook. The webhook contains to many information and we want to remove some part of the JSON POST call.
Try using the below
#set($dummy=$content.Payload.remove("card"))
I'm using the new Paypal Sync Api to get all transactions in a period of time. I need to be able to refund them too.
As all our transactions are always sales I need the sale ID to refund a transaction as stated in https://developer.paypal.com/docs/integration/direct/payments/refund-payments/#set-up-your-development-environment
Unfortunately I can not find anything in the transactions response which can be used as a sale ID
{
"transaction_details": [
{
"transaction_info": {
"paypal_account_id": "6EN7PS8P9DV6J",
"transaction_id": "05P62542TL3758730",
"transaction_event_code": "T0006",
"transaction_initiation_date": "2019-07-04T12:49:55+0000",
"transaction_updated_date": "2019-07-04T12:53:09+0000",
"transaction_amount": {
"currency_code": "USD",
"value": "10.00"
},
"fee_amount": {
"currency_code": "USD",
"value": "-0.49"
},
"transaction_status": "S",
"ending_balance": {
"currency_code": "USD",
"value": "0.00"
},
"available_balance": {
"currency_code": "USD",
"value": "0.00"
},
"protection_eligibility": "01"
},
"payer_info": {
"account_id": "6EN7PS8P9DV6J",
"email_address": "paypaltester3#test.de",
"address_status": "Y",
"payer_status": "Y",
"payer_name": {
"given_name": "Tester Three",
"surname": "Tester Three",
"alternate_full_name": "Tester Three Tester Three"
},
"country_code": "DE"
},
"shipping_info": {
"name": "Tester Three, Tester Three",
"address": {
"line1": "Teststreet. 1",
"city": "SomeCity",
"country_code": "DE",
"postal_code": "23434"
}
},
"cart_info": {
"item_details": [
{
"item_quantity": "1",
"item_unit_price": {
"currency_code": "USD",
"value": "10.00"
},
"item_amount": {
"currency_code": "USD",
"value": "10.00"
},
"total_item_amount": {
"currency_code": "USD",
"value": "10.00"
}
}
]
},
"store_info": {},
"auction_info": {},
"incentive_info": {}
}
],
"account_number": "96B45RCG6AX3E",
"start_date": "2019-07-03T00:00:00+0000",
"end_date": "2019-07-04T16:00:00+0000",
"last_refreshed_datetime": "2019-07-15T08:59:59+0000",
"page": 1,
"total_items": 15,
"total_pages": 1,
"links": [
{
"href": "https://api.sandbox.paypal.com/v1/reporting/transactions?start_date=2019-07-03T00%3A00%3A00Z&end_date=2019-07-04T16%3A00%3A00Z&fields=all&page_size=500&page=1",
"rel": "self",
"method": "GET"
}
]
}
I tested a refund call with a transactionId and with an invoiceId but both returned "INTERNAL_SERVICE_ERROR".
Is there any way to get the sale ID by an API after the customer has completed the payment or do I have to save the sale ID in a database after creating a payment?
Thanks
With T0006 you are lucky, it is a simple PayPal Checkout APIs. transaction, it would be harder with T0002. Anyway, no the response have much more data. The example is below.
You can find it in invoice_number, or invoice_id. If only you had this data during payment creation procedures.
"transaction_details":[
{
"transaction_info":{
"paypal_account_id":"HHHHH",
"transaction_id":"040404040404040",
"transaction_event_code":"T0006",
"transaction_initiation_date":"2019-10-31T06:21:37+0000",
"transaction_updated_date":"2019-10-31T06:31:02+0000",
"transaction_amount":{
"currency_code":"AUD",
"value":"96.90"
},
"fee_amount":{
"currency_code":"AUD",
"value":"-4.08"
},
"shipping_amount":{
"currency_code":"AUD",
"value":"17.90"
},
"transaction_status":"S",
"transaction_subject":"Full License",
"ending_balance":{
"currency_code":"AUD",
"value":"92.82"
},
"available_balance":{
"currency_code":"AUD",
"value":"92.82"
},
"invoice_id":"1110-US1",
"protection_eligibility":"01"
},
"payer_info":{
"account_id":"HHHHH",
"email_address":"test#paypal.com",
"address_status":"Y",
"payer_status":"Y",
"payer_name":{
"given_name":"First",
"surname":"Last",
"alternate_full_name":"First Last"
},
"country_code":"GB"
},
"shipping_info":{
"name":"Shaun Smith",
"address":{
"line1":"123, Some Road",
"line2":"Shop 4",
"city":"Melbourne",
"country_code":"AU",
"postal_code":"3185"
}
},
"cart_info":{
"item_details":[
{
"item_name":"Full License",
"item_description":"Full License",
"item_quantity":"1",
"item_unit_price":{
"currency_code":"AUD",
"value":"79.00"
},
"item_amount":{
"currency_code":"AUD",
"value":"79.00"
},
"total_item_amount":{
"currency_code":"AUD",
"value":"79.00"
},
"invoice_number":"1110-US1"
}
]
},
"store_info":{
},
"auction_info":{
},
"incentive_info":{
}
},
I have written a java class which hits the Azure REST API's and gets me the json response. for example below-
{
"value": [
{
"sku": {
"name": "Standard_LRS",
"tier": "Standard"
},
"kind": "Storage",
"id": "/subscriptions/cc60365e-3193-xxxx-b0f6-xxxxxxx/resourceGroups/WebServer-ubuntu/providers/Microsoft.Storage/storageAccounts/webserverubuntudiagxxx",
"name": "webserverubuntudiag4xx",
"type": "Microsoft.Storage/storageAccounts",
"location": "eastus",
"tags": {},
"properties": {
"networkAcls": {
"bypass": "AzureServices",
"virtualNetworkRules": [],
"ipRules": [],
"defaultAction": "Allow"
},
"trustedDirectories": [
"50228742-xxx-4cd2-xxx-3e50b7a62xx1"
],
"supportsHttpsTrafficOnly": false,
"encryption": {
"services": {
"file": {
"enabled": true,
"lastEnabledTime": "2018-03-26T15:02:26.8078850Z"
},
"blob": {
"enabled": true,
"lastEnabledTime": "2018-03-26T15:02:26.8078850Z"
}
},
"keySource": "Microsoft.Storage"
},
"provisioningState": "Succeeded",
"creationTime": "2018-03-26T15:02:26.6359768Z",
"primaryEndpoints": {
"blob": "https://webserverubuntudiagxx.blob.core.windows.net/",
"queue": "https://webserverubuntudiag4xx.queue.core.windows.net/",
"table": "https://webserverubuntudiagxx.table.core.windows.net/",
"file": "https://webserverubuntudiag4xx.file.core.windows.net/"
},
"primaryLocation": "eastus",
"statusOfPrimary": "available"
}
}
]
}
My questions is: Are there any model/pojo/bean already available to map this json to java object ?
I need to map the data from above json into the java beans/pojos to display into jsp pages, do I need to separately create custom pojos or there are any already in Azure libraries/sdk ? any suggestions?