I am using updateItem based on some condition. It works fine if condition is met, but it throws ConditionalCheckFailedException along with java InaccessibleObjectException if condition fails.
Why is it throwing InaccessibleObjectException? Also, how to handle it?
Update: Error also occurs in case of ValidationException
updateItemSpec = new UpdateItemSpec()
.withConditionExpression()
table.update(updateItemSpec).getItem()
You are using an old Java API for Amazon DynamoDB. To update a table, consider moving away from V1 and use the Enhanced Client - which is part of the AWS SDK for Java V2. More information here:
Mapping items in DynamoDB tables
Here is the code to update a table using the Enhanced Client.
public class EnhancedModifyItem {
public static void main(String[] args) {
String usage = "Usage:\n" +
" UpdateItem <key> <email> \n\n" +
"Where:\n" +
" key - the name of the key in the table (id120).\n" +
" email - the value of the modified email column.\n" ;
if (args.length != 2) {
System.out.println(usage);
System.exit(1);
}
String key = args[0];
String email = args[1];
Region region = Region.US_EAST_1;
DynamoDbClient ddb = DynamoDbClient.builder()
.region(region)
.build();
DynamoDbEnhancedClient enhancedClient = DynamoDbEnhancedClient.builder()
.dynamoDbClient(ddb)
.build();
String updatedValue = modifyItem(enhancedClient,key,email);
System.out.println("The updated name value is "+updatedValue);
ddb.close();
}
public static String modifyItem(DynamoDbEnhancedClient enhancedClient, String keyVal, String email) {
try {
//Create a DynamoDbTable object
DynamoDbTable<Customer> mappedTable = enhancedClient.table("Customer", TableSchema.fromBean(Customer.class));
//Create a KEY object
Key key = Key.builder()
.partitionValue(keyVal)
.build();
// Get the item by using the key and update the email value.
Customer customerRec = mappedTable.getItem(r->r.key(key));
customerRec.setEmail(email);
mappedTable.updateItem(customerRec);
return customerRec.getEmail();
} catch (DynamoDbException e) {
System.err.println(e.getMessage());
System.exit(1);
}
return "";
}
}
You can find all V2 DynamoDB examples here.
Related
I am using Spring Boot, Snowflake and AWS S3.
I have SQL query that queries two tables and gets the result. That result I have to write to S3 like CSV file and get URL for download in return.
I am doing that by creating temporary table and deleting it after data is copied to S3.
Here is code I have:
#Override
public void getUserTest(String userId) {
String q = "CREATE TEMPORARY TABLE \"TEST\".\"PUBLIC\".\"USER_TABLE_TEMP\" AS SELECT \"ID\", \"FIRST_NAME\", \"LAST_NAME\" from \"TEST\".\"PUBLIC\".\"USER_TABLE\"\n" +
" where \"ID\" = ?\n" +
" union all\n" +
" select \"ID\",\"ACCOUNT_NAME\", \"ACCOUNT_NUMBER\" from \"TEST\".\"PUBLIC\".\"ACCOUNT_DATA\"\n" +
" where \"ID\" = ?";
jdbcTemplate.query(q, s -> {}, userId, userId);
}
Method that writes to S3.
#Override
public URL writeToS3() {
String q = "copy into s3://snowflake171 from \"TEST\".\"PUBLIC\".\"USER_TABLE_TEMP\" storage_integration = s3_int file_format = CSV_TEST;\n";
jdbcTemplate.query(q, s -> {});
URL url = generateURL();
String dropTable = "drop table if exists \"TEST\".\"PUBLIC\".\"USER_TABLE_TEMP\"";
jdbcTemplate.query(dropTable, s -> {});
return url;
}
Method that generated URL:
public URL generateURL() {
try {
BasicAWSCredentials awsCreds = new BasicAWSCredentials(accessKey, secretKey);
final AmazonS3 s3Client = AmazonS3ClientBuilder.standard().withCredentials(new
AWSStaticCredentialsProvider(awsCreds)).withRegion(clientRegion).build();
// Set the presigned URL to expire after 2h.
java.util.Date expiration = new java.util.Date();
long expTimeMillis = Instant.now().toEpochMilli();
expTimeMillis += 1000 * 60 * 120;
expiration.setTime(expTimeMillis);
// Generate the presigned URL.
System.out.println("Generating pre-signed URL.");
GeneratePresignedUrlRequest generatePresignedUrlRequest =
new GeneratePresignedUrlRequest(bucket, objectKey)
.withMethod(HttpMethod.GET)
.withExpiration(expiration);
URL url = s3Client.generatePresignedUrl(generatePresignedUrlRequest);
System.out.println("Pre-Signed URL: " + url.toString());
return url;
} catch (AmazonServiceException e) {
// The call was transmitted successfully, but Amazon S3 couldn't process
// it, so it returned an error response.
e.printStackTrace();
} catch (SdkClientException e) {
// Amazon S3 couldn't be contacted for a response, or the client
// couldn't parse the response from Amazon S3.
e.printStackTrace();
}
return null;
}
Data is queried by userId I sent in. Everything works fine, but I am generating every time file with same name. And if I dont delete existing file in S3 I am not able to upload new one.
I should be able to upload different files for different userId.
How can I do this?
How can I give different names from files created in S3?
I have seen this in docs https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html but I don't know the best way to apply in code.
Is there a way I can add userId as prefix to filename?
You can add a custom object key with userId in your COPY query :
#Override
public URL writeToS3(userId) {
String q = "copy into s3://snowflake171/" + userId + " from \"TEST\".\"PUBLIC\".\"USER_TABLE_TEMP\" storage_integration = s3_int file_format = CSV_TEST;\n";
jdbcTemplate.query(q, s -> {});
I have a mega bot eg: projectID_stage with id=stage-aahedc and there are two sub agents: projectID_automated id=stageautomated-spiccu and projectC.
the projectID_automated has knowledge base enabled. and it's included in the projectID_stage.
The problem:
When I trying to do a detectIntent request with some part of knowledgebase set. it failed because of permission denied.
the code is similar with - https://github.com/googleapis/java-dialogflow/blob/master/samples/snippets/src/main/java/com/example/dialogflow/DetectIntentKnowledge.java (except I added two knowledge names)
import com.google.cloud.dialogflow.v2beta1.*;
import com.google.common.collect.*;
import java.io.*;
import java.util.*;
public class DetectIntentTexts {
// DialogFlow API Detect Intent sample with text inputs.
public static Map<String, QueryResult> detectIntentTexts(
String projectId, List<String> texts, String sessionId, String languageCode, String... kbNames)
throws IOException {
Map<String, QueryResult> queryResults = Maps.newHashMap();
// Instantiates a client
try (SessionsClient sessionsClient = SessionsClient.create()) {
// Set the session name using the sessionId (UUID) and projectID (my-project-id)
SessionName session = SessionName.of(projectId, sessionId);
System.out.println("Session Path: " + session.toString());
// Detect intents for each text input
for (String text : texts) {
// Set the text (hello) and language code (en-US) for the query
TextInput.Builder textInput =
TextInput.newBuilder().setText(text).setLanguageCode(languageCode);
// Build the query with the TextInput
QueryInput queryInput = QueryInput.newBuilder().setText(textInput).build();
QueryParameters queryParameters =
QueryParameters.newBuilder().addKnowledgeBaseNames(kbNames[0])
.addKnowledgeBaseNames(kbNames[1]).build();
DetectIntentRequest detectIntentRequest =
DetectIntentRequest.newBuilder()
.setSession(session.toString())
.setQueryInput(queryInput)
.setQueryParams(queryParameters)
.build();
// Performs the detect intent request
DetectIntentResponse response = sessionsClient.detectIntent(detectIntentRequest);
// Display the query result
QueryResult queryResult = response.getQueryResult();
System.out.println("====================");
System.out.format("Query Text: '%s'\n", queryResult.getQueryText());
System.out.format(
"Detected Intent: %s (confidence: %f)\n",
queryResult.getIntent().getDisplayName(), queryResult.getIntentDetectionConfidence());
System.out.format(
"Fulfillment Text: '%s'\n",
queryResult.getFulfillmentMessagesCount() > 0
? queryResult.getFulfillmentMessages(0).getText()
: "Triggered Default Fallback Intent");
queryResults.put(text, queryResult);
}
}
return queryResults;
}
}
// [END dialogflow_detect_intent_text]
and the main function is like below:
String projectID_automated = "stageautomated-spiccu";
String projectID_stage = "stage-aahedc";
String finalProjectID = projectID_stage;
String sessionID = "123456789";
String language = "en-US";
List<String> text = Arrays.asList("is it safe for my family and pets");
// when you trying to list out the kbs....
// listKbs(projectID);
// projects/<Project ID>/knowledgeBases/<Knowledge Base ID>`
String kb1 = "pest control-1", kb2 = "pest control-2";
String full1 = "projects/" + projectID_automated + "/knowledgeBases/MTE0NTc1ODUxNjIwNTM3NDY2ODg";
String full2 = "projects/" + projectID_automated + "/knowledgeBases/Njg0NTg5OTE0MzYyNjM1ODc4NA";
Map<String, QueryResult> r = DetectIntentTexts.detectIntentTexts(finalProjectID, text, sessionID, language, full1, full2);
System.out.println(r);
When using the projectID_automated for full1 and full2, it failed with below error:
Caused by: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: Project 'stageautomated-spiccu' in knowledge_base_names does not match project 'stage-aahedc' in session.
When using the projectID_stage for full1 and full2,
String full1 = "projects/" + projectID_stage + "/knowledgeBases/MTE0NTc1ODUxNjIwNTM3NDY2ODg";
String full2 = "projects/" + projectID_stage + "/knowledgeBases/Njg0NTg5OTE0MzYyNjM1ODc4NA";
it failed with below error in the query result
diagnostic_info {
fields {
key: "knowledge_connector_error"
value {
string_value: "INVALID_ARGUMENT: UNAUTHORIZED: You can\'t query KnowledgeBase under project: stage-aahedc, which is different than your agent project: stageautomated-spiccu"
}
}
Environment details
dialogflow#detectIntent
OS type and version: macos 10
Java version: jdk11
dialogflow version(s): dialogflow es + 'com.google.api.grpc:proto-google-cloud-dialogflow-v2beta1:0.99.5'
and 'com.google.cloud:google-cloud-dialogflow:4.1.5'
When creating a program to connect to cosmos db using Java, an error occurred and I asked a question.
When I executed the source code at the following URL,
The following error related to authentication occurred.
error message
Exception in thread "main" com.microsoft.azure.documentdb.DocumentClientException:
The input authorization token can't serve the request.
Please check that the expected payload is built as per the protocol, and check the key being used.
Server used the following payload to sign:
source
https://github.com/Azure/azure-documentdb-java#eclipse
import java.io.IOException;
import java.util.List;
import com.google.gson.Gson;
import com.microsoft.azure.documentdb.ConnectionPolicy;
import com.microsoft.azure.documentdb.ConsistencyLevel;
import com.microsoft.azure.documentdb.Database;
import com.microsoft.azure.documentdb.Document;
import com.microsoft.azure.documentdb.DocumentClient;
import com.microsoft.azure.documentdb.DocumentClientException;
import com.microsoft.azure.documentdb.DocumentCollection;
import com.microsoft.azure.documentdb.RequestOptions;
public class HelloWorld {
// Replace with your DocumentDB end point and master key.
private static final String END_POINT = "[YOUR_ENDPOINT_HERE]";
private static final String MASTER_KEY = "[YOUR_KEY_HERE]";
// Define an id for your database and collection
private static final String DATABASE_ID = "TestDB";
private static final String COLLECTION_ID = "TestCollection";
// We'll use Gson for POJO <=> JSON serialization for this sample.
// Codehaus' Jackson is another great POJO <=> JSON serializer.
private static Gson gson = new Gson();
public static void main(String[] args) throws DocumentClientException,
IOException {
// Instantiate a DocumentClient w/ your DocumentDB Endpoint and AuthKey.
DocumentClient documentClient = new DocumentClient(END_POINT,
MASTER_KEY, ConnectionPolicy.GetDefault(),
ConsistencyLevel.Session);
// Start from a clean state (delete database in case it already exists).
try {
documentClient.deleteDatabase("dbs/" + DATABASE_ID, null);
}
catch (Exception e) {
System.out.println(e.getMessage());
}
// Define a new database using the id above.
Database myDatabase = new Database();
myDatabase.setId(DATABASE_ID);
// Create a new database.
myDatabase = documentClient.createDatabase(myDatabase, null)
.getResource();
System.out.println("Created a new database:");
System.out.println(myDatabase.toString());
System.out.println("Press any key to continue..");
System.in.read();
// Define a new collection using the id above.
DocumentCollection myCollection = new DocumentCollection();
myCollection.setId(COLLECTION_ID);
// Set the provisioned throughput for this collection to be 1000 RUs.
RequestOptions requestOptions = new RequestOptions();
requestOptions.setOfferThroughput(1000);
// Create a new collection.
myCollection = documentClient.createCollection(
"dbs/" + DATABASE_ID, myCollection, requestOptions)
.getResource();
System.out.println("Created a new collection:");
System.out.println(myCollection.toString());
System.out.println("Press any key to continue..");
System.in.read();
// Create an object, serialize it into JSON, and wrap it into a
// document.
SomePojo allenPojo = new SomePojo("123", "Allen Brewer", "allen [at] contoso.com");
String allenJson = gson.toJson(allenPojo);
Document allenDocument = new Document(allenJson);
// Create the 1st document.
allenDocument = documentClient.createDocument(
"dbs/" + DATABASE_ID + "/colls/" + COLLECTION_ID, allenDocument, null, false)
.getResource();
System.out.println("Created 1st document:");
System.out.println(allenDocument.toString());
System.out.println("Press any key to continue..");
System.in.read();
// Create another object, serialize it into JSON, and wrap it into a
// document.
SomePojo lisaPojo = new SomePojo("456", "Lisa Andrews",
"lisa [at] contoso.com");
String somePojoJson = gson.toJson(lisaPojo);
Document lisaDocument = new Document(somePojoJson);
// Create the 2nd document.
lisaDocument = documentClient.createDocument(
"dbs/" + DATABASE_ID + "/colls/" + COLLECTION_ID, lisaDocument, null, false)
.getResource();
System.out.println("Created 2nd document:");
System.out.println(lisaDocument.toString());
System.out.println("Press any key to continue..");
System.in.read();
// Query documents
List<Document> results = documentClient
.queryDocuments(
"dbs/" + DATABASE_ID + "/colls/" + COLLECTION_ID,
"SELECT * FROM myCollection WHERE myCollection.email = 'allen [at] contoso.com'",
null).getQueryIterable().toList();
System.out.println("Query document where e-mail address = 'allen [at] contoso.com':");
System.out.println(results.toString());
System.out.println("Press any key to continue..");
System.in.read();
// Replace Document Allen with Percy
allenPojo = gson.fromJson(results.get(0).toString(), SomePojo.class);
allenPojo.setName("Percy Bowman");
allenPojo.setEmail("Percy Bowman [at] contoso.com");
allenDocument = documentClient.replaceDocument(
allenDocument.getSelfLink(),
new Document(gson.toJson(allenPojo)), null)
.getResource();
System.out.println("Replaced Allen's document with Percy's contact information");
System.out.println(allenDocument.toString());
System.out.println("Press any key to continue..");
System.in.read();
// Delete Percy's Document
documentClient.deleteDocument(allenDocument.getSelfLink(), null);
System.out.println("Deleted Percy's document");
System.out.println("Press any key to continue..");
System.in.read();
// Delete Database
documentClient.deleteDatabase("dbs/" + DATABASE_ID, null);
System.out.println("Deleted database");
System.out.println("Press any key to continue..");
System.in.read();
}
}
enviroment
java : openjdk 11
I would be grateful if you could give us any suggestions.
The error was due to the old version being built.
When built with the latest version 2.4.6, it works properly.
https://mvnrepository.com/artifact/com.microsoft.azure/azure-documentdb/2.4.6
I'm using Java to download HTML contents of websites whose URLs are stored in a database. I'd like to put their HTML into database, too.
I'm using Jsoup for this purpose:
public String downloadHTML(String byLink) {
String htmlInPage = "";
try {
Document doc = Jsoup.connect(byLink).get();
htmlInPage = doc.html();
} catch (org.jsoup.UnsupportedMimeTypeException e) {
// process this and some other exceptions
}
return htmlInPage;
}
I'd like to download websites concurrently and use this function:
public void downloadURL(int websiteId, String url,
String categoryName, ExecutorService executorService) {
executorService.submit((Runnable) () -> {
String htmlInPage = downloadHTML(url);
System.out.println("Category: " + categoryName + " " + websiteId + " " + url);
String insertQuery =
"INSERT INTO html_data (website_id, html_contents) VALUES (?,?)";
dbUtils.query(insertQuery, websiteId, htmlInPage);
});
}
dbUtils is my class based on Apache Commons DbUtils. Details are here: http://pastebin.com/iAKXchbQ
And I'm using everything mentioned above in a such way: (List<Object[]> details are explained on pastebin, too)
public static void main(String[] args) {
DbUtils dbUtils = new DbUtils("host", "db", "driver", "user", "pass");
List<String> categoriesList =
Arrays.asList("weapons", "planes", "cooking", "manga");
String sql = "SELECT lw.id, lw.website_url, category_name " +
"FROM list_of_websites AS lw JOIN list_of_categories AS lc " +
"ON lw.category_id = lc.id " +
"where category_name = ? ";
ExecutorService executorService = Executors.newFixedThreadPool(10);
for (String category : categoriesList) {
List<Object[]> sitesInCategory = dbUtils.select(sql, category );
for (Object[] entry : sitesInCategory) {
int websiteId = (int) entry[0];
String url = (String) entry[1];
String categoryName = (String) entry[2];
downloadURL(websiteId, url, categoryName, executorService);
}
}
executorService.shutdown();
}
I'm not sure if this solution is correct but it works. Now I want to modify code to save HTML not from all websites in my database, but only their fixed ammount in each category.
For example, download and save HTML of 50 websites from the "weapons" category, 50 from "planes", etc. I don't think it's necessary to use sql for this purpose: if we select 50 sites per category, it doesn't mean we save them all, because of possibly incorrect syntax and connection problems.
I've tryed to create separate class implementing Runnable with fields: counter and maxWebsitesPerCategory, but these variables aren't updated. Another idea was to create field Map<String,Integer> sitesInCategory instead of counter, put each category as a key there and increment its value until it reaches maxWebsitesPerCategory, but it didn't work, too. Please, help me!
P.S: I'll also be grateful for any recommendations connected with my realization of concurrent downloading (I haven't worked with concurrency in Java before and this is my first attempt)
How about this?
for (String category : categoriesList) {
dbUtils.select(sql, category).stream()
.limit(50)
.forEach(entry -> {
int websiteId = (int) entry[0];
String url = (String) entry[1];
String categoryName = (String) entry[2];
downloadURL(websiteId, url, categoryName, executorService);
});
}
sitesInCategory has been replaced with a stream of at most 50 elements, then your code is run on each entry.
EDIT
In regard to comments. I've gone ahead and restructured a bit, you can modify/implement the content of the methods I've suggested.
public void werk(Queue<Object[]> q, ExecutorService executorService) {
executorService.submit(() -> {
try {
Object[] o = q.remove();
try {
String html = downloadHTML(o); // this takes one of your object arrays and returns the text of an html page
insertIntoDB(html); // this is the code in the latter half of your downloadURL method
}catch (/*narrow exception type indicating download failure*/Exception e) {
werk(q, executorService);
}
}catch (NoSuchElementException e) {}
});
}
^^^ This method does most of the work.
for (String category : categoriesList) {
Queue<Object[]> q = new ConcurrentLinkedQueue<>(dbUtils.select(sql, category));
IntStream.range(0, 50).forEach(i -> werk(q, executorService));
}
^^^ this is the for loop in your main
Now each category tries to download 50 pages, upon failure of downloading a page it moves on and tries to download another page. In this way, you will either download 50 pages or have attempted to download all pages in the category.
I'm following this tutorial to write a workflow for 3 HTML forms (without using CQ5 form component and CQ5 workflow submit button). I use this code to write Process step handling for my workflow:
public class MyProcess implements WorkflowProcess {
public void execute(WorkItem item, WorkflowSession session,
MetaDataMap map) throws WorkflowException {
boolean advanced = false;
Boolean goBack = map.get("goBack", Boolean.class);
List<Route> routes = null;
if (goBack == null || goBack == false) {
routes = session.getRoutes(item);
} else {
routes = session.getBackRoutes(item);
}
for (Route route : routes) {
LOG.info("===============================");
LOG.info("Rout name: " + route.getName());
LOG.info("Destinations: ");
for (WorkflowTransition dest: route.getDestinations()) {
LOG.info("dest: " + dest.getTo().getTitle());
}
LOG.info("===============================");
if (route.hasDefault()) {
String fromTitle = item.getNode().getTitle();
String toTitle = route.getDestinations().get(0).getTo()
.getTitle();
session.complete(item, route);
LOG.info("===============================");
LOG.info(item.getId() + " advanced from " + fromTitle
+ " to " + toTitle);
LOG.info("===============================");
advanced = true;
}
}
// fallback if no route was marked as default
if (!advanced) {
session.complete(item, routes.get(0));
String fromTitle = item.getNode().getTitle();
String toTitle = routes.get(0).getDestinations().get(0).getTo()
.getTitle();
LOG.info("===============================");
LOG.info(item.getId() + " advanced from " + fromTitle + " to "
+ toTitle);
LOG.info("===============================");
}
}
}
My question is: after i use session.complete to advance to next step, how can i refresh the workflow session to reach the current step information.
I think #yashahuja is correct. I was poking around and on the aem 'working with workflow' page I found some information about persisting data throw workflow steps using MetaDataMap.
From this page: http://dev.day.com/docs/en/cq/current/workflows/wf-extending.html
"Use workflow metadata to persist information that is required during the lifetime of the workflow. A common requirement of workflow steps is to persist data for future use in the workflow, or to retrieve the persisted data.
Workflow metadata is stored in a MetaDataMap object. The Java API provides the Workflow.getMetaDataMap method that returns the MetaDataMap object. Also, the WorkItem.getWorkflowData method returns a WorkflowData object that provides the same getMetaDataMap object.
Therefore, the workflow MetaDataMap object is available to the OSGi service or ECMA script of a step component."
example:
public void execute(WorkItem item, WorkflowSession session, MetaDataMap args) throws WorkflowException {
MetaDataMap wfd = item.getWorkflow().getMetaDataMap();
wfd.put("mykey", "My Step Value");
Set<String> keyset = wfd.keySet();
Iterator<String> i = keyset.iterator();
while (i.hasNext()){
Object key = i.next();
log.info("The workflow medata includes key {} and value {}",key.toString(),wfd.get(key).toString());
}