Similar to: I cannot query my dynamodb table from aws lambda due to wrong filterexpression? and DynamoDB update error Invalid UpdateExpression: An expression attribute value used in expression is not defined
I am trying to code a way to query DynamoDB tables using partial matches on Partition Key / Sort Key in Java.
The DynamoDB table I am trying to access has a Partition key of "Type" (A restricted key word in DynamoDB, I know, but not my choice) and a Sort key of "Id". I know the "Type" but not the full Id, so I have researched the Query method using AWS SDK 2.x source code and have implemented as shown below:
DynamoDBClient dynamoDbClient = DynamoDbClient.builder()
.region(Region.EU_WEST_1)
.credentialsProvider(StaticCredentialsProvider.create(awsCredentials))
.build();
String idKey = "wholeIdKey";
String idValue = "partialIdValue";
String typeValue = "typeValue";
Map<String, String> expressionNames = new HashMap<>();
expressionNames.put("#t", "Type");
QueryRequest request = QueryRequest.builder()
.tableName(tableName)
.keyConditionExpression("begins_with ( " + idKey + ", :" + idValue + " )
AND #t = :" + typeValue)
.expressionAttributeNames(expressionNames)
.build();
QueryResponse response = dynamoDbClient.query(request);
However, when I run this code, I get the following error message:
Exception in thread "main" software.amazon.awssdk.services.dynamodb.model.DynamoDbException:
Invalid KeyConditionExpression: An expression attribute value used in expression is not defined; attribute value: :typeValue
It's as if it's not recognizing the fact that I have told the code use the Expression Attribute Names feature to replace the "#t" with "Type" (Which is a reserved keyword in DynamoDB)
Can anyone help?
EDIT: References for code:
https://docs.aws.amazon.com/code-samples/latest/catalog/javav2-dynamodb-src-main-java-com-example-dynamodb-Query.java.html
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.ExpressionAttributeNames.html
https://www.javadoc.io/static/software.amazon.awssdk/dynamodb/2.7.14/software/amazon/awssdk/services/dynamodb/model/QueryRequest.html#expressionAttributeNames--
The name is fine, but you're prefixing both values with ':'. That causes a lookup in ExpressionAttributeValues, which you did not provide.
Never try to write dynamic values directly into the query string.
Your expressionAttributeName looks fine, but you forgot to provide a value for :typeValue so dynamoDB cannot know what to look for.
In addition to what you did, you need to add an expressionAttributeValue where you can provide proper values. See documentation here
Fixed Code for whoever wants it in the future (Thanks to #aherve and #MattTimmermans)
DynamoDBClient dynamoDbClient = DynamoDbClient.builder()
.region(Region.EU_WEST_1)
.credentialsProvider(StaticCredentialsProvider.create(awsCredentials))
.build();
String idKey = "wholeIdKey";
String idValue = "partialIdValue";
String typeValue = "typeValue";
String typeKey = "typeKey";
Map<String, String> expressionNames = new HashMap<>();
expressionNames.put("#t", "Type");
expressionNames.put("#i", "Id");
Map<String, AttributeValue> expressionValues = new HashMap<>();
expressionValues.put(":typeName", AttributeValue.builder().s(typeValue).build());
expressionValues.put(":idName", AttributeValue.builder().s(idValue).build());
QueryRequest request = QueryRequest.builder()
.tableName(tableName)
.keyConditionExpression("#t = :typeName AND begins_with ( #i, :idName )")
.expressionAttributeNames(expressionNames)
.expressionAttributeValues(expressionValues)
.build();
response = dynamoDbClient.query(request);
I am trying to query a mongodb collection and retrieve certain documents based on a field value but also only retrieve a single field per record. I tried the following but no getting the solution I want:
MongoCollection<Document> collection =
database.getCollection("client_data");
//Document document = collection
// .find(new BasicDBObject("sampleUser", "myDb"))
//.projection(Projections.fields(Projections.include("address"),
//Projections.excludeId())).first();
BasicDBObject aQuery = new BasicDBObject();
aQuery.put("clientId",567);
FindIterable<Document> iterDoc = collection.find(aQuery);
The following retrieves all documents for clientid = 567, but I only want to show the address field.
The commented out code was also what I tried but not sure how to combine that with the query.
EDIT:
I am now able to iterate through all the results but would like to parse each document; I tried parsing the document into my class object but it immediately gives an error:
Unrecognized field "_id" (class
model.Client), not marked as ignorable
But _id is the very first field in the document:
Document{{_id=6216a7f64cedfd00011c35a5,
So I tried something else rather using the first document but then I don't know how to get the next document:
while(cursor.hasNext()) {
// System.out.println(cursor.next().toJson());
Client client = new Client();
try {
JsonParser jsonParser = new JsonFactory().createParser(cursor.next().toJson());
ObjectMapper mapper = new ObjectMapper();
ObjectNode rootNode = mapper.createObjectNode();
String customerInfo = fi.first().toJson();
JsonNode jobj = mapper.readTree(customerInfo);
// this gives the error// client = mapper.readValue(jsonParser,Client.class);
client.setId(jobj.path("_id").path("$oid").asText());
Please advise.
In order to:
retrieves all documents for clientid = 567, but I only want to show the address field
You would execute the following:
collection
.find(Filters.eq("clientId", 567))
.projection(Projections.fields(
Projections.include("address"),
Projections.excludeId())
).first()
Breaking it down:
.find(Filters.eq("clientId", 567)): apply the predicate 'where clientId = 567'
.projection(Projections.fields(Projections.include("address"), Projections.excludeId())): let the response include the address field and exclude the _id field
Pretty simple problem, but for some reason there is no correct documentation anywhere. Here is what I am implementing:
public static AttributeValue pushMessage(String ID,String message)
{
Map<String, AttributeValue> itemKey = mapKey(KEY, new AttributeValue(ID));
UpdateItemRequest request = new UpdateItemRequest();
request.setTableName(TABLE_NAME);
request.setKey(itemKey);
/*I don't know what to put for the update expression*/
//request.setUpdateExpression("ADD #"+LIST_NAME+" :append_value)");
request.setUpdateExpression("list_append("+LIST_NAME+", :append_value)");
request.setExpressionAttributeValues(Collections.singletonMap(":append_value", new AttributeValue(message)));
UpdateItemResult result = dynamoDB.updateItem(request);
}
I essentially just want to be able to append strings to a list I have in the items in my DB. I have tried everything the documentations have suggested(so basically 1 example) and numerous threads, but nothing works.
The error I am currently getting is:
Exception in thread "main" com.amazonaws.AmazonServiceException: Invalid UpdateExpression: Syntax error; token: "list_append", near: "list_append(" (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException;
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1372)
at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:919)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:697)
at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:449)
at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:411)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:360)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:2048)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:2018)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.updateItem(AmazonDynamoDBClient.java:1845)
at DBConnector.pushMessage(DBConnector.java:486)
at DBConnector.main(DBConnector.java:506)
Use the following UpdateExpression:
String.format("SET %s = list_append(%s, :append_value)", LIST_NAME, LIST_NAME)
I'm simply trying to update a customfield value in jira using java. I had created a method updateCustomField which accepts 3 parameters (customFieldCode, value, jiraId). Had tried using transition but all it did is change the jira status from "Open" to "Resolved 2". I googled everywhere but they suggest to use JSON which I have no idea how to apply.
here's my update method:
public void updateCustomField(String customFieldCode, String value, String jiraId) throws Exception {
final IssueRestClient issueRestClient = jiraClient.getIssueClient();
final Issue issue = issueRestClient.getIssue(jiraId).get();
FieldInput fieldInput = new FieldInput(customFieldCode, value);
List <FieldInput> fields = new ArrayList <FieldInput> ();
fields.add(fieldInput);
TransitionInput transision = new TransitionInput(1, fields);
issueRestClient.transition(issue, transision);
}
For those who want to simply update jira using java, you can try this jira-client library.
Here is a simple pojo:
public class Description {
private String code;
private String name;
private String norwegian;
private String english;
}
And please see the following code to apply an upsert to MongoDb via spring MongoTemplate:
Query query = new Query(Criteria.where("code").is(description.getCode()));
Update update = new Update().set("name", description.getName()).set("norwegian", description.getNorwegian()).set("english", description.getEnglish());
mongoTemplate.upsert(query, update, "descriptions");
The line to generate the Update object specifies every field of the Item class manually.
But if my Item object changes then my Dao layer breaks.
So is there a way to avoid doing this, so that all fields from my Item class are applied automatically to the update?
E.g.
Update update = new Update().fromObject(item);
Note that my pojo does not extend DBObject.
I found a pretty good solution for this question
//make a new description here
Description d = new Description();
d.setCode("no");
d.setName("norwegian");
d.setNorwegian("norwegian");
d.setEnglish("english");
//build query
Query query = new Query(Criteria.where("code").is(description.getCode()));
//build update
DBObject dbDoc = new BasicDBObject();
mongoTemplate.getConverter().write(d, dbDoc); //it is the one spring use for convertions.
Update update = Update.fromDBObject(dbDoc);
//run it!
mongoTemplate.upsert(query, update, "descriptions");
Plz note that Update.fromDBObject return an update object with all fields in dbDoc. If you just want to update non-null fields, you should code a new method to exclude null fields.
For example, the front-end post a doc like below:
//make a new description here
Description d = new Description();
d.setCode("no");
d.setEnglish("norwegian");
We only need to update the field 'language':
//return Update object
public static Update fromDBObjectExcludeNullFields(DBObject object) {
Update update = new Update();
for (String key : object.keySet()) {
Object value = object.get(key);
if(value!=null){
update.set(key, value);
}
}
return update;
}
//build udpate
Update update = fromDBObjectExcludeNullFields(dbDoc);
The solution for a new spring-data-mongodb version 2.X.X.
The API has evolved, since 2.X.X version there is:
Update.fromDocument(org.bson.Document object, String... exclude)
instead of (1.X.X):
Update.fromDBObject(com.mongodb.DBObject object, String... exclude)
The full solution:
//make a new description here
Description d = new Description();
d.setCode("no");
d.setName("norwegian");
d.setNorwegian("norwegian");
d.setEnglish("english");
Query query = new Query(Criteria.where("code").is(description.getCode()));
Document doc = new Document(); // org.bson.Document
mongoTemplate.getConverter().write(item, doc);
Update update = Update.fromDocument(doc);
mongoTemplate.upsert(query, update, "descriptions");
It works!
you can use save : (if non exist = insert else = upsert)
save(Object objectToSave, String collectionName)
read : javadoc
Just like previous answers said, use mongoTemplate.getConverter().write() and Update.fromDocument() functions. But i found Update.fromDocument() won't add "$set" key and won't work directly, the solution is to add "$set" yourself, like below (PS: I'm using 2.2.1.RELEASE version):
public static Update updateFromObject(Object object, MongoTemplate mongoTemplate) {
Document doc = new Document();
mongoTemplate.getConverter().write(object, doc);
return Update.fromDocument(new Document("$set", doc));
}
If you want to upsert Pojos incl. property String id; you have to exclude the _id field in the fromDBObject method Update.fromDBObject(dbDoc,"_id").
Otherwise you get the Exception:
org.springframework.dao.DuplicateKeyException: { "serverUsed" : "127.0.0.1:27017" , "ok" : 1 , "n" : 0 , "updatedExisting" : false , "err" : "E11000 duplicate key error collection: db.description index: _id_ dup key: { : null }" , "code" : 11000}; nested exception is com.mongodb.MongoException$DuplicateKey: { "serverUsed" : "127.0.0.1:27017" , "ok" : 1 , "n" : 0 , "updatedExisting" : false , "err" : "E11000 duplicate key error collection: db.description index: _id_ dup key: { : null }" , "code" : 11000}
because the _id field of the first is null
{
"_id" : null,
...
}
Fullcode based on #PaniniGelato answer would be
public class Description(){
public String id;
...
}
Description d = new Description();
d.setCode("no");
d.setName("norwegian");
d.setNorwegian("norwegian");
d.setEnglish("english");
//build query
Query query = new Query(Criteria.where("code").is(description.getCode()));
//build update
DBObject dbDoc = new BasicDBObject();
mongoTemplate.getConverter().write(d, dbDoc); //it is the one spring use for convertions.
Update update = Update.fromDBObject(dbDoc, "_id");
//run it!
mongoTemplate.upsert(query, update, "descriptions");
Then the upsert is working in the cases of insert and update. Corrections & thoughts are welcome ;)
This is what I am doing for the time being. Not so much elegant way to do it, but it does save a precious DB call:
import org.springframework.data.mongodb.core.MongoTemplate;
import org.springframework.data.mongodb.core.query.Query;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.mongodb.BasicDBObject;
import com.mongodb.DB;
import com.mongodb.DBCollection;
import com.mongodb.DBObject;
import com.mongodb.util.JSON;
/**
* Perform an upsert operation to update ALL FIELDS in an object using native mongo driver's methods
* since mongoTemplate's upsert method doesn't allow it
* #param upsertQuery
* #param object
* #param collectionName
*/
private void performUpsert(Query upsertQuery, Object object, String collectionName){
ObjectMapper mapper = new ObjectMapper();
try {
String jsonStr = mapper.writeValueAsString(object);
DB db = mongoTemplate.getDb();
DBCollection collection = db.getCollection(collectionName);
DBObject query = upsertQuery.getQueryObject();
DBObject update = new BasicDBObject("$set", JSON.parse(jsonStr));
collection.update(query, update, true, false);
} catch (IOException e) {
LOGGER.error("Unable to persist the metrics in DB. Error while parsing object: {}", e);
}
}
There are two cases here that need to be distinguished:
Update an item that was previously fetched from the DB.
Update or insert (upsert) an item you created by code.
In Case 1) You can simply use mongoTemplate.save(pojo, "collection"), because your POJO will already have a filled ObjectID in its id field.
In case 2) You have to explain to mongo what "already exists" means in case of your domain model: By default the mongoTemplate.save() method updates an existing item, if there is one with that same ObjectId. But with a newly instantiated POJO you do not have that id. Therefore the mongoTemplate.upsert() method has a query parameter that you can create like this:
MyDomainClass pojo = new MyDomainClass(...);
Query query = Query.query(Criteria.where("email").is("user1#domain.com"));
DBObject dbDoc = new BasicDBObject();
mongoTemplate.getConverter().write(pojo, dbDoc); //it is the one spring use for convertions.
dbDoc.removeField("_id"); // just to be sure to not create any duplicates
Update update = Update.fromDBObject(dbDoc);
WriteResult writeResult = mongoTemplate.upsert(query, update, UserModel.class);
I ran into the same problem. In het current Spring Data MongoDB version no such thing is available. You have to update the seperate fields by hand.
However it is possible with another framework: Morphia.
This framework has a wrapper for DAO functionality: https://github.com/mongodb/morphia/wiki/DAOSupport
You can use the DAO API to do things like this:
SomePojo pojo = daoInstance.findOne("some-field", "some-value");
pojo.setAProperty("changing this property");
daoInstance.save(pojo);
I think that:
Description add a property
#Id
private String id;
then get a document by the query condition,set Description's id by document's id.
and save
Just use ReflectionDBObject - if you make Description extend it, you should just get your object's fields transferred to Update reflectively, automagically. The note from above about null fields included in the update still holds true.
public void saveOrUpdate(String json) {
try {
JSONObject jsonObject = new JSONObject(json);
DBObject update1 = new BasicDBObject("$set", JSON.parse(json));
mongoTemplate.getCollection("collectionName").update(new Query(Criteria.where("name").is(jsonObject.getString("name"))).getQueryObject(), update1, true, false);
} catch (Exception e) {
throw new GenericServiceException("Error while save/udpate. Error msg: " + e.getMessage(), e);
}
}
this is very simple way to save json string into collection using mongodb
and spring.
This method can be override to use as JSONObject.
#Override
public void updateInfo(UpdateObject algorithm) {
Document document = new Document();
mongoTemplate.getConverter().write(algorithm, document);
Update update = Update.fromDocument(document);
mongoTemplate.updateFirst(query(where("_id").is(algorithm.get_id())), update, UpdateObject.class);
}
After upsert, I was Tring to fetch same record but it was given me the old one.
But in dB I am having new records.