I am trying to use ChangeStreams in MongoDB 4.4 and Camel 3.12.0. According to the Camel documents, the Exchange body will contain the full document of any change. I am building my route as below:
from("mongodb:mongoClient?consumerType=changeStreams&database=test&collection=accounts")
.process(new MongoIdProcessor())
.to("solrCloud://minikube:8983/solr?zkHost=minikube:2181,minikube:2182,minikube:2183&collection=accounts&autoCommit=true")
What I noticed is that if I issue an update (updateOne, updateMany) command on the "accounts" collection, there isn't any data in the Exchange object during processing.
Message message = exchange.getMessage(); // Null
Message in = exchange.getIn();
ObjectId objectId = in.getHeader("_id", ObjectId.class); // Present
Digging a little deeper, it seems that in MongoDbChangeStreamsThread.java, the collection being watched does not have the options set correctly.
#Override
protected MongoCursor initializeCursor() {
ChangeStreamIterable<Document> iterable = bsonFilter != null
? dbCol.watch(bsonFilter)
: dbCol.watch();
It should be this instead
#Override
protected MongoCursor initializeCursor() {
ChangeStreamIterable<Document> iterable = bsonFilter != null
? dbCol.watch(bsonFilter).fullDocument(FullDocument.UPDATE_LOOKUP)
: dbCol.watch().fullDocument(FullDocument.UPDATE_LOOKUP);
Do I really have to make the change in this class, or is there some configuration somewhere I can set? I'm concerned about maintaining code for this modified class of the Camel library.
Related
I'm trying to implement a java REST API allowing the UI to list documents from Firestore (potentially ordering it on multiple fields).
I'm following the official documentation but I'm struggling about how to handle/generate a next page token (since the UI will potentially need to iterate over and over) from the response. Is there any way to implement this behavior with the GRPC client? Should I switch to the REST client (which seems to expose a nextPageToken field)?
Here is a workaround I found to mime a pagination-like behavior:
public <T extends InternalModel> Page<T> paginate(#NonNull Integer maxResults, #Nullable String pageToken) {
try (Firestore db = getFirestoreService()) {
CollectionReference collectionReference = db.collection(getCollection(type));
Query query = collectionReference
.limit(maxResults);
// The page token is the id of the last document
if (!Strings.isNullOrEmpty(pageToken)) {
DocumentSnapshot lastDocument = collectionReference.document(pageToken).get().get();
query = query.startAfter(lastDocument);
}
List<InternalModel> items = (List<InternalModel>) query.get().get().toObjects(type);
String nextPageToken = "";
if (!CollectionUtils.isEmpty(items) && maxResults.equals(items.size())) {
nextPageToken = items.get(items.size() - 1).getId();
}
return Page.create(items, nextPageToken);
}
}
I'm opened to any better solution since this might not be the most optimal way.
I work on university project in java. I have to download attachments from new emails using GMAIL API.
I successfully connected to gmail account using OAuth 2.0 authorization.
private static final List<String> SCOPES = Collections.singletonList(GmailScopes.GMAIL_READONLY);
I tried to get unseen mails using
ListMessagesResponse listMessageResponse = service.users().messages().list(user).setQ("is:unseen").execute();
listMessageResponse is not null but when I call method .getResultSizeEstimate() it returns 0
also I tried to convert listMessageResponse to List < Message > (I guess this is more usable) using
List<Message> list = listMessageResponse.getMessages();
But list launches NullPointerException
Then tried to get each attachment with
for(Message m : list) {
List<MessagePart> part = m.getPayload().getParts();
for(MessagePart p: part) {
if(p.getFilename()!=null && p.getFilename().length()>0) {
System.out.println(p.getFilename()); // Just to check attachment filename
}
}
}
Is my approach correct (if not how to fix it) and how should I download those attachments.
EDIT 1:
Fixed q parameter, I mistakenly wrote is:unseen instead of is:unread.
Now app reaches unread mails successfully.
(For example there was two unread mails and both successfully reached, I can get theirs IDs easy).
Now this part trows NullPointerException
List<MessagePart> part = m.getPayload().getParts();
Both messages have attachments and m is not null (I get ID with .getID())
Any ideas how to overcome this and download attachment?
EDIT 2:
Attachments Downloading part
for(MessagePart p : parts) {
if ((p.getFilename() != null && p.getFilename().length() > 0)) {
String filename = p.getFilename();
String attId = p.getBody().getAttachmentId();
MessagePartBody attachPart;
FileOutputStream fileOutFile = null;
try {
attachPart = service.users().messages().attachments().get("me", p.getPartId(), attId).execute();
byte[] fileByteArray = Base64.decodeBase64(attachPart.getData());
fileOutFile = new FileOutputStream(filename); // Or any other dir
fileOutFile.write(fileByteArray);
fileOutFile.close();
}catch (IOException e) {
System.out.println("IO Exception processing attachment: " + filename);
} finally {
if (fileOutFile != null) {
try {
fileOutFile.close();
} catch (IOException e) {
// probably doesn't matter
}
}
}
}
}
Downloading working like charm, tested app with different type of emails.
Only thing left is to change label of unread message (that was reached by app) to read. Any tips how to do it?
And one tiny question:
I want this app to fetch mails on every 10 minutes using TimerTask abstract class. Is there need for manual "closing" of connection with gmail or that's done automatically after run() method iteration ends?
#Override
public void run(){
// Some fancy code
service.close(); // Something like that if even exists
}
I don't think ListMessagesResponse ever becomes null. Even if there are no messages that match your query, at least resultSizeEstimate will get populated in the resulting response: see Users.messages: list > Response.
I think you are using the correct approach, just that there is no message that matches your query. Actually, I never saw is:unseen before. Did you mean is:unread instead?
Update:
When using Users.messages: list only the id and the threadId of each message is populated, so you cannot access the message payload. In order to get the full message resource, you have to use Users.messages: get instead, as you can see in the referenced link:
Note that each message resource contains only an id and a threadId. Additional message details can be fetched using the messages.get method.
So in this case, after getting the list of messages, you have to iterate through the list, and do the following for each message in the list:
Get the message id via m.getId().
Once you have retrieved the message id, use it to call Gmail.Users.Messages.Get and get the full message resource. The retrieved message should have all fields populated, including payload, and you should be able to access the corresponding attachments.
Code sample:
List<Message> list = listMessageResponse.getMessages();
for(Message m : list) {
Message message = service.users().messages().get(user, m.getId()).execute();
List<MessagePart> part = message.getPayload().getParts();
// Rest of code
}
Reference:
Class ListMessagesResponse
Users.messages: list > Response
For the below mentioned code, I'm getting Trust Boundary Violation in the CheckMarx report.
Error description -
Method 'getResponse' gets user input from element request. This element’s value flows through the code without being properly sanitized or validated and is eventually stored in the server-side Session object, in 'parseRequest' method.**
Code -
#Context
HttpHeaders httpHeader;
void parseRequest(SomeRequestType inputRequest) {
HashMap<String, Data> requestData = inputRequest.getRequestData(httpHeader);
if (requestData != null) {
if (Strings.isNullOrEmpty(inputRequest.getId())) {
Data data = requestData.get("data");
var dataID = data.getID();
if ((dataID != null) && Pattern.matches("[0-9]+", dataID)) {
inputRequest.setId(dataID);
ThreadContext.put("ID", dataID);
}
}
}
}
I am getting checkmarx vulnerability at below line for without being properly sanitized or validated
ThreadContext.put("ID", dataID);
Could some please help me, how to properly sanitize the above line.
If you know for sure that dataID is a number, convert it to integer/long right away, like this:
int dataIDasNumber = Integer.parseInt(dataID);
And use it like int/long here:
inputRequest.setId(dataIDasNumber);
ThreadContext.put("ID", dataIDasNumber);
Then you don't need to do this:
Pattern.matches...
And your checkmarx violation should go away.
I have a Java application running on Spring and defining multiple gRPC endpoints. These endpoints are meant to be queried from multiple clients, one of which being in PHP, so I used the PHP lib for gRPC. Now I wonder how to properly get the metadata from the server in case of an invalid request, this metadata containing mostly constraint violations built by the Java validator and transformed into a collection of gRPC FieldViolation objects. In this example, the server is supposed to return one single field violation as metadata, with the key "violationKey" and the description "violationDescription":
try {
// doStuff
} catch (ConstraintViolationException e) {
Metadata trailers = new Metadata();
trailers.put(ProtoUtils.keyForProto(BadRequest.getDefaultInstance()), BadRequest
.newBuilder()
.addFieldViolations(FieldViolation
.newBuilder()
.setField("violationKey")
.setDescription("violationDescription")
.build()
)
.build()
);
responseObserver.onError(Status.INVALID_ARGUMENT.asRuntimeException(trailers));
}
On the PHP side, this is the implementation to retrieve the metadata:
class Client extends \Grpc\BaseStub
{
public function callService()
{
$call = $this->_simpleRequest(
'MyService/MyAction',
$argument,
['MyActionResponse', 'decode'],
$metadata, $options
);
list($response, $status) = $call->wait();
var_dump($status->metadata); // A
var_dump($call->getMetadata()); // B
}
}
Result: "A" outputs an empty array, "B" outputs the proper metadata, formatted as follows:
array(1) {
["google.rpc.badrequest-bin"]=>
array(1) {
[0]=>
string(75) "
I
testALicense plate number is not in a valid format for country code FR"
}
}
Why is the metadata in the status empty, and why is the metadata retrieved by $call->getMetadata() is formatted that way ("I" followed by the violation key, then "A" and finally the violation description) ? How can I avoid to make potentially tedious transformation of this metadata client-side?
Can you please log an issue on our grpc/grpc Github repo so that we can better follow up there? Thanks.
I am trying to delete all documents from a database but i want to preserve the views. So i tried
//First, get all DesignDocument for the current database
List<DesignDocument> dbDesigns = cloudant.getDesignDocumentManager().list();
//Now, we delete the database
cloudantClient.deleteDB(_DatabaseName);
//now we create the database again
cloudant = cloudantClient.database(_DatabaseName, true);
//finally, try to add the DesignDocuments back
if (dbDesigns != null && dbDesigns.size() > 0) {
for (DesignDocument dDoc : dbDesigns) {
Response response = cloudant.getDesignDocumentManager().put(dDoc);
System.out.println(response);
}
}
but i get error at
Response response = cloudant.getDesignDocumentManager().put(dDoc);
java.lang.IllegalArgumentException: rev should be null
at com.cloudant.client.org.lightcouch.internal.CouchDbUtil.assertNull(CouchDbUtil.java:72)
at com.cloudant.client.org.lightcouch.CouchDbClient.put(CouchDbClient.java:410)
at com.cloudant.client.org.lightcouch.CouchDbClient.put(CouchDbClient.java:394)
at com.cloudant.client.org.lightcouch.CouchDatabaseBase.save(CouchDatabaseBase.java:196)
at com.cloudant.client.api.Database.save(Database.java:710)
at com.cloudant.client.api.DesignDocumentManager.put(DesignDocumentManager.java:122)
is there any other way to preserve the views?
I'm suspecting the error is raised because the document revision property (_rev) is set in dDoc. However, since a document with a matching id is not found in the database the put method raises an error. Try setting the revision to null using the setRevision method prior to invoking put
dDoc.setRevision(null);
Response response = cloudant.getDesignDocumentManager().put(dDoc);