We Keep Coding

Find not working for GridFS after updating metadata

I added a step in my application to persist files via GridFS and added a metadata field called "processed" to work as a flag for a scheduled task that retrieves the new file and sends it on for processing. Since the Java driver for GridFS doesn't have a method allowing metadata to be updated I used MongoCollection for the "fs.files" collection to update "metadata.processing" to true.
I use GridFSBucket.find(eq("metadata.processed", false) to get the new files for processing and then update metadata.processed to true once processing is completed. This works if I add a new file while the application is running. However, if I have an existing file with "metadata.processed" set to false and start the application, the above find call returns no results. Similarly if I have a file that was already processed and I set the "metadata.processed" field back to false, the above find call also ceases working.
private static final String FILTER_STR = "'{'\"filename\" : \"{0}\"'}'";
private static final String UPDATE_STR =
"'{'\"$set\": '{'\"metadata.processed\": \"{0}\"'}}'";
#Autowired
private GridFSBucketFactory gridFSBucketFactory;
#Autowired
private MongoCollectionFactory mongoCollectionFactory;
public void storeFile(String filename, DateTime publishTime,
InputStream inputStream) {
if (fileExists(filename)) {
LOGGER.info("File named {} already exists.", filename);
} else {
uploadToGridFS(filename, publishTime, inputStream);
LOGGER.info("Stored file named {}.", filename);
}
}
public GridFSDownloadStream getFile(BsonValue id) {
return gridFSBucketFactory.getGridFSBucket().openDownloadStream(id);
}
public GridFSDownloadStream getFile(String filename) {
final GridFSFile file = getGridFSFile(filename);
return file == null ? null : getFile(file.getId());
}
public GridFSFindIterable getUnprocessedFiles() {
return gridFSBucketFactory.getGridFSBucket()
.find(eq("metadata.processed", false));
}
public void setProcessed(String filename, boolean isProcessed) {
final BasicDBObject filter =
BasicDBObject.parse(format(FILTER_STR, filename));
final BasicDBObject update =
BasicDBObject.parse(format(UPDATE_STR, isProcessed));
if (updateOne(filter, update)) {
LOGGER.info("Set metadata for {} to {}", filename, isProcessed);
}
}
private void uploadToGridFS(String filename, DateTime publishTime,
InputStream inputStream) {
gridFSBucketFactory.getGridFSBucket().uploadFromStream(filename,
inputStream, createMetadata(publishTime));
}
private GridFSUploadOptions createMetadata(DateTime publishTime) {
final Document metadata = new Document();
metadata.put("processed", false);
// metadata.put("publishTime", publishTime.toString());
return new GridFSUploadOptions().metadata(metadata);
}
private boolean fileExists(String filename) {
return getGridFSFile(filename) != null;
}
private GridFSFile getGridFSFile(String filename) {
return gridFSBucketFactory.getGridFSBucket()
.find(eq("filename", filename)).first();
}
private boolean updateOne(BasicDBObject filter, BasicDBObject update) {
try {
mongoCollectionFactory.getFsFilesCollection().updateOne(filter,
update, new UpdateOptions().upsert(true));
} catch (final MongoException e) {
LOGGER.error(
"The following failed to update, filter:{0} update:{1}",
filter, update, e);
return false;
}
return true;
}
Any idea what I can do to ensure:
GridFSBucket.find(eq("metadata.processed", false)
returns the proper results for existing files and/or files that have had the metadata changed?

The issue was due to setting the metadata.processed value as a String vs a boolean.
When initially creating the metadata I set its value with a boolean:
private GridFSUploadOptions createMetadata(DateTime publishTime) {
final Document metadata = new Document();
metadata.put("processed", false);
// metadata.put("publishTime", publishTime.toString());
return new GridFSUploadOptions().metadata(metadata);
}
And later I check for a boolean:
public GridFSFindIterable getUnprocessedFiles() {
return gridFSBucketFactory.getGridFSBucket()
.find(eq("metadata.processed", false));
}
But when updating the metadata using the "fs.files" MongoCollection I incorrectly added quotes around the boolean value here:
private static final String UPDATE_STR =
"'{'\"$set\": '{'\"metadata.processed\": \"{0}\"'}}'";
Which caused the metadata value to be saved as a String vs a boolean.

AbstractGoogleClient - Application name is not set. Call Builder#setApplicationName

How can i set GCP Application name?
2018-Nov-22 11:18:18.283 WARN [http-nio-8080-exec-9]
c.g.a.c.g.s.AbstractGoogleClient - Application name is not set.
Call
Builder#setApplicationName.
I have following constructor in my class.
private final GoogleCredential googleCredential;
private final String projectId;
private final String zone;
private final String region;
private GcpClient(GcpAccountSettings settings) throws
GcpClientException {
try {
this.googleCredential =
getCredential(settings.getClientEmail(),
settings.getPrivateKey());
} catch (GeneralSecurityException | IOException e) {
throw new GcpClientException(e);
}
this.projectId = settings.getProjectId();
this.zone = settings.getZone();
this.region = settings.getRegions();
}
Im really hoping your help.

We can set application name like following.
Compute compute =
new Compute.Builder(googleCredential.getTransport(),
googleCredential.getJsonFactory(), googleCredential)
.setApplicationName(projectId).build();
if you need more details please follow below link
how to set application name

Using AWS Java's SDKs, how can I terminate the CloudFormation stack of the current instance?

Uses on-line decomentation I come up with the following code to terminate the current EC2 Instance:
public class Ec2Utility {
static private final String LOCAL_META_DATA_ENDPOINT = "http://169.254.169.254/latest/meta-data/";
static private final String LOCAL_INSTANCE_ID_SERVICE = "instance-id";
static public void terminateMe() throws Exception {
TerminateInstancesRequest terminateRequest = new TerminateInstancesRequest().withInstanceIds(getInstanceId());
AmazonEC2 ec2 = new AmazonEC2Client();
ec2.terminateInstances(terminateRequest);
}
static public String getInstanceId() throws Exception {
//SimpleRestClient, is an internal wrapper on http client.
SimpleRestClient client = new SimpleRestClient(LOCAL_META_DATA_ENDPOINT);
HttpResponse response = client.makeRequest(METHOD.GET, LOCAL_INSTANCE_ID_SERVICE);
return IOUtils.toString(response.getEntity().getContent(), "UTF-8");
}
}
My issue is that my EC2 instance is under an AutoScalingGroup which is under a CloudFormationStack, that is because of my organisation deployment standards though this single EC2 is all there is there for this feature.
So, I want to terminate the entire CloudFormationStack from the JavaSDK, keep in mind, I don't have the CloudFormation Stack Name in advance as I didn't have the EC2 Instance Id so I will have to get it from the code using the API calls.
How can I do that, if I can do it?

you should be able to use the deleteStack method from cloud formation sdk
DeleteStackRequest request = new DeleteStackRequest();
request.setStackName(<stack_name_to_be_deleted>);
AmazonCloudFormationClient client = new AmazonCloudFormationClient (<credentials>);
client.deleteStack(request);
If you don't have the stack name, you should be able to retrieve from the Tag of your instance
DescribeInstancesRequest request =new DescribeInstancesRequest();
request.setInstanceIds(instancesList);
DescribeInstancesResult disresult = ec2.describeInstances(request);
List <Reservation> list = disresult.getReservations();
for (Reservation res:list){
List <Instance> instancelist = res.getInstances();
for (Instance instance:instancelist){
List <Tag> tags = instance.getTags();
for (Tag tag:tags){
if (tag.getKey().equals("aws:cloudformation:stack-name")) {
tag.getValue(); // name of the stack
}
}

At the end I've achieved the desired behaviour using the set of the following util functions I wrote:
/**
* Delete the CloudFormationStack with the given name.
*
* #param stackName
* #throws Exception
*/
static public void deleteCloudFormationStack(String stackName) throws Exception {
AmazonCloudFormationClient client = new AmazonCloudFormationClient();
DeleteStackRequest deleteStackRequest = new DeleteStackRequest().withStackName("");
client.deleteStack(deleteStackRequest);
}
static public String getCloudFormationStackName() throws Exception {
AmazonEC2 ec2 = new AmazonEC2Client();
String instanceId = getInstanceId();
List<Tag> tags = getEc2Tags(ec2, instanceId);
for (Tag t : tags) {
if (t.getKey().equalsIgnoreCase(TAG_KEY_STACK_NAME)) {
return t.getValue();
}
}
throw new Exception("Couldn't find stack name for instanceId:" + instanceId);
}
static private List<Tag> getEc2Tags(AmazonEC2 ec2, String instanceId) throws Exception {
DescribeInstancesRequest describeInstancesRequest = new DescribeInstancesRequest().withInstanceIds(instanceId);
DescribeInstancesResult describeInstances = ec2.describeInstances(describeInstancesRequest);
List<Reservation> reservations = describeInstances.getReservations();
if (reservations.isEmpty()) {
throw new Exception("DescribeInstances didn't returned reservation for instanceId:" + instanceId);
}
List<Instance> instances = reservations.get(0).getInstances();
if (instances.isEmpty()) {
throw new Exception("DescribeInstances didn't returned instance for instanceId:" + instanceId);
}
return instances.get(0).getTags();
}
// XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
// Example of usage from the code:
deleteCloudFormationStack(getCloudFormationStackName());
// XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

How to change .properties file using apache.commons.configuration

I have an app where I filter messages according to some rules(existing some keywords or regexps). These rules are to be stored in .properties file(as they must be persistent). I've figured out how to read data from this file. here is the part of the code:
public class Config {
private static final Config ourInstance = new Config();
private static final CompositeConfiguration prop = new CompositeConfiguration();
public static Config getInstance() {
return ourInstance;
}
public Config(){
}
public synchronized void load() {
try {
prop.addConfiguration(new SystemConfiguration());
System.out.println("Loading /rules.properties");
final PropertiesConfiguration p = new PropertiesConfiguration();
p.setPath("/home/mikhail/bzrrep/DLP/DLPServer/src/main/resources/rules.properties");
p.load();
prop.addConfiguration(p);
} catch (ConfigurationException e) {
e.printStackTrace();
}
final int processors = prop.getInt("server.processors", 1);
// If you don't see this line - likely config name is wrong
System.out.println("Using processors:" + processors);
}
public void setKeyword(String customerId, String keyword){
}
public void setRegexp(String customerId, String regexp)
{}
}
as you see I'm going to add values to some properties. Here is the .properties file itself:
users = admin, root, guest
users.admin.keywords = admin
users.admin.regexps = test-5, test-7
users.root.keywords = root
users.root.regexps = *
users.guest.keywords = guest
users.guest.regexps =
I have a GUI for user to add keywords and regexps to this config. so, how to implement methods setKeyword and setRegexp?

The easyest way I found is to read the current values of the property to the String[], add there a new value and set property.
props.setProperty(fieldName, values);

Wicket image component not found on page

I get the following problem when trying to display a list of items. For each item, I have to display an image which is dynamically loaded via a Wicket WebResource. The items are loaded step by step — 50 at a time — upon user scrolling, using an Ajax scroll.
[ERROR] 2011-04-19 09:58:18,000 btpool0-1 org.apache.wicket.RequestCycle.logRuntimeException (host=, request=, site=):
org.apache.wicket.WicketRuntimeException: component documentList:scroller:batchElem:666:content:item:3:batchItemContent:linkToPreview:imageThumbnail not found on page com.webapp.document.pages.DocumentListPage[id = 1]
listener interface = [RequestListenerInterface name=IResourceListener, method=public abstract void org.apache.wicket.IResourceListener.onResourceRequested()]
org.apache.wicket.protocol.http.request.InvalidUrlException: org.apache.wicket.WicketRuntimeException: component documentList:scroller:batchElem:666:content:item:3:batchItemContent:linkToPreview:imageThumbnail
not found on page com.webapp.document.pages.DocumentListPage[id = 1] listener interface = [RequestListenerInterface name=IResourceListener, method=public abstract void org.apache.wicket.IResourceListener.onResourceRequested()]
at org.apache.wicket.protocol.http.WebRequestCycleProcessor.resolve(WebRequestCycleProcessor.java:262)
at org.apache.wicket.RequestCycle.step(RequestCycle.java:1310)
at org.apache.wicket.RequestCycle.steps(RequestCycle.java:1428)
at org.apache.wicket.RequestCycle.request(RequestCycle.java:545)
at org.apache.wicket.protocol.http.WicketFilter.doGet(WicketFilter.java:479)
at org.apache.wicket.protocol.http.WicketFilter$$EnhancerByGuice$$51619816.CGLIB$doGet$6()
at org.apache.wicket.protocol.http.WicketFilter$$EnhancerByGuice$$51619816$$FastClassByGuice$$6d42bf5d.invoke()
at com.google.inject.internal.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:64)
at com.freiheit.monitoring.PerformanceMonitoringMethodInterceptor.invoke(PerformanceMonitoringMethodInterceptor.java:115)
at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:64)
at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:44)
at org.apache.wicket.protocol.http.WicketFilter$$EnhancerByGuice$$51619816.doGet()
at org.apache.wicket.protocol.http.WicketFilter.doFilter(WicketFilter.java:312)
at org.apache.wicket.protocol.http.WicketFilter$$EnhancerByGuice$$51619816.CGLIB$doFilter$4()
How can this problem be solved?
Here is the part of the code responsible for adding the image:
previewLink.add(createThumbnailSmall("imageThumbnail", documentModel));
in
createThumbnailSmall(final String id, final IModel<BaseDocument> documentModel) {
// thumbnailResource is an object that contains the path of the image
if (thumbnailResource != null) {
final WebResource resource = getWebResource(thumbnailResource);
final Image image = new Image(id, resource);
return image;
}
return new InvisibleContainer(id);
}
WebResource getWebResource(final DocumentResource documentResource) {
return new WebResource() {
private static final long serialVersionUID = 1L;
#Override
public IResourceStream getResourceStream() {
return new BaseStreamResource(documentResource);
}
};
}
where BaseStreamResource is the following:
public class BaseStreamResource extends AbstractResourceStream {
private InputStream _fileInputStream = null;
private DocumentResource _resource = null;
public BaseStreamResource(final DocumentResource documentResource) {
_resource = documentResource;
}
#Override
public InputStream getInputStream() throws ResourceStreamNotFoundException {
if (_fileInputStream == null) {
try {
if (_resource == null) {
throw new ResourceStreamNotFoundException("Resource was null");
}
_fileInputStream = _resource.getFileInputStream();
} catch (final ResourceNotAvailableException ex) {
throw new ResourceStreamNotFoundException(ex);
}
}
return _fileInputStream;
}
In HTML:
<a wicket:id="linkToPreview" href="#">
<img wicket:id="imageThumbnail" alt="Attachment"></img></a>

The code added hasn't really added any clues for me, but maybe I can help narrow it down a bit anyway.
The stacktrace includes a reference to com.webapp.document.pages.DocumentListPage, which is likely calling some of the code you've posted. The error indicates a bad url, so debugging into that class, adding debug prints, and looking at the values of any field containing a url might be worthwhile.
It might even help to modify the code in DocumentListPage (maybe temporarily for debugging) to catch org.apache.wicket.protocol.http.request.InvalidUrlException and adding debugging prints specifically when the exception is caught.
This isn't really an answer, but it's too big for a comment, and maybe it'll help you get closer to an answer.

The following solution solved the problem:
- extend WebResource class
- add extended class as a resource to application shared resources
Ex:
public class MyWebResource extends WebResource {
final ValueMap map = new ValueMap();
#Override
public IResourceStream getResourceStream() {
String fileName = getFileName();
File file = new File(basePath, fileName);
if (!file.exists()) {
LOG.error("File does not exist: " + file);
throw new IllegalStateException("File does not exist: " + file);
}
return new FileResourceStream(file);
}
public final void addResource() {
Application.get().getSharedResources().add(getClass().getName(), this);
}
protected String getFileName() {
return getParameters().getString("id");
}
public final String urlFor(final String fileName) {
final ResourceReference resourceReference = new ResourceReference(getClass().getName());
final String encodedValue = WicketURLEncoder.QUERY_INSTANCE.encode(fileName);
map.add("id", encodedValue);
final CharSequence result = RequestCycle.get().urlFor(resourceReference, map);
if (result == null) {
throw new IllegalStateException("The resource was not added! "
+ "In your Application class add the following line:"
+ "MyConcreteResource.INSTANCE.addResource()");
}
String absoluteUrl = RequestUtils.toAbsolutePath(result.toString());
return absoluteUrl;
}
}
In Application class, in init(), I have added MyWebResource to shared resources:
public void init() {
...
new MyWebResource().addResource();
...
}