Title of the question may give you the impression that it is duplicate question, but according to me it is not.
I am just a few months old in Java and a month old in MongoDB, SpringBoot and REST.
I have a Mongo Collection with 3 fields in a document, _id (default field), appName and appKey. I am using list to iterate through all the documents and find one document whose appName and appKey matches with the one that is passed. This collection right now has only 4 entries, and thus it is running smoothly. But I was reading a bit about collections and found that if there will be a higher number of documents in a collection then the result with list will be much slower than hashMap.
But as I have already said that I am quite new to Java, I am having a bit of trouble converting my code to hashMap, so I was hoping if someone can guide me through this.
I am also attaching my code for reference.
public List<Document> fetchData() {
// Collection that stores appName and appKey
MongoCollection<Document> collection = db.getCollection("info");
List<Document> nameAndKeyList = new ArrayList<Document>();
// Getting the list of appName and appKey from info DB
AggregateIterable<Document> output = collection
.aggregate(Arrays.asList(new BasicDBObject("$group", new BasicDBObject("_id",
new BasicDBObject("_id", "$id").append("appName", "$appName").append("appKey", "$appKey"))
)));
for (Document doc : output) {
nameAndKeyList.add((Document) doc.get("_id"));
}
return nameAndKeyList;
}// End of Method
And then I am calling it in another method of the same class:
List<Document> nameAndKeyList = new ArrayList<>();
//InfoController is the name of the class
InfoController obj1 = new InfoController();
nameAndKeyList = obj1.fetchData();
// Fetching and checking if the appName & appKey pair
// is present in the DB one by one.
// If appName & appKey mismatches, it increments the value
// of 'i' and check them with the other values in DB
for (int i = 0; i < nameAndKeyList.size(); i++) {
"followed by my code"
And if I am not wrong then there will be no need for the above loop also.
Thanks in advance.
You just need a simple find query to get the record you need directly from Mongo DB.
Document document = collection
.find(new Document("appName", someappname).append("appKey", someappkey)).first();
First of all a list is not much slower or faster than an HashMap. A Hasmap is commonly used to save key-pair values such as "ID", "Name" or something like that. In your case I see you are using ArrayList without a specified size for the list. better use a linked list when you do not know the size because an arraylist is holding a array behind and extending this by copying. If you want to generate a Hasmap out of the List or use a Hasmap you need to map an ID and the value to the records.
HashMap<String /*type of the identifier*/, String /*type of value*/> map = new HashMap<String,String>();
for (Document doc : output) {
map.put(doc.get("_id"), doc.get("_value"));
}
First, avoid premature optimization (lookup the expression if you don’t know what it is). Put a realistic number of thousands of items containing near-realistic data in your list. Try to retrieve an item that isn’t there. This will force your for loop to traverse the entire list. See how long it takes. Try a number of times to get an impression of whether you get impatient. If you don’t, you’re done.
If you find out that you need a speed-up, I agree that HashMap is one of the obvious solutions to try. One of the first things to consider with this is a key type for you HashMap. As I understand, what you need to search for is an item where appName and appKey are both right. The good solution is to write a simple class with these two fields and equals and hashCode methods (I’ll call it DocumentHashMapKey for now, think of a better name). For hashCode(), try Objects.hash(appName, appKey). If it doesn’t give satisfactory performance with the data you have, consider alternatives. Now you are ready to build your HashMap< DocumentHashMapKey, Document>.
If you’re lazy or just want a first impression of how a HashMap performs, you may also build your keys by concatenating appName + "$##" + appKey (where the string in the middle is something that is unlikely to be part of a name or key) and use HashMap<String, Document>.
Everything I said can be refined depending on your needs. This was just to get you started.
Thanks everyone for your help, without which I would not have got to a solution.
public HashMap<String, String> fetchData() {
// Collection that stores appName and apiKey
MongoCollection<Document> collection = db.getCollection("info");
HashMap<String, String> appKeys = new HashMap<String, String>();
// Getting the list of appName and appKey from info DB
AggregateIterable<Document> output = collection
.aggregate(Arrays.asList(new BasicDBObject("$group", new BasicDBObject("_id",
new BasicDBObject("_id", "$id").append("appName", "$appName").append("appKey", "$appKey"))
)));
String appName = null;
String appKey = null;
for (Document doc : output) {
Document temp = (Document) doc.get("_id");
appName = (String) temp.get("appName");
appKey = (String) temp.get("appKey");
appKeys.put(appName, appKey);
}
return appKeys;
Calling the above method into another method of the same class.
InfoController obj = new InfoController();
//Fetching the values of 'appName' & 'appKey' sent from 'info' DB
HashMap<String, String> appKeys = obj.fetchData();
storedAppkey = appKeys.get(appName);
//Handling the case of mismatch
if (storedAppkey == null || storedApikey.compareTo(appKey)!=0)
{//Then the response and further processing that I need to do.
Now what HashMap has done is that it has made my code more readable and the 'for' loop that I was using for iterating is gone, although it might not make much difference in the performance as of now.
Thanks once again to everyone for your help and support.
Related
I have a table Book with bookId and lastBorrowed as hash and range keys, respectively.
Let's say each time a book is borrowed, a new row is created.
(Yes, this is NOT sufficient and I can just add a column to keep track of the count and update lastBorrowed date. But let's just say I'm stuck with this design there's nothing I can do about it.)
Given a set of bookIds (or hashKeys), I would like to be able to query the last time each book is borrowed.
I attempted to use QueryRequest, but kept getting com.amazonaws.AmazonServiceException: Attempted conditional constraint is not an indexable operation
final Map<String, Condition> keyConditions =
Collections.singletonMap(hashKeyFieldName, new Condition()
.withComparisonOperator(ComparisonOperator.IN)
.withAttributeValueList(hashKeys.stream().map(hashKey -> new AttributeValue(hashKey)).collect(Collectors.toList())));
I also tried using BatchGetItemRequest, but it didn't work, either:
final KeysAndAttributes keysAndAttributes = new KeysAndAttributes() .withConsistentRead(areReadsConsistent);
hashKeys.forEach(hashKey -> { keysAndAttributes.addExpressionAttributeNamesEntry(hashKeyFieldName, hashKey); });
final Map<String, KeysAndAttributes> requestedItemsByTableName = newHashMap();
requestedItemsByTableName.put(tableName, keysAndAttributes);
final BatchGetItemRequest request = new BatchGetItemRequest().withRequestItems(requestedItemsByTableName);
Any suggestion would be much appreciated!
Or if someone can tell me this is currently not supported at all, then I guess I'll just move on!
You can do this, in fact its very easy. All you have to do is execute a Query for your bookId and then take the first result.
By the way, your table design sounds absolutely fine, the only problem is the attribute should probably be called borrowed rather than last borrowed.
You can have multiple results for a single bookId, but because lastBorrowed is your range key, the results will come back ordered by that attribute.
You seem to be using Legacy Functions, are you editing old code?
If not, execute your Query something like this:
//Setting up your DynamoDB connection
AmazonDynamoDB client = AmazonDynamoDBClientBuilder.standard()
.withRegion(Regions.US_WEST_2).build();
DynamoDB dynamoDB = new DynamoDB(client);
Table table = dynamoDB.getTable("YOURTABLE");
//Define the Query
QuerySpec spec = new QuerySpec()
.withKeyConditionExpression("bookId = :book_id)
.withValueMap(new ValueMap()
.withString(":book_id", "12345")
.withScanIndexForward(true);
//Execute the query
ItemCollection<QueryOutcome> items = table.query(spec);
//Print out your results - take the first item
Iterator<Item> iterator = items.iterator();
while (iterator.hasNext()) {
System.out.println(iterator.next().toJSONPretty());
}
I have a HashMap with key of type Double and my custom object as value.
It looks like this:
private static Map<Double, Incident> incidentHash = new HashMap<>();
The Incident object has following attributes: String date, String address, String incidentType.
Now I have a String date that I get from the user as input and I want to check if there exists any incident in the HashMap with that user inputted date. There can be many Incidents in the HashMap with the given date but as long as there's at least one Incident with the given date, I can do *
something.
I can just iterate over all the values in the HashMap and check if a given date exists but I was wondering if there is any better and more efficient way possible without modifying the data structure.
Given your HashMap, NO, there is not another way of doing so without iterating that HashMap.
As for changing the structure, you could do as Map<String, List<Incident>> that way you would have a date as key and a List of incidents for that date, given your requirement: There can be many Incidents in the HashMap with the given date.
So this would be a O(1)
//considering that the key is added when you have at least one incident
if (yourHash.get("yourDateStringWhatEverTheFormatIs") != null)
You can use streams API (from Java8) as shown in the below code with inline comments:
String userInput="10-APR-2017";
Optional<Map.Entry<Double, Incident>> matchedEntry =
incidentHash.entrySet().stream().
//filter with the condition to match
filter(element -> element.getValue().getDate().equals(userInput)).findAny();
//if the entry is found, do your logic
matchedEntry.ifPresent(value -> {
//do something here
});
If you are looking for something prior to JDK1.8, you can refer the below code:
String userInput="10-APR-2017";
Set<Map.Entry<Double, Incident>> entries = incidentHash.entrySet();
Map.Entry<Double, Incident> matchedEntry = null;
for(Iterator<Map.Entry<Double, Incident>> iterator = entries.iterator();
iterator.hasNext();) {
Map.Entry<Double, Incident> temp = iterator.next();
if(temp.getValue().getDate().equals(userInput)) {
matchedEntry = temp;
break;
}
}
You can use a TreeMap with your custom Comparator. In your Comparator compare the values of dates.
You would have to iterate through the map until you find a data that matches. Since you only need to know if any occurrences exist you can simply exit the loop when you find a match instead of iterating the rest of the map.
You can only keep a second Hash/TreeMap that matches the attribute to the object, so you can also check this attibute qickly. But you have to curate one such map for each attribute you want to access quickly. This makes it a bit more complex and use more memory, but can be much much faster.
If this is not an option the stream API referenced in other answers is a nice and tidy way to iterate over all objects to search for an attribute.
private static Map<Double, Incident> incidentHash = new HashMap<>();
private static Map<String, List<Incident>> incidentsPerDayMap = new HashMap<>();
Given that you don't want to iterate the Map and currently it's the only way to get the required value, I would recommend recomment another Map that contains Date as key and List<Incident> as value. It can be a TreeMap, e.g.:
Map<Date, List<Incident>> incidents = new TreeMap<>();
You can put the entry in this Map whenever an entry is added into the original Map, e.g.:
Incident incident = ;// incident object
Date date; //Date
incidents.computeIfAbsent(date, t -> new ArrayList<>()).add(incident);
Once the user inputs the Date, you can get all the incidents belonging to this date just by incidents.get(). Although that will give you a list and you still need to iterate over it, it will contain a lot less elements and get method in TreeMap will guarantee you log n complexity as it is sorted. So, your search operation will be much more efficient.
I'd like to imagine there's existing API functionality for this. Suppose there was Java code that looks something like this:
JavaRDD<Integer> queryKeys = ...; //values not particularly important
List<Document> allMatches = db.getCollection("someDB").find(queryKeys); //doesn't work, I'm aware
JavaPairRDD<Integer, Iterator<ObjectContainingKey>> dbQueryResults = ...;
Goal of this: After a bunch of data transformations, I end up with an RDD of integer keys that I'd like to make a single db query with (rather than a bunch of queries) based on this collection of keys.
From there, I'd like to turn the query results into a pair RDD of the key and all of its results in an iterator (making it easy to hit the ground going again for the next steps I'm intending to take). And to clarify, I mean a pair of the key and its results as an iterator.
I know there's functionality in MongoDB capable of coordinating with Spark, but I haven't found anything that'll work with this yet (it seems to lean towards writing to a database rather than querying it).
I managed to figure this out in an efficient enough manner.
JavaRDD<Integer> queryKeys = ...;
JavaRDD<BasicDBObject> queries = queryKeys.map(value -> new BasicDBObject("keyName", value));
BasicDBObject orQuery = SomeHelperClass.buildOrQuery(queries.collect());
List<Document> queryResults = db.getCollection("docs").find(orQuery).into(new ArrayList<>());
JavaRDD<Document> parallelResults = sparkContext.parallelize(queryResults);
JavaRDD<ObjectContainingKey> results = parallelResults.map(doc -> SomeHelperClass.fromJSONtoObj(doc));
JavaPairRDD<Integer, Iterable<ObjectContainingKey>> keyResults = results.groupBy(obj -> obj.getKey());
And the method buildOrQuery here:
public static BasicDBObject buildOrQuery(List<BasicDBObject> queries) {
BasicDBList or = new BasicDBList();
for(BasicDBObject query : queries) {
or.add(query);
}
return new BasicDBObject("$or", or);
}
Note that there's a fromJSONtoObj method that will convert your object back from JSON into all of the required field variables. Also note that obj.getKey() is simply a getter method associated to whatever "key" it is.
I have just started using Mongo Db . Below is my data structure .
It has an array of skillID's , each of which have an array of activeCampaigns and each activeCampaign has an array of callsByTimeZone.
What I am looking for in SQL terms is :
Select activeCampaigns.callsByTimeZone.label,
activeCampaigns.callsByTimeZone.loaded
from X
where skillID=50296 and activeCampaigns.campaign_id= 11371940
and activeCampaigns.callsByTimeZone='PT'
The output what I am expecting is to get
{"label":"PT", "loaded":1 }
The Command I used is
db.cd.find({ "skillID" : 50296 , "activeCampaigns.campaignId" : 11371940,
"activeCampaigns.callsByTimeZone.label" :"PT" },
{ "activeCampaigns.callsByTimeZone.label" : 1 ,
"activeCampaigns.callsByTimeZone.loaded" : 1 ,"_id" : 0})
The output what I am getting is everything under activeCampaigns.callsByTimeZone while I am expecting just for PT
DataStructure :
{
"skillID":50296,
"clientID":7419,
"voiceID":1,
"otherResults":7,
"activeCampaigns":
[{
"campaignId":11371940,
"campaignFileName":"Aaron.name.121.csv",
"loaded":259,
"callsByTimeZone":
[{
"label":"CT",
"loaded":6
},
{
"label":"ET",
"loaded":241
},
{
"label":"PT",
"loaded":1
}]
}]
}
I tried the same in Java.
QueryBuilder query = QueryBuilder.start().and("skillID").is(50296)
.and("activeCampaigns.campaignId").is(11371940)
.and("activeCampaigns.callsByTimeZone.label").is("PT");
BasicDBObject fields = new BasicDBObject("activeCampaigns.callsByTimeZone.label",1)
.append("activeCampaigns.callsByTimeZone.loaded",1).append("_id", 0);
DBCursor cursor = coll.find(query.get(), fields);
String campaignJson = null;
while(cursor.hasNext()) {
DBObject campaignDBO = cursor.next();
campaignJson = campaignDBO.toString();
System.out.println(campaignJson);
}
the value obtained is everything under callsByTimeZone array. I am currently parsing the JSON obtained and getting only PT values . Is there a way to just query the PT fields inside activeCampaigns.callsByTimeZone .
Thanks in advance .Sorry if this question has already been raised in the forum, I have searched a lot and failed to find a proper solution.
Thanks in advance.
There are several ways of doing it, but you should not be using String manipulation (i.e. indexOf), the performance could be horrible.
The results in the cursor are nested Maps, representing the document in the database - a Map is a good Java-representation of key-value pairs. So you can navigate to the place you need in the document, instead of having to parse it as a String. I've tested the following and it works on your test data, but you might need to tweak it if your data is not all exactly like the example:
while (cursor.hasNext()) {
DBObject campaignDBO = cursor.next();
List callsByTimezone = (List) ((DBObject) ((List) campaignDBO.get("activeCampaigns")).get(0)).get("callsByTimeZone");
DBObject valuesThatIWant;
for (Object o : callsByTimezone) {
DBObject call = (DBObject) o;
if (call.get("label").equals("PT")) {
valuesThatIWant = call;
}
}
}
Depending upon your data, you might want to add protection against null values as well.
The thing you were looking for ({"label":"PT", "loaded":1 }) is in the variable valueThatIWant. Note that this, too, is a DBObject, i.e. a Map, so if you want to see what's inside it you need to use get:
valuesThatIWant.get("label"); // will return "PT"
valuesThatIWant.get("loaded"); // will return 1
Because DBObject is effectively a Map of String to Object (i.e. Map<String, Object>) you need to cast the values that come out of it (hence the ugliness in the first bit of code in my answer) - with numbers, it will depend on how the data was loaded into the database, it might come out as an int or as a double:
String theValueOfLabel = (String) valuesThatIWant.get("label"); // will return "PT"
double theValueOfLoaded = (Double) valuesThatIWant.get("loaded"); // will return 1.0
I'd also like to point out the following from my answer:
((List) campaignDBO.get("activeCampaigns")).get(0)
This assumes that "activeCampaigns" is a) a list and in this case b) only has one entry (I'm doing get(0)).
You will also have noticed that the fields values you've set are almost entirely being ignored, and the result is most of the document, not just the fields you asked for. I'm pretty sure you can only define the top-level fields you want the query to return, so your code:
BasicDBObject fields = new BasicDBObject("activeCampaigns.callsByTimeZone.label",1)
.append("activeCampaigns.callsByTimeZone.loaded",1)
.append("_id", 0);
is actually exactly the same as:
BasicDBObject fields = new BasicDBObject("activeCampaigns", 1).append("_id", 0);
I think some of the points that will help you to work with Java & MongoDB are:
When you query the database, it will return you the whole document of
the thing that matches your query, i.e. everything from "skillID"
downwards. If you want to select the fields to return, I think those will only be top-level fields. See the documentation for more detail.
To navigate the results, you need to know that a DBObjects are returned, and that these are effectively a Map<String,
Object> in Java - you can use get to navigate to the correct node,
but you will need to cast the values into the correct shape.
Replacing while loop from your Java code with below seems to give "PT" as output.
`while(cursor.hasNext()) {
DBObject campaignDBO = cursor.next();
campaignJson = campaignDBO.get("activeCampaigns").toString();
int labelInt = campaignJson.indexOf("PT", -1);
String label = campaignJson.substring(labelInt, labelInt+2);
System.out.println(label);
}`
mongodb query is db.test.find({"col1":{"$ne":""}}).count(), I have tried many sources to find the solution, the "col1" must be populated from list array, please help me
I have pasted a part of my code
`
List<String> likey = new ArrayList<String>();
for (DBObject o : out.results())
{
likey.add(o.get("_id").toString());
}
Iterator<String>itkey = likey.iterator();
DBCursor cursor ;
//cursor = table.find();
HashMap<String, String> hashmap = new HashMap<String, String>();
while (itkey.hasNext())
{
System.out.println((String)itkey.next());
String keys = itkey.next().toString();
//System.out.println("keys --> "+keys);
String nullvalue = "";
Boolean listone = table.distinct(keys).contains(nullvalue);
hashmap.put(keys, listone.toString());
//System.out.println("distinct --> "+keys+" "+listone);
//System.out.println("proper str --- >"+ '"'+keys+'"');
}
Iterator<String> keyIterator = hashmap.keySet().iterator();
Iterator<String> valueIterator = hashmap.values().iterator();
while (keyIterator.hasNext()) {
//System.out.println("key: " + keyIterator.next());
while (valueIterator.hasNext()) {
//System.out.println("value: " + valueIterator.next());
//System.out.println("Key: " + keyIterator.next() +""+"value: "+valueIterator.next());
String hashkey = valueIterator.next();
}
}
`
When you post code, it helps if you indent it, so it is more readable. As I mentioned to you on another forum, you need to go back and review the Java collection classes, since you have multiple usage errors in the above code.
Here are a few things you need to do to clean up your code:
1) You don't need to use the itkey iterator. Instead, use:
for (String key : likey)
and get rid of all the itkey.next calls. Your current code only processes every second element of the List. The other ones are printed out.
2) Your HashMap will map a key to a Boolean. Is that what you want? You said you want to count the number of non-zero values for the key. So, the line:
Boolean listone = table.distinct(keys).contains(nullvalue);
is almost certainly in error.
3) When you iterate over the HashMap, you don't need the valueIterator. Instead, get the key (either from the keyIterator, or a variable you define using the simpler iterator syntax above), then use the key to get the matching value using hashmap.get(key).
This will not make your code work, but it will clean it up somewhat - at the moment it is difficult to understand what you are intending it to do.