I need to insert a list of objects with a predefined _id (Long) into a collection.
insert(object) method for a single object from AdvancedDatastore works great. The trouble begins when i try to use the insert() method which accepts an Iterable. Here is a sample piece of code:
try {
advancedDatastore.insert("collection_name", feeds, WriteConcern.ERRORS_IGNORED);
} catch (Exception e) {
e.printStackTrace();
}
I guess that this code is supposed to ignore errors (an object with a duplicate id already exists in the collection) and just continue with the next item, but it does not. And no exception is raised.
Thanks!
Update:
This code inserts all the elements, but "1" is not printed out.
try {
System.err.println(0);
advancedDatastore.insert("collection_name", feeds, WriteConcern.ERRORS_IGNORED.continueOnErrorForInsert(true));
System.err.println(1);
} catch (Exception e) {
e.printStackTrace();
}
Update2:
Sorry, the code completes properly and "1" is printed out, but it takes tremendously more time than single inserts. In my case 35_000 inserts 1 by one - 3 seconds, in batch - 100+ seconds
Update3:
So far the best way to deal with the issue for me is to use native java driver for mongodb.
1st I convert my object list to DBObject list:
final List<DBObject> dbObjects = new ArrayList<DBObject>();
for (MyObject object: objectList) {
dbObjects.add(morphia.toDBObject(object));
}
Then I insert through mongo DB instance:
db.getCollection("collection_name").insert(dbObjects, WriteConcern.UNACKNOWLEDGED.continueOnErrorForInsert(true));
Performace for inserting 150_000 objects:
Native DB insert: 2-3 seconds
via Morphia's insert(object): 15+ seconds
via Morphia's insert(Iterable): 400+ seconds
A better way would be appreciated.
It works to me in this way
final List<DBObject> dbObjects = new ArrayList<DBObject>();
try {
TypedQuery<RegistroCivil> consulta = em.createQuery("select p from RegistroCivil p", RegistroCivil.class);
List<RegistroCivil> lista = consulta.getResultList();
for (RegistroCivil object : lista) {
dbObjects.add(morphia.toDBObject(object));
}
long start = System.currentTimeMillis();
ds.getCollection(RegistroCivil.class).insert(dbObjects);
//ds.save(lista);
long end = System.currentTimeMillis();
tmongo = end - start;
Related
Im trying to use the findDistinct function from mongoTemplate but i always retrieve an empty result list.
Can you help me out to spot the problem ? Or maybe you have a simpliest way to do it
NB:
I do have data in my collection
(on a basic find, i fetch more than 300 results in the list but all of this result are the same excepting on one key, i want all the distinct object from their NAME value for instance )
I tryied this :
List<DiffusionListImpl> list = new ArrayList<>();
try{
query = new Query(Criteria.where("CUSTOMERNUMBER").is(1));
list = mongoTemplate.findDistinct(query, KeyWhereIWantTheDistinct, collectionName,
KlassResultModel.class);
} catch (MongoException e) {
logger.error("MongoException: " + e);
} catch (Exception e) {
logger.error("Error: " + e);
}
return list;
My bad, i misread the documentation.
But i find it akward to have this kind of comportement of this function.
I have to make a call to the DB to fetch a list of distinct value and then make another Call of the same DB to retrieve the object.
Is there any way to do it in one call? (Performance issue)
It can be done in one DB call, use below code.
final List<DiffusionListImpl> result =
IteratorUtils.toList(this.mongoTemplate.getCollection("collectionName")
.distinct("fieldName", query.getQueryObject(), DiffusionListImpl.class)
.iterator());
for IteratorUtils you can use apache
import org.apache.commons.collections4.IteratorUtils;
If we wanna get an object ID we should do this:
String objectId = gameScore.getObjectId();
but what if we wanna get an object ID by a query? Like this:
ParseQuery<ParseObject> query = ParseQuery.getQuery("mytable");
query.whereEqualTo("Title", "Adrians Book");
List<ParseObject> results = null;
try {
results = query.find();
if(!results.isEmpty()) {
String objectId = results.getObjectId();
}
} catch (com.parse4cn1.ParseException e) {
Dialog.show("Err", "Something went wrong.", "OK", null);
}
Sounds interesting don't you think? I wish it could be possible. As you can see in this example the query will get a value from a specific object in the table which could track for the object ID then returning it as well. ParseQuery class should be implemented with getObjectId(). Because by this way applications always could have access to object IDs from the query even after applications get restarted so in the first example the gameScore which is actually an instance of ParseObject would lost reference to the Database after restarting. Getting object IDs by the query it would be able to program applications to get object IDs automatically without the need of doing it manually nor depending on instances of ParseObject.
#Shai Almog: Thank you very much for taking your time to look at the ParseQuery documentation.
I accidentally figured out how to get this done!
ParseQuery<ParseObject> query = ParseQuery.getQuery("mytable");
query.whereEqualTo("Title", "Adrians Book");
List<ParseObject> results = null;
try {
results = query.find();
if(!results.isEmpty()) {
String objectId = results.get(0).getObjectId();
System.out.println(objectId);
}
} catch (com.parse4cn1.ParseException e) {
Dialog.show("Err", "Something went wrong.", "OK", null);
}
Yep, after adding the method .get(index) it allows you to access the method .getObjectId() since results is a list of a ParseObject, then the respective objectId of your query result will be printed in the console! I'm pretty glad it's working because I won't need to serialize each object for now which would be a pain.
Also if you wanna set an instance of ParseObject with an existing objectId in case you need to update something in your Database, you can use this example:
ParseObject po = ParseObject.create("mytable");
po.setObjectId(//YOUR DESIRED OBJECTID HERE, AS LONG AS IT EXISTS IN THE DATABASE);
As far as I know you need to get the whole object then query it's ID. I don't see a query id method here https://github.com/sidiabale/parse4cn1/blob/41fe491699e604fc6de46267479f47bc422d8978/src/com/parse4cn1/ParseQuery.java
I have some method in my DAO class:
public void insertAVAYAcmCDRs(List<AvayaCmCdr> cdrList) {
AvayaCmCdr aCdrList1 = null;
try {
em.getTransaction().begin();
for (AvayaCmCdr aCdrList : cdrList) {
aCdrList1 = aCdrList;
em.persist(aCdrList);
}
em.getTransaction().commit();
em.clear();
} catch (Exception e) {
logger.log(Level.INFO, "Exception in task time={0}. Exception message = {1}.", new Object[]{aCdrList1.getDate(), e.getMessage()});
}
}
I tried save all array entities to DB. But in DB i have uniqe index - it does not allow to insert duplicate rows. It work normaly on DB side but i have some error in java.
a different object with the same identifier value was already associated with the session:
I get this error on 2 step of cycle. I print this object and found dublicate in DB.
I want ignore this error and continue insert data or somehow handle the error.
if this row already in the database i want ignore and skip it and continue insert
Why are you assigning this aCdrList1 = aCdrList ? Is there any specific reason?
you can save aCdrList object. Use below one
em.saveOrUpdate(aCdrList);
or
em.merge(aCdrList);
i set an 'notimeout' option to a dbcursor in java:
BasicDBObject nearbyQueries = new BasicDBObject("$gt", 0)
.append("$lte", 2);
DBCursor trueClassInstances = locationsCollection.find(new BasicDBObject("distanceFromHotel", nearbyQueries)).addOption(Bytes.QUERYOPTION_NOTIMEOUT).limit(100000);
double counter = 0;
int currentPresent = 0;
for (DBObject instance : trueClassInstances) {
...
}
even with this option i set, this exception is thrown:
Exception in thread "main" com.mongodb.MongoCursorNotFoundException: Query failed with error code -5 and error message 'Cursor 1876954464377 not found on server XXXXXX:27017' on server XXXXXXXX:27017
at com.mongodb.connection.GetMoreProtocol.receiveMessage(GetMoreProtocol.java:115)
at com.mongodb.connection.GetMoreProtocol.execute(GetMoreProtocol.java:68)
at com.mongodb.connection.GetMoreProtocol.execute(GetMoreProtocol.java:37)
at com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:155)
at com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:219)
at com.mongodb.connection.DefaultServerConnection.getMore(DefaultServerConnection.java:194)
at com.mongodb.operation.QueryBatchCursor.getMore(QueryBatchCursor.java:197)
at com.mongodb.operation.QueryBatchCursor.hasNext(QueryBatchCursor.java:93)
at com.mongodb.MongoBatchCursorAdapter.hasNext(MongoBatchCursorAdapter.java:46)
at com.mongodb.DBCursor.hasNext(DBCursor.java:152)
at locationExtraction.DistanceClassification.FeatureAnalyzer.main(FeatureAnalyzer.java:27)
FeatureAnalyzer.java:27 is the for loop line.
this problem appear in other project with similar setting...
what am i doing wrong? maybe my choice of 'for' loop instead of this kind of iteration can cause this strange behavior?
while(cursor.hasNext())
{
DBObject next = cursor.next();
}
Thanks
Looks like you are not able to process each batch within time limit. Try reducing batch size so that each batch could be consumed before time runs out. This should help.
cursor.addOption(com.mongodb.Bytes.QUERYOPTION_NOTIMEOUT).batchSize(100)
so the problem is solved.
this is very strange but there is a problem with using 'for' loop for iterating on cursor. so dont do it like i did it, use 'while' loop:
while(cursor.hasNext())
{
DBObject next = cursor.next();
}
before use cursor.hasNext() and cursor.next() to do business logical, just before you get the mongo cursor, invoke FindIterable object's noCursorTimeout(true) method. for example:
FindIterable<Document> findIterable = sentenceColl.find(condition);
// set no timeout
findIterable.noCursorTimeout(true);
MongoCursor<Document> mongoCursor = findIterable.iterator();
while (mongoCursor.hasNext()) {
mongoCursor.next();
}
try this:
Iterator<BasicDBObject> it = null;
it = coll.find("{"field": {$in:#}}", fieldList).with(
new QueryModifier() {
public void modify(DBCursor cursor) {
cursor.setOptions(Bytes.QUERYOPTION_NOTIMEOUT);
}
}
).as(BasicDBObject.class);
{
"question":"what is your color?",
"choices":[{"option":"yello"},{"option":"blue"}],
"creation-date":"2014-04-13",
"expiry date":"2014-04-14"
}
In order to retrive list of polls
public List<Poll> getPolls()
{
try {
mongoClient=new MongoClient("NavDeep",27017);
db=mongoClient.getDB("sms-voting");
collection=db.getCollection("pollsCollection");
} catch(Exception e){
e.printStackTrace();
}
List<Poll> polls = new ArrayList<Poll>();
DBCursor cursor=collection.find();
while(cursor.hasNext())
{
DBObject object = cursor.next();
what should i write in order to retrieve List<Poll>????
}
}
mongoClient.close();
return polls;
}
but i am getting null pointer exception near BasicDBList pollList=(BasicDBList)object.get("pollsCollection");
Can any body please Help me out.What should i actually write inside get()??
Thnks,
deepthi
I think the problem could be that you're doing cursor.next() twice in the lines above, pulling two records at a time.
What if you try:
polls.add(object)
instead of
polls.add(cursor.next())
Rather than iterating the cursor you appear to want the .toArray() method:
DBCursor cursor = collection.find();
List<DBObject> list = cursor.toArray();
Generally cursors are a good idea, and you probably should be building any array type results from within that loop. But this is a way to change the cursor.