We Keep Coding

Google Appengine Datastore Timeout Exception

We are fetching the list of namespaces from datastore which counts upto 30k.
The cron to fetch namespaces runs daily. But one day it works fine and other day it throws datastore timeout exception.
com.google.appengine.api.datastore.DatastoreTimeoutException: The
datastore operation timed out, or the data was temporarily
unavailable.
Related Code :
DatastoreService ds = DatastoreServiceFactory.getDatastoreService();
FetchOptions options = FetchOptions.Builder.withChunkSize(150);
Query q = new Query(Entities.NAMESPACE_METADATA_KIND);
for (Entity e : ds.prepare(q).asIterable(options)){
// A nonzero numeric id denotes the default namespace;
// see Namespace Queries, below
if (e.getKey().getId() != 0){
continue;
}else{
namespaces.add(e.getKey().getName());
}
}
What could be the issue?

According to official documentation:
DatastoreTimeoutException is thrown when a datastore operation times
out. This can happen when you attempt to put, get, or delete too many
entities or an entity with too many properties, or if the datastore is
overloaded or having trouble.
This means that datastore having troubles with your request. Try to handle that error like:
import com.google.appengine.api.datastore.DatastoreTimeoutException;
try {
// Code that could result in a timeout
} catch (DatastoreTimeoutException e) {
// Display a timeout-specific error page
}

How ignore hibernate error and continue insert data?

I have some method in my DAO class:
public void insertAVAYAcmCDRs(List<AvayaCmCdr> cdrList) {
AvayaCmCdr aCdrList1 = null;
try {
em.getTransaction().begin();
for (AvayaCmCdr aCdrList : cdrList) {
aCdrList1 = aCdrList;
em.persist(aCdrList);
}
em.getTransaction().commit();
em.clear();
} catch (Exception e) {
logger.log(Level.INFO, "Exception in task time={0}. Exception message = {1}.", new Object[]{aCdrList1.getDate(), e.getMessage()});
}
}
I tried save all array entities to DB. But in DB i have uniqe index - it does not allow to insert duplicate rows. It work normaly on DB side but i have some error in java.
a different object with the same identifier value was already associated with the session:
I get this error on 2 step of cycle. I print this object and found dublicate in DB.
I want ignore this error and continue insert data or somehow handle the error.
if this row already in the database i want ignore and skip it and continue insert

Why are you assigning this aCdrList1 = aCdrList ? Is there any specific reason?
you can save aCdrList object. Use below one
em.saveOrUpdate(aCdrList);
or
em.merge(aCdrList);

How do i do a Save operation on a MongoDB collection using the Java driver?

I just switch over from Python, and need to continue my work with a MongoDB database. One particular task is to save an incoming document (in this case, a tweet) into a collection for archiving. A tweet could comes in multiple times, so I prefer to use save() over insert() since the former do not raise an error if the document already exists in the collection. But it seems the Java driver for MongoDB does not support the save operation. Am I missing something?
EDIT: for reference, i'm using this library 'org.mongodb:mongodb-driver:3.0.2'
Example code:
MongoCollection<Document> tweets = db.getCollection("tweets");
...
Document tweet = (Document) currentDocument.get("tweet");
tweets.insertOne(tweet);
The last line raise this error when the tweet already exists:
Exception in thread "main" com.mongodb.MongoWriteException: insertDocument :: caused by :: 11000 E11000 duplicate key error index: db.tweets.$_id_ dup key: { : ObjectId('55a403b87f030345e84747eb') }

Using the 3.x MongoDB Java driver you can use MongoCollection#replaceOne(Document, Document, UpdateOptions) like this:
MongoClient mongoClient = ...
MongoDatabase database = mongoClient.getDatabase("myDB");
MongoCollection<Document> tweets = db.getCollection("tweets");
...
Document tweet = (Document) currentDocument.get("tweet");
tweets.replaceOne(tweet, tweet, new UpdateOptions().upsert(true));
This will avoid the duplicate key error. However, this is not exactly the same as using DBCollection#save(DBObject), since it uses the whole Document as filter instead of just the _id field. To mirror the old save method, you would have to write something like this:
public static void save(MongoCollection<Document> collection, Document document) {
Object id = document.get("_id");
if (id == null) {
collection.insertOne(document);
} else {
collection.replaceOne(eq("_id", id), document, new UpdateOptions().upsert(true));
}
}

creating unique index in mongoDB

I am using a java program for mongo db insertion trying to create a unique index for a field. product_src is a field in my collection and I want to set it as unique index for avoiding the duplicate insertion. I am trying the following code but showing syntax error what is the problem with this.
DB db;
try {
sample = new MongoClient("myIP",PORT);
db = sample.getDB("client_mahout");
t = db.getCollection("data_flipkart_in_avoid_duplicate_checking");
System.out.println("enter the system ip");
db.t.ensureIndex({"product_src":1});
} catch (Exception e) {}
t is the collection. there is problem with line db.t.ensureIndex({"product_src":1});
Please give me a sample code how to create unique index in mongo DB

For future reference, the way to handle this in the Java Mongo driver v3.0+ is by:
public void createUniqueIndex() {
Document index = new Document("field", 1);
MongoCollection<Document> collection = client.getDatabase("db").getCollection("Collection");
collection.createIndex(index, new IndexOptions().unique(true));
}

You need to pass a DBObject to the ensureIndex() method.
db.t.ensureIndex(new BasicDBObject("product_src",1))
But, the ensureIndex method has been deprecated since version 2.12, you need to use createIndex() instead.
db.t.createIndex(new BasicDBObject("product_src",1));

Java Neo4j Deletion and Internal 500 Errors

When I delete my neo4j database after my tests like this
public static final DatabaseOperation clearDatabaseOperation = new DatabaseOperation() {
#Override public void performOperation(GraphDatabaseService db) {
//This is deprecated on the GraphDatabaseService interface,
// but the alternative is not supported by implementation (RestGraphDatabase)
for (Node node : db.getAllNodes()) {
for (Relationship relationship : node.getRelationships()) {
relationship.delete();
}
boolean notTheRootNode = node.getId() != 0;
if (notTheRootNode) {
node.delete();
}
}
When querying the database through an ajax search (i.e searching on an empty database it returns an internal 500 error)
localhost:9000/search-results?keywords=t 500 Internal Server Error
197ms
However if I delete the database manually like this
start r=relationship(*) delete r;
start n=node(*) delete n;
No exception is thrown
Its most likely an issue with my code at a lower level in the call and return.
Just wandering why the error only works on one of the scenarios above and not both

Use cypher,
you should probably state more obviously that you use the rest-graph-database.
Are you querying after the deletion or during it?
Please check your logs in data/graph.db/messages.log and data/log/console.log to find the error cause.
Perhaps you can also look at the response body of the http-500 request

As per your error I guess your data is getting corrupted after deletion.
I have used same code like yours and deleted the nodes, except I put the Iterator in transaction and shut down the database after opetation.
e.g.
Transaction _tx = _db.beginTx();
try {
for ( your conditions){
Your code
}
_tx.success();
} catch (Exception e) {
_logger.error(e.getMessage());
}finally{
_tx.finish();
_db.shutdown();
graphDbFactory.cleanUp();
}
Hope it will work for you.