why is my appengine IdGeneratorStrategy generating huge numbers? - java

I just moved my code from one machine to another, released it and suddenly it created an entry with a key of "576728208506880" so I re-released the exact same code from the original machine and created another field and this time the key created was "21134006"
Can anyone shed any light on why this might be?!
Thanks,
J

It's perfectly normal. App Engine generates numeric IDs between 0 and 2^53, and scatters them out throughout the entire range:
http://googlecloudplatform.blogspot.ca/2013/05/update-on-datastore-auto-ids.html
You can hack around it a bit by using the legacy auto id policy in your settings.

Appengine datastore IDs are not generated sequentially.
(Imagine that you had a burst of 1,000 new entities created in the same second - the short answer is that AppEngine needs a strategy to generate IDs that won't collide).
See this answer for more details and a potential solution.
See "Assigning Identifiers" of the AppEngine docs for more information.

Related

Google Places API - saving place_id and violation of terms and conditions

I want to build an app which shows places around user using Google Places based on user interests. As mentioned here:
Place IDs are exempt from the caching restrictions stated in Section
10.5.d of the Google Maps APIs Terms of Service. You can therefore store place ID values indefinitely.
So, can I save place_id in cloud database and perform any analytics operation over it? For example; if I gather place_ids added in each user's favorite places table and from analytics; I can know which place_id are the most ones added to favorites? or can I show something like 'Trending Places' in app from gathered place_ids in responses?
Will it violate the terms and conditions? I read the whole page of terms but couldn't find the answer.
can anyone help me out? Thanks.
Yes you can 100% store the place_id indefinitely and reuse it.
See Referencing a Place with a Place ID.
Please note one thing that
A single place ID refers to only one place, but a place can have
multiple place IDs
These terms and conditions are kind of self explanatory. Except your requirement which will be clarified after the below link is read carefully. As per your requirement , inorder to prevent calling services next time with same query which user had done with an intention of saving network calls is acceptable.
No caching or storage: You will not pre-fetch, cache, index, or store any Content to be used outside the Service, except that you may store limited amounts of Content solely for the purpose of improving the performance of your Maps API Implementation due to network latency (and not for the purpose of preventing Google from accurately tracking usage), and only if such storage
1) is temporary (and in no event more than 30 calendar days)
2) is secure 3)
does not manipulate or aggregate any part of the Content or Service 4) and
does not modify attribution in any way. Go through this Section 10.5 Intellectual Property Restrictions. Subsection (B)
You'll need to contact Google to get a 100% answer.
That being said, from my experience it looks like the clause you included is intended exactly for the kind of thing you want to do.
Again, I want to reiterate that contacting Google directly is something you should do if you still have concerns.
You can store place ID values indefinitely.
Just What part of
You can therefore store place ID Values indefinitely.
Don't you understand?
Indefinitely requires a server.

Android: Random Generation in a Networked Situation (Multiplayer)

I wrote a class that, given a seed and difficulty, will return a playing field to my game. The generation is consistent (no matter what, the same seed & difficulty level will always result in the same play field). As far as I know all android devices use Java 1.6 so here goes my question(s):
Is it safe to send only the seed and difficulty to other devices in a multiplayer environment?
Do I need to worry about when Google updates Java version level form 1.6? or will they likely update all android devices to that version level (I am assuming the Random class will have been changed)? And if not what would be a good way to detect if Random class is different?
Rephrased, what precautionary measures should be in place to ensure that the class java.util.Random, which my field generation class uses heavily, will result in the same play field for every device? Or, alternatively, would it be more wise to consider sending all play field data to the non-hosting device(s)?
I could probably accomplish the latter with a reliable message with size of:
byte[ROWS * COLUMNS]
In advance, I appreciate any guidance/suggestions in this matter. This is a difficult issue to search for so some links for future views may be appropriate.
There are a few options here, but I guess I was hoping for some magic JVM property defining the java.util.Random class revision version.
First option is to check the java version and compare it against the other device's version. If they are the same it is safe (as far as I know) to assume that the Random class is the same and thus the seed and difficulty can be sent. If, however, they are different you either send all the data or check the documentation/version release notes yourself to see when the Random class was changed and then determine if all the data should be sent based on previously acquired java version identifier.
The second option is to simply always send all the data. Which is what I will personally be doing.
If you're not as lucky as I and your data exceeds the value of Multiplayer.MAX_RELIABLE_MESSAGE_LEN (in bytes) you may have to break the data into multiple messages which could get ugly but is entirely doable.

UUID Generated randomly is having duplicates

I'm using the below function to generate UUID
UUID.randomUUID().toString()
In production we have 50+ servers (application server - each is a JVM on its own) and for requests that land in these servers, as a first step we generate a UUID which essentially uniquely identifies a transaction.
What we are observing is that in Server 6 and Server 11, the UUIDs generated are matching at least for 10 to 15 messages per day which is strange because given the load i.e. about 1 million transactions a day, these UUIDs being duplicate within the same day is very odd.
This is what we have done so far
Verified the application logs - we didn't find anything fishy in there, all logs are as normal
Tried replicating this issue in the test environment with similar load in production and with 50+ servers - but this didn't happen in the test environment
Checked the application logic - this doesn't seem to be an issue because all other 48 servers except 6 and 11 which have a copy of the same code base is working perfectly fine and they are generating unique UUIDs per transaction.
So far we haven't been able to trace the issue, my question is basically if there is something at JVM level we are missing or UUID parameter that we need to set for this one off kind of an issue?
Given time, I'm sure you'll find the culprit. In the meantime, there was a comment that I think deserves to be promoted to answer:
You are generating pseudo random UUIDs at multiple locations. If you don't find other bugs, consider either generating all the pseudo random UUIDs at one location, or generate real random UUIDs
So create a UUID server. It is just a process that churns out blocks of UUIDs. Each block consists maybe 10,000 (or whatever is appropriate) UUIDs. The process writes each block to disk after the process verifies the block contains no duplicates.
Create another process to distribute the blocks of UUIDs. Maybe it is just an a web service that returns an unused block when it gets a request. The transaction server makes a request for a block and then consumes those UUIDs as it creates transactions. When the server has used most of its assigned UUIDs, it requests another block.
I wouldn't waste time wondering how UUID.randomUUID() is generating a few duplicate UUIDs per day. The odds of that happening by chance are infinitesimal. (Generating a whole series of duplicates is possible—if the underlying RNG state is duplicated, but that doesn't seem to be the case.)
Instead, look for places where a UUID stored by one server could be clobbering one stored by another. Why does this only happen between 2 servers out of 50? That has something to do with the details of your environment and system that haven't been shared.
As stated above, the chances of a legit collision are impossibly small. A more likely possibly is if the values are ever transferred between objects in an improper way.
For languages like Java that behave as pass by reference, consider the following scenario
saveObject1.setUUID(initObj.getUUID())
initObj.setUUID(UUID.randomUUID());
saveObject2.setUUID(initObj.getUUID())
In this case saveObject1 & saveObject2 will have the same value, because they are both pointed to the same object reference (initObj's UUID reference).
An issue like this seems more likely than the actual UUIDs being a collision, esp if you can reproduce it. Naturally if it doesn't happen all the time it's probably something more complex, like a rare race condition where initObj doesn't get reinitialized in time, causing saveObject1 & 2 to share the same object reference.

Setting ttl on a mongoDB collection - in application or shell?

I would like to set the ttl for a collection once, what is the idiomatic way of achieving this when building a java application that uses mongoDB? Do ppl simply apply settings like these in the shell? Or in the application code is it normal to check if a collection is already in the DB, if it is not then create it with the desired options?
Thanks!
I never do index building in my application code anymore.
I confess that I used to. Everytime my application started up I would ensure all my indexes, until suddenly one day a beginner developer got hold of my code and accidently deleted a character within one of my index sequences.
Consequently the entire cluster froze and went down due to processing, in the foreground, this index building. Fortunately I had a number of delayed and non-index building slaves to repair from but still, I lost about 12 hours all in all and in turn 12 hours of business.
I would recommend you don't do your index building in the application code but instead carfully within your mongo console. That goes for any operation like this, even TTL indexing.
You can set a TTL on a collection as documented here.
Using the Java driver, I would try:
theTTLCollection.ensureIndex(new BasicDBObject("status", 1), new BasicDBObject("expireAfterSeconds", 3600));
hth.
Setting a TTL
is an index operation, so I guess that it would not be wise performance wise to do it every time your code is running.

Best way to store small, alternating, public data that updates every couple of hours?

The essence of my problem is that there are too many solutions, and I would like to find which one wins out in pros and cons before I build an infrastructure around it.
(Simplified for the purpose of this forum) This is an auction site where five auctions are stored in a rank #1-5, #1 being the currently featured auction. The other four are simply "on deck." After either a couple hours or the completion of that auction, #2-5 move up to #1-4 and a new one is chosen to be #5
I'm using a dedicated server and I've been considering just storing the data in the servlet or maybe adding a column in the database as a boolean for each auction...like "isFeatured = 1"
Suffice it to say the data is read about 5 times+ more often than it is written, which is why I'm leaning towards good old SQL.
When you can retrieve the relevant auctions from DB with a simple query with ORDER BY and TOP or something similar then try this. If no performance issues occur then KISS and you're done.
Otherwise when these 5 auctions are valid for a while then cache them in memory. Have a singleton holding these auctions and provide methods for updating for example. Maybe you want to use a caching lib instead. Update these Top5 whenever necessary but serve them directly out of memory without hiting a DB or something similar expensive.
What kind of scale are you looking for? How many application servers need access to the data?
I think you're probably making this more complicated than it is. Just use a database, take a hit of ACID, and move onto whatever else you need to work on. :P
Have you taken a look at SQLite? It allows for "good old SQL" without all of the hassles of setting up a separate database server. As long as the data isn't too huge (to be fair, I haven't tested the size limits, but I've skimmed blog entries mentioning the use of SQLite to process files of several dozen MB in size quickly and with no problems), you should be fine.
It isn't a perfect solution for all needs (frankly, I sometimes find the dynamic typing to be a pain), but since it relies on locally stored files, reads will be much faster than firing up a network connection to talk to a more "traditional" RDBMS.

Categories