Summary
Really globally unique IDs in flash and/or javascript clients. Can I do this with an RNG available in current browsers/flash or must I build a composite ID with server-side randomness?
Details
I need to generate globally unique identifiers for objects. I have multiple server-side "systems" written in java that need to be able to exchange ids; each of these systems also has a set of flex/javascript clients that actually generate the IDs for new objects. I need to guarantee global uniqueness across the set of unrelated systems; for instance I need to be able to merge/sync the databases of two independent systems. I must guarantee that there is never a collision between these ids and that I never need to change the id of an object once created. I need to be able to generate ids in flash and javascript clients without contacting the server for every id. A solution that relies on some server provided seed or system id is fine as long as the server isn't contacted too often. A solution that works completely disconnected is preferable. Similarly a solution that requires no upfront registration of systems is preferable to one that relies on a central authority (like the OUI in a MAC address).
I know the obvious solution is "use a UUID generator," such as UIDUtil in flash. This function specifically disclaims global uniqueness. In general, I'm concerned about relying on a PRNG to guarantee global uniqueness.
Proposed solutions
Rely entirely on a secure random number generator in the client.
Flash 11+ has flash.crypto.generateRandomBytes; Javascript has window.crypto but it's pretty new and not supported in IE. There are solutions like sjcl that use the mouse to add entropy.
I understand that given a perfect RNG the possibility of collision for a 2122 random UID is meteorite tiny, but I'm worried that I won't actually get this degree of randomness in a javascript or flash client. I'm further concerned that the typical use case for even a cryptographic RNG is different from mine: for session keys, etc, collisions are acceptable as long as they are unpredictable by an attacker. In my case, collisions are completely unacceptable. Should I really rely on the raw output of a secure RNG for a unique ID?
Generate a composite ID that includes system, session and object IDs.
An obvious implementation would be to create a system UUID at server install time, keep a per-client-login session id (eg in a database), and then send the system and session ids to the client which would keep a per-session counter. The uid would be the triple: system ID, session ID, client counter.
I could imagine directly concatenating these or hashing them with a cryptographic hash. I'm concerned that the hashing itself may potentially introduce collisions, particularly if the input to the hash is about the same size as the output. But the hash would obscure the system id and counters which could leak information.
Instead of generating the system ID at install time, another solution would be to have a central registry that handed out unique system IDs, kind of like what DOI does. This requires more coordination however, but I guess is the only way to really guarantee global uniquness.
Key questions
Random or composite based?
Include system ID?
If system id: generate a random system ID or use a central registry?
Include timestamp or some other nonce?
To hash or not to hash?
The simplest answer is to use a server assigned client ID which is incremented for each client, and a value on each client which is incremented for each fragment on that client. The pair of client ID and fragment ID becomes the globally unique ID for that piece of content.
Another simple approach is to generate a set of unique IDs (say 2k at a time) on the server and send them in a batch to each client. When the client runs out of IDs it contacts the server for more.
Client IDs should be stored in a central repository accessible to all the servers.
It may help looking at methods for distributed hashing which is used to uniquely identify and locates fragments within a peer-to-peer environment. This may be overkill considering you have a server which can intervene to assert uniqueness.
To answer your questions you need to determine the benefit that the added complexity of a system ID, nonce or hash would bring.
System ID:
A system ID would typically be used to uniquely identify the system within a domain. So if you don't care who the user is, or how many sessions are open, but only want to make sure you know who the device is, then use a system ID. This is usually less useful in a user-centric environment such as JavaScript or Flash, where the user or session may be relevant.
Nonce:
A nonce/salt/random seed would be used to obfuscate or otherwise scramble the ID. This is important when you don't want others to be able to guess the original value of the ID. If this is necessary then it may be better to encrypt the ID with a private encryption key, and pass a public decryption key to each consumer who needs to read the ID.
Timestamp: Considering the variability of the client's clock (ie you cannot guarantee it adheres to any time or time zone), a timestamp would need to be treated as a pseudo-random value for this application.
Hash: While hashes are often (ab)used to create unique keys, their real purpose is to map a large (possibly infinite) domain to a smaller, more manageable one. For example, MD5 is typically used to generate a unique ID from a timestamp, random number, and/or nonce data. What is actually happening is that the MD5 function is mapping an infinite range of data into a space of 2^128 possibilities. While this is a massive space, it is not infinite, so logic tells you that there will be (even if only in theory) the same hash assigned to two different fragments. On the other hand perfect hashing attempts to assign a unique identifier to each piece of data, however this is entirely unnecessary if you just assign a unique identifier to each client fragment to start with.
Something quick and dirty and also may not work out for your use case --
Using Java's UUID and coupling that with something like , say clientName.
This should solve the multiple client and multiple server issue.
The rationale behind this is that the possibility of getting 2 calls at the same nanosecond are low, refer to the links provided below. Now, by coupling the clientName with the UUID you are ensuring unique IDs across clients and that should leave only handling the use case of the same client calling twice within the same nanosecond.
You could write a java module to generate the IDs and then get Flash to talk to this module.
For your reference, you could refer to --
Is unique id generation using UUID really unique?
Getting java and flash to talk to each other
A middle ground builds on #ping's answer:
Use client name, high-resolution time, and optionally some other pseudo-random seed
Hash the data to produce the UID (or, just go directly to using UUIDs)
Log the resulting to a central server for entry into a database
Treat any collision as a prominently flagged bug, rather than as a situation that deserves special code.
With a UUID or reasonably long hash, the chances of a duplicate or nil. So either:
A) You'll get no duplicates for the life of the application, life is good.
B) You'll see a duplicate, or maybe two (freaky!), over a few decades. Intervene manually to deal with those cases; if you're running servers with your client, you can afford it.
C) If you get a third collision, then there is something fundamentally wrong with the code, and this can be investigated and measures taken to avoid a repetition.
This way, the ID is generated at the client, contacts to the server are one-way and operationally non-critical, the seeds don't have to be random, the hashing obscures the origins of the ID and so avoids constructed collisions, and you can be confident that there have been no collisions. (If you test that collision detection code!) Even UUIDs could be plenty adequate in this scenario.
The only way hashing increases the likelihood of collisions is if your information content in the original seed information approaches the size of the hash. That's extremely unlikely, but if true and you're still thinking about micrometeorites, just increase the size of the hashed value.
My two cents.. Each server locks a DB table and get an id from it, and increments it. this will be the server unique id.
Each client connecting will get this id, coupled with a unique identifier issued by the server. This unique key has to be unique for this server, but another server might issue the same id to a different client.
Finally, each client will generate a unique id for each request.
Coupling all three will guarantee a true unique global id over the entire system, the final id will look something like:
[server id][client id][request id]
Related
Requirement:
we need to assign a random, long enough number/string to merchants so that they are uniquely identifiable(do not want someone to guess the identifier ). This is required because we are printing this number/string as QR code and giving it to merchants so that our users can read the QR code and get the info about the merchant.
current Implementation:
we cannot print id's as they will be sequential, so we have introduced a new field (externalId) to store a unique id generated by TimeBasedGenerator of JUG UUID generator. As I researched more I found there are performance issues with UUID. All the articles talks about UUId as primary key, I am not using UUID as primary key but as a secondary identifier. I am not worried about inserts and updates, as they will be fewer as compared to search. As user will be sending us the UUID read from the printed QR code and we will need to search the merchant using this UUID field, Hence there will be huge impact on performance.
solution as per me:
instead of UUID we will give a combination of ID:UUID, so that when we get the request we can split and fetch the merchant using ID and check if the externalId(UUID) present in our DB matches with the one provided, if it matches we return the merchant other wise not.
Will this solution increase the performance significantly or the time taken by string operations will nullify the effect.
Is there any better approach to do this?
Thanks
UUIDs are generated in various flavors. As you are perhaps already aware, RFC 4122 outlines a variant (i.e. UUID schema) that defines five different versions. I suggest that you look into version 1 of RFC 4122.
Version 1 uses a MAC address of the machine generating the UUID as the last 12 digits of the generated UUID.
Assume you go to a second-hand computer recycler and purchase an obsolete ethernet adaptor; power it up and get the MAC address of that adaptor; and then destroy the adaptor (shredder, shot-gun, acid, etc.). You can now be assured that no other computer in the universe will -ever- generate a V1 UUID using that MAC address.
Now you could write a UUID (RFC 4122, V1) generator that would use harvested MAC address.
You could harvest a separate MAC address (from additional cards) for each merchant, and then you would be able to identify UUIDs generated for each.
BONUS: You might experiment with V1 UUIDs using Mahonri Moriancumer's UUID and GUID Generator and Forensics.
The best implementation would depend on your UUID'S character distribution but what I did when faced with similar situation is use only the UUID for the end user. In the database I added a column filled with the first 2 chars of the UUID and indexed that column.
Either this or, as you proposed, the ID would serve the same purpose. The benefits of indexing a partial UDID are that
you can tweak how many characters to use to improve performance
the id is not used externaly potentially causing issues.
On the other hand, an integer index is probably using less memory and in no need of tweeking
Performance wise, for my implementation we were facing search queries of 1 in a million taking 7-8 seconds. With this solution we dropped to a few milliseconds
I want to make a javaEE application when users can register and confirm their email when receiving a email with a link after inserting their data in registration form (name, mail...)
To do that I am going to generate a long and unique key with java.util.UUID, store in a database and then send an email to the user with that key being part of the URL (Example: www.mysite.com/account.xhtml?id=KEY). Then the user will click the link, I extract the key from the URL and check if that key is stored in the DB. If it is, the user registration will be completed.
My question is, when creating that key with java.util.UUID, how can I know that it is a unique key? Should I check if there is another equal key in the DB and if so create a new one until the created key is unique?
What's the chance that a randomly-generated 128-bit integer will be equal to another randomly-generated integer?
If you just need peace of mind, use a primary key and if the insert fails due to a key collision, re-create a new UUID and retry the insert.
There are couple of ways you can do UUID in Java.
Java 5 onwards better practice is using java.util.UUID It is size of the string 36 characters. This link gives you simple example.
This discussion will give you answer to your question. It is very strong. I have never came across someone is complaining about its uniqueness.
But if you adding into DB or using in storage or using through network, size may be matters. So converting to other formats - Bases is good solution (Base64, Base85 etc). Please check this discussion here. You can use apache library org.apache.commons.codec.binary.Base64. Base85 is not safe for URLs.
My recommendation is, if you have may application/session beans/web services (many interconnections other applications and data transfers etc) are creating UUIDs, I prefer to do unique application name padding too. Like APP1, APP2 etc and then decode to other bases. If UUID is 6fcb514b-b878-4c9d-95b7-8dc3a7ce6fd8, then APP1-6fcb514b-b878-4c9d-95b7-8dc3a7ce6fd8 like that...
Though it is off the topic here, BUT When you use a URL like this www.mysite.com/account.xhtml?id=KEY, beware about SQL injection hacking attacks.
I am building a distributed application on top of Java and Cassandra. To generate unique sequential 32bit & 64 bit IDs, is an approach like using Flickr's ticket servers to generate primary IDs, a good one? I am particularly excited about this as it can help me reduce the size of the IDs to 32 bits or 64 bits as required, which otherwise may go up to 128 bits with UUIDs. I do not want these IDs to be perfectly sequential, but increasing at least!
Using a single database server may however introduce a single point of failure that was eliminated by Cassandra. However this may be OK for the initial stage of our application. Later we may introduce two servers to alleviate those problems.
Does this sound like a good strategy? In short, we are mixing MYSQL and Cassandra in one application. I know, if mySQL is down for some reason then we can't go ahead with Cassandra alone.
We have looked to other solutions like snowflake however it did not perfectly matched our requirements.
EDIT : I am seeking advice on whether using MySQL for generating unique primary IDs to key the data/ entities stored in Cassandra database is a good approach. What are the downsides, if any, of an approach like Flickr's Ticket servers?
I'm not a big fan of trying to attach meaning to surrogate keys (which you're trying to do if you want them to increase over time). As you're seeing, it makes your problem of generating keys more complicated. Assuming that you want the keys to increase over time simply so that you can sort data, why not include a timestamp of when the object was created and store that in your data store? This simplifies the key generation significantly and allows you to do pretty much everything you could do with keys that increase over time, with the added bonus of the fact it will be crystal clear to whoever has to maintain your code how objects should be sorted.
In general, you can't have both "always increasing" and "no SPOF and no complex synchronization".
If you want to have several ID-generators which do not have to ask each other on every new ID, each of them really need a separate ID-pool.
A really simple example is mentioned in the article linked by you, where one server creates odd ones while the other one create even ones. (You can expand this to more servers trivially). Of course, then you can't be sure that one server doesn't run ahead of the other, which can lead to a non-increasing sequence like 111, 120, 113, 122, 115, 124 ...
If you only want "roughly increasing", you can implement a scheme where each server in some intervals (like each minute or each 10000 IDs) tells the other one(s) his current ID, and the other one then jumps its own ID (only forward) if he hangs back too far. This should be done in a way which does not interrupt the ID-generation, for robustness if the other server is down.
Ah, for the "free bits at the end", simply multiply your ID by some number (the same one each time, and a power of two if you really want "free bits" and not only "space for data"), then add your data (which should be less than number). But of course then you'll run out of ID space quite a bit earlier (by factor number).
I am working on a web application related to Discussion forums using Java and Cassandra database.
I need to construct 'keys' for the rows storing the user's details and & another set of rows storing the content posted by the user.
One option is to get the randomly generated UUID provided by Java language, but these are 16 bytes long. and since NoSQL database involves heavy denormalization, I am concerned whether I would be wasting lots of disk space, RAM and other resources if the key could be generated in smaller sizes.
I need to generate two types of keys, one for the Users & other for Content Posted by Users.
For the Content posted by users, would timestamp+userId be a good key. where timestamp is the server time at which content was posted and userId refers to key of user row.
Any suggestions, comments appreciated ..
Thanks
Marcos
Is this a distributed application?
Then you could use a simple synchronized counter and initialize it on startup with the next available id.
On the other hand a database should be able to handle the UUID hashes as created by java.
This is a standard for creating things like sessionIds, that need to be unique.
Your problem is somewhat similar since a session in your context would represent a set of user input.
I came across a question recently that was for "Generating primary key in a clustered environment of 5 App-Servers - [OAS Version 10] without using database".
Usually we generate PK by a DB sequence, or storing the values in a database table and then using a SP to generate the new PK value...However current requirement is to generate primary key for my application without referencing the database using JDK 1.4.
Need expert's help to arrive on better ways to handle this.
Thanks,
Use a UUID as your primary key and generate it client-side.
Edit:
Since your comment I felt I should expand on why this is a good way to do things.
Although sequential primary keys are the most common in databases, using a randomly generated primary key is frequently the best choice for distributed databases or (particularly) databases that support a "disconnected" user interface, i.e. a UI where the user is not continuously connected to the database at all times.
UUIDs are the best form of randomly generated key since they are guaranteed to be very unique; the likelyhood of the same UUID being generated twice is so extremely low as to be almost completely impossible. UUIDs are also ubiquitous; nearly every platform has support for the generation of them built in, and for those that don't there's almost always a third-party library to take up the slack.
The biggest benefit to using a randomly generated primary key is that you can build many complex data relationships (with primary and foreign keys) on the client side and (when you're ready to save, for example) simply dump everything to the database in a single bulk insert without having to rely on post-insert steps to obtain the key for later relationship inserts.
On the con side, UUIDs are 16 bytes rather than a standard 4-byte int -- 4 times the space. Is that really an issue these days? I'd say not, but I know some who would argue otherwise. The only real performance concern when it comes to UUIDs is indexing, specifically clustered indexing. I'm going to wander into the SQL Server world, since I don't develop against Oracle all that often and that's my current comfort zone, and talk about the fact that SQL Server will by default create a clustered index across all fields on the primary key of a table. This works fairly well in the auto-increment int world, and provides for some good performance for key-based lookups. Any DBA worth his salt, however, will cluster differently, but folks who don't pay attention to that clustering and who also use UUIDs (GUIDs in the Microsoft world) tend to get some nasty slowdowns on insert-heavy databases, because the clustered index has to be recomputed every insert and if it's clustered against a UUID, which could put the new key in the middle of the clustered sequence, a lot of data could potentially need to be rearranged to maintain the clustered index. This may or may not be an issue in the Oracle world -- I just don't know if Oracle PKs are clustered by default like they are in SQL Server.
If that run-on sentence was too hard to follow, just remember this: if you use a UUID as your primary key, do not cluster on that key!
You may find it helpful to look up UUID generation.
In the simple case, one program running one thread on each machine, you can do something such as
MAC address + time in nanseconds since 1970.
If you cannot use database at all, GUID/UUID is the only reliable way to go. However, if you can use database occasionally, try HiLo algorithm.
You should consider using ids in the form of UUID. Java5 has a class for representing them (and must also have a factory to generate them). With this factory class, you can backport the code to your anticated Java 1.4 in order to have the identifiers you require.
Take a look at these strategies used by Hibernate (section 5.1.5 in the link). You will surely find it useful.
It explains several methods, its pros and cons, also stating if they are safe in a clustered environment.
Best of all, there is available code that already implements it for you :)
If it fits your application, you can use a larger string key coupled with a UUID() function or SHA1(of random data).
For sequential int's, I'll leave that to another poster.
You can generate a key based on the combination of below three things
The IP address or MAC address of machine
Current time
An incremental counter on each instance (to ensure same key does not get generated twice on one machine as time may appear same in two immediate key creations because of underlying time precision)
by using Statement Object you can called statement.getGeneratedKeys(); method to retrieve the auto-generated key(s) generated by the execution of this Statement object.
Java doc
Here is how it's done in MongoDB: http://www.mongodb.org/display/DOCS/Object+IDs
They include a timestamp.
But you can also install Oracle Express and select sequences, you can select in bulk:
SQL> select mysequence.nextval from dual connect by level < 20;
NEXTVAL
1
2
3
4
5
..
20
Why are you not allowed to use the database? Money (Oracle express is free) or single point of failure? Or do you want to support other databases than Oracle in the future?
Its shipped OOB in many Spring-based applications like Hybris-
The typeCode is the name of your table like, User, Address, etc.
private PK generatePkForCode(final String typeCode)
{
final TypeInfoMap persistenceInfo = Registry.getCurrentTenant().getPersistenceManager().getPersistenceInfo(typeCode);
return PK.createCounterPK(persistenceInfo.getItemTypeCode());
}