I'm currently using Hibernate as JPA provider and I want to switch IDs generation from the persistence layer to the application layer.
My schema (on MySQL 5.7) is currently using a BIGINT(20) data-type for IDs, but I don't want to refactor to go for UUIDs.
So I thought that something like a "System" UID should be enough:
public static long getUID()
{
long key = getSystemKey() << 56; // like 0x0100000000000000L;
long ts = System.currentTimeMillis() << 12;
int r = new Random().nextInt(0xFFF);
long id = key + ts + r;
return id;
}
The generated id is in the form
KK TTTTTTTTTTT RRR
where getSystemKey() [K] returns a unique fixed byte for each "machine" the application is running on (it's declared inside the configuration file).
timestamp ts [T] is using 11 nybbles, ensuring enough millis to 2527-06-23 08:20:44.415
random r [R] is used to add randomness per machine per millis (last 3 nybbles).
So I'm wondering if this way is consistent enough, what are pros and cons and if there's a better way.
Thanks
UPDATE
I tested this method with 100 threads and 10,000 executions:
public static void main(String[] args) throws Exception
{
List<Callable<Long>> runners = new ArrayList<>();
for(int i = 0; i < 10000; i++)
{
runners.add(SUID::random);
}
ExecutorService pool = Executors.newFixedThreadPool(100);
List<Future<Long>> results = pool.invokeAll(runners);
pool.shutdown();
int dups = 0;
Set<Long> ids = new HashSet<>();
for(Future<Long> future : results)
{
if(!ids.add(future.get()))
{
dups++;
}
}
System.out.println(dups);
}
I got around 6% of collisions.
So, the only way seems to use some synchronization:
public final class SUID
{
private static final AtomicLong SEQUENCE = new AtomicLong(Config.getSystemKey() << 56 | System.currentTimeMillis() << 12);
private SUID()
{
super();
}
public static long generate()
{
return SEQUENCE.incrementAndGet();
}
}
Related
For my current java project, I am trying to generate random ID's for registered users. So far I have been using min +(int) (Math.random()*((max-min)+1)) as my formula to generate the random number. The problem that I am facing is that sometimes the numbers repeat themselves and my application wouldn't work with them.
int min = 1001;
int max = 1050;
for (int i=1; i<=1; i++)
{
int a = min +(int) (Math.random()*((max-min)+1));
}
I have tried using and incorporating
Integer[] arr = new Integer[100];
for (int i = 1; i < arr.length; i++) {
arr[i] = i;
}
Collections.shuffle(Arrays.asList(arr));
but numbers generated would constantly come out as "null" and it would repeat the loop a few hundred times and flood my txt file.
In general, random generators Random or Math.random() are not the correct ways to generate a unique id. As you mentioned, it could be repeated (and it will definitely be).
I would recommend two ways of generating ID.
The first one is to use AtomicInteger. This is good when your ID should be unique but not random.
private static final AtomicInteger ID = new AtomicInteger(0);
public static String generateUniqueId() {
return String.valueOf(ID.incrementAndGet());
}
The second one, which is preferable to me, is to use UUID. This is good when your ID should be as unique as random.
public static String generateUniqueId() {
return String.valueOf(UUID.randomUUID());
}
Another one, I can mention is to use System.nanoTime().
public static String generateUniqueId() {
return String.valueOf(System.nanoTime());
}
Long ago I had some investigation and find out that this is pretty stable for normal payload. But in general, it could retrieve the same value if you build such a system, that should generate ID so often.
Instead of generating numbers I would recommend to generate UUID. The chance of a is collision is close to impossible.
UUID id = UUID.randomUUID();
Otherwise if you want to stick with numbers I would recommend you to implement yourself some Sequence service within your application.
import java.util.concurrent.atomic.AtomicLong;
public class SequenceService {
private final AtomicLong ids;
public SequenceService() {
long initialValue = getInitialValue();
this.ids = new AtomicLong(initialValue);
}
public long generateNextId() {
return ids.incrementAndGet();
}
private long getInitialValue() {
// this methods reads the last known leased id (e.g. from the file system)
}
}
I do not have a background in CS. I am really new to parallel programming and I do not know how exactly the hardware works when running a program. However, I have noticed the following. Say I have:
public class Counter {
private static int parallelCount = 0;
private static int sequentialCount = 0;
public static void main(String[] args) {
int n = 1000;
// I count in parallel:
IntStream.range(0, n).parallel().forEach(i -> {
parallelCount++;
});
// I count sequentially:
for (int i = 0; i < n; i++) {
sequentialCount++;
}
System.out.println("parallelCount = " + parallelCount);
System.out.println("sequentialCount = " + sequentialCount);
}
}
why I may get:
parallelCount = 984
sequentialCount = 1000
I guess this has to do with the hardware and the way the compiler access memory. I am really interested to know why this happens. And what is one possible solution?
Whenever more than one threads can access a value that is mutable then the system goes out of sync meaning the kind of problem that you are facing. No one can be sure what the result will be, and many a times the result will be wrong. You cannot guarantee which thread will write the value last.
Therefore, you need to synchronize the access to the shared resource (the integer you are incrementing) so that all threads get the latest updated value and the answer is always correct.
Coming to your program you can try making the parallelCount variable an Atomic Integer like AtomicInteger parallelCount = new AtomicInteger(); An Atomic Integer is thread safe meaning that they can be concurrently updated without running the system out of sync.
import java.util.concurrent.atomic.AtomicInteger;
import java.util.stream.IntStream;
public class Counter {
private static AtomicInteger parallelCount = new AtomicInteger();
private static int sequentialCount = 0;
public static void main(String[] args) {
int n = 1000;
// I count in parallel:
IntStream.range(0, n).parallel().forEach(i -> {
parallelCount.getAndIncrement();
});
// I count sequentially:
for (int i = 0; i < n; i++) {
sequentialCount++;
}
System.out.println("parallelCount = " + parallelCount);
System.out.println("sequentialCount = " + sequentialCount);
}
}
As you can expect standard for loop will increment sequentialCount 1000 times
Regarding parallel stream, the application will try to open multiple threads which need to execute your function on parallel. In this situation, multiple threads can increment value at the same time and store value to int.
For example, suppose that we have two threads that working parallel and want to increment value from variable parallelCount. If parallelCount contains value 50. Both threads will read value 50 and calculate the new value 51 and store it to memory.
This approach can produce other concurrent problems. In order to solve this problem, you can use synchronization, locking, atomic classes, or another approach.
Multiple theads do an operation that is not atomic (incrementing a value).
The code you wrote translates to byte code and might cause something like this:
To avoid this, you need to synchronize the access to that critical code.
But note, that if all of your code is critical code, then it's redundant to use multiple threads.
AtomicInteger
We can make use of AtomicInteger class from Java concurrency package while working with parallel streams as the behavior can be unpredictable while using primitive data type
import java.util.stream.IntStream;
import java.util.concurrent.atomic.AtomicInteger;
public class Main
{
private static AtomicInteger parallelCount = new AtomicInteger();
private static int sequentialCount = 0;
public static void main(String[] args) {
System.out.println("Hello World");
int n = 100000;
// I count in parallel:
IntStream.range(0, n).parallel().forEach(i -> {
parallelCount.incrementAndGet();
});
// I count sequentially:
for (int i = 0; i < n; i++) {
sequentialCount++;
}
System.out.println("parallelCount = " + parallelCount);
System.out.println("sequentialCount = " + sequentialCount);
}
}
I have large Map where I store some objects. The Map is large: it has around 200k objects. When I try to run some methods, that require to read map values, the program freezes. When I debug it, it seems that my IDE is 'collecting data' (picture). It has never completed the task.
I have 16GB RAM.
What can I do to speed this up?
I get performance issues around 61 million elements.
import java.util.*;
public class BreakingMaps{
public static void main(String[] args){
int count = Integer.MAX_VALUE>>5;
System.out.println(count + " objects tested");
HashMap<Long, String> set = new HashMap<>(count);
for(long i = 0; i<count; i++){
Long l = i;
set.put(l, l.toString());
}
Random r = new Random();
for(int i = 0; i<1000; i++){
long k = r.nextInt()%count;
k = k<0?-k:k;
System.out.println(set.get(k));
}
}
}
I run the program with java -Xms12G -Xmx13G BreakingMaps
I suspect your problem is not the map, but circumstances surrounding the map. If I write the same program, but use a class with hashcode colisions then the program cannot handle 200K elements.
static class Key{
final long l;
public Key(long l){
this.l = l;
}
#Override
public int hashCode(){
return 1;
}
#Override
public boolean equals(Object o){
if(o!=null && o instanceof Key){
return ((Key)o).l==l;
}
return false;
}
}
Look at this - as the solution you can increase the heap size for your app:
java -Xmx6g myprogram.
But it's not very good. I'd suggest you to try to rework your data processing approach. Maybe you can apply some filtering before fetching the data to decrease the data size or implement some calculation on database level.
Can someone give me the design of how to maintain all the hardcoded values sepratly.
I'm currently using some design patterns, are these fine? And is there any thing more I can do for better design of my applicaiton code.
The design patters I use are (My application is having multiple screens similar wizard):
Using a ResourceBundle for all the strings, which are visible to user (for internationalizaiton)
Using the properties file to save all the values of a particular screen to make accessible for other screens (These values are for internal purpose of program, not visible to user). I'm continuously loading these properties in program at various places to get the updated values like, while going to second screen (Panel) from first screen, to get the values of first screen in second screen.
I'm Thinking to externalize the logging messages using Resource bundles.
Is there any better design approaches? to seperate from program code, A. the "messges visible to user", B. log messages, C. Usability values (screen sizes, Fonts, etc), values user has input in the screens, directory/file paths...
I would check out the Java Preferences API, which allows you to store system-wide defaults, per-user defaults and make use of hard-coded defaults in the absence of configured values.
Resource bundles for I18N are exactly the right thing. Properties files do well, but you'll have to repackage and redeploy if you change them. It's often more flexible to put other items in a database, because you can change the app by pushing data without repackaging and redeploying.
Your GUI should have a complete data model. This model can read I18n resource files and regular resource files to populate part of the model. A data model is one or more Java classes that hold the data important to your application while the application is running.
As an example, here is a stopwatch GUI that I created.
And here's the data model class associated with the stopwatch GUI.
import java.util.ArrayList;
import java.util.List;
import javax.swing.table.DefaultTableModel;
public class StopwatchModel {
protected boolean isSplitTime;
protected long startTime;
protected long endTime;
protected DefaultTableModel tableModel;
protected List<Long> splitTimes;
protected String[] columnNames = {"", "Increment", "Cumulative"};
public StopwatchModel() {
this.splitTimes = new ArrayList<Long>();
this.isSplitTime = false;
this.startTime = 0;
this.endTime = 0;
setTableModel();
}
public void resetTimes() {
this.splitTimes.clear();
this.isSplitTime = false;
this.startTime = 0;
this.endTime = 0;
}
public boolean isSplitTime() {
return isSplitTime;
}
public long getStartTime() {
return startTime;
}
public long getEndTime() {
return endTime;
}
public long getLastSplitTime() {
int size = splitTimes.size();
if (size < 1) {
return getStartTime();
} else {
return splitTimes.get(size - 1);
}
}
public long getPenultimateSplitTime() {
int size = splitTimes.size();
if (size < 2) {
return getStartTime();
} else {
return splitTimes.get(size - 2);
}
}
public DefaultTableModel getTableModel() {
return tableModel;
}
public int getTableModelRowCount() {
return tableModel.getRowCount();
}
public void clearTableModel() {
tableModel.setRowCount(0);
}
public int addTableModelRow(long startTime, long previousSplitTime,
long currentSplitTime, int splitCount) {
String[] row = new String[3];
row[0] = "Split " + ++splitCount;
row[1] = formatTime(previousSplitTime, currentSplitTime, false);
row[2] = formatTime(startTime, currentSplitTime, false);
tableModel.addRow(row);
return splitCount;
}
public void setStartTime() {
if (getStartTime() == 0L) {
this.startTime = System.currentTimeMillis();
} else {
long currentTime = System.currentTimeMillis();
int size = splitTimes.size();
if (size > 0) {
long splitTime = splitTimes.get(size - 1);
splitTime = splitTime - getEndTime() + currentTime;
splitTimes.set(size - 1, splitTime);
}
this.startTime = currentTime - getEndTime() + getStartTime();
}
}
protected void setTableModel() {
this.tableModel = new DefaultTableModel();
this.tableModel.addColumn(columnNames[0]);
this.tableModel.addColumn(columnNames[1]);
this.tableModel.addColumn(columnNames[2]);
}
public void setSplitTime() {
this.splitTimes.add(System.currentTimeMillis());
isSplitTime = true;
}
public void setEndTime() {
Long split = System.currentTimeMillis();
if (isSplitTime) {
this.splitTimes.add(split);
}
this.endTime = split;
}
public String formatTime(long startTime, long time, boolean isTenths) {
long elapsedTime = time - startTime;
int seconds = (int) (elapsedTime / 1000L);
int fraction = (int) (elapsedTime - ((long) seconds * 1000L));
fraction = (fraction + 5) / 10;
if (fraction > 99) {
fraction = 0;
}
if (isTenths) {
fraction = (fraction + 5) / 10;
if (fraction > 9) {
fraction = 0;
}
}
int hours = seconds / 3600;
seconds -= hours * 3600;
int minutes = seconds / 60;
seconds -= minutes * 60;
StringBuilder builder = new StringBuilder();
builder.append(hours);
builder.append(":");
if (minutes < 10) builder.append("0");
builder.append(minutes);
builder.append(":");
if (seconds < 10) builder.append("0");
builder.append(seconds);
builder.append(".");
if ((!isTenths) && (fraction < 10)) builder.append("0");
builder.append(fraction);
return builder.toString();
}
}
The model would also contain transient data that gets you from one JPanel to the next JPanel. Transient data is data that only needs to exist for a short period of time, usualy as long as the GUI is active. It doesn't have to be saved after the GUI is exited.
The reason that you use a model / view to build a GUI is separation of concerns. The rest of your application accesses the data model, rather than any GUI components.
For "Usability values" I've used a singleton pattern. Create a class called Configuration and then a static method, Configuration.getConfig() that calls the private constructor. All of the objects that your program needs can be stored as fields in this class. The constructor loads the values using the preferences API. I posted some example code for this in a recent S.O. post.
The advantage of placing all of your configuration data in a single class which is responsable for loading it is that now the method in which you are storing this data is abstracted away from the rest of your code. If you decide to use a database or ResourceBundle for storage, only Configuration needs to change.
I need to generate monotonically increasing integers.
Could I use the timestamp to somehow generate such type of integer sequence?
I would request an integer at a time, and I won't be requesting more than an integer in the same second's interval - if I do, I won't mind if it passes me the same integer within that second's interval.
You can use an AtomicInteger object to maintain a thread safe counter. Then use getAndIncrement() when you need the next integer.
Since monotonically increasing integers do not need to be contiguous (ie there can be gaps, as long as the number keeps increasing), and it sounds like you want all calls made in the same second to return the same integer, a method that returns how many seconds the JVM has been up would do nicely.
Here's a simple implementation that does that:
private static long startTime = System.currentTimeMillis();
public static int secondsSinceStart() {
return (int) TimeUnit.SECONDS.convert(
System.currentTimeMillis() - startTime, TimeUnit.MILLISECONDS);
}
FYI, this would last 68 years before rolling over.
This is my self made generator...
public final class IdGenerator {
private static final int MAGNITUDE = 10000;
private static long previousTimestamp;
private static int counter = 0;
private IdGenerator() {
}
public synchronized static long generateId() {
final long timeMillis = System.currentTimeMillis();
if (previousTimestamp != timeMillis) {
counter = 0;
}
previousTimestamp = timeMillis;
final int counterValue = counter++;
if (counterValue >= MAGNITUDE) {
//just to be sure
throw new IllegalStateException("too many id generated for a single timestamp!");
}
return timeMillis * MAGNITUDE + counterValue;
}
}