I would like to have a persistent, distributed counter. My idea is to use a database sequence. Only sequence. I do not want to have a table, because I will not populate the table. I just need a sequence of unique numbers.
I don't want to use naive select mys-seq.nextval from dual (or org.springframework.jdbc.support.incrementer.OracleSequenceMaxValueIncrementer) because I would like to use sequence caching ability - I do not want to hit the database every time I need a new number.
I guess I should use org.hibernate.id.enhanced.SequenceStyleGenerator, but I cannot find any example of how to use it "standalone", without entity.
Unfortunately, all examples I found describes how to configure entity id generation with the sequence.
PS. I have the Spring Boot app.
I found simple solution for my problem: I can treat each number from database sequence as a range of numbers to use. For example, if sequence returns 5, it means that reserved range for my counter is 5000 - 5999.
In that solution I will hit database once for thousand numbers.
Initially I thought I have to utilize internal database level sequence number caching, but the same result I can achieve with trivial application level caching
Related
The problem is that I need to generate unique value for the one of my POJOs field in Java. I am using Hibernate.
It is pretty simple to solve it using sequence generator, but in my reality I need to make it random every time and I need it to have value between 10^5 and 10^6 — it must be 6 digits number.
So, it is also not a problem to generate it locally in Java service, but what if I have 2 more same services. What if they will generate same number, which break my numbers uniqueness. That's why I need that number to be generated by database.
Please, advice me best approach to achieve this. What strategy I need to follow. Thanks
There is no need to do this at ORM level. Just let the database engine auto increment it.
You can set the field to be unique and auto increment from 100,000 AUTO_INCREMENT = 100000; when you are creating the table.
Your numbers will then guaranteed to be unique.
You could also do this manually at service level if you put both checking for the highest existing number and the insert in the same transaction.
You could also do it via a trigger.
I am writing an application which will be deployed on n number of nodes. The applications entity classes used the SEQUENCE generation strategy to generate the primary keys. Since, there would be bulk inserts; we shall be giving an allocation size as well.
The concern is when the application will be deployed on n nodes and if two nodes simultaneously requests next sequence from the defined sequence in database:
Wouldn't there be any race condition?
Or is that sequence also has some light weight locking mechanism to serve the requests sequentially, as it happens in IDENTITY strategy?
Or sequence is not the right solution to this problem?
Kindly help. Thank you !
Think of Sequence as a table with one column storing an integer representing the current id. Each time you insert a new entry, the next operations happen in a transaction:
The current value from SEQUENCE table is read
That value is assigned as ID to the new entry
The value from SEQUENCE is incremented
To answer your questions
The concurrency issues are addressed by the database.
Since inserts happen in a transaction (both simple and bulk inserts), the consistency on ID generation is enforced by the database engine via transactions (by the isolation level of the transaction to be more precise). Make sure your database engine supports transactions.
Sequence is the right solution, assuming your database engine supports transactions.
I have a requirement in my project.
I want to generate an increasing unique sequence number which will be mapped to a specific field(interchange id ) for the output xml file.
The customer wants to generate some alert around this number. So they are very specific about the number should be
1. Unique 2. Increasing.
So now I have two approach to this case
I will generate a sequence with the help of oracle sequence. But the problem is again they do not want to unnecessarily hit the database.
Will genearate in java with the help of static variable. But I feel its not full proof. I think if my application or server restarts then the static variable will start form 0 again. In that case the number will not be unique.
So my question is, if we can get this something easily in mulesoft or any idea is apreciated
TIA.
Use the static field, but grab the value from the database. Given that there's no requirement that the numbers need to be consecutive, you can increment the database value by f.e. 100 each time, then you need to hit the database only every 100th number that you need to produce.
Obtaining a sequence value from the database does not qualify as "hitting the database unnecessarily".
It is necessary in order to obtain a unique sequence value in an efficient and scalable way.
While sequence contention isn't unheard of in an Oracle database it is usually not the biggest problem you have in a busy database. And one of the easiest things to fix: increase the sequence cache.
If you do know (as opposed to just assuming) that you will get a performance problem, then you might think about increasing the sequence increment to a very high number. Then, when you start your application you call nextval and get the upper limit of numbers you can hand out inside your Java code without risking anything. If you reach that limit, call nextval again to get the next slice of numbers. Essentially combining your static variable with the sequence persistence that Oracle offers
But again: I doubt that calling the sequence for each number will get you into trouble any time soon (and if it does, you probably will have other performance problems that are far bigger)
Another point as to why it is preferable to use something like the database for this is because a static variable will only work while you have a single JVM instance.
If you ever need to scale through more nodes this becomes a very brittle pattern (unless you are able to avoid by using GUID's instead - but this won't fulfil your incrementing requirement in this case).
Instead of 'the' database you could use another efficient mechanism like memcache or Redis?
I want to store different kinds of counters for my user.
Platform: Java
E.g. I have identified:
currentNumRecords
currentNumSteps
currentNumFlowsInterval1440
currentNumFlowsInterval720
currentNumFlowsInterval240
currentNumFlowsInterval60
currentNumFlowsInterval30
etc.
Each of the counters above needs to be reset at the beginning of each month for each user. The value of each counter can be unpredictably high with peaks etc. (I mean that a lot of things are counted, so I want to think about a scalable solution).
Now my question is what approach to take to:
a) Should I have separate columns for each counter on the user table and doing things like 'Update set counterColumn = counterColumn+ 1' ?
b) put all the values in some kind of JSON/XML and put it in a single column? (in this case I always have to update all values at once)
The disadvantage I see is row locking on the user table everytime a single counter is incremented.
c) having an separate counter table with 3 columns (userid, name, counter) and doing one INSERT for each count + having a background job doing aggregates which are written to the User table? In this case would it be ok to store the aggregated counters as JSON inside a column in the user table?
d) Doing everything in MySQL or also use another technology? I also thought about using another solution for storing counters and only keeping the aggregates in MySQL. E.g. I have experimented with Apache Cassandra's distributed counters. My concerns are about the Transactions which cassandra does not have.
I need the counters to be exact because they are used for billing, thus I don't know if Cassandra is a good fit here, although the scalability of Cassandra seems tempting.
What about Redis for storing the counters + writing the aggregates in MySQL? Does Redis have stuff which helps me here? Or should I just store everything in a simple Java HashMap in-memory and have a aggregation background thread and don't use another technology?
In summary I am concerned about:
reduce row locking
have exact counters (transactions?)
Thanks for your ideas :)
You're sort of saying contradictory things.
The number of counts can be huge or at least unpredictable per user.
To me this means they must be uniform, like an array. It is not possible to have an unbounded number of heterogenous data, unless you have an unbounded amount of code and an unbounded number of developer hours to expend.
If they are uniform they should be flattened into a table user_counter where each row is of the form (user_id, counter_name, counter_value). However you will need to think carefully about what sort of indices you will need, etc. Updating at the beginning of the month if they are all set to zero or some default value is one SQL query.
Basically (c). (a) and (b) are most absurd and MySQL is still a suitable technology for this.
Your requirement is not so untypical. In general this is statistical session/user/... bound written data.
The first thing is to split things if not already done so. Make a mostly readonly database, and separately collect these data. So a separated user table for the normal properties.
The statistical data could be held in an in-memory table. You could also use means other than a database, a message queue, session attributes.
I am using Hibernate 3.0 in my application with Postgres database. It is a monitoring application and gathers data every minute. So we have thousands of rows in some tables every month.
Currently i am using sequence for generating Id in hibernate. Is there any better option according to this scenario?
Any suggestion will be appreciated.
IMHO sequence is the best approach because it gives you more flexibility although you may also use identity (auto-increment) column. I think it postgres it is called serial and there is also a way to store ids in sepearate table. To address these 3 approach you may use
appropriately :
#GeneratedValue(strategy=GenerationType.TABLE)
#GeneratedValue(strategy=GenerationType.SEQUENCE)
#GeneratedValue(strategy=GenerationType.IDENTITY)
As for your previous question whether it is good to use single sequence for all tables. I wouldn't recommend this approach becasue db must assert that all sequence numbers are unique that is why each sequence generated value needs to be synchronized by the db server. If you have single sequence per db it may cause performace issues when multiple requests from multiple tables asks for next id value. I would rather recommend to have single sequence per table.
While I am not sure if there is a better alternative than using a sequence, I am pretty sure that you would want to look at using StatelessSession if this is just for gathering data. You can get rid of all the overhead for e.g 1st level cache, transactional write-behind etc