I need for a particular business scenario to set a field on an entity (not the PK) a number from a sequence (the sequence has to be a number between min and max
I defined the sequence like this :
CREATE SEQUENCE MySequence
MINVALUE 65536
MAXVALUE 4294967296
START WITH 65536
INCREMENT BY 1
CYCLE
NOCACHE
ORDER;
In Java code I retrieve the number from the sequence like this :
select mySequence.nextval from dual
My question is :
If I call this "select mySequence.nextval from dual" in a transaction and in the same time in another transaction same method is called (parallel requests) it is sure that the values returned by the sequence are different ?
Is not possible to have like read the uncommitted value from the first transaction ?
Cause let's say I would have not used sequence and a plain table where I would increment myself the sequence, then the transaction 2 would have been able to read same value if the trasactinalitY was the default "READ COMMITTED".
The answer is NO.
Oracle guarantees that numbers generated by sequence are different. Even if parallel requests are issued, RAC environment or rollback and commits are mixed.
Sequences have nothing to do with transactions.
See here the docs:
Use the CREATE SEQUENCE statement to create a sequence, which is a
database object from which multiple users may generate unique
integers. You can use sequences to automatically generate primary key
values.
When a sequence number is generated, the sequence is incremented,
independent of the transaction committing or rolling back. If two
users concurrently increment the same sequence, then the sequence
numbers each user acquires may have gaps, because sequence numbers are
being generated by the other user. One user can never acquire the
sequence number generated by another user. After a sequence value is
generated by one user, that user can continue to access that value
regardless of whether the sequence is incremented by another user.
Sequence numbers are generated independently of tables, so the same
sequence can be used for one or for multiple tables. It is possible
that individual sequence numbers will appear to be skipped, because
they were generated and used in a transaction that ultimately rolled
back. Additionally, a single user may not realize that other users are
drawing from the same sequence.
Oracle guarantees sequence numbers will be different. Even if your transaction is rolled back, the sequence is 'used' and not reissued to another query.
Edit: Adding additional information after requirements around "no gaps" were stated in comments by Cris
If your requirements are for a sequence of numbers without gaps then oracle sequences will probably not be a suitable solution, as there will be gaps when transactions roll back, or when the database restarts or any other number of scenarios.
Sequences are primarily intended as a high performance generation tool for unique numbers (e.g. primary keys) without regard to gaps and transaction context constraints.
If your design / business / audit requirements need to account for every number then you would need instead to design a solution that uses a predetermined number within the transaction context. This can be tricky and prone to performance / locking issues in a multi-threaded environment. It would be better to try to redefine your requirement so that gaps don't matter.
sequence.nextval never returns the same value (before cycled) for the concurrent request. Perhaps you should check the following URL:
http://docs.oracle.com/cd/B19306_01/server.102/b14220/schema.htm#sthref883
Unfortunately you have to implement you're 'own wheel' - transactional sequence. It is rather simple - just create the table like sequence_name varchar2, value, min_value number, max_value number, need_cycle char and mess around 'select value into variable from your sequence table for update wait (or nowait - it depends from your scenario)'. After it issue update set value = variable from previous step + 1 where sequence_name = the name of your sequence and issue the commit statement from client side. That's it.
Related
Example:
suppose that entity E has id generated by sequence e_seq
suppose that value of sequence is initially 0 on the database, and increment is configured to be 50
when hibernate starts, it gets the next value of the sequence (i.e. 0+50=50) and keeps an internal cache of the available values (i.e. those in the interval 0-50)
as long as the cache has available values, no further requests to the dbms are performed to get next value of sequence
only after you create 50 instances of entity E the 50 ids are consumed and hibernate asks the next value to the dbms.
suppose that the hibernate cache has still 50 ids available
suppose that a low-level procedure (like data migrations) inserts let's say 100 entities of type E in the database using SQL statements (not using hibernate APIs), with ids from 1 to 100 and then resets the sequence value to 100
if application tries to create a new entity from its APIs, it will use an id taken from the hibernate cache but which has already being used by the low-level procedure, hence causing a duplicate id exception
I need therefore to find a way to tell Hibernate to "reset its ids cache", or in other words "force hibernate to contact again the dbms to get the current sequence value".
a low-level procedure [...] inserts let's say 100 entities [...] with ids from 1 to 100
Why is that low-level procedure generating the IDs on its own? Why is it NOT using the sequence?
The whole point of Hibernate's pooled and pooled-lo ID generating mechanisms, which you appear to be using (and definitely should, if you're not), is to be able to safely cache IDs even on the face of any external processes making use of the sequence outside of Hibernate's control.
If that external process used the sequence too, your problem would disappear, since none of Hibernate's cached values would get used; and the next batch of cached values would start from whatever sequence value was last generated by the external process, avoiding conflicts:
Hibernate caches values 0-49. sequence.NEXTVAL would be 50.
External process inserts 100 rows. sequence.NEXTVAL would be 5050.
Hibernate ends up using all cached values, and asks for the next sequence value.
Hibernate caches values 5050-5099. sequence.NEXTVAL would be 5100.
Etc.
The solution to your issue, assuming you're using Hibernate's pooled(-lo) ID strategy, is not to disable or reset Hibernate's cache and hinder your application performance; the solution is to make any external processes use NEXTVAL() too to generate the appropriate IDs for the entities when inserting data into that table, instead of providing their own values.
Concerns:
"But then I would end up with gaps in my IDs!"
So what?
There's no problem whatsoever in your ID column having gaps. Your goal here is avoiding ID conflicts and ensuring that your application does not make 2 trips to DB (one for the sequence, one for the actual insert) every time you create an entity. If not having a neat, perfectly sequential set of IDs is the price to pay for that, so be it! Quite a deal, if you ask me ;)
"But then entities that were created later using Hibernate's cached values would have a lower ID than those created by the external process before!"
So what?
The primary goal of having an ID column is to be able to uniquely identify a row via a single value. Discerning order of creation should not be a factor in how you manage your ID values; a timestamp column is better suited for that.
"But the ID value would grow up too fast! I just inserted 50 rows and it's already by the thousands! I'll run out of numbers!"
Ok, legitimate concern here. But if you're using sequences, chances are you're using either Oracle or PostgreSQL, maybe SQL Server. Am I right?
Well, PostgreSQL's MAXVALUE for a bigint sequence is 9223372036854775807. Same goes for SQL Server. If your process inserted a new row each millisecond non-stop, it would still take it 5 million years to reach the end of the sequence. Oracle's MAXVALUE for a sequence is 999999999999999999999999999, several orders of magnitude greater than that.
So... As long as the datatype of your ID column and sequence is aptly chosen, you're safe on that regard.
Have you tried to clear the current session and create a new one?
This forces Hibernate to re-query the database for the current sequence value.
In other words you can use the method Session.flush() and Session.clear():
Session session = sessionFactory.openSession();
Transaction transaction = session.beginTransaction();
// Perform some operations that use the id cache
session.flush();
session.clear();
// Perform some more operations that use the id cache
transaction.commit();
session.close();
Or you could use EntityManager.refresh() which will refresh the state of the instance from the database, and in the process, update the internal cache with the current sequence value:
EntityManager em = entityManagerFactory.createEntityManager();
em.getTransaction().begin();
// Perform some operations that use the id cache
em.refresh(entity);
// Perform some more operations that use the id cache
em.getTransaction().commit();
em.close();
May this link will help
https://www.baeldung.com/hibernate-identifiers#3-sequence-generation
#Entity
public class User {
#Id
#GeneratedValue(generator = "sequence-generator")
#GenericGenerator(
name = "sequence-generator",
strategy = "org.hibernate.id.enhanced.SequenceStyleGenerator",
parameters = {
#Parameter(name = "sequence_name", value = "user_sequence"),
#Parameter(name = "initial_value", value = "4"),
#Parameter(name = "increment_size", value = "1")
}
)
private long userId;
// ...
}
In my springboot application, I noticed one strange issue when inserting new rows.
My ids are generated by sequence, but after I restart the application it starts from 21.
Example:
First launch, I insert 3 rows - ids generated by sequence 1,2,3
After restart second launch, I insert 3 rows ids generated from 21. So ids are 21,22 ...
Every restart It increased to 20. - This increasing pattern always 20
Refer my database table (1,2 after restart 21)
My JPA entity
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
#Column(unique = true, nullable = false)
private Long id;
I tried some stackoverflow solutions, it's not working
I tried this, not working
spring.jpa.properties.hibernate.id.new_generator_mappings=false
I want to insert rows by sequence like 1,2,3,4. Not like this 1,2,21,22, How to resolve this problem?
Although I think the question comments already provide all the information necessary to understand the problem, please, let me try explain some things and try fixing some inaccuracies.
According to your source code you are using the IDENTITY id generation strategy:
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
#Column(unique = true, nullable = false)
private Long id;
You are using an Oracle database and this is a very relevant information for the question.
Support for IDENTITY columns was introduced in Oracle 12c, probably Release 1, and in Hibernate version - I would say 5.1 although here in SO is indicated that you need at least - 5.3.
Either way, IDENTITY columns in Oracle are supported by the use of database SEQUENCEs: i.e., for every IDENTITY column a corresponding sequence is created. As you can read in the Oracle documentation this explain why, among others, all the options for creating sequences can be applied to the IDENTITY column definition, like min and max ranges, cache size, etc.
By default a sequence in Oracle has a cache size of 20 as indicated in a tiny note in the aforementioned Oracle documentation:
Note: When you create an identity column, Oracle recommends that you
specify the CACHE clause with a value higher than the default of 20 to
enhance performance.
And this default cache size is the reason that explains why you are obtaining this non consecutive numbers in your id values.
This behavior is not exclusive to Hibernate: please, just issue a simple JDBC insert statement or SQL commands with any suitable tool and you will experiment the same.
To solve the issue create your table indicating NOCACHE for your IDENTITY column:
CREATE TABLE your_table (
id NUMBER GENERATED BY DEFAULT ON NULL AS IDENTITY NOCACHE,
--...
)
Note you need to use NOCACHE and not CACHE 0 as indicated in the question comments and now in a previous version of other answers, which is an error because the value for the CACHE option should be at least 2.
Probably you could modify your column without recreating the whole table as well:
ALTER TABLE your_table MODIFY (ID GENERATED BY DEFAULT ON NULL AS IDENTITY NOCACHE);
Having said all that, please, be aware that in fact the cache mechanism is an optimization and not a drawback: in the end, and this is just my opinion, those ids are only non natural assigned IDs and, in a general use case, the cache benefits outweigh the drawbacks.
Please, consider read this great article about IDENTITY columns in Oracle.
The provided answer related to the use of the hilo optimizer could be right but it requires explicitly using the optimizer in your id field declaration which seems not to be the case.
It is related to Hi/Lo algorithm that Hibernate uses for incrementing the sequence value. Read more in this example: https://www.baeldung.com/hi-lo-algorithm-hibernate.
This is an optimization used by Hibernate, which consumes some values from the DB sequence into a pool (Java runtime) and uses them while executing appropriate INSERT statements on the table. If this optimization is turned off and set allocationSize=1, then the desired behavior (no gaps in ids) is possible (with a certain precision, not always), but for the price of making two requests to DB for each INSERT.
Examples give the idea of what is going on in the upper level of abstraction.
(Internal implementation is more complex, but here we don't care)
Scenario: user makes 21 inserts during some period of time
Example 1 (current behavior allocationSize=20)
#1 insert: // first cycle
- need next MY_SEQ value, but MY_SEQ_PREFETCH_POOL is empty
- select 20 values from MY_SEQ into MY_SEQ_PREFETCH_POOL // call DB
- take it from MY_SEQ_PREFETCH_POOL >> remaining=20-1
- execute INSERT // call DB
#2-#20 insert:
- need next MY_SEQ value,
- take it from MY_SEQ_PREFETCH_POOL >> remaining=20-i
- execute INSERT // call DB
#21 insert: // new cycle
- need next MY_SEQ value, but MY_SEQ_PREFETCH_POOL is empty
- select 20 values from MY_SEQ into MY_SEQ_PREFETCH_POOL // call DB
- take it from MY_SEQ_PREFETCH_POOL >> remaining=19
- execute INSERT // call DB
Example 2 (current behavior allocationSize=1)
#1-21 insert:
- need next MY_SEQ value, but MY_SEQ_PREFETCH_POOL is empty
- select 1 value from MY_SEQ into MY_SEQ_PREFETCH_POOL // call DB
- take it from MY_SEQ_PREFETCH_POOL >> remaining=0
- execute INSERT // call DB
Example#1: total calls to DB is 23
Example#2: total calls to DB is 42
Manual declaration of the sequence in the database will not help in this case, because, for instance in this statement\
CREATE SEQUENCE ABC START WITH 1 INCREMENT BY 1 CYCLE NOCACHE;
we control only "cache" used in the DB internal runtime, which is not visible to Hibernate. It affects sequence gaps in situations when DB stopped and started again, and this is not the case.
When Hibernate consumes values from the sequence it implies that the state of the sequence is changed on DB side. We may treat it as hotel rooms booking: a company (Hibernate) booked 20 rooms for a conference in a hotel (DB), but only 2 participants arrived. Then 18 rooms will stay empty and cannot be used by other guests. In this case the "booking period" is forever.
More details on how to configure Hibernate work with sequences is here:
https://ntsim.uk/posts/how-to-use-hibernate-identifier-sequence-generators-properly
Here is a short answer for older version of Hibernate. Still it has relevant ideas:
https://stackoverflow.com/a/5346701/2774914
I have a column (certificate number) in DB, wherein I have to save the value of certificate number for 3 different products, each products series starts from a different number (1st product series starts from 001, 2nd product starts with 2000 and so on so forth), I have to update the same column in the table from the previously saved certificate number of that particular product series.is there any way I can achieve this using spring boot, JPA, with PostgreSQL?
Thanks in advance.
It's quite easy to do with PostgreSQL SEQUENCE.
First you create a set of required sequences, i.e.
CREATE SEQUENCE product1 START 1; // You may need to define max value as well
CREATE SEQUENCE product2 START 2000;
etc.
Then you can use nextval(sequence_name) in your SQL statements, i.e.
INSERT INTO table_name VALUES (nextval('product1'), 'bla-bla');
You can find more info here https://www.postgresql.org/docs/13/sql-createsequence.html
If you want to use JPA, then you need to look at two annotations.
In order to create a sequence for an entity you use
#SequenceGenerator(name="product2", initialValue=2000)
and on the field definition you use
#GeneratedValue(strategy=GenerationType.SEQUENCE, generator="product2")
Edit: I see the issue you may have with JPA, you can't choose which sequence to use for an operation.
I created a PostgreSQL sequence on a PostgreSQL 10.7 dB called markush_seq
I read from the seq
select nextval('markush_seq’)` )
using a java web service:
When I run the web service on eclipse (using java 1.8.161) or call the sequence direct from SQL developer, it works fine and the sequence increments by 1 each time eg:
http://localhost:8086/wipdbws/read-markush-seq
21767823690
21767823691
21767823692
However when I run the webservice on AWS (which uses java 1.8.252) and read from the seq using:
https://aws-location/wipdbws/read-markush-seq
I get the sequence number returned as eg:
21767823692
21767823702
21767823693
21767823703
21767823694
21767823704
The sequence in AWS appears to be a combination of 2 incrementing sequences, 10 apart.
It’s the same java code, the only thing that has changed is:
The location of the webservice
a. AWS – USWEST
b. Eclipse - London
The java version:
a. 1.8.161 in London
b. 1.8.252 in US WEST
The seq details are:
SELECT * FROM information_schema.sequences
where sequence_name='markush_seq';
select * from pg_sequences where sequencename='markush_seq';
Any suggestion appreciated.
Likely due to multiple sessions accessing the sequence and sequence cache settings.
Documentation says:
although multiple sessions are guaranteed to allocate distinct
sequence values, the values might be generated out of sequence when
all the sessions are considered. For example, with a cache setting of
10, session A might reserve values 1..10 and return nextval=1, then
session B might reserve values 11..20 and return nextval=11 before
session A has generated nextval=2. Thus, with a cache setting of one
it is safe to assume that nextval values are generated sequentially;
with a cache setting greater than one you should only assume that the
nextval values are all distinct, not that they are generated purely
sequentially. Also, last_value will reflect the latest value reserved
by any session, whether or not it has yet been returned by nextval.
I need advise regarding the pattern to use for the following problem.
There are many rows -- let us call it messages(identified by MSG_ID in DB)-- in a table which corresponds to a file. Means, the file has been split into many pieces and put into database.
So parts corresponding to file can be identified using a GROUP_ID column, and MSG_ID corresponds to individual message. The primary key is a combination of GROUP_ID and MSG_ID.
Now, each message consists of n number of logical records(which are typically payment instructions(k x 128 bytes of data)). Where current reading payment instruction ends can be said only after reading the next 128 characters. Also parts of a payment instruction can be in consecutive messages. Which means a complete payment instruction can be spread across end of MSG_ID n and start of MSG_ID n+1.
We are using spring batch to do the processing.
I have tried querying the table and writing all the records to a flat file one by one and start the spring batch from there.
I would like to know whether there's any pattern which I can use to achieve the requirement without using a flat file.
Like,
Read MSG_ID 1 and GROUP_ID "ABC" from db
Seperate the payment instructions and give each instruction to the processor.
When end of MSG_ID 1 is reached check whether the final record in hand form a Payment Instruction, if not read MSG_ID 2 and GROUP_ID "ABC" and append to the previous left over record.
Read till MSG_ID==N where N is known before starting the read process.
Is there any ItemReaders in Spring or Iterator patterns in java which I can use ?
To be more clear, there are patterns for handling, "IF your logical record is spread in multiple rows". Is there any pattern for "IF one row in DB contains 'M' number logical records, where M may not be an integer, use this type of Iterator or ItemReaders"
Thanks.
I believe you should write a custom Reader (processing logic) for your scenario. It certainly doesn't seem as a common case.
The algorithm you proposed seems OK. You should have no trouble writing a Reader which reads a complete payment instruction and hands it off for further processing.