I've been trying for two days now to store an array list with about six million entries in my Postgres database with Spring-Data-JPA.
The whole thing works, but it's very slow. I need about 27 minutes for everything.
I've already played around with the batch size, but that didn't bring much success. I also noticed that saving takes longer and longer the bigger the table gets. Is there a way to speed it up ?
I've done the whole thing with SQLite before, there I only needed about 15 seconds for the same amount.
My Entity
#Data
#Entity
#Table(name = "commodity_prices")
public class CommodityPrice {
#Id
#Column( name = "id" )
#GeneratedValue( strategy = GenerationType.SEQUENCE )
private long id;
#Column(name = "station_id")
private int station_id;
#Column(name = "commodity_id")
private int commodity_id;
#Column(name = "supply")
private long supply;
#Column(name = "buy_price")
private int buy_price;
#Column(name = "sell_price")
private int sell_price;
#Column(name = "demand")
private long demand;
#Column(name = "collected_at")
private long collected_at;
public CommodityPrice( int station_id, int commodity_id, long supply, int buy_price, int sell_price, long demand,
long collected_at ) {
this.station_id = station_id;
this.commodity_id = commodity_id;
this.supply = supply;
this.buy_price = buy_price;
this.sell_price = sell_price;
this.demand = demand;
this.collected_at = collected_at;
}
}
My insert Class
#Slf4j
#Component
public class CommodityPriceHandler {
#Autowired
CommodityPriceRepository commodityPriceRepository;
#Autowired
private EntityManager entityManager;
public void inserIntoDB() {
int lineCount = 0;
List<CommodityPrice> commodityPrices = new ArrayList<>( );
StopWatch stopWatch = new StopWatch();
stopWatch.start();
try {
Reader reader = new FileReader( DOWNLOAD_SAVE_PATH + FILE_NAME_COMMODITY_PRICES );
Iterable<CSVRecord> records = CSVFormat.EXCEL.withFirstRecordAsHeader().parse( reader );
for( CSVRecord record : records ) {
int station_id = Integer.parseInt( record.get( "station_id" ) );
int commodity_id = Integer.parseInt( record.get( "commodity_id" ) );
long supply = Long.parseLong( record.get( "supply" ) );
int buy_price = Integer.parseInt( record.get( "buy_price" ) );
int sell_price = Integer.parseInt( record.get( "sell_price" ) );
long demand = Long.parseLong( record.get( "demand" ) );
long collected_at = Long.parseLong( record.get( "collected_at" ) );
CommodityPrice commodityPrice = new CommodityPrice(station_id, commodity_id, supply, buy_price, sell_price, demand, collected_at);
commodityPrices.add( commodityPrice );
if (commodityPrices.size() == 1000){
commodityPriceRepository.saveAll( commodityPrices );
commodityPriceRepository.flush();
entityManager.clear();
commodityPrices.clear();
System.out.println(lineCount);
}
lineCount ++;
}
}
catch( IOException e ) {
log.error( e.getLocalizedMessage() );
}
commodityPriceRepository.saveAll( commodityPrices );
stopWatch.stop();
log.info( "Successfully inserted " + lineCount + " lines in " + stopWatch.getTotalTimeSeconds() + " seconds." );
}
}
My application.properties
# HIBERNATE
spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.PostgreSQLDialect
spring.jpa.properties.hibernate.jdbc.lob.non_contextual_creation=true
spring.jpa.hibernate.ddl-auto = update
spring.jpa.properties.hibernate.jdbc.batch_size=1000
spring.jpa.properties.hibernate.order_inserts=true
While you are doing your insert in batch, your sequence generation strategy still requires you to issue one statement for each record you insert. Thus, for a batch size of 1000 records you issue 1001 statements, which is clearly not what is expected.
My recommendations:
enable sql logging to see what statements are sent to your db. I personally use datasource-proxy, but use anything you are happy with.
modify your sequence generator. At a minimum, use
#Id
#Column( name = "id" )
#GeneratedValue(generator = "com_pr_generator", strategy = GenerationType.SEQUENCE )
#SequenceGenerator(name="com_pr_generator", sequenceName = "book_seq", allocationSize=50)
private long id;
Read about different generation strateges and fine tune your sequence generator.
A beginner’s guide to Hibernate enhanced identifier generators
Hibernate pooled and pooled-lo identifier generators
Related
I have a dataframe named timeDF which has the schema below:
root
|-- Id: long (nullable = true)
|-- Model: timestamp (nullable = true)
|-- Prevision: timestamp (nullable = true)
I want to add a new row at the end of timeDF by transforming two Calendar objects c1 & c2 to Timestamp. I know I can do it by first converting them to Timestamp like so :
val t1 = new Timestamp(c1.getTimeInMillis)
val t2 = new Timestamp(c2.getTimeInMillis)
However, I can't figure out how I then write those variables to timeDF as a new row, and how to let spark increase the Id column value ?
Should I create a List object with t1 and t2 and make a temporary dataframe from this list to then union the two dataframes ? If so how do I manage the Id column ? Isn't it too much a mess for such a simple operation ?
Can someone explain me please ?
Thanks.
Here is a solution you can try, in a nutshell:
Ingest your file.
Create a new dataframe with your data and unionByName().
Correct the id.
Clean up.
Create the extra record
First you create the extra record from scratch. As you mix several types, I used a POJO, here is the code:
List<ModelPrevisionRecord> data = new ArrayList<>();
ModelPrevisionRecord b = new ModelPrevisionRecord(
-1L,
new Timestamp(System.currentTimeMillis()),
new Timestamp(System.currentTimeMillis()));
data.add(b);
Dataset<ModelPrevisionRecord> ds = spark.createDataset(data,
Encoders.bean(ModelPrevisionRecord.class));
timeDf = timeDf.unionByName(ds.toDF());
ModelPrevisionRecord is a very basic POJO:
package net.jgp.labs.spark.l999_scrapbook.l000;
import java.sql.Timestamp;
public class ModelPrevisionRecord {
public long getId() {
return id;
}
public void setId(long id) {
this.id = id;
}
public Timestamp getModel() {
return model;
}
public void setModel(Timestamp model) {
this.model = model;
}
public Timestamp getPrevision() {
return prevision;
}
public void setPrevision(Timestamp prevision) {
this.prevision = prevision;
}
private long id;
private Timestamp model;
private Timestamp prevision;
public ModelPrevisionRecord(long id, Timestamp model, Timestamp prevision) {
this.id = id;
this.model = model;
this.prevision = prevision;
}
}
Correct the Id
The id is -1, so the id is to create a new column, id2, with the right id:
timeDf = timeDf.withColumn("id2",
when(
col("id").$eq$eq$eq(-1), timeDf.agg(max("id")).head().getLong(0)+1)
.otherwise(col("id")));
Cleanup the dataframe
Finally, clean up your dataframe:
timeDf = timeDf.drop("id").withColumnRenamed("id2", "id");
Important notes
This solution will only work if you add one record at a time, otherwise, you will end up having the same id.
You can see the whole example here: https://github.com/jgperrin/net.jgp.labs.spark/tree/master/src/main/java/net/jgp/labs/spark/l999_scrapbook/l000, it might be easier to clone...
If your first dataframe can be sorted by ID and you need to add rows one by one you can find maximum ID in your list:
long max = timeDF.agg(functions.max("Id")).head().getLong(0);
and then increment and add it to your dataframe by Union. To do this, follow the following example which age can act like id. people.json is a file in spark examples.
Dataset<Row> df = spark.read().json("H:\\work\\HadoopWinUtils\\people.json");
df.show();
long max = df.agg(functions.max("age")).head().getLong(0);
List<Row> rows = Arrays.asList(RowFactory.create(max+1, "test"));
StructType schema = DataTypes.createStructType(Arrays.asList(
DataTypes.createStructField("age", DataTypes.LongType, false, Metadata.empty()),
DataTypes.createStructField("name", DataTypes.StringType, false, Metadata.empty())));
Dataset<Row> df2 = spark.createDataFrame(rows, schema);
df2.show();
Dataset<Row> df3 = df.union(df2);
df3.show();
I tried this but I don't know why, when printing the table saved, it only keep the last 2 rows, all others being deleted.
This is how I init the delta table :
val schema = StructType(
StructField("Id", LongType, false) ::
StructField("Model", TimestampType, false) ::
StructField("Prevision", TimestampType, false) :: Nil
)
var timestampDF = spark.createDataFrame(sc.emptyRDD[Row], schema)
val write_format = "delta"
val partition_by = "Model"
val save_path = "/mnt/path/to/folder"
val table_name = "myTable"
spark.sql("DROP TABLE IF EXISTS " + table_name)
dbutils.fs.rm(save_path, true)
timestampDF.write.partitionBy(partition_by)
.format(write_format)
.save(save_path)
spark.sql("CREATE TABLE " + table_name + " USING DELTA LOCATION '" + save_path + "'")
And this how I add a new item to it
def addTimeToData(model: Calendar, target: Calendar): Unit = {
var timeDF = spark.read
.format("delta")
.load("/mnt/path/to/folder")
val modelTS = new Timestamp(model.getTimeInMillis)
val targetTS = new Timestamp(target.getTimeInMillis)
var id: Long = 0
if (!timeDF.head(1).isEmpty) {
id = timeDF.agg(max("Id")).head().getLong(0) + 1
}
val newTime = Arrays.asList(RowFactory.create(id, modelTS, targetTS))
val schema = StructType(
StructField("Id", LongType, false) ::
StructField("Model", TimestampType, false) ::
StructField("Prevision", TimestampType, false) :: Nil
)
var newTimeDF = spark.createDataFrame(newTime, schema)
val unionTimeDF = timeDF.union(newTimeDF)
timeDF = unionTimeDF
unionTimeDF.show
val save_path = "/mnt/datalake/Exploration/Provisionning/MeteoFrance/Timestamps/"
val table_name = "myTable"
spark.sql("DROP TABLE IF EXISTS " + table_name)
dbutils.fs.rm(save_path, true)
timeDF.write.partitionBy("Model")
.format("delta")
.save(save_path)
spark.sql("CREATE TABLE " + table_name + " USING DELTA LOCATION '" + save_path + "'")
}
I'm not very familiar with delta tables so I don't know if I can just use SQL on it to add values like so :
spark.sql("INSERT INTO 'myTable' VALUES (" + id + ", " + modelTS + ", " + previsionTS + ")");
And I don't if just putting the timestamps variable like so will work.
I am back with a bug/problem that came to sunlight now. Usually I test the local development and changes on an H2DB but as I know, this has to work on Oracle and MSSQL too.
Now testing on oracle again this problem occurred:
The Key COR_VIEWSETTINGSCOR_USERSETTINGS_FK0 and COR_VIEWSETTINGSCOR_USERSETTINGS_FK1 are generated automatic and are way too long for an oracle db.
To know how these keys are created I will now show you the entities UserSettings and UserViewSettings.
hint: you can overlook the entities and go further to the edits if they confuse you. maybe you can still help me.
UserSettings
/**
The Class UserSettings.
*/
#org.hibernate.envers.Audited
#DataObject( value = UserSettings.DATA_OBJECT_NAME )
#CRUDDefinition( supportsRead = true, supportsCreate = true, supportsUpdate = true, supportsDelete = true )
#Entity( name = UserSettings.DATA_OBJECT_NAME )
#NamedQuery( name = UserSettings.DATA_OBJECT_NAME, query = "from userSettings e where e.name = :name" )
#javax.persistence.Inheritance( strategy = javax.persistence.InheritanceType.TABLE_PER_CLASS )
#AttributeOverrides( { #AttributeOverride( name = "id", column = #Column( name = "USERSETTINGS_ID" ) )
} )
#Table( name = "COR_USERSETTINGS", indexes = {
#javax.persistence.Index( name="COR_USERSETTINGS_FK0", columnList = "SETTINGSTYPE_ID" ),
#javax.persistence.Index( name="COR_USERSETTINGS_FK1", columnList = "USER_ID" ),
}
)
public class UserSettings extends NamedRevisionEntity implements NameSettingsType, NameSettings
{
/** The Constant serialVersionUID. */
private static final long serialVersionUID = 1L;
/** The Constant DATA_OBJECT_NAME. */
public static final String DATA_OBJECT_NAME = "userSettings";
#javax.persistence.Basic( fetch = javax.persistence.FetchType.EAGER, optional = false )
#Column( name = "SETTINGS", nullable = false, unique = false, insertable = true, updatable = true )
#javax.persistence.Lob
private java.lang.String settings;
#javax.persistence.ManyToOne( fetch = javax.persistence.FetchType.EAGER, optional = false )
#javax.persistence.JoinColumn( name = "SETTINGSTYPE_ID", nullable = false, unique = false, insertable = true, updatable = true )
private SettingsType settingsType;
#javax.persistence.ManyToOne( fetch = javax.persistence.FetchType.EAGER, optional = true )
#javax.persistence.JoinColumn( name = "USER_ID", nullable = true, unique = false, insertable = true, updatable = true )
private User user;
public SettingsType getSettingsType()
{
return settingsType;
}
public void setSettingsType( SettingsType settingsType )
{
this.settingsType = settingsType;
}
public User getUser()
{
return user;
}
public void setUser( User user )
{
this.user = user;
}
public java.lang.String getSettings()
{
return settings;
}
public void setSettings( java.lang.String settings )
{
this.settings = settings;
}
#Override
public String getDataObjectName()
{
return DATA_OBJECT_NAME;
}
#Override
public String toString()
{
StringBuilder builder = new StringBuilder( super.toString() );
builder.append( ", " );
try
{
builder.append( ToStringUtils.referenceToString( "settingsType", "SettingsType", this.settingsType ) );
}
catch( Exception ex )
{
builder.append( ex.getClass().getName() );
builder.append( ": " );
builder.append( ex.getMessage() );
}
builder.append( ", " );
try
{
builder.append( ToStringUtils.referenceToString( "user", "User", this.user ) );
}
catch( Exception ex )
{
builder.append( ex.getClass().getName() );
builder.append( ": " );
builder.append( ex.getMessage() );
}
builder.append( "]" );
return builder.toString();
}
}
UserViewSettings
/**
The Class UserViewSettings.
*/
#org.hibernate.envers.Audited
#DataObject( value = UserViewSettings.DATA_OBJECT_NAME )
#CRUDDefinition( supportsRead = true, supportsCreate = true, supportsUpdate = true, supportsDelete = true )
#Entity( name = UserViewSettings.DATA_OBJECT_NAME )
#AttributeOverrides( { #AttributeOverride( name = "id", column = #Column( name = "VIEWSETTINGS_ID" ) )
} )
#Table( name = "COR_VIEWSETTINGS", uniqueConstraints = {
#javax.persistence.UniqueConstraint( name="COR_VIEWSETTINGS_UNQ1", columnNames = { "NAME", "SETTINGSTYPE_ID", "VIEW_NAME", "VIEWTYPE_ID" } ),
}
, indexes = {
#javax.persistence.Index( name="COR_VIEWSETTINGS_FK0", columnList = "VIEWTYPE_ID" ),
}
)
public class UserViewSettings extends UserSettings implements NameViewName, NameViewType
{
/** The Constant serialVersionUID. */
private static final long serialVersionUID = 1L;
/** The Constant DATA_OBJECT_NAME. */
public static final String DATA_OBJECT_NAME = "userViewSettings";
#javax.persistence.Basic( fetch = javax.persistence.FetchType.EAGER, optional = false )
#Column( name = "VIEW_NAME", nullable = false, unique = false, insertable = true, updatable = true )
private java.lang.String viewName;
#javax.persistence.ManyToOne( fetch = javax.persistence.FetchType.EAGER, optional = true )
#javax.persistence.JoinColumn( name = "VIEWTYPE_ID", nullable = true, unique = false, insertable = true, updatable = true )
private ViewType viewType;
public java.lang.String getViewName()
{
return viewName;
}
public void setViewName( java.lang.String viewName )
{
this.viewName = viewName;
}
public ViewType getViewType()
{
return viewType;
}
public void setViewType( ViewType viewType )
{
this.viewType = viewType;
}
#Override
public String getDataObjectName()
{
return DATA_OBJECT_NAME;
}
#Override
public String toString()
{
StringBuilder builder = new StringBuilder( super.toString() );
builder.append( ", " );
builder.append( "viewName" );
builder.append( "=" );
builder.append( this.viewName );
builder.append( ", " );
try
{
builder.append( ToStringUtils.referenceToString( "viewType", "ViewType", this.viewType ) );
}
catch( Exception ex )
{
builder.append( ex.getClass().getName() );
builder.append( ": " );
builder.append( ex.getMessage() );
}
builder.append( "]" );
return builder.toString();
}
}
Starting Wildfly 10.0.0 with Hibernate 5.2 and an Oracle 11 Database then results in the error that the automatic generated Keys COR_VIEWSETTINGSCOR_USERSETTINGS_FK0 and COR_VIEWSETTINGSCOR_USERSETTINGS_FK1 are naturally too long for the database.
I took a look at the NamingStrategies for Hibernate and even tried some but they didn't change the error for me.
How can I impact the generation of these keys?
EDIT:
So turning on DEBUG gave me this:
2016-11-29 09:22:03,190 DEBUG [org.hibernate.SQL] (ServerService Thread Pool -- 58) create index COR_USERSETTINGS_FK0 on COR_USERSETTINGS (SETTINGSTYPE_ID)
2016-11-29 09:22:03,190 DEBUG [org.hibernate.SQL] (ServerService Thread Pool -- 58) create index COR_USERSETTINGS_FK1 on COR_USERSETTINGS (USER_ID)
2016-11-29 09:22:03,190 DEBUG [org.hibernate.SQL] (ServerService Thread Pool -- 58) create index COR_VIEWSETTINGSCOR_USERSETTINGS_FK0 on COR_VIEWSETT INGS(SETTINGSTYPE_ID)
2016-11-29 09:22:03,190 DEBUG [org.hibernate.SQL] (ServerService Thread Pool -- 58) create index COR_VIEWSETTINGSCOR_USERSETTINGS_FK1 on COR_VIEWSETTINGS (USER_ID)
2016-11-29 09:22:03,190 DEBUG [org.hibernate.SQL] (ServerService Thread Pool -- 58) create index COR_VIEWSETTINGS_FK0 on COR_VIEWSETTINGS (VIEWTYPE_ID)
Now I found the Class ImplicitIndexNameSource in the package org.hibernate.boot.model.naming but the internet doesn't really give examples what I can do with this and it seems to be an empty class for a long since a long time.
EDIT 2:
The previous edit seems to be a wrong path. I found the place where these logs are created. It's StandardIndexExporter which gets called from SchemaCreatorImpl. So I need to dig even deeper into the framework but if somebody sees this. Is this the right path? Can I modify code so that He will do the thing I want? It seems to be the hbm2ddl that is the culprit since the index get's created in StandardIndexExport in these lines:
final String indexNameForCreation;
if ( dialect.qualifyIndexName() ) {
indexNameForCreation = jdbcEnvironment.getQualifiedObjectNameFormatter().format(
new QualifiedNameImpl(
index.getTable().getQualifiedTableName().getCatalogName(),
index.getTable().getQualifiedTableName().getSchemaName(),
jdbcEnvironment.getIdentifierHelper().toIdentifier( index.getName() )
),
jdbcEnvironment.getDialect()
);
}
else {
indexNameForCreation = index.getName();
}
final StringBuilder buf = new StringBuilder()
.append( "create index " )
.append( indexNameForCreation )
.append( " on " )
.append( tableName )
.append( " (" );
boolean first = true;
Iterator<Column> columnItr = index.getColumnIterator();
while ( columnItr.hasNext() ) {
final Column column = columnItr.next();
if ( first ) {
first = false;
}
else {
buf.append( ", " );
}
buf.append( ( column.getQuotedName( dialect ) ) );
}
buf.append( ")" );
return new String[] { buf.toString() };
I would appreciate help a lot. This is getting really frustrating
So I got it working.
Answering for future people that might find this and have the same issue.
The index key gets created by the dialect of oracle that hibernate is referrencing to. So what had to be done was implementing an custom OracleDialect that overrides the method getIndexExporter and points to the custom IndexExporter. In this IndexExporter you can then modify the way the keys are created. In my case I fixed the solution like this:
/**
* Gets the correct index name if it is a index for a TABLE_PER_CLASS inheritance and longer than
* 30 chars.
*
* #param index the index to decide for
* #return the correct index name
*/
private String getCorrectIndexName( Index index )
{
if ( index.getTable() instanceof DenormalizedTable && index.getName().length() > 30 )
{
String prefixedTable = index.getTable().getName();
String tableName = prefixedTable.substring( prefixedTable.indexOf( '_' ) + 1, prefixedTable.length() );
tableName = shortenName( tableName );
Iterator<Column> columnItr = index.getColumnIterator();
String reference;
if ( columnItr.hasNext() )
{
reference = extractReference( columnItr.next() );
}
else
{
/** backup strategy to prevent exceptions */
reference = shortenName( NamingHelper.INSTANCE.hashedName( index.getName() ) );
}
return tableName + "_" + reference;
}
return index.getName();
}
/**
* Extract the reference column of the index and hash the full name before shortening it with
* shortenName().
*
* #param index the index to extract the reference from.
* #return the reference with an appended _FK(hashedReference).
*/
private String extractReference( Column column )
{
String reference = column.getQuotedName( dialect );
String md5Hash = NamingHelper.INSTANCE.hashedName( reference );
md5Hash = md5Hash.substring( md5Hash.length() - 4, md5Hash.length() );
reference = shortenName( reference );
return reference + "_FK" + md5Hash;
}
/**
* Shorten the name to a maximum of 11 chars if it's longer.
*
* #param reference the reference to shorten
* #return the shortened string
*/
private static String shortenName( String reference )
{
if ( reference.length() > 11 )
{
return reference.substring( 0, 11 );
}
return reference;
}
this had to be called in the Overriden function getSqlCreateStrings. the changed lines look like this:
String indexName = getCorrectIndexName( index );
indexNameForCreation = jdbcEnvironment.getQualifiedObjectNameFormatter()
.format(
new QualifiedNameImpl( index.getTable().getQualifiedTableName().getCatalogName(),
index.getTable().getQualifiedTableName().getSchemaName(), jdbcEnvironment.getIdentifierHelper().toIdentifier( indexName ) ),
jdbcEnvironment.getDialect() );
I hope that helps someone.
I have the following Oracle table definition.
CREATE TABLE "SIAS"."OPERATION_REG"
(
"ID" NUMBER CONSTRAINT "CT_OPERATION_REG_ID" NOT NULL ENABLE,
"OPERATION_NAME" VARCHAR2(30 BYTE),
"APPLICATION_NAME" VARCHAR2(30 BYTE),
"EXECUTION_DATE" DATE,
"EXECUTION_USER" VARCHAR2(80 BYTE),
"RESULT" VARCHAR2(20 BYTE),
CONSTRAINT "PK_OPERATION_REG_ID" PRIMARY KEY ("ID") USING INDEX PCTFREE 10
INITRANS 2 MAXTRANS 255 COMPUTE STATISTICS NOLOGGING STORAGE(INITIAL 65536 NEXT
1048576 MINEXTENTS 1 MAXEXTENTS 2147483645 PCTINCREASE 0 FREELISTS 1 FREELIST
GROUPS 1 BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
TABLESPACE "SIAS_DAT" ENABLE
)
SEGMENT CREATION IMMEDIATE PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255
NOCOMPRESS LOGGING STORAGE
(
INITIAL 65536 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS 2147483645 PCTINCREASE
0 FREELISTS 1 FREELIST GROUPS 1 BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT
CELL_FLASH_CACHE DEFAULT
)
TABLESPACE "SIAS_DAT" ;
CREATE UNIQUE INDEX "SIAS"."IDX_OPERATION_REG_ID" ON "SIAS"."OPERATION_REG"
(
"ID"
)
PCTFREE 10 INITRANS 2 MAXTRANS 255 COMPUTE STATISTICS NOLOGGING STORAGE
(
INITIAL 65536 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS 2147483645 PCTINCREASE 0
FREELISTS 1 FREELIST GROUPS 1 BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT
CELL_FLASH_CACHE DEFAULT
)
TABLESPACE "SIAS_DAT" ;
CREATE OR REPLACE TRIGGER "SIAS"."BI_OPERATION_REG" BEFORE
INSERT ON OPERATION_REG REFERENCING NEW AS NEW OLD AS OLD FOR EACH ROW BEGIN
:NEW.ID := SEQ_OPERATION_REG.NEXTVAL;
EXCEPTION
WHEN OTHERS THEN
RAISE_APPLICATION_ERROR
(
-20255, 'ERROR EN TRIGGER BI_OPERATION_REG'
)
;
END;
/
ALTER TRIGGER "SIAS"."BI_OPERATION_REG" ENABLE;
I have this trigger enabled to auto generate the value of the ID column when a new row is being created.
create or replace
TRIGGER BI_OPERATION_REG BEFORE INSERT
ON OPERATION_REG
REFERENCING NEW AS NEW OLD AS OLD
FOR EACH ROW
BEGIN
:NEW.ID := SEQ_OPERATION_REG.NEXTVAL;
EXCEPTION
WHEN OTHERS
THEN
RAISE_APPLICATION_ERROR (-20255, 'ERROR EN TRIGGER BI_OPERATION_REG');
END;
This is the sequence definition to generate the values for ID
CREATE SEQUENCE "SIAS"."SEQ_OPERATION_REG" MINVALUE 1 MAXVALUE
999999999999999999999999999 INCREMENT BY 1 START WITH 37 NOCACHE NOORDER NOCYCLE ;
I have no control over the database, because the DBA team is out of my scope, so I have to deal with those definitions. I have created a JPA Entity that maps OPERATION_REG table. This is the ID property method mapping for column ID.
#Id
#GeneratedValue(strategy = GenerationType.AUTO, generator = "G1")
#SequenceGenerator(name = "G1", sequenceName = "SEQ_OPERATION_REG")
#Column(name = "ID")
public int getId() {
return id;
}
This is the full code of my entity mapping
import org.hibernate.annotations.GenericGenerator;
import javax.persistence.*;
import java.sql.Timestamp;
import java.util.Collection;
#Entity
#Table(name = "OPERATION_REG")
public class OperationRegEntity extends BaseEntity {
private int id;
private String operationName;
private String applicationName;
private Timestamp executionDate;
private String executionUser;
private String result;
private Collection<TokenRegEntity> tokenRegsById;
private Collection<TraceRegEntity> traceRegsById;
#Id
#GeneratedValue(generator="select-generator")
#GenericGenerator(name="select-generator", strategy="select", parameters = #org.hibernate.annotations.Parameter(name="key", value="ID"))
// #GeneratedValue(strategy = GenerationType.AUTO, generator = "G1")
// #SequenceGenerator(name = "G1", sequenceName = "SEQ_OPERATION_REG")
#Column(name = "ID")
public int getId() {
return id;
}
public void setId(int id) {
this.id = id;
}
#Basic
#Column(name = "OPERATION_NAME")
public String getOperationName() {
return operationName;
}
public void setOperationName(String operationName) {
this.operationName = operationName;
}
#Basic
#Column(name = "APPLICATION_NAME")
public String getApplicationName() {
return applicationName;
}
public void setApplicationName(String applicationName) {
this.applicationName = applicationName;
}
#Basic
#Column(name = "EXECUTION_DATE")
public Timestamp getExecutionDate() {
return executionDate;
}
public void setExecutionDate(Timestamp executionDate) {
this.executionDate = executionDate;
}
#Basic
#Column(name = "EXECUTION_USER")
public String getExecutionUser() {
return executionUser;
}
public void setExecutionUser(String executionUser) {
this.executionUser = executionUser;
}
#Basic
#Column(name = "RESULT")
public String getResult() {
return result;
}
public void setResult(String result) {
this.result = result;
}
#Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
OperationRegEntity that = (OperationRegEntity) o;
if (id != that.id) return false;
if (applicationName != null ? !applicationName.equals(that.applicationName) : that.applicationName != null)
return false;
if (executionDate != null ? !executionDate.equals(that.executionDate) : that.executionDate != null)
return false;
if (executionUser != null ? !executionUser.equals(that.executionUser) : that.executionUser != null)
return false;
if (operationName != null ? !operationName.equals(that.operationName) : that.operationName != null)
return false;
if (result != null ? !result.equals(that.result) : that.result != null) return false;
return true;
}
#Override
public int hashCode() {
int result1 = id;
result1 = 31 * result1 + (operationName != null ? operationName.hashCode() : 0);
result1 = 31 * result1 + (applicationName != null ? applicationName.hashCode() : 0);
result1 = 31 * result1 + (executionDate != null ? executionDate.hashCode() : 0);
result1 = 31 * result1 + (executionUser != null ? executionUser.hashCode() : 0);
result1 = 31 * result1 + (result != null ? result.hashCode() : 0);
return result1;
}
#OneToMany(mappedBy = "operationRegByOperationRegId")
public Collection<TokenRegEntity> getTokenRegsById() {
return tokenRegsById;
}
public void setTokenRegsById(Collection<TokenRegEntity> tokenRegsById) {
this.tokenRegsById = tokenRegsById;
}
#OneToMany(mappedBy = "operationRegByOperationRegId")
public Collection<TraceRegEntity> getTraceRegsById() {
return traceRegsById;
}
public void setTraceRegsById(Collection<TraceRegEntity> traceRegsById) {
this.traceRegsById = traceRegsById;
}
}
I have a problem, because, when I create a new object and persist it on the database, I follow this strategy
#Autowired
OperationRegService operationregservice;
#Transactional(propagation = Propagation.REQUIRES_NEW)
public OperationRegEntity createOperationReg(GenericRequestParameters parameters) {
OperationRegEntity oper = new OperationRegEntity();
oper.setApplicationName(parameters.getApplication());
oper.setExecutionUser(parameters.getApplicationUser());
oper.setOperationName(parameters.getSIASOperationName());
oper.setExecutionDate(new Timestamp(Calendar.getInstance().getTime().getTime()));
oper.setResult("INITIATED");
operationregservice.persist(oper);
return oper;
}
When I analyse the information of oper.getID(), the value is different from the actual value created in the database, in particular, always 1 point below. For example, the java entity has ID value of 34 and the table row entity has an ID value of 35, as if the sequence is getting called twice. Any ideas?
You shouldn't use a #SequenceGenerator, because that's used when you want Hibernate to call the sequence upon persisting an entity.
In your use case the database does the call so you need to use the select identifier generator strategy:
#Id
#GeneratedValue(generator="select-generator")
#GenericGenerator(name="select-generator",
strategy="select",
parameters = #org.hibernate.annotations.Parameter(name="key", value="ID")
)
#Column(name = "ID")
public int getId() {
return id;
}
Ok, I figure out the problem, and it was in the way the trigger generated the sequence. The key was to generate the sequence if no ID value was set already. That way, Hibernate will call the sequence, set the ID value, and the trigger will check if the value was set, if so, it will not call the sequence. If no value was set, then the trigger calls the sequence and sets the value
This is the valid trigger
create or replace
TRIGGER BI_OPERATION_REG BEFORE INSERT
ON OPERATION_REG
REFERENCING NEW AS NEW OLD AS OLD
FOR EACH ROW
BEGIN
IF :NEW.ID IS NULL THEN SELECT SEQ_OPERATION_REG.NEXTVAL INTO :NEW.ID FROM dual; END IF;
EXCEPTION
WHEN OTHERS
THEN
RAISE_APPLICATION_ERROR (-20255, 'ERROR EN TRIGGER BI_OPERATION_REG');
END;
What's the methodology of testing #SequenceGenerator from Hibernate? I want to be sure that every sequence is perfectly mapped, no mistake in spelling, and incrementing is done by 1. Is there any way to do this dynamically for all sequences?
Here's the sample of my sequence mapping:
#Column(name = "ADDRESS_ID", nullable = false, precision = 20)
#Id
#SequenceGenerator(name = "AddressSeq", sequenceName = "ADDRESS_SEQ", allocationSize = 1)
#GeneratedValue(strategy = GenerationType.SEQUENCE, generator = "AddressSeq")
private Long addressId;
You can use this answer to obtain a list of your entity classes:
List<ClassLoader> classLoadersList = new LinkedList<ClassLoader>();
classLoadersList.add(ClasspathHelper.contextClassLoader());
classLoadersList.add(ClasspathHelper.staticClassLoader());
reflections = new Reflections(
new ConfigurationBuilder()
.setScanners(new SubTypesScanner(false),
new ResourcesScanner())
.setUrls(
ClasspathHelper.forClassLoader(classLoadersList
.toArray(new ClassLoader[0])))
.filterInputsBy(
new FilterBuilder().include(FilterBuilder
.prefix("me.business.model"))));
And get your DDL with:
ClassPathResource cpr = new ClassPathResource("db/schema.sql");
schemaContent = new String(FileCopyUtils.copyToByteArray(cpr
.getInputStream())).toLowerCase();
And get the sequence for each class witH;
private String getSequenceName(Class<?> clazz) {
for (Field f : clazz.getDeclaredFields()) {
if (f.isAnnotationPresent(SequenceGenerator.class)) {
SequenceGenerator sg = f.getAnnotation(SequenceGenerator.class);
return sg.sequenceName();
}
}
return null;
}
The test is simple:
Set<Class<?>> entities = reflections.getSubTypesOf(Object.class);
for (Class<?> clazz : entities) {
String name = getSequenceName(clazz);
if (name == null)
continue;
if (!schemaContent.contains(name.toLowerCase())) {
fail("The clazz " + clazz.getSimpleName()
+ " has a sequence called: " + name
+ " and it doesn't exits");
}
}
You can see it here
If you want to see if it works, change the sequence name in one of your entitites and run the test.
I've got an Entity class named Fee. After performing an initial query, if more than one Fee is returned, I'd like to add some (WHERE) conditions only on the rows returned by the first query. Since a code snippet is better than a thousand words, here it is:
// Results of initial query
List<Fee> fees = queryFindFees.getResultList();
if (fees == null || fees.size() <= 0)
return null;
if (fees.size() == 1) {
Fee f = fees.get(0);
jpa.refresh(f);
return f;
}
// More than one fee found
String sqlBase = "SELECT f FROM Fee f WHERE f IN :fees";
String sqlComplete = sqlBase/* + " AND f.isValid = ?1"*/;
queryFindFees = jpa.createQuery(sqlComplete);
queryFindFees.setParameter("fees", fees);
// Commented out for test
// queryFindFees.setParameter(1, "Y");
List<Fee> specFees = queryFindFees.getResultList();
Now, since I'm actually asking for the EntityManager to return all the Fees already returned in the previous query, I would expect the same resultset. Instead the list specFees is always empty.
What am I doing wrong?
Thanks in advance,
Luca
EDIT 1: Details of Entity class Fee
#Entity
public class Fee implements Serializable {
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
#Column(unique = true, nullable = false)
private long id;
#Column(nullable = false, length = 1)
private String isValid;
...
#Override
public boolean equals(Object arg0) {
if (!(arg0 instanceof Fee))
return false;
Fee f0 = (Fee) arg0;
return f0.getId() == this.getId();
}
}
The generated SQL (sqlComplete variable) is as expected:
SELECT f FROM Fee f WHERE f IN :fees
EDIT 2: As suggested by Deepak, using the collection of IDs works:
String sqlBase = "SELECT f FROM Fee f WHERE f.id IN :feesIds";
String sqlComplete = sqlBase/* + " AND f.isValid = ?1"*/;
List<Long> feesIds = new ArrayList<Long>();
for (Fee f : fees) {
feesIds.add(f.getId());
}
queryFindFees = jpa.createQuery(sqlComplete);
queryFindFees.setParameter("feesIds", feesIds);
Using this code the query works (returns all the original Fees), however if possible I would like to avoid using the for cycle because the number of Fee instances may be very large...
Looks like your overridden equals method is creating problem .Make your id Long Object and use the below overridden method .Alternatively u can use this query also
SELECT f FROM Fee f WHERE f.id IN :fees
this time your fees parameter will contain list of ids of Fee objects
#Entity
public class Fee implements Serializable {
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
#Column(unique = true, nullable = false)
private Long id;
#Column(nullable = false, length = 1)
private String isValid;
...
#Override
public boolean equals(Object arg0) {
if (!(arg0 instanceof Fee))
return false;
Fee f0 = (Fee) arg0;
return f0.getId().equals(this.getId());
}
}