We Keep Coding

pgloader - How to import a longblob as oid?

In a nutshell
How do you migrate a longblob from MySQL to Postgres using pgloader s.t. Hibernate is happy if the column is annotated #Lob and #Basic(fetch= FetchType.LAZY)?
Full story
So I'm migrating (or trying to, at least) a MySQL DB to postgres. And I'm now trying to move this table correctly:
My current pgloader script is fairly simple:
LOAD DATABASE
FROM mysql://foo:bar#localhost:3306/foobar
INTO postgresql://foo:bar#localhost:5432/foobar
CAST
type int to integer drop typemod,
type bigint with extra auto_increment to bigserial drop typemod,
type bigint to bigint drop typemod
ALTER TABLE NAMES MATCHING 'User' RENAME TO 'users'
ALTER TABLE NAMES MATCHING ~/./ SET SCHEMA 'public'
;
This is sufficient to load the data and have the foreign keys working.
The postgres table looks like this:
The File, however, is a java entity and its content is annotated #Lob:
#Entity
#Inheritance(strategy= InheritanceType.JOINED)
public class File extends BaseEntity {
#NotNull
private String name;
#Column
#Size(max = 4096)
private String description;
#NotNull
private String mimeType;
#Lob
#Basic(fetch= FetchType.LAZY)
private transient byte[] content;
...
}
which is why the application fails to connect to the migrated database with error:
Schema-validation: wrong column type encountered in column [content] in table [File];
found [bytea (Types#BINARY)], but expecting [oid (Types#BLOB)]
How do I get this migration to work?
I did try setting
spring.jpa.properties.hibernate.jdbc.use_streams_for_binary=false
as suggested in proper hibernate annotation for byte[] but that didn't do anything.

Hm ... I guess I can just create blobs after the fact, as suggested by Migrate PostgreSQL text/bytea column to large object?
Meaning the migration script will get an extension:
LOAD DATABASE
FROM mysql://foo:bar#localhost:3306/foobar
INTO postgresql://foo:bar#localhost:5432/foobar
CAST
type int to integer drop typemod,
type bigint with extra auto_increment to bigserial drop typemod,
type bigint to bigint drop typemod
ALTER TABLE NAMES MATCHING 'User' RENAME TO 'users'
ALTER TABLE NAMES MATCHING ~/./ SET SCHEMA 'public'
AFTER LOAD DO
$$
ALTER TABLE file RENAME COLUMN content TO content_bytes;
$$,
$$
ALTER TABLE file ADD COLUMN content OID;
$$,
$$
UPDATE file SET
content = lo_from_bytea(0, content_bytes::bytea),
content_bytes = NULL
;
$$,
$$
ALTER TABLE file DROP COLUMN content_bytes
$$
;

Hibernate Generation Type for SQL Server and Others

I wrote program that uses different databases like sql server, oracle etc. My problem is that I can't handle GenerationType and insert correct row into table. Using GenerationType.AUTO and hibernate.id.new_generator_mappings := false in sql server, my program is able to insert new row into table, but ID is always null, same problem is when GenerationType is IDENTITY.
I tried to add auto-incrementation only for sql server, but Liquibase yells at me that it's not supported for mssql. When I use Sequences for Oracle as well SQL Server my program is trying to get "next value" from generator but it cannot and do infinite loop. Even if I set default value for ID it won't increment this value.
Thats my code :
#Id
#GeneratedValue(strategy = GenerationType.AUTO, generator = "name")
#SequenceGenerator(name = "name", sequenceName = "SEQ", allocationSize = 1)
private Long id;
I would like to be able to add auto-incrementing indices into table and it should work for SQL Server databases and I don't want to use Table strategy for generation because it needs additional table in db.

Problem solved. I add condition in Liquibase xml file that checks whether db is mssql type and if it's true script drops ID column and adds it with IDENTITY(1,1) option.
The only problem is that now I have to switch aforementioned "hibernate.id.new_generator_mappings" setting.

Force hibernate to leave id empty

So I am using Postgres and Hibernate 4.2.2 and with entity like this
#Entity(name = "Users")
#Check(constraints = "email ~* '^[A-Za-z0-9._%-]+#[A-Za-z0-9.-]+[.][A-Za-z]+$'")
#DynamicInsert
public class Users {
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
#Column(name = "id_user",unique = true)
#Index(name = "user_pk")
private Integer idUser;
Hibernate still inserts some id that is already in the table, instead of leaving it emtpy for the database to fill it in. Also hibernate forces ids based on its cache not even checking the database whether it has the lates id.
How can I force it so I can leave id blank and let the database insert it?
First I thought it was because I was using int and that int is by default 0 but even when using object it just forces the id there from its cache.
So my goal is to let the database fill the ids instead of hibernate or at least Hibernate before filling it in to check the database for id first.

So the error I was getting wasCaused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "users_pkey" Detail: Key (id_user)=(1) already exists.
And it wasn't caused by Hibernate and caching but by import of data at creation of database, where I inserted with given ids eg: INSERT INTO users(id_user,email,password,tag) VALUES (1,'a#b.c','***','Adpleydu');
and the sequence for generating wasn't updated so if I inserted with pure SQL via console I got the same error.

Seeding the data is the problem. However you can still seed with pure sequal and have the sequence "keep up".
1) Assure your primary key is of type SERIAL.
CREATE TABLE table_name(
id SERIAL
);
2) Add this 'setval' line to assure the sequence is updated.
select setval('table_name_id_seq',COALESCE((select max(id) + 1 from table_name), 1));
Reference:
https://www.postgresqltutorial.com/postgresql-serial/

How do I convert a column of large objects to long text?

I'm using Spring Boot with Hibernate, JPA and PostgreSQL. I'm wanting to convert database large objects into text content. Previously I was defining my long text in my JPA entity as #Lob:
#Lob
String description;
I then discovered that often problems are created using #Lob's and decided to rather change them to:
#Type(type="org.hibernate.type.StringClobType")
String description;
Which is represented in the database as a text type. Unfortunately, now the reference numbers (oid's) of the previous large objects are stored in my rows instead of the actual content. For example:
id | description
---------------------
1 | 463784 <- This is a reference to the object rather than the content
instead of:
id | description
---------------------
1 | Once upon a time, in a galaxy...
My question is now that we have thousands of rows of data in the database, how do I write a function or perform a query to replace the large object id with the actual text content stored in the large object?

Special thanks to #BohuslavBurghardt for pointing me to this answer. For your convenience:
UPDATE table_name SET column_name = lo_get(cast(column_name as bigint))

I needed some additional conversion:
UPDATE table_name SET text_no_lob = convert_from(lo_get(text::oid), 'UTF8');

I had the same problem with Spring, Postgres and JPA (Hibernate). I had a payload field that was like below
#NotBlank
#Column(name = "PAYLOAD")
private String payload;
I wanted to change the data type to text to support large data. So I used #Lob and I got the same error. To resolve that I first changed my field in my Entity like below:
#NotBlank
#Column(name = "PAYLOAD")
#Lob
#Type(type = "org.hibernate.type.TextType")
private String payload;
And because my data in this column was some scalar(Number) I have changed it to normal text with below command in Postgres:
UPDATE MYTABLE SET PAYLOAD = lo_get(cast(PAYLOAD as bigint))
Thanks a lot #Sipder.

Hibernate: #GeneratedValue(strategy = GenerationType

I am using DB2 for my application. I run some insertion script after creating database. That insertion script generates records in table with id's given in insertion script.
Suppose for abc table insertion script creates a record with id = 3. As id’s are set to auto generated in hibernate so while saving third record from application I got exception.
Caused by: com.ibm.websphere.ce.cm.DuplicateKeyException: One or
more values in the INSERT statement, UPDATE statement, or foreign
key update caused by a DELETE statement are not valid
because the primary key, unique constraint or unique
index identified by "1" constrains table
I am using #GeneratedValue(strategy = GenerationType.AUTO)
What strategy = GenerationType I should use to overcome this problem.

There are issues with certain Databases and Hibernate when you use GenerationType.IDENTITY. Try using a sequence and explicitlly configure everything for it:
#Id
#SequenceGenerator(name = "DEPARTMENT_ID_GENERATOR", sequenceName="department_sequence", allocationSize=100)
#GeneratedValue(strategy=GenerationType.SEQUENCE, generator = "DEPARTMENT_ID_GENERATOR")
#Column(unique = true, nullable = false)
protected Long id;

For DB2 #GeneratedValue(strategy = GenerationType.IDENTITY) should work correctly.

If ids are provided in file then you don't need #GeneratedValue at all as there is no id to generate. And make sure to clean database as #SjB suggested.
Also, without knowing much about DB2, the error message suggests that there may be other violation than just duplicate id on insert. Are there any foreign keys involved?

Nothing works except this query.
alter table TABLE_NAME alter column ID set GENERATED BY DEFAULT RESTART WITH 10000;
DB2 should choose available ID itself but not doing so.