In a nutshell
How do you migrate a longblob from MySQL to Postgres using pgloader s.t. Hibernate is happy if the column is annotated #Lob and #Basic(fetch= FetchType.LAZY)?
Full story
So I'm migrating (or trying to, at least) a MySQL DB to postgres. And I'm now trying to move this table correctly:
My current pgloader script is fairly simple:
LOAD DATABASE
FROM mysql://foo:bar#localhost:3306/foobar
INTO postgresql://foo:bar#localhost:5432/foobar
CAST
type int to integer drop typemod,
type bigint with extra auto_increment to bigserial drop typemod,
type bigint to bigint drop typemod
ALTER TABLE NAMES MATCHING 'User' RENAME TO 'users'
ALTER TABLE NAMES MATCHING ~/./ SET SCHEMA 'public'
;
This is sufficient to load the data and have the foreign keys working.
The postgres table looks like this:
The File, however, is a java entity and its content is annotated #Lob:
#Entity
#Inheritance(strategy= InheritanceType.JOINED)
public class File extends BaseEntity {
#NotNull
private String name;
#Column
#Size(max = 4096)
private String description;
#NotNull
private String mimeType;
#Lob
#Basic(fetch= FetchType.LAZY)
private transient byte[] content;
...
}
which is why the application fails to connect to the migrated database with error:
Schema-validation: wrong column type encountered in column [content] in table [File];
found [bytea (Types#BINARY)], but expecting [oid (Types#BLOB)]
How do I get this migration to work?
I did try setting
spring.jpa.properties.hibernate.jdbc.use_streams_for_binary=false
as suggested in proper hibernate annotation for byte[] but that didn't do anything.
Hm ... I guess I can just create blobs after the fact, as suggested by Migrate PostgreSQL text/bytea column to large object?
Meaning the migration script will get an extension:
LOAD DATABASE
FROM mysql://foo:bar#localhost:3306/foobar
INTO postgresql://foo:bar#localhost:5432/foobar
CAST
type int to integer drop typemod,
type bigint with extra auto_increment to bigserial drop typemod,
type bigint to bigint drop typemod
ALTER TABLE NAMES MATCHING 'User' RENAME TO 'users'
ALTER TABLE NAMES MATCHING ~/./ SET SCHEMA 'public'
AFTER LOAD DO
$$
ALTER TABLE file RENAME COLUMN content TO content_bytes;
$$,
$$
ALTER TABLE file ADD COLUMN content OID;
$$,
$$
UPDATE file SET
content = lo_from_bytea(0, content_bytes::bytea),
content_bytes = NULL
;
$$,
$$
ALTER TABLE file DROP COLUMN content_bytes
$$
;
Related
I am using Springboot JPAs to generate a MySql8 table. It works fine unless I try to use the columndefinition in the Column annotation.
#Column(columnDefinition = "boolean default false")
private Boolean example;
I tried replacing boolean with TINYINT but MySql8 should support boolean as far as I am aware.
I get an error during the table generation:
Caused by: java.sql.SQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'boolean default false at line 5
To me it looks like the generated SQL Syntax has additional quotation marks but I am unsure how to fix the issue.
create table `exampleTable` (
`id` bigint not null auto_increment,
`example` `boolean default false`,
primary key (`id`)
)
EDIT:
I found my Issue in this question:
Spring JPA globally_quoted_identifiers incorrectly quoting column type TEXT
if you want to set the default value of the example to be false then you can write it like this and remove columnDefintion from there
private Boolean example=false;
Disable quotes by adding spring.jpa.properties.hibernate.globally_quoted_identifiers=false to application.properties file
But after disabling, you will not be able to use column and table names that match mysql keywords (eg desc). Workaround by adding an underscore:
#Column(name = "desc_")
String desc;
When I insert data with special characters with liquibase from SQL file it works.
When I insert data with special characters from IntelliJ console via SQL insert it works.
When I insert data with special characters by persisting an entity it gives the following
error:
(Quarkus Main Thread) Incorrect string value: '\xE1\xE1\xE1\xE1\xE1\xE1...' for column 'LEGAL_CLASSIFICATION' at row 1
The data I would like to insert:
product.setLegalClassification("ááááááásdaásdáasáá");
product.persist();
Properties:
hibernate-orm:
dialect: "org.hibernate.dialect.MySQLInnoDBDialect"
datasource:
db-kind: mysql
username: ${DB_USER}
password: ${DB_PASSWORD}
jdbc:
driver: "com.mysql.cj.jdbc.Driver"
url: "jdbc:mysql://localhost:3306/my-db?useUnicode=true&characterEncoding=utf8"
I've already tried with URLs:
"jdbc:mysql://localhost:3306/my-db?characterEncoding=utf8"
"jdbc:mysql://localhost:3306/my-db?useUnicode=true&characterEncoding=UTF-8"
"jdbc:mysql://localhost:3306/my-db?useUnicode=true&characterEncoding=utf8"
"jdbc:mysql://localhost:3306/my-db?useUnicode=true;characterEncoding=utf8;"
"jdbc:mysql://localhost:3306/my-db?useUnicode=yes&characterEncoding=utf8"
"jdbc:mysql://localhost:3306/my-db?useUnicode=true&characterEncoding=UTF-8"
"jdbc:mysql://localhost:3306/my-db?useUnicode=true;characterEncoding=UTF-8;"
"jdbc:mysql://localhost:3306/my-db?useUnicode=yes&characterEncoding=UTF-8"
and driver:
"org.hibernate.dialect.MySQLInnoDBDialect"
"org.hibernate.dialect.MySQLDialect"
The column type is set like this:
<column name="LEGAL_CLASSIFICATION"
type="LONGTEXT"/>
and in a MySQL dbms SQL changeset:
ALTER TABLE PRODUCT MODIFY COLUMN LEGAL_CLASSIFICATION LONGTEXT CHARACTER SET utf8mb4 COLLATE utf8mb4_hungarian_ci;
I tried to query database info with the following results:
SELECT SCHEMA_NAME 'database', default_character_set_name 'charset', DEFAULT_COLLATION_NAME 'collation' FROM information_schema.SCHEMATA;
gives
database, charset, collation
information_schema, utf8mb3, utf8_general_ci
my-db, utf8mb4, utf8mb4_0900_ai_ci
show variables like 'character%';
gives
variable_name, value
character_set_client, utf8mb4
character_set_connection, utf8mb4
character_set_database, utf8mb4
character_set_filesystem, binary
character_set_results, utf8mb4
character_set_server, utf8mb4
character_set_system, utf8mb3
character_sets_dir, /usr/share/mysql-8.0/charsets/
show variables like 'collation%';
gives
variable_name, value
collation_connection, utf8mb4_unicode_ci
collation_database, utf8mb4_0900_ai_ci
collation_server, utf8mb4_0900_ai_ci
SELECT DEFAULT_CHARACTER_SET_NAME, DEFAULT_COLLATION_NAME FROM information_schema.SCHEMATA S WHERE schema_name = 'my-db';
gives
default_character_set_name, default_collation_name
utf8mb4, utf8mb4_0900_ai_ci
Before I set the collation and character set on the columns I could not even insert data from SQL file via liquibase but altering tables fixed it, although I still can't insert through hibernate...
Does anybody have an idea what I should do to make it work?
I really appreciate any help you can provide.
UPDATE (solution/workaround)!!
I've found out that even without jdbc url parameters varchar and longvarbinary columns are working correctly from SQL file, terminal and even through hibernate.
So the only column that's not encoded correctly is the one with LONGTEXT. Unfortunately for me varchar is too short for the job, so currently I solved this issue by using type="JAVA.SQL.TYPES.LONGVARBINARY" as column type (with this I don't even need the column alteration) and I put #Lob on a byte[] entity field and convert the data while I'm mapping the field to a dto.
My Entity:
#Column(name = "LEGAL_CLASSIFICATION")
#Lob
private byte[] legalClassification;
Liquibase column:
<column name="LEGAL_CLASSIFICATION"
type="JAVA.SQL.TYPES.LONGVARBINARY"/>
Mapping:
#Mapping(source = "legalClassification", target = "legalClassification", qualifiedByName = "toUtf8")
ProductModel toModel(Product entity);
#Mapping(source = "legalClassification", target = "legalClassification", qualifiedByName = "fromUtf8")
Product toEntity(ProductModel entity);
#Named("toUtf8")
default String toUtf8(byte[] entity) {
return entity == null ? null : new String(entity, StandardCharsets.UTF_8);
}
#Named("fromUtf8")
default byte[] fromUtf8(String model) {
return model == null ? null : model.getBytes(StandardCharsets.UTF_8);
}
Now Everything is working fine, with good 4byte utf8 encoding. Altough I did not find any documentation how this works in MySQL or hibernate and why I cannot store 4byte utf8 encoded strings in TEXT and LONGTEXT type columns. I would appreciate an explanation on that.
Getting an error when trying to fetch table meta data.
Using:
JOOQ 3.14.0,
PostgeSQL 12,
PostgreSQL driver 42.2.10
Example:
#RestController
#ApiVersion("v2")
public class TypeResource {
private final DSLContext ctx;
#GetMapping(value = "/test")
public void test(#RequestParam String table) {
ctx.meta().getTables(table);
}
}
Error (This is not the whole error but the rest is similar):
17:16:44.808 [http-nio-8181-exec-5] [WARN ] org.jooq.impl.MetaImpl - Default value : Could not load default value: '{}'::character varying[] for type: varchar ([Ljava.lang.String;)
org.jooq.exception.DataTypeException: Cannot convert from '{}'::character varying[] (class java.lang.String) to class [Ljava.lang.String;
at org.jooq.tools.Convert$ConvertAll.fail(Convert.java:1303)
at org.jooq.tools.Convert$ConvertAll.from(Convert.java:1192)
at org.jooq.tools.Convert.convert0(Convert.java:392)
at org.jooq.tools.Convert.convert(Convert.java:384)
at org.jooq.tools.Convert.convert(Convert.java:458)
at org.jooq.impl.AbstractDataType.convert(AbstractDataType.java:534)
at org.jooq.impl.DefaultDataType.convert(DefaultDataType.java:86)
at org.jooq.impl.DSL.val(DSL.java:24373)
at org.jooq.impl.DSL.inline(DSL.java:23891)
at org.jooq.impl.MetaImpl$MetaTable.init(MetaImpl.java:910)
at org.jooq.impl.MetaImpl$MetaTable.<init>(MetaImpl.java:560)
at org.jooq.impl.MetaImpl$MetaSchema.getTables(MetaImpl.java:421)
at org.jooq.impl.MetaImpl.getTables0(MetaImpl.java:217)
at org.jooq.impl.AbstractMeta$4.iterator(AbstractMeta.java:194)
at org.jooq.impl.AbstractMeta.get(AbstractMeta.java:340)
at org.jooq.impl.AbstractMeta.initTables(AbstractMeta.java:191)
at org.jooq.impl.AbstractMeta.getTables(AbstractMeta.java:172)
at org.jooq.impl.AbstractMeta.getTables(AbstractMeta.java:167)
17:16:44.843 [http-nio-8181-exec-5] [WARN ] org.jooq.impl.MetaImpl - Default value : Could not load default value: '{{0,0}}'::numeric[] for type: numeric ([Ljava.math.BigDecimal;)
org.jooq.exception.DataTypeException: Cannot convert from '{{0,0}}'::numeric[] (class java.lang.String) to class [Ljava.math.BigDecimal;
at org.jooq.tools.Convert$ConvertAll.fail(Convert.java:1303)
at org.jooq.tools.Convert$ConvertAll.from(Convert.java:1192)
at org.jooq.tools.Convert.convert0(Convert.java:392)
at org.jooq.tools.Convert.convert(Convert.java:384)
at org.jooq.tools.Convert.convert(Convert.java:458)
at org.jooq.impl.AbstractDataType.convert(AbstractDataType.java:534)
at org.jooq.impl.DefaultDataType.convert(DefaultDataType.java:86)
at org.jooq.impl.DSL.val(DSL.java:24373)
at org.jooq.impl.DSL.inline(DSL.java:23891)
at org.jooq.impl.MetaImpl$MetaTable.init(MetaImpl.java:910)
at org.jooq.impl.MetaImpl$MetaTable.<init>(MetaImpl.java:560)
at org.jooq.impl.MetaImpl$MetaSchema.getTables(MetaImpl.java:421)
at org.jooq.impl.MetaImpl.getTables0(MetaImpl.java:217)
at org.jooq.impl.AbstractMeta$4.iterator(AbstractMeta.java:194)
at org.jooq.impl.AbstractMeta.get(AbstractMeta.java:340)
at org.jooq.impl.AbstractMeta.initTables(AbstractMeta.java:191)
at org.jooq.impl.AbstractMeta.getTables(AbstractMeta.java:172)
at org.jooq.impl.AbstractMeta.getTables(AbstractMeta.java:167)
Example table create script :
CREATE TABLE public.user_role
(
id bigint NOT NULL DEFAULT nextval('user_role_id_seq'::regclass),
user_id bigint NOT NULL,
role_id bigint NOT NULL,
ts_created timestamp without time zone NOT NULL DEFAULT CURRENT_TIMESTAMP,
ts_updated timestamp without time zone NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_by_user character varying(256) COLLATE pg_catalog."default" NOT NULL DEFAULT 'n/a'::character varying,
updated_by_process character varying(64) COLLATE pg_catalog."default" NOT NULL DEFAULT 'n/a'::character varying,
CONSTRAINT user_role_pkey PRIMARY KEY (id),
CONSTRAINT x_role_id_fk FOREIGN KEY (role_id)
REFERENCES public.role (id) MATCH FULL
ON UPDATE NO ACTION
ON DELETE NO ACTION,
CONSTRAINT x_user_id_fk FOREIGN KEY (user_id)
REFERENCES public."user" (id) MATCH FULL
ON UPDATE NO ACTION
ON DELETE NO ACTION
)
TABLESPACE pg_default;
ALTER TABLE public.user_role
OWNER to postgres;
CREATE UNIQUE INDEX x_role_user_id_role_id_uq
ON public.user_role USING btree
(user_id ASC NULLS LAST, role_id ASC NULLS LAST)
TABLESPACE pg_default;
Initially my goal was to get the table indexes and this is working fine.
I tried to update JOOQ from 3.13.1 to 3.14.0 and tested with different tables in the database but without any luck.
This is a bug which will be fixed in jOOQ 3.15 via https://github.com/jOOQ/jOOQ/issues/8469. jOOQ currently assumes that DatabaseMetaData column descriptions produce values for DEFAULT expressions, instead of expressions.
The bug is "cosmetic", as it only affects your logs. The field is still produced correctly (without any DEFAULT expression). You can mute the message in your logger configuration until the above bug is fixed.
mySQL Table generated using:
CREATE TABLE `actors` (
`actorID` INT(11) NOT NULL,
`actorName` VARCHAR(255) NOT NULL,
PRIMARY KEY AUTO_INCREMENT (actorID)
);
Mapped class:
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
#Column(name = "actorID", length = 11, unique = true, nullable = false)
private int actorID;
....
Error:
ERROR: Field 'actorID' doesn't have a default value
This error is hit when trying to create a new Actor object, and saving it to the database.
I've tried using other generation strategies, and dropping/rebuilding tables / entire database, as per the answers to similar questions. No luck so far.
Thank you for your time,
Samuel Smith
EDIT: Dropping the table and recreating it using the SQL syntax shown by shippi seemed to solve the problem for me.
Just try to use
#GeneratedValue(strategy=GenerationType.AUTO)
#Id
#Column(name = "actorID")
private int actorID;
This should fix your issue. Leave the actorID null if saving.
Ensure also that the DB works fine, try to write an insert statement without inserting the ID and lets see whether the pure insert is working. If the direct insert is working to the database you could start to troubleshoot hibernate.
And a small point, I use to define the PH and Auto increment in the same line where defining the column:
actorID INT(11) NOT NULL AUTO_INCREMENT PRIMARY KEY,
but it should work either way.
Your error is in your table definition.
Must be:
CREATE TABLE `actors` (
`actorID` INT(11) NOT NULL AUTO_INCREMENT,
`actorName` VARCHAR(255) NOT NULL,
PRIMARY KEY AUTO_INCREMENT (actorID)
);
You missed AUTO_INCREMENT in actorID.
I had the same issue, Solved it empty the entire database to empty and re run the application. Everything is no working fine.
I've got an entity Case that has an id CaseId (unfortunately a string due to compability with a legacy system). This id is foreign key in the table Document, and each Case can have many documents (onetomany). I've put the following in my Case entity:
#Id
#Column(name = "CaseId", length = 20, nullable = false)
private String caseId;
#OneToMany(fetch=FetchType.EAGER)
#JoinColumns ( {
#JoinColumn(name="caseId", referencedColumnName="CaseId")
} )
private Set<Document> documents;
The table for Document contains "CaseId varchar(20) not null". Right now, in the database, all cases have six documents. Yet when I do myCase.documents().size, I only ever get a single document. What should I do to get all the documents?
Cheers
Nik
The mapping looks correct. But it would be interesting to see:
the Document entity (and its equals/hashCode)
the SQL performed (see this previous answer to activate SQL logging)