We Keep Coding

Bulk/Batch update using PostgreSQL java

I am doing insertion/Updation into table using below command .
insertResult = ((InsertReturningStep) ctx.insertInto(jOOQEntity, insertFields)
.values(insertValue).onDuplicateKeyUpdate()
.set(tableFieldMapping.duplicateInsertMap)).returning().fetch();
But using above command I am able to insert/update one record at a time.
I want to update multiple record at a single command.
For this i am sending List of values for same fields into value but i am getting below error .
"java.lang.IllegalArgumentException: The number of values must match the number of fields"
Is there any solution to update bulk records at a one shot

Write to dynamic partition Java-Spark

I've created the following table in Hive:
CREATE TABLE mytable (..columns...) PARTITIONED BY (load_date string) STORED AS ...
And I'm trying to insert data to my table with spark as follow:
Dataset<Row> dfSelect = df.withColumn("load_date","15_07_2018");
dfSelect.write().mode("append").partitionBy("load_date").save(path);
And also make the following configuration:
sqlContext().setConf("hive.exec.dynamic.partition","true");
sqlContext().setConf("hive.exec.dynamic.partition.mode","nonstrict");
And after I make the write command I see on HDFS the directory /myDbPath/load_date=15_07_2018, which contains the file that I've written but when I make query like:
show partitions mytable
or
select * from mytable where load_date="15_07_2018"
I get 0 records.
What happened and how can I fix this?
EDIT
If I run the following command in Hue:
msck repair table mytable
I solve the problem, how can I do it in my code?

Hive stores a list of partitions for each table in its metastore. If, however, new partitions are directly added to HDFS (say by using hadoop fs -put command (or) .save..etc), the metastore (and hence Hive) will not be aware of these partitions unless the user runs either of the below commands
Meta store check command (msck repair table)
msck repair table <db.name>.<table_name>;
(or)
ALTER TABLE table_name ADD PARTITION commands on each of the newly added partitions.
We can also add partition by using alter table statement by using this way we need to add each and every newly created partition to the table
alter table <db.name>.<table_name> add partition(load_date="15_07_2018") location <hdfs-location>;
Run either of the above statements and then check the data again for load_date="15_07_2018"
For more details refer these links add partitions and msck repair table

Liquibase + H2 + Junit Primary Key Sequence starts over

I managed to integrate Liquibase into our Maven build to initialize a H2 inmemory database with a few enrys. Those rows have the primary key generated using a sequence table which works as expected (BigInt incremented values starting from 1).
My issue is that when i try to persist a new entity into that table from within a Junit integration test i get a "unique key constraint violation" because that new entity has the same primary key as the very first row inserted using the Liquibase changelog-xmls.
So the initialisation works perfectly fine as expected. The maven build uses the liquibase changelog-xmls
For now i just wipe the according tables completly before any integration tests with an own Runner... but that wont be a possibility in the furture. Its currently quite a chalange to investigate such issues since there is not yet much specific information on Liquibase available.
Update Workaround
While id prefer below answer using H2 brings up the problem that below changeset wont work because the required minValue is not supported.
<changeSet author="liquibase-docs" id="alterSequence-example">
<alterSequence
incrementBy="1"
maxValue="371717"
minValue="40"
ordered="true"
schemaName="public"
sequenceName="seq_id"/>
As a simple workaround i now just drop the existing sequence that was used to insert my testdata in a second changeSet:
<changeSet id="2" author="Me">
<dropSequence
sequenceName="SEQ_KEY_MY_TBL"/>
<createSequence
sequenceName="SEQ_KEY_MY_TBL"
incrementBy="1"
startValue="40"/>
</changeSet>
This way the values configured in the changelog-*.xml will be inserted using the sequence with an initial value of 1. I insert 30 rows so Keys 1-30 are used. After that the sequence gets dropped and recreated with a higher startValue. This way when persisting entities from within a Junit based integration Test the new entities will have primary keys starting from 40 and the previous unique constraint problem is solved.
Not H2 will probably soon release a version supporting minValue/maxValue since the according patch already exists.
Update:
Maybe we should mention this still is just a Workaround, anyone knows if H2 supports a Sequence with Liquibase that wont start over after DB-Init?

You should instruct liquibase to set the start value for those sequences to a value beyond those you have used for the entries you created. Liquibase has an alterSequence element for this. You can add such elements at the end of your current liquibase script.

Spring's JdbcDaoSupport (using MySQL Connector/J) fails after executing sql that adds FK

I am using Spring's JdbcDaoSupport class with a DriverManagerDataSource using the MySQL Connector/J 5.0 driver (driverClassName=com.mysql.jdbc.driver). allowMultiQueries is set to true in the url.
My application is an in-house tool we recently developed that executes sql scripts in a directory one-by-one (allows us to re-create our schema and reference table data for a given date, etc, but I digress). The sql scripts sometime contain multiple statements (hence allowMultiQueries), so one script can create a table, add indexes for that table, etc.
The problem happens when including a statement to add a foreign key constraint in one of these files. If I have a file that looks like...
--(column/constraint names are examples)
CREATE TABLE myTable (
fk1 BIGINT(19) NOT NULL,
fk2 BIGINT(19) NOT NULL,
PRIMARY KEY (fk1, fk2)
);
ALTER TABLE myTable ADD CONSTRAINT myTable_fk1
FOREIGN KEY (fk1)
REFERENCES myOtherTable (id)
;
ALTER TABLE myTable ADD CONSTRAINT myTable_fk2
FOREIGN KEY (fk2)
REFERENCES myOtherOtherTable (id)
;
then JdbcTemplate.execute throws an UncategorizedSqlException with the following error message and stack trace:
Exception in thread "main" org.springframework.jdbc.UncategorizedSQLException: StatementCallback; uncategorized SQLException for SQL [ THE SQL YOU SEE ABOVE LISTED HERE ];
SQL state [HY000]; error code [1005]; Can't create table 'myDatabase.myTable' (errno: 150); nested exception is java.sql.SQLException: Can't create table 'myDatabase.myTable' (errno: 150)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:83)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:80)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:80)
and the table and foreign keys are not inserted.
Also, especially weird: if I take the foreign key statements out of the script I showed above and then place them in their own script that executes after (so I now have 1 script with just the create table statement, and 1 script with the add foreign key statements that executes after that) then what happens is:
tool executes create table script, works fine, table is created
tool executes add fk script, throws the same exception as seen above (except errno=121 this time), but the FKs actually get added (!!!)
In other words, when the create table/FK statements are in the same script then the exception is thrown and nothing is created, but when they are different scripts a nearly identical exception is thrown but both things get created.
Any help on this would be greatly appreciated. Please let me know if you'd like me to clarify anything more.
Some more info:
1) This only happens on my box. My coworker does not get the same problem.
2) The script that forces the tool to error works fine when executed from the mysql command line using the "script" command

My God.
http://bugs.mysql.com/bug.php?id=41635
and
[2nd link removed because spam filter isn't letting me add 2 links. Search Google for "mysql connector / j errno 150" and it's the 3rd result]
...
Looks like mySql5.1 has a bug with its jdbc connector where it bombs where an alter statement to add a FK is in a script with any other statement.
When I broke out my 3 statements into 3 scripts, it worked (the way I was trying before with the 2 fk statements in their own script still bombed because they were sharing a script!!). Also, my coworker is using MySql5.0, so it didn't affect him.
Holy Cow, that was a fun 5 hours.

The best way to import(merge)-export java db database

I have let's say two pc's.PC-a and PC-b which both have the same application installed with java db support.I want from time to time to copy the data from the database on PC-a to database to PC-b and vice-versa so the two PC's to have the same data all the time.
Is there an already implemented API in the database layer for this(i.e 1.export-backup database from PC-a 2.import-merge databases to PC-b) or i have to do this in the sql layer(manually)?

As you mention in the comments that you want to "merge" the databases, this sounds like you need to write custom code to do this, as presumably there could be conficts - the same key in both, but with different details against it, for example.

In short: You can't do this without some work on your side. SalesLogix fixed this problem by giving everything a site code, so here's how your table looked:
Customer:
SiteCode varchar,
CustomerID varchar,
....
primary key(siteCode, CustomerID)
So now you would take your databases, and match up each record by primary key. Where there are conflicts you would have to provide a report to the end-user, on what data was different.
Say machine1:
SiteCode|CustomerID|CustName |phone |email
1 XXX |0001 |Customer1 |555.555.1212 |darth#example.com
and on machine2:
SiteCode|CustomerID|CustName |phone |email
2 XXY |0001 |customer2 |555.555.1213 |darth#nowhere.com
3 XXX |0001 |customer1 |555.555.1212 |darth#nowhere.com
When performing a resolution:
Record 1 and 3 are in conflict, because the PK matches, but the data doesnt (email is different).
Record 2 is unique, and can freely exist in both databases.
There is NO way to do this automatically without error or data corruption or referential integrity issues.

I guess you are using Java DB (aka Derby) - in which case, assuming you just can't use a single instance, you can do a backup/restore.

Why dont you have the database on one pc. and have all other pc's request data from the host pc