How to create a database with flyway? - java

Question: Is it possible to create a new DB in a migration script and then connect to it? How?
My Scenario:
I'm trying to use flyway in my Java project (RESTful application using Jersey2.4 + tomcat 7 + PostgreSQL 9.3.1 + EclipseLink) for managing the changes between different developers which are using git. I wrote my init script and ran it with:
PGPASSWORD='123456' psql -U postgres -f migration/V1__initDB.sql
and it worked fine. The problem is that I can't create new DB with my scripts. when I include the following line in my script:
CREATE DATABASE my_database OWNER postgres ENCODING 'UTF8';
I get this error:
org.postgresql.util.PSQLException: ERROR: CREATE DATABASE cannot run inside a transaction block
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2157)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1886)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:555)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:403)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:395)
at com.googlecode.flyway.core.dbsupport.JdbcTemplate.executeStatement(JdbcTemplate.java:230)
at com.googlecode.flyway.core.dbsupport.SqlScript.execute(SqlScript.java:89)
at com.googlecode.flyway.core.resolver.sql.SqlMigrationExecutor.execute(SqlMigrationExecutor.java:72)
at com.googlecode.flyway.core.command.DbMigrate$2.doInTransaction(DbMigrate.java:252)
at com.googlecode.flyway.core.command.DbMigrate$2.doInTransaction(DbMigrate.java:250)
at com.googlecode.flyway.core.util.jdbc.TransactionTemplate.execute(TransactionTemplate.java:56)
at com.googlecode.flyway.core.command.DbMigrate.applyMigration(DbMigrate.java:250)
at com.googlecode.flyway.core.command.DbMigrate.access$700(DbMigrate.java:47)
at com.googlecode.flyway.core.command.DbMigrate$1.doInTransaction(DbMigrate.java:189)
at com.googlecode.flyway.core.command.DbMigrate$1.doInTransaction(DbMigrate.java:138)
at com.googlecode.flyway.core.util.jdbc.TransactionTemplate.execute(TransactionTemplate.java:56)
at com.googlecode.flyway.core.command.DbMigrate.migrate(DbMigrate.java:137)
at com.googlecode.flyway.core.Flyway$1.execute(Flyway.java:872)
at com.googlecode.flyway.core.Flyway$1.execute(Flyway.java:819)
at com.googlecode.flyway.core.Flyway.execute(Flyway.java:1200)
at com.googlecode.flyway.core.Flyway.migrate(Flyway.java:819)
at ir.chom.MyApp.<init>(MyApp.java:28)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.glassfish.hk2.utilities.reflection.ReflectionHelper.makeMe(ReflectionHelper.java:1117)
at org.jvnet.hk2.internal.Utilities.justCreate(Utilities.java:867)
at org.jvnet.hk2.internal.ServiceLocatorImpl.create(ServiceLocatorImpl.java:814)
at org.jvnet.hk2.internal.ServiceLocatorImpl.createAndInitialize(ServiceLocatorImpl.java:906)
at org.jvnet.hk2.internal.ServiceLocatorImpl.createAndInitialize(ServiceLocatorImpl.java:898)
at org.glassfish.jersey.server.ApplicationHandler.createApplication(ApplicationHandler.java:300)
at org.glassfish.jersey.server.ApplicationHandler.<init>(ApplicationHandler.java:279)
at org.glassfish.jersey.servlet.WebComponent.<init>(WebComponent.java:302)
at org.glassfish.jersey.servlet.ServletContainer.init(ServletContainer.java:167)
at org.glassfish.jersey.servlet.ServletContainer.init(ServletContainer.java:349)
at javax.servlet.GenericServlet.init(GenericServlet.java:160)
at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1280)
at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1091)
at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5176)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5460)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.apache.catalina.core.StandardContext.reload(StandardContext.java:3954)
at org.apache.catalina.loader.WebappLoader.backgroundProcess(WebappLoader.java:426)
at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1345)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1530)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1540)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1540)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1519)
at java.lang.Thread.run(Thread.java:724)
It seems that this is a problem with JDBC that uses autocommit option. This option can be disabled with something like this:
Connection connection = dataSource.getConnection();
Connection.setAutoCommit(false); // Disables auto-commit.
but I don't know how to pass this option to flyway connection. Also if I solve this I think I will have problem with passing password to \c command.

Flyway always operates within the database used in the jdbc connection string.
Once connected, all scripts run within a transaction. As CREATE DATABASE is not supported within transactions, you will not be able to accomplish what you want.
What you can do however, is create a schema instead. Flyway will even do this for you, if you point it at a non-existing one.

I dont know if this is even possible to do in flyway.
Flyway is intended to connect to an already existing database (whether it is empty or not). It also would be a good practice to keep your database creation separate from your database migrations.

Here is a workaround that worked for me (assuming the use of the Maven plugin):
Configure the plugin with two executions. The first execution creates the database. The second execution migrates an existing database.
<plugin>
<groupId>org.flywaydb</groupId>
<artifactId>flyway-maven-plugin</artifactId>
<version>${flyway.version}</version>
<executions>
<execution>
<id>create-db</id>
<goals>
<goal>migrate</goal>
</goals>
<configuration>
<driver>org.postgresql.Driver</driver>
<url>jdbc:postgresql://database-server/</url>
<user>postgres</user>
<password>password</password>
<placeholders>
<DATABASE.NAME>MyDatabase</DATABASE.NAME>
</placeholders>
<locations>
<location>com/foo/bar/database/create</location>
</locations>
</configuration>
</execution>
<execution>
<id>migrate-db</id>
<goals>
<goal>migrate</goal>
</goals>
<configuration>
<driver>org.postgresql.Driver</driver>
<url>jdbc:postgresql://database-server/MyDatabase</url>
<user>postgres</user>
<password>password</password>
<locations>
<location>com/foo/bar/database/migrate</location>
</locations>
</configuration>
</execution>
</executions>
<dependencies>
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<version>${postgresql.version}</version>
</dependency>
</dependencies>
</plugin>
Then add V1__Created_database.sql to the com/foo/bar/database/create directory. This file contains:
CREATE DATABASE ${DATABASE.NAME}

Flyway can't create database for you.
It can create schema if you didn't create one by
flyway.schemas: schema1,schema2

You can try what is suggested in this issue: https://github.com/flyway/flyway/issues/2556, use the createDatabaseIfNotExist parameter in the mysql url configured in flyway, as below:
<groupId>org.flywaydb</groupId>
<artifactId>flyway-maven-plugin</artifactId>
<version>6.4.1</version>
<configuration>
<url>jdbc:mysql://localhost:3306/<databaseName>?createDatabaseIfNotExist=true</url>
<user>root</user>
<password>root</password>
</configuration>

If u have schema database creation command in V1 of your scripts, flyway can generate it but not database:
flyway -baselineOnMigrate=true -url=jdbc:mysql://localhost/ -schemas=test_db -user=root -password=root_pass -locations=filesystem:/path/to/scrips/ migrate
and similar to this in the script file:
DROP SCHEMA IF EXISTS `test_db` ;
CREATE SCHEMA `test_db` COLLATE utf8_general_ci ;

Related

Flyway is unable to create and find database in Spring Boot [duplicate]

Question: Is it possible to create a new DB in a migration script and then connect to it? How?
My Scenario:
I'm trying to use flyway in my Java project (RESTful application using Jersey2.4 + tomcat 7 + PostgreSQL 9.3.1 + EclipseLink) for managing the changes between different developers which are using git. I wrote my init script and ran it with:
PGPASSWORD='123456' psql -U postgres -f migration/V1__initDB.sql
and it worked fine. The problem is that I can't create new DB with my scripts. when I include the following line in my script:
CREATE DATABASE my_database OWNER postgres ENCODING 'UTF8';
I get this error:
org.postgresql.util.PSQLException: ERROR: CREATE DATABASE cannot run inside a transaction block
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2157)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1886)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:555)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:403)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:395)
at com.googlecode.flyway.core.dbsupport.JdbcTemplate.executeStatement(JdbcTemplate.java:230)
at com.googlecode.flyway.core.dbsupport.SqlScript.execute(SqlScript.java:89)
at com.googlecode.flyway.core.resolver.sql.SqlMigrationExecutor.execute(SqlMigrationExecutor.java:72)
at com.googlecode.flyway.core.command.DbMigrate$2.doInTransaction(DbMigrate.java:252)
at com.googlecode.flyway.core.command.DbMigrate$2.doInTransaction(DbMigrate.java:250)
at com.googlecode.flyway.core.util.jdbc.TransactionTemplate.execute(TransactionTemplate.java:56)
at com.googlecode.flyway.core.command.DbMigrate.applyMigration(DbMigrate.java:250)
at com.googlecode.flyway.core.command.DbMigrate.access$700(DbMigrate.java:47)
at com.googlecode.flyway.core.command.DbMigrate$1.doInTransaction(DbMigrate.java:189)
at com.googlecode.flyway.core.command.DbMigrate$1.doInTransaction(DbMigrate.java:138)
at com.googlecode.flyway.core.util.jdbc.TransactionTemplate.execute(TransactionTemplate.java:56)
at com.googlecode.flyway.core.command.DbMigrate.migrate(DbMigrate.java:137)
at com.googlecode.flyway.core.Flyway$1.execute(Flyway.java:872)
at com.googlecode.flyway.core.Flyway$1.execute(Flyway.java:819)
at com.googlecode.flyway.core.Flyway.execute(Flyway.java:1200)
at com.googlecode.flyway.core.Flyway.migrate(Flyway.java:819)
at ir.chom.MyApp.<init>(MyApp.java:28)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.glassfish.hk2.utilities.reflection.ReflectionHelper.makeMe(ReflectionHelper.java:1117)
at org.jvnet.hk2.internal.Utilities.justCreate(Utilities.java:867)
at org.jvnet.hk2.internal.ServiceLocatorImpl.create(ServiceLocatorImpl.java:814)
at org.jvnet.hk2.internal.ServiceLocatorImpl.createAndInitialize(ServiceLocatorImpl.java:906)
at org.jvnet.hk2.internal.ServiceLocatorImpl.createAndInitialize(ServiceLocatorImpl.java:898)
at org.glassfish.jersey.server.ApplicationHandler.createApplication(ApplicationHandler.java:300)
at org.glassfish.jersey.server.ApplicationHandler.<init>(ApplicationHandler.java:279)
at org.glassfish.jersey.servlet.WebComponent.<init>(WebComponent.java:302)
at org.glassfish.jersey.servlet.ServletContainer.init(ServletContainer.java:167)
at org.glassfish.jersey.servlet.ServletContainer.init(ServletContainer.java:349)
at javax.servlet.GenericServlet.init(GenericServlet.java:160)
at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1280)
at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1091)
at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5176)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5460)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.apache.catalina.core.StandardContext.reload(StandardContext.java:3954)
at org.apache.catalina.loader.WebappLoader.backgroundProcess(WebappLoader.java:426)
at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1345)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1530)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1540)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1540)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1519)
at java.lang.Thread.run(Thread.java:724)
It seems that this is a problem with JDBC that uses autocommit option. This option can be disabled with something like this:
Connection connection = dataSource.getConnection();
Connection.setAutoCommit(false); // Disables auto-commit.
but I don't know how to pass this option to flyway connection. Also if I solve this I think I will have problem with passing password to \c command.
Flyway always operates within the database used in the jdbc connection string.
Once connected, all scripts run within a transaction. As CREATE DATABASE is not supported within transactions, you will not be able to accomplish what you want.
What you can do however, is create a schema instead. Flyway will even do this for you, if you point it at a non-existing one.
I dont know if this is even possible to do in flyway.
Flyway is intended to connect to an already existing database (whether it is empty or not). It also would be a good practice to keep your database creation separate from your database migrations.
Here is a workaround that worked for me (assuming the use of the Maven plugin):
Configure the plugin with two executions. The first execution creates the database. The second execution migrates an existing database.
<plugin>
<groupId>org.flywaydb</groupId>
<artifactId>flyway-maven-plugin</artifactId>
<version>${flyway.version}</version>
<executions>
<execution>
<id>create-db</id>
<goals>
<goal>migrate</goal>
</goals>
<configuration>
<driver>org.postgresql.Driver</driver>
<url>jdbc:postgresql://database-server/</url>
<user>postgres</user>
<password>password</password>
<placeholders>
<DATABASE.NAME>MyDatabase</DATABASE.NAME>
</placeholders>
<locations>
<location>com/foo/bar/database/create</location>
</locations>
</configuration>
</execution>
<execution>
<id>migrate-db</id>
<goals>
<goal>migrate</goal>
</goals>
<configuration>
<driver>org.postgresql.Driver</driver>
<url>jdbc:postgresql://database-server/MyDatabase</url>
<user>postgres</user>
<password>password</password>
<locations>
<location>com/foo/bar/database/migrate</location>
</locations>
</configuration>
</execution>
</executions>
<dependencies>
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<version>${postgresql.version}</version>
</dependency>
</dependencies>
</plugin>
Then add V1__Created_database.sql to the com/foo/bar/database/create directory. This file contains:
CREATE DATABASE ${DATABASE.NAME}
Flyway can't create database for you.
It can create schema if you didn't create one by
flyway.schemas: schema1,schema2
You can try what is suggested in this issue: https://github.com/flyway/flyway/issues/2556, use the createDatabaseIfNotExist parameter in the mysql url configured in flyway, as below:
<groupId>org.flywaydb</groupId>
<artifactId>flyway-maven-plugin</artifactId>
<version>6.4.1</version>
<configuration>
<url>jdbc:mysql://localhost:3306/<databaseName>?createDatabaseIfNotExist=true</url>
<user>root</user>
<password>root</password>
</configuration>
If u have schema database creation command in V1 of your scripts, flyway can generate it but not database:
flyway -baselineOnMigrate=true -url=jdbc:mysql://localhost/ -schemas=test_db -user=root -password=root_pass -locations=filesystem:/path/to/scrips/ migrate
and similar to this in the script file:
DROP SCHEMA IF EXISTS `test_db` ;
CREATE SCHEMA `test_db` COLLATE utf8_general_ci ;

Wildfly Maven Plugin - commands seem to have no effect

I am using the Wildfly Maven Plugin and it is working, in that it turns on, runs web application, however I am having trouble with my custom configurations, namely:
Setting the Root logger and Console logger to debug mode
Allowing connections from 0.0.0.0:8080 or something other than localhost.
Here is my set up:
<plugin>
<groupId>org.wildfly.plugins</groupId>
<artifactId>wildfly-maven-plugin</artifactId>
<version>2.0.2.Final</version>
<executions>
<execution>
<phase>install</phase>
<goals>
<goal>run</goal>
</goals>
</execution>
</executions>
<configuration>
<java-opts>
<java-opt>-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=1044</java-opt>
</java-opts>
<commands>
<!-- **These are the commands that aren't going through** -->
<command>/subsystem=logging/root-logger=ROOT:write-attribute(name="level", value="DEBUG") </command>
<command>/subsystem=logging/console-handler=CONSOLE:write-attribute(name="level", value="DEBUG")</command>
<command>/subsystem=logging/file-handler=debug:add(level=DEBUG,autoflush=true,file={"relative-to"=>"jboss.server.log.dir", "path"=>"debug.log"})</command>
<command>/subsystem=logging/logger=org.jboss.as:add(level=DEBUG,handlers=[debug])</command>
<command>/subsystem-----Enable Remote access here?</command>
</commands>
<add-user>
<users>
<user>
<username>admin</username>
<password>admin.1234</password>
</user>
<user>
<username>admin-user</username>
<password>user.1234</password>
<groups>
<group>admin</group>
<group>user</group>
</groups>
<application-user>true</application-user>
</user>
<user>
<username>default-user</username>
<password>user.1234</password>
<groups>
<group>user</group>
</groups>
<application-user>true</application-user>
</user>
</users>
</add-user>
</configuration>
</plugin>
I know when starting from terminal, one would use this: ./standalone.sh -b 0.0.0.0 -bmanagement 0.0.0.0 However I am running demos straight from Maven and need to access my webapp from a separate machine.
Note - from within the Wildfly Management page, I can manually set the Root Logger and Console Logger to debug mode and then the proper debug logs will flow out.
For example, manually I could go here: http://127.0.0.1:9990/console/index.html#logging-configuration and then manually change the logging from the default INFO level to DEBUG:
So my question is, along with allowing remote access, is how to change the logging level as a command into the maven wildfly plugin.
You'd need to upgrade the plugin version to 2.1.0.Beta1 to get that to work. The 2.0.x versions do no have the ability to execute CLI commands from the run or deploy goals.
If you need to stick with the version you're using you'd need to define the execute-commands goal. Then you could use the embedded server to configure the server.
<commands>
<!-- **These are the commands that aren't going through** -->
<command>embed-server</command>
<command>/subsystem=logging/root-logger=ROOT:write-attribute(name="level", value="DEBUG") </command>
<command>/subsystem=logging/console-handler=CONSOLE:write-attribute(name="level", value="DEBUG")</command>
<command>/subsystem=logging/file-handler=debug:add(level=DEBUG,autoflush=true,file={"relative-to"=>"jboss.server.log.dir", "path"=>"debug.log"})</command>
<command>/subsystem=logging/logger=org.jboss.as:add(level=DEBUG,handlers=[debug])</command>
<command>/subsystem-----Enable Remote access here?</command>
<command>stop-embedded-server</command>
</commands>

Java Hibernate, Liquibase supports cross databases and SQLite?

I'm using Spring Boot 1.5.2 with Hibernate 5 and try to support as many databases as possible (i.e: Hibernate will create all the tables in SQLite 3, then I use Liquibase as an abstract layer to generate the XML change-log files for all kind of supported databases which Liquibase claimed: supported databases).
so I added the dependency for Liquibase in pom.xml (Maven).
<!-- Database schema versions migration -->
<dependency>
<groupId>org.liquibase</groupId>
<artifactId>liquibase-core</artifactId>
<version>3.5.3</version>
</dependency>
<dependency>
<groupId>org.liquibase</groupId>
<artifactId>liquibase-maven-plugin</artifactId>
<version>3.4.1</version>
</dependency>
and a plugin to generate the changelog XML file from created database of Hibernate
<!-- Database schema versions migration -->
<plugin>
<groupId>org.liquibase</groupId>
<artifactId>liquibase-maven-plugin</artifactId>
<version>3.5.3</version>
<configuration>
<propertyFile>src/main/resources/liquibase.properties</propertyFile>
</configuration>
</plugin>
I have some configuration in liquibase.properties to connect to SQLite3 datab file, then I can run this command to create a changelog file.
mvn liquibase:generateChangeLog
The output changelog I cannot recreate in another different SQLite 3 db file, due to the addPrimaryKey element:
<changeSet author="rasdaman (generated)" id="1497363976895-86">
<addPrimaryKey columnNames="address_id" tableName="address"/>
</changeSet>
and the error in Java Spring Boot when it starts:
addPrimaryKey is not supported on sqlite,
classpath:/database_versions/db.changelog-master.xml::1497366115846-62::rasdaman (generated)
at liquibase.changelog.DatabaseChangeLog.validate(DatabaseChangeLog.java:266)
at liquibase.Liquibase.update(Liquibase.java:210)
at liquibase.Liquibase.update(Liquibase.java:192)
at liquibase.integration.spring.SpringLiquibase.performUpdate(SpringLiquibase.java:431)
at liquibase.integration.spring.SpringLiquibase.afterPropertiesSet(SpringLiquibase.java:388)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1687)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1624)
If I use the generated output file from SQLite by Liquibase and allow Spring Boot to starts with Postgresql datasource, I got another error:
org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'liquibase' defined in class path resource
[org/springframework/boot/autoconfigure/liquibase/LiquibaseAutoConfiguration$LiquibaseConfiguration.class]: Invocation of init method failed; nested exception is liquibase.exception.MigrationFailedException: Migration failed for change set classpath:/database_versions/db.changelog-master.xml::1497366115846-1:: (generated):
Reason: liquibase.exception.DatabaseException: ERROR: syntax error at or near "("
Position: 72 [Failed SQL: CREATE TABLE public."HT_abstract_coverage" (abstract_coverage_id BIGINT(2000000000, 10) NOT NULL, hib_sess_id CHAR(36))]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1628)
So it seems I don't have another tool to generate an abstract data file like XML automatically which can be imported to any common databases without problems? If you have any suggestion, please advise!
Thanks,
The problem is Liquibase does not support SQLite very well http://www.liquibase.org/documentation/changes/add_primary_key.html , also Hibernate which needs a third party SQLite dialect and the performance for write is not good, so I think Liquibase could work well with another databases like: postgresql, mysql, hyperSQL,...

Connecting to remote HBase service using Java

I have a small sample code in which I try to establish a connection to a remote HBase entity. The code runs on a windows machine without HBase installed and I try to connect to a remote Ubuntu Server that has it installed and running. The IP in the below snippet is of course just a placeholder.
The code is as follows:
public static void main(String[] args) {
Configuration conf = HBaseConfiguration.create();
HBaseAdmin admin = null;
String ip = "10.10.10.10";
String port = "2181";
conf.set("hbase.zookeeper.quorum", ip);
conf.set("hbase.zookeeper.property.clientPort", port);
try {
admin = new HBaseAdmin(conf);
boolean bool = admin.tableExists("sensor_data");
System.out.println("Table exists? " + bool);
} catch (IOException e) {
e.printStackTrace();
}
}
But for some reason I get this error:
org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator
at org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:229)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:202)
at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:155)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:811)
at org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:602)
at org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:366)
at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:303)
at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:313)
at com.twoBM.Tests.HBaseWriter.main(HBaseWriter.java:26)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
Caused by: java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator
at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:596)
at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:580)
at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:559)
at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1185)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1152)
at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:300)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:61)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
... 15 more
I am using Gradle to build my project and currently I am only using the two following dependencies:
compile 'org.apache.hive:hive-jdbc:2.1.0'
compile 'org.apache.hbase:hbase:1.1.6'
Does anyone know to fix this problem? I have tried googling this problem, but without any of the found links providing an actual solution.
Best regards
This is definitely Google Guava's dependency conflict. The default constructor of Stopwatch class became private since Guava v.17 and marked deprecated even earlier.
So to HBase Java client works properly you need Guava v.16 or earlier. Check the way you build your application (Maven/Gradle/Classpath) and find the dependency which uses Guava v.17+. After that, you can resolve the conflict.
I received the same error and had to spend for 5 days to know the issue.
I added following dependency and its gone.
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>15.0</version>
</dependency>
You can use maven shade plugin to solve this issue. That a look at this blog post. Here is an example (actually a snippet from my working pom.)
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.0.0</version>
<executions>
<execution>
<id>assemble-all</id>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<!--<finalName>PROJECT_NAME-${project.version}-shaded</finalName>-->
<relocations>
<relocation>
<pattern>com.google.common</pattern>
<shadedPattern>shaded.com.google.common</shadedPattern>
</relocation>
<relocation>
<pattern>com.google.protobuf</pattern>
<shadedPattern>shaded.com.google.protobuf</shadedPattern>
</relocation>
</relocations>
<artifactSet>
<includes>
<include>*:*</include>
</includes>
</artifactSet>
</configuration>
</execution>
</executions>
</plugin>

JAXB XJC Possible to suppress comment creation in generated classes?

Our project uses XJC to generate Java classes from an XSD. I'm using JAVA EE 6.
When all the XSDs we have are re-generated, the generated classes include this comment at the top of the file:
// Generated on: 2011.02.23 at 02:17:06 PM GMT
Is it possible to suppress this comment? The reason is that we use SVN for version control, and every time we regenerate our classes, every single file shows as being changed in SVN, even though the only thing that differs is this comment. So I'd like to remove the comment altogether if possible.
There is a -no-header directive, but I don't want to remove the entire header, so that future generations know that it's a file generated from a tool, and that modifications will be overwritten. I only want to remove the timestamp. (Or alternatively, I'd remove the inbuilt header and then insert my own header somehow.)
I am using this Maven plugin which replaces the // Generated on: 2011.02.23 at 02:17:06 PM GMT line:
<plugin>
<groupId>com.google.code.maven-replacer-plugin</groupId>
<artifactId>maven-replacer-plugin</artifactId>
<version>1.3.8</version>
<executions>
<execution>
<phase>prepare-package</phase>
<goals>
<goal>replace</goal>
</goals>
</execution>
</executions>
<configuration>
<includes>
<include>src/main/java/jaxb/*.java</include>
</includes>
<token>^// Generated on.*$</token>
<value>// Generated on: [TEXT REMOVED by maven-replacer-plugin]</value>
<regexFlags>
<regexFlag>MULTILINE</regexFlag>
</regexFlags>
</configuration>
</plugin>
I'm late to the party, but since version 2.0 of the jaxb2-maven-plugin, there's a noGeneratedHeaderComments configuration option. (see the JAXB-2 Maven Plugin Docs)
You can use it like this:
...
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>jaxb2-maven-plugin</artifactId>
<version>2.3.1</version>
<executions>
<execution>
<id>xjc</id>
<goals>
<goal>xjc</goal>
</goals>
</execution>
</executions>
<configuration>
<target>2.1</target>
<sources>
<source>FirstXSD.xsd</source>
<source>SecondXSD.xsd</source>
</sources>
<xjbSources>
<xjbSource>OptionalBindings.xjb</xjbSource>
</xjbSources>
<noGeneratedHeaderComments>true</noGeneratedHeaderComments>
</configuration>
<dependencies>
<dependency>
<groupId>org.glassfish.jaxb</groupId>
<artifactId>jaxb-xjc</artifactId>
<version>${jaxb.version}</version>
</dependency>
</dependencies>
</plugin>
</plugins>
...
So no need for another plugin or script to run.
If you want to keep a disclaimer, you can use one of the techniques already mentioned to inject it where wanted.
If you use ant, the following snippet may be useful for replacing the comments:
<replaceregexp
match="^// Generated on:.*$"
replace="// Generated on: [date removed]"
byline="true">
<fileset dir="src">
<include name="**/*.java"/>
</fileset>
</replaceregexp>
I know this is 2 years after the fact, but because the classes are generated they aren't necessarily needed in SVN. What needs to be in SVN is the schema or whatever file you use for source to generate the classes. As long as you have the source and the tools to generate the classes, the classes in SVN are redundant and as you saw, problematic in SVN or any SCCS. So put the schema file in SVN and avoid the issue altogether.
If it's not possible using an option you can post-process the generated files yourself.
For a very specific use-case we had to do it that way on our project...
We use Maven and we execute a specific script after the Java classes have been generated and before we compile and package them to a distriuable JAR.
To build on cata's answer (upvoted) the maven-replacer-plugin is the way to go. I've come up with the following that strips out the entire comment (not just the timestamp) which you can replace with your file comment (license etc.).
<plugin>
<groupId>com.google.code.maven-replacer-plugin</groupId>
<artifactId>maven-replacer-plugin</artifactId>
<executions>
<execution>
<phase>prepare-package</phase>
<goals>
<goal>replace</goal>
</goals>
</execution>
</executions>
<configuration>
<!-- assumes your xjc is putting source code here -->
<includes>
<include>src/main/java/**/*.java</include>
</includes>
<regex>true</regex>
<regexFlags>
<regexFlag>MULTILINE</regexFlag>
</regexFlags>
<replacements>
<replacement>
<token>(^//.*\u000a|^\u000a)*^package</token>
<value>// your new comment
package</value>
</replacement>
</replacements>
</configuration>
</plugin>
The one gotcha to watch out for is that the <value> element treats the text literally. So if you want a line break in your replacement text you need to put a line break in your pom.xml file (as I've demonstrated above).
What you should you :
Generate your classes in target :
${project.build.directory}/generated-sources
If you add target to ignore list (svn), that's all.
I also want to have text header with warning about classes was auto-generated and should not be modified manually, but because I place such files into git I do not want there always changed date of generation.
That header generated in com.sun.tools.xjc.Options#getPrologComment method. So essentially it call:
return Messages.format(
Messages.FILE_PROLOG_COMMENT,
dateFormat.format(new Date()));
Messages.FILE_PROLOG_COMMENT defined as Driver.FilePrologComment. With futher debugging I found it use standard Java localization bundles.
So, to change header format we just may provide our properties override for their values from MessageBundle.properties.
We can do it in two way:
Just copy that file (from repo by link, or just from jar of appropriate version what you are using) into src/main/resources/com/sun/tools/xjc/MessageBundle.properties in your project and change key Driver.FilePrologComment as you wish.
But first case have some drawbacks - first you copy-paste many code which you do not change, second you should update it when you update XJC dependency. So better I recommend place it as src/main/resources/com/sun/tools/xjc/MessageBundle_en.properties (note _en suffix in filename) file and place there only properties you really want to change. Something like:
# We want header, but do NOT willing there `Generated on: {0}` part because want commit them into git!
Driver.FilePrologComment = \
This file was generated by the JavaTM Architecture for XML Binding(JAXB) Reference Implementation, v2.4.0-b180830.0438 \n\
See https://javaee.github.io/jaxb-v2/ \n\
Any modifications to this file will be lost upon recompilation of the source schema. \n
Ensure that file in compiler classpath, especially if you call it from some plugins.
That is common mechanism for translation. See related answer: JAXB english comments in generated file
If you are using maven-jaxb2-plugin there is an tag noFileHeader set it to true. It will prevent jaxb to generate the header that includes that date line on it.
<noFileHeader>true</noFileHeader>

Categories