Trying to run Java-based (compiled) migration files which are not in the project where Flyway is configured. Can anyone tell me is it possible to do so?
I've created a jar which use flyway to do migrations. Jar expects an argument which is migration scripts' location. Migration scripts are in a different location/project. So far all scripts are SQL based. (i.e. XXX.sql). Need to add a java based migration scripts to it, to do some complex logic.
Tried to add pom.xml to the script location and a sample java migration script in db/migration folder. Java-based migration file is ignored by Flyway. Is it due to checksum validation fail?
Java-based migrations are compiled and .class files are in the classpath. My folder structure as below.
C:/
└── database-migration-scripts
└── src/main/java/
└── db
└── migration
└── V1__m1.sql
└── V2__m2.sql
└── V3__SampleJava_script.java
└── target/classes/
└── db
└── migration
└── V3__SampleJava_script.class
└── pom.xml
W:/
└── someLocationOutsideProject
└── flyway-database-migration.jar
NOTE: flyway runner (jar) and scripts will be in different locations in the same or different machine. In the example above, migration scripts in C:/ and jar in W:/
In order to be discovered, Java migrations should go under src/main/java with the package db.migration
e.g.
package db.migration; // <-- classpath:db/migration
import org.flywaydb.core.api.migration.jdbc.JdbcMigration;
import java.sql.Connection;
public class V3__SampleJava_script implements JdbcMigration {
public void migrate(Connection connection) throws Exception {
// your code...
}
}
Difficult to diagnose without seeing your pom.xml and how your jar is packaged but given the folder structure of your target directory above, perhaps either the V3__SampleJava_script.class is added to the jar under classpath:resources/db/migration or is just not included at all.
To check, try unzipping the jar file:
jar -xf flyway-database-migration.jar
or just listing the contents:
jar -tf flyway-database-migration.jar
It is also worth noting that if you have overridden the locations setting with a filesystem: path, the documentation states that the directory "may only contain sql migrations", any java migrations will just be ignored.
Update 2018-06-03
Given that the flyway-database-migration.jar is being pointed to a filesystem: location, only sql migrations will be discovered, not java ones. The database-migration-scripts directory needs to be added to the classpath and the flyway location set to classpath:db/migration.
java -cp C:\database-migration-scripts;<existing classpath> ...
Update 2018-06-09
Because of the way that the flyway-database-migrations.jar is packaged, using the Spring Boot Executable Jar format, all the application dependencies are placed in the BOOT-INF/lib directory inside the executable jar and are loaded by a separate classloader from the org.springframework.boot.loader.JarLauncher main class. So, contrary to the above, I'm not sure if it's possible to pass additional classpath entries to the application using the -cp command line option. I think you would need to remove spring boot and package it as a regular jar file. I certainly encountered class visibility issues and decided to try a different approach.
I downloaded the Flyway Command Line Runner and updated the <RUNNER_DIR>/conf/flyway.conf settings with the following:
flyway.url=jdbc:mariadb://localhost:3306/flyway?createDatabaseIfNotExist=true
flyway.user=root
flyway.password=yourPassword
flyway.locations=classpath:db/migration
I created a directory under <RUNNER_DIR>/jars/ called database-migration-scripts.jar with the following structure (the .jar is important as Flyway will only add files or directories with this suffix to the classpath):
database-migration-scripts.jar/
└── db
└── migration
├── V1__m1.sql
├── V2__m2.sql
└── V3__SampleJava_script.class
Finally, I added all of the runtime dependencies for the database-migration-scripts project to <RUNNER_DIR>/lib:
lib/
├── animal-sniffer-annotations-1.14.jar
├── checker-compat-qual-2.0.0.jar
├── checker-qual-2.3.0.jar
├── error_prone_annotations-2.1.3.jar
├── flyway-commandline-5.1.1.jar
├── flyway-core-5.1.1.jar
├── guava-23.6-jre.jar
├── j2objc-annotations-1.1.jar
├── jcl-over-slf4j-1.7.25.jar
├── jsr305-1.3.9.jar
├── jul-to-slf4j-1.7.25.jar
├── log4j-over-slf4j-1.7.25.jar
├── logback-classic-1.1.11.jar
├── logback-core-1.1.11.jar
├── slf4j-api-1.7.25.jar
├── snakeyaml-1.17.jar
├── spring-aop-4.3.13.RELEASE.jar
├── spring-beans-4.3.13.RELEASE.jar
├── spring-boot-1.5.9.RELEASE.jar
├── spring-boot-autoconfigure-1.5.9.RELEASE.jar
├── spring-boot-starter-1.5.9.RELEASE.jar
├── spring-boot-starter-jdbc-1.5.9.RELEASE.jar
├── spring-boot-starter-logging-1.5.9.RELEASE.jar
├── spring-context-4.3.13.RELEASE.jar
├── spring-core-4.3.13.RELEASE.jar
├── spring-expression-4.3.13.RELEASE.jar
├── spring-jdbc-4.3.13.RELEASE.jar
├── spring-tx-4.3.13.RELEASE.jar
├── tomcat-jdbc-8.5.23.jar
└── tomcat-juli-8.5.23.jar
After that I was able to successfully run:
./flyway migrate
And was able to verify that both sql and java migrations had been successfully applied:
./flyway info
+-----------+---------+-------------------+-------------+---------------------+---------+
| Category | Version | Description | Type | Installed On | State |
+-----------+---------+-------------------+-------------+---------------------+---------+
| Versioned | 1 | m1 | SQL | 2018-06-09 07:41:57 | Success |
| Versioned | 2 | m2 | SQL | 2018-06-09 07:41:57 | Success |
| Versioned | 3 | SampleJava script | SPRING_JDBC | 2018-06-09 07:47:56 | Success |
+-----------+---------+-------------------+-------------+---------------------+---------+
Phew! This is much harder work than packaging the migrations with the application, in my opinion.
One of the other downsides of this approach is that if someone adds a new java migration with an additional dependency (e.g. commons-lang3, or whatever else) that dependency needs to be added to the <RUNNER_DIR>/lib directory as well.
Hope this helps!
Related
I have a .jar that I built following the Oracle docs, using jar cfm hangman.jar Manifest.txt src/classes/app/Main.class. The manifest.txt file contains Main-Class as classes.app.Main, telling where my Main class is. When executed, ClassNotFoundException is thrown, saying it couldn't find classes.app.Main. I need help trying to understand what's wrong here. Is it the main class or maybe a missing classpath?
Here's the project tree:
.
├── hangman.jar
├── Manifest.txt
├── README.md
└── src
├── app
│ ├── Main.java
│ ├── Player.java
│ ├── Players.java
│ ├── Play.java
│ ├── Themes.java
│ ├── Word.java
│ └── Words.java
└── classes
└── app
├── Main.class
├── Play.class
├── Player.class
├── Players.class
├── Themes.class
├── Word.class
└── Words.class
You don't show the code, but it is extremely likely that the package for your class is just app not classes.app, and classes is only a directory name to contain the class files, not actually part of the package hierarchy. The name of a class file entry in a jar, OR the name of a class file relative to a classpath directory, must be exactly a directory path equal to the package hierarchy (if any) plus the class name and the suffix .class, with nothing added or removed. This means your jar should be created by going to the classes directory and then adding the file(s) relative to that directory:
jar cfm hangman.jar Manifest.txt -C classes app/Main.class
and the Main-class entry in the manifest should be app.Main. If you only need main-class in the manifest and nothing else (except version, IIRC), you can have jar create it for you:
jar cfe hangman.jar app.Main -C classes app/Main.class
Also I note that there are other classes in your source tree. If these classes are called or referenced from the Main class, directly or indirectly (i.e. nested), they must also be in the jar. You probably want to use app/* instead, although it is possible you want or even need to be more selective.
Meta: I thought this was covered in the standard tutorial, but although most of the pieces are there they aren't really pulled together anyplace I could find and refer to.
I am trying to get a better understanding of how the new java module system works, especially when it comes to bundling together java packages to their respective modules.
I could not find a way of asking this question in an easy to understand way without tying it to a specific example.
If I have the following project structure:
.
└── src
└── main
└── java
├── module-info.java
└── org
└── lb
└── app
└── App.java
where module-info.java is:
module lb.module {}
and App.java is:
package org.lb.app;
public class App {
public static void main (String[] args) {
System.out.println("Hello, world!");
}
}
and then I run the command from the . directory:
javac -d build/classes $(find src/main/ -name "*.java")
which gives me the following result:
.
├── build
│ └── classes
│ ├── module-info.class
│ └── org
│ └── lb
│ └── app
│ └── App.class
└── src
└── ...
and finally I run the command:
java --module-path build/classes/ --module lb.module/org.lb.app.App
My questions are:
How does the javac tool knows that module-info.class should be in the root directory folder build/classes if there is no "package information" (e.g. like there are in normal java files), and not, for instance in a folder that mimics the original location (e.g. build/classes/src/main/java/module-info.class
I did not expect it to work, but it does. How does the JVM knows that the org.lb.app package belongs to the lb.module module? I purposefully "misnamed" the module, making it not have same name as the directory that contains it, and also did not export the package. It seems obvious for a human reader, but I want to understand what the java program assumes in order to bundle these two together.
Thank you.
How can I import a library from maven central into a project with the ceylon import-jar command?
Please show the full command.
E.g. for "joda-time-2.9.4.jar" from "http://repo.maven.apache.org/maven2/" into a local directory.
I guess it must be:
./ceylon-1.2.3/bin/ceylon import-jar --rep "http://repo.maven.apache.org/maven2/" --verbose --out localdir "joda-time:joda-time/2.9.4" "joda-time-2.9.4.jar"
But as far as I can see the tool is not working (ceylon versions 1.2.2 and 1.2.3).
Working with maven central is essential.
This question is linked with The ceylon copy tool because both tools present me with a riddle.
I understand you are asking about the ceylon import-jar tool specifically, but would like to offer a different solution that is easier if your goal is to import a jar from a remote repository.
I would suggest you use the Ceylon Gradle Plugin, which I wrote.
It knows how to grab dependencies from repositories (including JCenter and Maven Central, but many others), and it will run the ceylon -import-jar tool for you automatically.
Full Example:
Run the following command to create a new test project (enter simple for the folder name):
ceylon new simple --module-name=com.athaydes.test --module-version=1.0
Enter the new project name and have a look at what's in it (minimum Ceylon project):
cd simple
tree # or use Finder, Window Explorer or whatever
You'll see this:
└── source
└── com
└── athaydes
└── test
├── module.ceylon
├── package.ceylon
└── run.ceylon
Edit module.ceylon so it has the following contents (add whatever dependencies you want):
module com.athaydes.test "1.0" {
native("jvm")
import joda_time.joda_time "2.9.4";
}
Notice the name of the module must be a valid Ceylon identifier! So, the Gradle plugin replaces invalid characters with _, generating a valid Ceylon identifier from the Maven artifact name.
Create a build.gradle file at the root of the project so the Gradle plugin can work, with the following contents:
plugins {
id "com.athaydes.ceylon" version "1.2.0"
}
repositories {
jcenter()
}
ceylon {
module = "com.athaydes.test"
flatClasspath = false
importJars = true
forceImports = true // necessary to bypass optional dependencies issues in Maven poms
}
dependencies {
ceylonCompile "joda-time:joda-time:2.9.4"
}
We must declare this dependency here as a normal Maven dependency so Gradle knows where to get the Jars from.
Done... now just run importJars:
gradle importJars
Or, to just see the actual command generated (will not actually run it):
gradle -P get-ceylon-command importJars
Here's the generated command:
ceylon import-jar
--force
--descriptor=/Users/renato/programming/experiments/ceylon-gradle/simple/build/module-descriptors/joda_time_2.9.4.properties
--out=/Users/renato/programming/experiments/ceylon-gradle/simple/modules
--rep=aether:/Users/renato/programming/experiments/ceylon-gradle/simple/build/maven-settings.xml
--rep=/Users/renato/programming/experiments/ceylon-gradle/simple/modules
joda_time.joda_time/2.9.4
/Users/renato/.gradle/caches/modules-2/files-2.1/joda-time/joda-time/2.9.4/1c295b462f16702ebe720bbb08f62e1ba80da41b/joda-time-2.9.4.jar
The jars will be imported to the default location, modules (but you can configure that):
── build
│ ├── dependency-poms
│ │ └── joda-time-2.9.4.pom
│ ├── maven-repository
│ │ └── joda-time
│ │ └── joda-time
│ │ └── 2.9.4
│ │ ├── joda-time-2.9.4.jar
│ │ └── joda-time-2.9.4.pom
│ ├── maven-settings.xml
│ └── module-descriptors
│ └── joda_time_2.9.4.properties
├── build.gradle
├── modules
│ └── joda_time
│ └── joda_time
│ └── 2.9.4
│ ├── joda_time.joda_time-2.9.4.jar
│ ├── joda_time.joda_time-2.9.4.jar.sha1
│ └── module.properties
└── source
└── com
└── athaydes
└── test
├── module.ceylon
├── package.ceylon
└── run.ceylon
Now you can run the Ceylon code with the runCeylon task (or just run if there's no other task with this name):
gradle run
NOTE:
Unfortunately, actually importing the specific Jar you chose into the Ceylon repo is impossible with its original name... because in Ceylon, joda-time is an illegal identifier... so you need to change the name of the module when imported by Ceylon. The Gradle plugin does it for you.. but you need to know what the valid identifier will be to be able to write the import statement in the module file (you can just let the plugin run and it will tell you what the name will be).
A much simpler approach
If you want to avoid the complexity of this approach, you can just use the default Gradle plugin approach to NOT import Maven jars into the Ceylon repository and just use the simple Java classpath (which means you relinquish using the Ceylon modules system!).
If you do that, your build.gradle file will look like this:
plugins {
id "com.athaydes.ceylon" version "1.2.0"
}
repositories {
jcenter()
}
ceylon {
module = "com.athaydes.test"
}
And the module.ceylon file:
module com.athaydes.test "1.0" {
native("jvm")
import "joda-time:joda-time" "2.9.4";
}
Notice that we don't need to mess up with the dependency name using this approach. From Ceylon 1.2.3, you should prepend the dependency with the maven: qualifier to avoid warnings.
That simple!
1. As a (partial) answer to my question, this turned out to work:
$ ../bin/ceylon import-jar --rep flat:"../flat/" Jama/1.0.3 ../flat/Jama-1.0.3.jar
I downloaded the jar (in this case Jama-1.0.3.jar) by hand and then I was able to import it.
I had to try a lot to find out where to put the prefix "flat:", i.e. either to put it after "import" in the module descriptor "module.ceylon" or on the command line. The latter turned out to be the right choice.
But still, I haven't been able to find out how to import the jar from maven directly using the import-jar tool.
2. More detailed documentation is needed about managing modules. Specifically, there should be a clarification what the term "legacy repository" means.
Does "legacy" mean "deprecated"?
3. I hope that the following way to import dependencies into a project is not considered as "legacy" or "deprecated":
a) Rename the jar file, so that the name relfects the compressed directory structure within the jar.
b) Put the jar into a directory structure that again reflects the directory structure within the jar.
c) Put all that into the modules directory of the project, merging directories if necessary.
This seems to be the most explicit and reliable way to include dependencies into a project and I hope this way will not be deprecated or considered "legacy" at any time.
I have a gradle-based java project with this structure
.
├── myproject
│ ├── src
│ | └── main
│ | ├── java
│ | └── resources
│ | └── myresource.xml
| ├── build
| | ├── classes
| | | └── main
│ | | └── myresource.xml
| | ├── resources
I'm trying to access some files in the resources folder using a ClassLoader, like this
ClassLoader.getSystemClassLoader().getResoure("/myresource.xml");
but it does not find the file.
The only way I have found to access those files is by exploring the known structure of the project
Path resourcesPath= FileSystems.getDefault().getPath(System.getProperty("user.dir"), "/src/main/resources/");
Any idea on what am I doing wrong?
Well, it seems my difficulties came from another problem (resources not being copied to the proper places). Once I solved that problem, the ClassLoader was able to find my resources using either of these two forms:
ClassLoader.getSystemClassLoader().getResource("./myresource.xml");
ClassLoader.getSystemClassLoader().getResource("myresource.xml");
Edit: When using the jar embedded in other applications, the former solution do not work, use this in that case:
Thread.currentThread().getContextClassLoader().getResource("myresource.xml")
http://docs.oracle.com/javase/7/docs/api/java/lang/Class.html#getResource(java.lang.String)
For example something like MyMain.class.getResource("/config.txt") or use relative path when appropriate.
Maybe you should use it this way:
Thread.currentThread().getContextClassLoader().getResource("myresource.xml")
If i use ClassLoader.getSystemClassLoader().getResource("myresource.xml")
When I use it as a jar package embedded in other applications, I still have no access to resources.
Another solution to access Java resources in the build.gradle with in Gradle in Groovy is to do for example to read VERSION file in the Java resources:
static String getVersion(File rootFile) {
File versionFile = new File(rootFile, "src/main/resources/VERSION")
byte[] versionBytes = Files.readAllBytes(versionFile.toPath())
String version = new String(versionBytes, StandardCharsets.UTF_8).trim()
version
}
version = getVersion(project.projectDir)
It avoids manipulating the ClassLoader.
Note: sourceSets.main.output.resourcesDir does not exist yet in function of the task.
I am creating WLST scripts which will be run against a server running weblogic which is fairly locked down. I only have permission to view logs and read only access to the console unless a pre-authorised change. I don't have permission to access the entire weblogic domain and run WLST.
I would like to run WLST on a differernt server which has Java installed but not Weblogic.
I was initially hoping to add a weblogic jar to the classpath then run the tool but it seems a bit more complicated that that.
I have been following these instructions without success.
My current setup is this directory structure with jars taken from weblogic 12.1.1:
├── launch.sh
├── lib
│ ├── com.bea.core.utils.full_2.0.0.0.jar
│ ├── com.bea.core.xml.xmlbeans_2.2.0.0.jar
│ ├── com.oracle.cie.comdev_6.4.0.0.jar
│ ├── com.oracle.cie.config_7.2.0.0.jar
│ ├── com.oracle.cie.config-wls_7.2.0.0.jar
│ ├── com.oracle.cie.config-wls-schema_10.3.6.0.jar
│ ├── com.oracle.cie.wizard_6.1.0.0.jar
│ ├── com.oracle.core.weblogic.msgcat_1.3.0.0.jar
│ ├── jython.jar
│ ├── weblogic.jar
│ ├── weblogic.server.modules.jsf2.0_12.1.1.0.jar
│ ├── wlclient.jar
│ ├── wlfullclient.jar
│ └── wlthint3client.jar
└── props.txt
props.txt is empty described on the Oracle forum and launch.sh contains:
java -cp $(echo lib/*.jar | tr ' ' ':') -Dprod.props.file=props.txt -Dbea.home= -Dweblogic.home= weblogic.WLST
on running launch.sh, i get the error:
sam#ubuntu64vm:~/Desktop/scripts$ ./launch.sh
Initializing WebLogic Scripting Tool (WLST) ...
Problem invoking WLST - java.lang.NullPointerException
Is it possible to create a minimal / portable WLST application?
Standalone WSLT works for Weblogic 10.3.4 by running the following command (see Note 3, this stopped the java.lang.NullPointerException):
java -cp lib/wlfullclient.jar;lib/com.bea.core.xml.xmlbeans_2.2.0.0.jar;lib/com.oracle.cie.comdev_6.3.0.0.jar;lib/com.oracle.cie.config-wls-schema_10.3.4.0.jar;lib/com.oracle.cie.config-wls_7.1.0.0.jar;lib/com.oracle.cie.config_7.1.0.0.jar;lib/com.oracle.cie.wizard_6.1.0.0.jar;lib/com.oracle.core.weblogic.msgcat_1.1.0.0.jar;lib/jython.jar;lib/weblogic.jar -Dprod.props.file=lib/props.txt -Dbea.home= -Dweblogic.home=c:/users/username/wls10 weblogic.WLST your-script.py
Notes about script:
My Example setup of WLST is run from c:/users/username/wls10
The required jar are in c:/users/username/wls10/lib.
The weblogic.home must be set to an absolute path e.g. c:/users/username/wls10'.
You must create a blank props.txt in /lib directory.
I also followed same instructions as above.
my list of libraries needed is following:
coherence.jar
com.bea.core.xml.xmlbeans.jar
com.oracle.cie.comdev_7.7.0.0.jar
com.oracle.cie.config_8.4.0.0.jar
com.oracle.cie.config-external_8.4.0.0.jar
com.oracle.cie.config-owsm_8.4.0.0.jar
com.oracle.cie.config-security_8.4.0.0.jar
com.oracle.cie.config-wls_8.4.0.0.jar
com.oracle.cie.config-wls-external_8.4.0.0.jar
com.oracle.cie.config-wls-schema_8.4.0.0.jar
com.oracle.cie.dependency_1.7.0.0.jar
com.oracle.cie.encryption_2.4.0.0.jar
com.oracle.cie.service-table_1.4.0.0.jar
com.oracle.cie.wizard_7.7.0.0.jar
com.oracle.core.weblogic.msgcat.jar
com.oracle.glcm.common-logging_1.5.0.0.jar
com.oracle.glcm.encryption_2.6.0.0.jar
com.oracle.weblogic.lifecycle.provisioning.api.jar
com.oracle.weblogic.lifecycle.provisioning.core.jar
com.oracle.weblogic.lifecycle.provisioning.wlst.jar
cryptoj.jar
jython-modules.jar
weblogic.jar
wlfullclient.jar
wls-api.jar
wlst-impl.jar
I used jarscan utility to locate all missing classes by trying to run several wlst scripts. Total size is cca 150MB
My launch.sh is following:
#!/bin/bash
MYDIR=$(dirname $0)
CP=$(echo $MYDIR/lib/*.jar | tr ' ' ':')
echo $CP
java -cp $CP -Dprod.props.file=$MYDIR/props.txt -Dbea.home=`pwd` -Dweblogic.home=`pwd` weblogic.WLST $#
Not perfect, but working for me.
I made a script to retrieve all the required files for a Standalone WLST tool
The repository is https://github.com/cheloide/wlst-standalone
The script extracts and install Weblogic Server in a temporary location, creates wlfullclient.jar and then copies the required resources to a directory of your choosing or the working path
I also made another script in the same repo to use the Weblogic.Deployer tool
Currently the tool only works with GNU/Linux; should work with Mac-OS with some tweaks.
I recently faced the same issue on WLS 12.1.3 and ended up with a different set of dependent jars. Here's what I needed:
wlfullclient.jar
weblogic.jar
com.bea.core.xml.xmlbeans_1.0.0.0_2-6-0.jar
com.oracle.cie.comdev_7.1.0.0.jar
com.oracle.cie.config-owsm_8.1.0.0.jar
com.oracle.cie.config-security_8.1.0.0.jar
com.oracle.cie.config-wls-schema_12.1.3.0.jar
com.oracle.cie.config-wls_8.1.0.0.jar
com.oracle.cie.config_8.1.0.0.jar
com.oracle.cie.dependency_1.1.0.0.jar
com.oracle.cie.encryption_2.1.0.0.jar
com.oracle.cie.service-table_1.1.0.0.jar
com.oracle.cie.wizard_7.1.0.0.jar
com.oracle.core.weblogic.msgcat_3.0.0.0.jar
jython-modules.jar
This has been tested for start/stop server and undeploy/deploy application.