I have a hadoop job that gets deployed to Amazon EMR. Because EMR provides a hadoop-core jar, I've configured the dependency in my pom as provided:
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>0.20.2-cdh3u6</version>
<scope>provided</scope>
</dependency>
The problem is that when I want to debug the jar within intellij, it can't find one of the classes in this jar, org.apache.hadoop.util.RunJar.
Is it possible to configure Intellij to include this jar on the classpath when debugging/running the jar within intellij, but still to exclude this jar from the output jar?
The path to the actual jar on my local machine is /Users/jbrown/.m2/repository/org/apache/hadoop/hadoop-core/0.20.2-cdh3u6/hadoop-core-0.20.2-cdh3u6.jar - I've tried adding -cp /Users/jbrown/.m2/repository/org/apache/hadoop/hadoop-core/0.20.2-cdh3u6/hadoop-core-0.20.2-cdh3u6.jar to the VM options in the run config, but then the job couldn't run at all.
Assuming that you are using maven from your dependency example, you can use multiple maven profiles to tackle this problem.
You can have a prod and debug profiles which include a dependencies section. The debug one can skip the provided tag for all hadoop related dependencies, which would include the dependencies in the final JAR.
You can read more about maven profiles here.
Related
I am using maven in my project. In my project, I am using a Java library testcontainers-keycloak. In the pom.xml of that library https://github.com/dasniko/testcontainers-keycloak/blob/2.1.2/pom.xml, many dependencies have been mentioned. I have noticed one thing for the dependencies with the <scope>test</scope> . I am unable to see these dependencies in my .m2 directory. I was curious about it. Can any one tell me that how maven manages the dependencies with the <scope>test</scope> . In which directory, they are downloaded. I know that test scope indicates that the dependency is not required for normal use of the application, and is only available for the test compilation and execution phases but I do not know that where and when maven downloads such dependencies before executing our tests.
I am new to maven, I have some .hbs (handle-bar template files), I have "handlebars-proto", which includes premade handlebar server.
https://github.com/jknack/handlebars.java (search for handlebars-proto)
I've added the dependency to my pom.xml as
<dependency>
<groupId>com.github.jknack</groupId>
<artifactId>handlebars-proto</artifactId>
<version>${current-version}</version>
</dependency>
Documentation states, I should be able to run it using,
java -jar handlebars-proto-${current-version}.jar -dir myTemplates
How can I create a Intellij Run Configuration to do same as above. I tried creating JAR task runner & using "handlevar-proto-{version}.jar" but it's unable to find the jar file. In general how to run a jar from the maven in IntelliJ.
There is option in run configurations to configure JAR Application
This is the first time I am using maven, I want to implement a processor for apache-nifi. Now for this I am using a proprietary jar file which is an SDK. It is not on the repositories. Therefore i have put it in the pom.xml as follows.
<dependency>
<groupId>KS</groupId>
<artifactId>En-SDK</artifactId>
<version>1.0</version>
<scope>system</scope>
<systemPath>${project.basedir}/lib/En-SDK-Java.jar</systemPath>
</dependency>
I am using intelliJ which seem to have no problem importing this dependency. So is the maven build procedure.
I use mvn clean install as explained in this tutorial. Which gives me with a nar package as the output. (no build errors)
This nar , which is supposed to be put in the $NIFI_HOME/lib directory does not bundle the said local jar.
If i place this nar file in the required directory and start apache-nifi,
INFO [main] org.apache.nifi.nar.NarClassLoaders Loaded NAR file: /..././work/nar/extensions/myNar.nar
Seems to be loaded by the NarClassLoaders , but following that I get an exception and nifi doesn't start.
java.lang.NoClassDefFoundError: com/kls/../../SubscriptionInterface
The SubscriptionInterface is a class from the said local jar.
If I look at the nar file's folder structure, inside the META-INF/bundled-dependencies/ i see every other dependant jar files defined in the pom but not this local jar i used.
How to overcome this?
The following Stackoverflow question seems to be similar to what you have asked.
Maven 2 assembly with dependencies: jar under scope "system" not included
Try to put the dependency in a local repository and declare it in the pom.
I want dependencies that are having system scope to be part of my project final executable jar. I tried maven-assembly, maven-shade and maven-dependency plugin. But using these plugins, only those dependency of my project which were present in my local maven repository were getting added. Dependency with system scope (not present in my local maven repo) are not getting added in the final executable jar.
I tried searching over google, but most of the links are suggesting to add it local maven repo first. I have some limitations so I cannot add those dependency on local repo. I want it to picked from file system directly, and wanted it to be part of final executable jar.
<!-- Teradta jdbc dependency -->
<dependency>
<groupId>org.teradata</groupId>
<artifactId>teradata</artifactId>
<version>4.0</version>
<scope>system</scope>
<systemPath>${basedir}/../../../lib/terajdbc4.jar</systemPath>
</dependency>
Above dependency is not getting added in the final jar that maven is building.
Please suggest me the right plugin with its usage for this use case.
Any help on this would be really appreciated.
There is a jar file lets say "abc.jar" which maven dependency does not exist(ie created a jar by using java command of own classes). I want to add this jar as maven dependency so that at build time it will automatically copy that jar in lib folder as like other maven dependency. how i will do. please help .
Add it as a dependency with a system scope. See the docs here.
However, rather than adding it as a system dependency it might be better to mavenize the jar itself, then you can build and install it into your dependency management system.
Also, see this question: Can I add jars to maven 2 build classpath without installing them?
You can use the systemPath attribute in the dependency tag in the POM file of your project.
In your pom.xml, use the following snippet corresponding to abc.jar:
<dependencies>
<!-- Other dependencies -->
<dependency>
<groupId>abc</groupId>
<artifactId>x</artifactId>
<version>1.0</version>
<scope>system</scope>
<systemPath>{path_to_abc.jar}</systemPath>
</dependency>
</dependencies>
The scope parameter corresponding to this artifact must be set to system, for the artifact to be picked up from the specified systemPath.
Hope this helps!
A normal maven dependency is always resolved by looking into a repository. So you must put your JAR file into a repository.
You could install your JAR into your local repository. Have a look at the install plugin. The install-file goal is your friend.
If other developers also need this JAR (because they are working with the same project), they either need to install it locally too, or - better - you deploy the JAR to a remote repository. Have a look at the deploy plugin. Here the deploy-file goal is your friend. For deploying artifacts, you need a repository manager like Nexus or Artifactory.
However, a dependency could also have the system scope (look at the other answers).