We Keep Coding

How to run a Java/Maven application with external libraries in Heroku?

I've been building a Java application, which runs as expected when executing it locally. Now I'm trying the next step, to move it into an external platform. I chose Heroku because of their free plans for a starter, and also because it lets me have a small PostGreSQL database, which I use along with the application.
Due to the way Heroku works, I was forced to integrate Java with Maven, which I know little to none about. But I was still able to work my way there until I was able to compile and deploy the application (source code is in GitHub), albeit with errors.
For once, I'm using a couple external libraries:
jsoup-1.13.1.jar – Jsoup, for easier parsing of HTML files
mail.jar – Java mail, for emailing reports
Whenever the application is deployed in Heroku, there's an Exception in thread "main" java.lang.NoClassDefFoundError: org/jsoup/Jsoup, ultimately triggered by the Jsoup.connect invocation. From what I understand, Maven only add the references to the program but does not include the actual libraries there for runtime, which I think may be the cause of the issue.
I've tried modifying my pom.xml file by adding these dependencies manually:
<dependencies>
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<version>42.2.18</version>
</dependency>
<dependency>
<groupId>com.sun.mail</groupId>
<artifactId>javax.mail</artifactId>
<version>1.6.2</version>
</dependency>
<dependency>
<groupId>org.jsoup</groupId>
<artifactId>jsoup</artifactId>
<version>1.13.1</version>
</dependency>
</dependencies>
And also these plugins in the build section:
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>3.0.0</version>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<finalName>checker</finalName>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.4</version>
<executions>
<execution>
<id>copy-dependencies</id>
<phase>package</phase>
<goals>
<goal>copy-dependencies</goal>
</goals>
</execution>
</executions>
</plugin>
What am I doing wrong? I would be very grateful for hints to help me get my application running, at least resolving the reason why Jsoup isn't working at all in Heroku. Thanks in advance!

How to run jar file of a maven project generated after build?

I made a simple project with a few tests in it and I want to be able to launch tests on other computers. I built the project using Run As -> Maven build... -> Goals: package in Eclipse, and I found a jar file in target folder of the project. But when I try to run it in cmd using java -jar project.jar I get the following error:
Error: Main method not found in class com.example.TestPurchase, please define the main method as:
public static void main(String[] args)
or a JavaFX application class must extend javafx.application.Application
As far as I know, TestNG doesn't need any Main method, because of the annotations.
That brings me to some questions:
Is something wrong with how I build my project?
Did I understand the method of executing tests via jar file correct?
Do I even have to use that jar file? Because I can run tests from command line being in project folder using mvn test
Here is my pom.xml file:
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.example</groupId>
<artifactId>Sightsy</artifactId>
<version>0.0.1-SNAPSHOT</version>
<properties>
<java.version>1.8</java.version>
<selenium.version>3.12.0</selenium.version>
<testng.version>6.13.1</testng.version>
<javafaker.version>0.14</javafaker.version>
<guava.version>23.2-jre</guava.version>
<extentreports.version>3.0.7</extentreports.version>
<extenttestng.version>1.3.1</extenttestng.version>
<assertj.version>3.8.0</assertj.version>
<maven.compiler.version>3.7.0</maven.compiler.version>
<commons.version>3.7</commons.version>
<commons.io.version>2.6</commons.io.version>
</properties>
<dependencies>
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-java</artifactId>
<version>${selenium.version}</version>
</dependency>
<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<version>${testng.version}</version>
</dependency>
<dependency>
<groupId>com.github.javafaker</groupId>
<artifactId>javafaker</artifactId>
<version>${javafaker.version}</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>${guava.version}</version>
</dependency>
<dependency>
<groupId>com.aventstack</groupId>
<artifactId>extentreports</artifactId>
<version>${extentreports.version}</version>
</dependency>
<dependency>
<groupId>com.vimalselvam</groupId>
<artifactId>testng-extentsreport</artifactId>
<version>${extenttestng.version}</version>
</dependency>
<dependency>
<groupId>org.assertj</groupId>
<artifactId>assertj-core</artifactId>
<version>${assertj.version}</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>${commons.version}</version>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>${commons.io.version}</version>
</dependency>
<dependency>
<groupId>com.opencsv</groupId>
<artifactId>opencsv</artifactId>
<version>3.4</version>
</dependency>
<dependency>
<groupId>org.apache.poi</groupId>
<artifactId>poi</artifactId>
<version>3.12</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.poi</groupId>
<artifactId>poi-ooxml</artifactId>
<version>3.12</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-lambda-java-core</artifactId>
<version>1.2.0</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>${maven.compiler.version}</version>
<configuration>
<source>${java.version}</source>
<target>${java.version}</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.2.1</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>com.example.TestPurchase</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.21.0</version>
<configuration>
<suiteXmlFiles>
<suiteXmlFile>src/main/resources/suites/testng.xml</suiteXmlFile>
</suiteXmlFiles>
</configuration>
</plugin>
</plugins>
</build>

I'm not familiar with TestNG framework, but my understanding is as following:
When "mvn package" is executed, your project is "validated", "compiled" and then "packed". During compilation, your code is changed to bytecode and can be interpreted (executed) by java virtual machine (JVM). This compiled code has only your application (I assume, that this is an application that is executed on Java server).
During the "package" step, these compiled classes (without tests) are put together in jar file. This jar file should contain only your application (again, without tests) - this is what you want because your jar file is smaller and contain only what is really needed.
Now when "mvn test" is executed, classes with test code are compiled and then executed. TestNG add the main function "automatically" so JVM knows what to do. Please note that running again "mvn package" will still NOT include these tests in jar file.
To address your questions directly:
Ad 1. No
Ad 2. Yes, "mvn test" is the correct way.
Ad 3. No, you should not use jar file during testing. It is used later during deployment.

First of all, you must have a separate class with "main" method, where you will specify your xml suite files.
package com.example;
import java.util.ArrayList;
import java.util.List;
import org.testng.TestNG;
public class MainClass {
public static void main(String[] args) {
TestNG testSuite = new TestNG();
List<String> suites = new ArrayList<String>();
suites.add("path_to_your_xml_suite_file_in_target_folder");
testSuite.setTestSuites(suites);
testSuite.run();
}
}
Then, you will need to specify this class with "main" method in configuration of maven-shade-plugin in pom.xml
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.2.1</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>com.example.TestNGMainClass</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
Then, when you will execute mvn build package without performing tests, you will have a fully working executable jar file in target folder, which you can launch in command line using java -jar name_of_you_jar_file.jar. It will need drivers and xml suite files to work

Spark: ClassNotFoundException while reading/ writing CSV

I'm trying to write a DataFrame as follows to a CSV file on HDFS
df.write()
.format("com.databricks.spark.csv")
.option("header", "true")
.save("/user/cloudera/csv");
but I get the following error
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/csv/CSVFormat
...
Caused by: java.lang.ClassNotFoundException: org.apache.commons.csv.CSVFormat
... 21 more
My pom.xml has the following dependencies
<dependency>
<groupId>com.databricks</groupId>
<artifactId>spark-csv_2.10</artifactId>
<version>1.5.0</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-csv</artifactId>
<version>1.5</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.10</artifactId>
<version>1.6.0</version>
</dependency>
I use spark 1.6.0 with scala 2.10.5 and use the following command to submit the job
spark-submit --jars /path/spark-csv_2.10-1.5.0.jar --class com.iris.Begin /path/CsvSolver.jar
I also have commons-csv/1.1 and commons-csv/1.5 in .m2 repository.
Could someone help me with this?

Just try to add those need jars to jars folder located in spark folder ...\spark\jars\

<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-csv</artifactId>
<version>1.5</version>
</dependency>
Try adding this to the pom. If that doesn't work manually download the JAR from here https://mvnrepository.com/artifact/org.apache.commons/commons-csv/1.5 and add using --jars to spark submit. That would definitely solve the problem

It's better to build fat jar that will include all your dependencies (spark-core should be marked as provided) & submit only this jar without any additional --jars options.
In Maven you can generate fat jar by using Maven Assembly plugin with predefined profile jar-with-dependencies. Something like:
<build>
<plugins>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>3.1.0</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
</plugin>
</plugins>
</build>

How to use dependencies from a jar file?

I am trying to import a jar file and to use its dependencies into a Maven project.
The jar file I want to import is itself another Maven project and includes all of the dependencies of it.
I managed to import my jar file and to use the java code (from my packages in src/main in the Maven project jar file) but the project where I imported mt jar file still does not recognize the dependencies of the jar.
For example, I am trying to import org.json.JSONObject (whom dependency is defined in the jar file) but there is a compiling error.
Is there any way to do what I want or any other solution ?
Thank you !
(Sorry for the bad level of explanation, I am French and it is quite hard form me to explain in details my problem)
EDIT
Here are samples of my pom from my jar file :
<!-- One of the dependency I want to use -->
<dependency>
<groupId>org.json</groupId>
<artifactId>json</artifactId>
<version>20160212</version>
<scope>test</scope>
</dependency>
<!-- The plugin I used to create my jar file with dependencies -->
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.2</version>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
I then included the jar file using an "in-project" repository as explained here : http://charlie.cu.cc/2012/06/how-add-external-libraries-maven/
The error message is a compiling error message saying that the org.json.JSONObject is not recognized :
package org.json does not exist

You add the json lib only for test scope.
Remove the scope and the error should gone.
<!-- One of the dependency I want to use -->
<dependency>
<groupId>org.json</groupId>
<artifactId>json</artifactId>
<version>20160212</version>
</dependency>

Jackson ObjectMapper giving me a VerifyError only when compiled/packaged with Maven

I'm a relatively new Java programmer, and I've only worked with Maven and Jackson on a couple projects.
In my current project, I am attempting to build a process I can run on Heroku (which I'm brand new to) once every 24 hours. Part of this program uses Jackson to deserialize json to use in an array. When run through my IDE (IntelliJ), everything works fine. However, when I run the program after a mvn package I get the following error:
Exception in thread "main" java.lang.VerifyError:
(class: org/codehaus/jackson/map/ObjectMapper, method: <init> signature: (Lorg/codehaus/jackson/JsonFactory;Lorg/codehaus/jackson/map/SerializerProvider;Lorg/codehaus/jackson/map/DeserializerProvider;Lorg/codehaus/jackson/map/SerializationConfig;Lorg/codehaus/jackson/map/DeserializationConfig;)V) Bad type in putfield/putstatic
at com.example.project.Main.main(Main.java:47)
This is obviously a showstopper since it keeps me from deploying to Heroku.
The line throwing the error is when I initialize an ObjectMapper:
ObjectMapper mapper = new ObjectMapper();
As for my POM, I was following this tutorial from Heroku. Here are the relevant sections from my POM:
...
<build>
<sourceDirectory>src/main/java</sourceDirectory>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>appassembler-maven-plugin</artifactId>
<version>1.1.1</version>
<configuration>
<assembleDirectory>target</assembleDirectory>
<programs><program>
<mainClass>com.example.project.Main</mainClass>
<name>main</name>
</program></programs>
</configuration>
<executions>
<execution>
<phase>package</phase><goals><goal>assemble</goal></goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.4</version>
<executions>
<execution>
<id>copy-dependencies</id>
<phase>package</phase>
<goals><goal>copy-dependencies</goal></goals>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<configuration>
<source>1.6</source>
<target>1.6</target>
</configuration>
</plugin>
</plugins>
</build>
<dependencies>
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-mrbean</artifactId>
<version>1.9.13</version>
</dependency>
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-core-asl</artifactId>
<version>1.9.13</version>
</dependency>
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-asl</artifactId>
<version>0.9.5</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
<version>2.2.2</version>
</dependency>
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-mapper-asl</artifactId>
<version>1.9.13</version>
</dependency>
...
</dependencies>
...
So my process is a mvn clean install and a mvn package (where both succeed), and then sh target/bin/main (where we get the error).
If I can provide any more useful information, please let me know! Thanks in advance!
EDIT:
After running my program differently...
java -cp target/classes:"target/dependency/*" com.example.project.Main
I found that I essentially ran into the same problem as #Jithin from this post:
Jackson + Tomcat - java.lang.VerifyError: Cannot inherit from final class
The error was coming from having conflicting Jackson versions in my classpath/dependency tree (i.e., one of my other dependencies was also using Jackson). I found that, in my case, the only Jackson dependency I required was jackson-annotations. The program now runs like a charm! Hopefully this helps someone down the road.