I'm trying to load data from Google BigQuery into Spark running on Google Dataproc (I'm using Java). I tried to follow instructions on here: https://cloud.google.com/dataproc/docs/tutorials/bigquery-connector-spark-example
I get the error: "ClassNotFoundException: Failed to find data source: bigquery."
My pom.xml looks like this:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.virtualpairprogrammers</groupId>
<artifactId>learningSpark</artifactId>
<version>0.0.3-SNAPSHOT</version>
<packaging>jar</packaging>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<java.version>1.8</java.version>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.3.2</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>2.3.2</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>com.google.cloud.spark</groupId>
<artifactId>spark-bigquery_2.11</artifactId>
<version>0.9.1-beta</version>
<classifier>shaded</classifier>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.5.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<artifactId>maven-jar-plugin</artifactId>
<version>3.0.2</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
<archive>
<manifest>
<mainClass>com.virtualpairprogrammers.Main</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
</plugins>
</build>
</project>
After adding the dependency to my pom.xml it was downloading a lot to build the .jar, so I think I should have the correct dependency? However, Eclipse is also warning me that "The import com.google.cloud.spark.bigquery is never used".
This is the part of my code where I get the error:
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import com.google.cloud.spark.bigquery.*;
public class Main {
public static void main(String[] args) {
SparkSession spark = SparkSession.builder()
.appName("testingSql")
.getOrCreate();
Dataset<Row> data = spark.read().format("bigquery")
.option("table","project.dataset.tablename")
.load()
.cache();
I think you only added BQ connector as compile time dependency, but it is missing at runtime. You need to either make a uber jar which includes the connector in your job jar (the doc needs to be updated), or include it when you submit the job gcloud dataproc jobs submit spark --properties spark.jars.packages=com.google.cloud.spark:spark-bigquery_2.11:0.9.1-beta.
I faced the same issue and updated the format from "bigquery" to "com.google.cloud.spark.bigquery" and that worked for me.
Specifying the dependency in the build.sbt and using "com.google.cloud.spark.bigquery" in format as suggested by Peter resolved the issue for me.
Related
I am trying to deploy a simple Rest API to Google Cloud Functions. Per this documentation, I need the target to always be org.springframework.cloud.function.adapter.gcp.GcfJarLauncher
However when I deploy it using the provided code, I get:
ERROR: (gcloud.alpha.functions.deploy) OperationError: code=3, message=Build failed:
build succeeded but did not produce the class "org.springframework.cloud.function.adapter.gcp.GcfJarLauncher"
specified as the function target: Error: class not found: org.springframework.cloud.function.adapter.gcp.GcfJarLauncher;
Error ID: d818fd83
Here is the code I am running in the Cloud CLI:
gcloud alpha functions deploy function-sample-gcp-http --entry-point org.springframework.cloud.function.adapter.gcp.GcfJarLauncher --runtime java11 --trigger-http --source target/deploy --memory 512MB
Here my code repo but I will include some relevant bits below
pom.xml:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.7.5</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<groupId>com.divr-fx-test</groupId>
<artifactId>divr-function</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>divr-function</name>
<description>Function testing for DIVR</description>
<properties>
<java.version>11</java.version>
<spring-cloud-function.version>4.0.0-SNAPSHOT</spring-cloud-function.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-function-web</artifactId>
<version>3.2.8</version>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-function-web</artifactId>
<version>3.2.8</version>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-function-dependencies</artifactId>
<version>3.2.8</version>
<type>pom</type>
</dependency>
<dependency>
<groupId>com.google.cloud.functions</groupId>
<artifactId>functions-framework-api</artifactId>
<version>1.0.4</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-function-adapter-gcp</artifactId>
<version>3.2.8</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<!-- Build an executable JAR -->
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<configuration>
<archive>
<manifest>
<addClasspath>true</addClasspath>
<classpathPrefix>lib/</classpathPrefix>
<mainClass>com.mypackage.MyClass</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<outputDirectory>target/deploy</outputDirectory>
</configuration>
</plugin>
<plugin>
<groupId>com.google.cloud.functions</groupId>
<artifactId>function-maven-plugin</artifactId>
<version>0.9.1</version>
<configuration
<functionTarget>org.springframework.cloud.function.adapter.gcp.GcfJarLauncher</functionTarget>
<port>8080</port>
</configuration>
</plugin>
</plugins>
</build>
</project>
Main:
package com.fkgcp;
import java.util.function.Function;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
#SpringBootApplication
public class DivrFunctionApplication {
public static void main(String[] args) {
SpringApplication.run(DivrFunctionApplication.class, args);
}
// this part I don't actually want to use, but it is present
// in every tutorial, so I included it to see if it would be
// triggered instead of my REST functions
#Bean
public Function<String, String> uppercase() {
return value -> value.toUpperCase();
}
}
Also, I added a MANIFEST.MF, is it in the right location?
divr-function/src/main/resources/META-INF/MANIFEST.MF
Contents:
Main-Class: com.fkgcp.DivrFunctionApplication
What am I missing?
Reviewing your source code repository, I think the issue can be caused because you need to reference the spring-cloud-function-adapter-gcp library as a dependency of the spring-boot-maven-plugin in addition to include it as a dependency in your pom.xml file:
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<outputDirectory>target/deploy</outputDirectory>
</configuration>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-function-adapter-gcp</artifactId>
</dependency>
</dependencies>
</plugin>
According to the Spring Cloud documentation:
Notice that we also reference spring-cloud-function-adapter-gcp as a
dependency of the spring-boot-maven-plugin. This is necessary because it
modifies the plugin to package your function in the correct JAR format for
deployment on Google Cloud Functions.
In addition, and although probably unrelated, consider remove your maven-jar-plugin configuration from your pom.xml: the spring-boot-maven-plugin already provides all the necessary functionality to generate the jar you need to deploy and the configuration provided by the maven-jar-plugin may be a cause of errors.
On the other hand, your MANIFEST.MF file looks fine to me and it is placed in the right location.
I am planning to use Cucumber Test with Junit 5 in Maven. So I followed cucumber to install different maven dependency. I added a runner class to execute my cucumber tests
package pirate;
import org.junit.platform.suite.api.ConfigurationParameter;
import org.junit.platform.suite.api.IncludeEngines;
import org.junit.platform.suite.api.SelectClasspathResource;
import org.junit.platform.suite.api.Suite;
import static io.cucumber.junit.platform.engine.Constants.GLUE_PROPERTY_NAME;
#Suite
#IncludeEngines("cucumber")
#SelectClasspathResource("pirate")
#ConfigurationParameter(key = GLUE_PROPERTY_NAME, value = "pirate")
public class Runner {}
I also created a new folder name pirate under resources folder and move all .feature files into that new folder.
But when I execute mvn clean install, the command fails at testCompile:
It seems like the compiler can't read the package name?
Below is my pom.xml
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.example</groupId>
<artifactId>onetwothree</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>
<name>onetwothree</name>
<url>http://maven.apache.org</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<build>
<testSourceDirectory>src/test/java</testSourceDirectory>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.1</version>
<configuration>
<source>11</source>
<target>11</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0-M5</version>
<configuration>
<properties>
<configurationParameters>
cucumber.junit-platform.naming-strategy=long
</configurationParameters>
</properties>
</configuration>
</plugin>
</plugins>
</build>
<dependencies>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter</artifactId>
<version>5.9.0</version>
</dependency>
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-java</artifactId>
<version>7.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-junit-platform-engine</artifactId>
<version>7.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.platform</groupId>
<artifactId>junit-platform-suite</artifactId>
<version>1.9.0</version>
<scope>test</scope>
</dependency>
</dependencies>
</project>
My project structure is:
but if I comment out cucumber-junit-platform-engine dependency in pom.xml, the error is gone but it won't run the cucumber tests. Did I miss something here?
Not sure if it's related, but one of the error messages is like:
[ERROR] error reading /Users/xx/.m2/repository/org/junit/platform/junit-platform-engine/1.9.1/junit-platform-engine-1.9.1.jar; zip file is empty
[ERROR] /Users/xx/Desktop/zz/src/test/java/pirate/Runner.java:[1,1] cannot access pirate
ZipException opening "junit-platform-engine-1.9.1.jar": zip END header not found
Maven : error in opening zip file when running maven
following the procedure fixes the "cannot access " problem.
did you tried with a different scope than test for the cucumber-junit-platform-engine dependency?
I want to use Java code in the web. For this I want to convert Java to WASM and use this wasm-file in JavaScript. For converting Java to WebAssembly, I am using TeaVM.
First, I created an archetype with this command: mvn archetype:generate -DarchetypeGroupId=org.teavm.flavour -DarchetypeArtifactId=teavm-flavour-application -DarchetypeVersion=0.2.0
In addition, I added these two dependencies (according to http://blog.dmitryalexandrov.net/webassembly-for-java-developers/):
<dependency>
<groupId>org.teavm</groupId>
<artifactId>teavm-jso-apis</artifactId>
<version>${teavm.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.teavm</groupId>
<artifactId>teavm-interop</artifactId>
<version>${teavm.version}</version>
</dependency>
and added the following in the plugin section:
<targetType>WEBASSEMBLY</targetType>
<optimizationLevel>FULL</optimizationLevel>
<heapSize>8</heapSize>
My Java file:
#BindTemplate("templates/client.html")
public class Client extends ApplicationTemplate {
private String userName = "ABC";
public static void main(String[] args) {
Client client = new Client();
client.bind("application-content");
}
#Export(name = "getUserName")
public String getUserName() {
return userName;
}
}
But when I am doing mvn clean package, I am getting to following error (but a wasm file is created):
my complete pom:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>my.company</groupId>
<artifactId>java_wasm</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>war</packaging>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<java.version>1.8</java.version>
<flavour.version>0.2.0</flavour.version>
<teavm.version>0.6.0</teavm.version>
<jackson.version>2.5.4</jackson.version>
</properties>
<dependencies>
<dependency>
<groupId>org.teavm</groupId>
<artifactId>teavm-classlib</artifactId>
<version>${teavm.version}</version>
</dependency>
<dependency>
<groupId>org.teavm</groupId>
<artifactId>teavm-metaprogramming-impl</artifactId>
<version>${teavm.version}</version>
</dependency>
<dependency>
<groupId>org.teavm.flavour</groupId>
<artifactId>teavm-flavour-widgets</artifactId>
<version>${flavour.version}</version>
</dependency>
<dependency>
<groupId>org.teavm.flavour</groupId>
<artifactId>teavm-flavour-rest</artifactId>
<version>${flavour.version}</version>
</dependency>
<dependency>
<groupId>org.teavm</groupId>
<artifactId>teavm-jso-apis</artifactId>
<version>${teavm.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.teavm</groupId>
<artifactId>teavm-interop</artifactId>
<version>${teavm.version}</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
<version>${jackson.version}</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<configuration>
<source>${java.version}</source>
<target>${java.version}</target>
</configuration>
</plugin>
<plugin>
<artifactId>maven-war-plugin</artifactId>
<version>2.4</version>
<configuration>
<webResources>
<resource>
<directory>${project.build.directory}/generated/js</directory>
</resource>
</webResources>
</configuration>
</plugin>
<plugin>
<groupId>org.teavm</groupId>
<artifactId>teavm-maven-plugin</artifactId>
<version>${teavm.version}</version>
<executions>
<execution>
<id>web-client</id>
<phase>prepare-package</phase>
<goals>
<goal>compile</goal>
</goals>
<configuration>
<targetDirectory>${project.build.directory}/generated/js/teavm</targetDirectory>
<mainClass>my.company.Client</mainClass>
<minifying>true</minifying>
<debugInformationGenerated>true</debugInformationGenerated>
<sourceMapsGenerated>true</sourceMapsGenerated>
<sourceFilesCopied>true</sourceFilesCopied>
<optimizationLevel>ADVANCED</optimizationLevel>
<targetType>WEBASSEMBLY</targetType>
<optimizationLevel>FULL</optimizationLevel>
<heapSize>8</heapSize>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
How can I create a complete WASM without errors? Thank you in advance!
Wasm backend of TeaVM does not support JSO interop layer. It also supports subset of features available in JavaScript backend. So there's no way to make TeaVM Flavour work in Wasm, instead your should prefer JavaScript target. If you want to learn how to deal with Wasm BE, you can take a look at example.
Wasm has proven to be extremely inappropriate to run Java, so I recommend to use JavaScript BE of TeaVM. Also, please note that official site (htts://teavm.org) lists links where you can get help (google groups, gitter, direct email). I don't follow StackOverflow questions about TeaVM and don't receive notifications from SO.
I could not find any helping document regarding this error hence I am posting this question.
I am trying to use locally built Eclipse JDT core jar in my project. I followed the flowing steps.
Created an eclipse workspace as described in Link
Built eclipse.jdt.core using the mvn -P build-individual-bundles package
Added the resulting jar file that is created in the target folder to my project (i.e., project A) as a maven dependency.
After the above steps, I could successfully compile project A and now it gives me the following runtime error.
Exception in thread "pool-2-thread-1" java.lang.NoSuchMethodError: org.eclipse.core.runtime.SubMonitor.split(I)Lorg/eclipse/core/runtime/SubMonitor;
at org.eclipse.jdt.core.dom.ASTParser.createAST(ASTParser.java:820)
at utils.JavaASTUtil.parseSource(JavaASTUtil.java:87)
at change.CFile.<init>(CFile.java:32)
at change.RevisionAnalyzer.buildGitModifiedFiles(RevisionAnalyzer.java:310)
at change.RevisionAnalyzer.analyzeGit(RevisionAnalyzer.java:130)
at change.ChangeAnalyzer.analyzeGit(ChangeAnalyzer.java:243)
at change.ChangeAnalyzer.analyzeGit(ChangeAnalyzer.java:228)
at main.MainChangeAnalyzer$2.run(MainChangeAnalyzer.java:99)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
pom.xml of the project is given below
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>AtomicASTChangeMining</groupId>
<artifactId>AtomicASTChangeMining</artifactId>
<version>0.0.1-SNAPSHOT</version>
<repositories>
<repository>
<id>local-maven-repo</id>
<url>file:///Users/xx/Documents/Research_Topic_2/</url>
</repository>
</repositories>
<build>
<sourceDirectory>src</sourceDirectory>
<plugins>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.5.1</version>
<configuration>
<source>11</source>
<target>11</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>11</source>
<target>11</target>
</configuration>
</plugin>
</plugins>
</build>
<dependencies>
<!--Below is the locally built jdt core jar -->
<dependency>
<groupId>org.eclipse.jdt</groupId>
<artifactId>org.eclipse.jdt.core</artifactId>
<version>3.23.0-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>org.eclipse.core</groupId>
<artifactId>runtime</artifactId>
<version>3.10.0-v20140318-2214</version>
</dependency>
<dependency>
<groupId>org.eclipse.birt.runtime</groupId>
<artifactId>org.eclipse.core.resources</artifactId>
<version>3.10.0.v20150423-0755</version>
</dependency>
<dependency>
<groupId>org.eclipse</groupId>
<artifactId>osgi</artifactId>
<version>3.10.0-v20140606-1445</version>
</dependency>
</dependencies>
</project>
Does anybody have any idea about the runtime error?
Thanks a lot!
org.eclipse.core.runtime.SubMonitor is actually in the org.eclipse.equinox.common plug-in.
According to the Javadoc the split methods were added to SubMonitor in version 3.8 of org.eclipse.equinox.common. This corresponds to Eclipse release 4.6.
Do experiments on org.eclipse.core version. You might need to upgrade the version of org.eclipse.birt.runtime. Select the version that contains the method SubMonitor.split().
I am trying to follow the Maven in 5 Minutes tutorial for my project but for some reason I can't seem to actually run my project when it builds. I have the following pom.xml file:
<?xml version="1.0"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.my-group</groupId>
<artifactId>my-artifact</artifactId>
<version>0.0.1-SNAPSHOT</version>
<dependencies>
<dependency>
<groupId>com.sun</groupId>
<artifactId>tools</artifactId>
<version>1.8.0</version>
<scope>system</scope>
<systemPath>${java.home}/../lib/tools.jar</systemPath>
</dependency>
<dependency>
<groupId>commons-cli</groupId>
<artifactId>commons-cli</artifactId>
<version>1.3.1</version>
</dependency>
<dependency>
<groupId>com.oracle</groupId>
<artifactId>javafx</artifactId>
<version>2.2.3</version>
<scope>system</scope>
<systemPath>${project.basedir}/jfxrt.jar</systemPath>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.5.1</version>
<configuration>
<source>1.7</source>
<target>1.7</target>
<manifest>
<mainClass>com.my.package.MyClass</mainClass>
</manifest>
</configuration>
</plugin>
</plugins>
</build>
</project>
The tutorial seems to imply that I can do this from my project root:
mvn package
java -cp target/my-artifact-0.0.1-SNAPSHOT.jar com.my.package.MyClass
However, if I try to do this, I get the following error message:
Error: Could not find or load main class com.my.package.MyClass
If I run java -cp ../my-artifact-0.0.1-SNAPSHOT.jar com.my.package.MyClass from the target/classes directory I still get the same message, which is weird because I've verified that the .class file is there. Why am I not able to run my project after building it with the above steps?