In short: In Spark 2.0 spark-submit prefers his version of Guava library (14.0.1) - but I want to use recent jar version (19.0).
Question: How convince Spark to use version provided in my pom.xml file?
My suspicion: I can use spark.driver.userClassPathFirst=true option. But it is experimental feature (Spark 2.0.0 doc) - so maybe there is better solution?
Problem detailed explanation:
I'm using Spark 2.0.0 (hadoop2.7) and Elasticsearch 2.3.4. And I'm fighting with very simple application which tries use Spark Streaming and Elasticsearch together. Here it is:
SparkConf sparkConf = new SparkConf().setAppName("SampleApp");
JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, Durations.milliseconds(500));
jssc.checkpoint("/tmp");
JavaDStream<String> messages = jssc.textFileStream("/some_directory_path");
TransportClient client = TransportClient.builder().build()
.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName("localhost"), 9300));
messages.foreachRDD(rdd -> {
XContentBuilder builder = jsonBuilder()
.startObject()
.field("words", "some words")
.endObject();
clientTu.prepareIndex("indexName", "typeName")
.setSource(builder.string())
.get();
});
jssc.start();
jssc.awaitTermination();
The project is build with Maven. Here is part of pom.xml
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.0.0</version>
<scope>provided</scope>
<exclusions>
<exclusion>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.11</artifactId>
<version>2.0.0</version>
<scope>provided</scope>
<exclusions>
<exclusion>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch</artifactId>
<version>2.3.4</version>
<exclusions>
<exclusion>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>2.6.2</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>19.0</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<createSourcesJar>true</createSourcesJar>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>com.abc.App</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
As you can see I've done some exclusions.
And everything seems great, but after execution using command:
spark-submit --class com.abc.App --master local[2] /somePath/superApp-0.0.1-SNAPSHOT.jar
I gain the exception:
Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;
at org.elasticsearch.threadpool.ThreadPool.<clinit>(ThreadPool.java:190)
at org.elasticsearch.client.transport.TransportClient$Builder.build(TransportClient.java:131)
at com.abc.App.main(App.java:44)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:729)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
After adding --conf spark.driver.userClassPathFirst=true to spark-submit command, the application seems to work correctly. But I'm not sure this is the right way to manage this problem, because this option is marked in documentation as experimental.
In other words, Spark prefers libraries from his runtime environment and ignores those provided in "uber" jar (assembled jar). So I want to know how change this behavior in proper way?
Question again: What I have to do to be sure that particular jar defined in my POM will be used in runtime (with defined version)?
Edit: In more complex application which tries to use both Spark Streaming and Elasticsearch there're the same problems with other libraries (for example io.netty:netty). And in such situation, simple spark.driver.userClassPathFirst option activation doesn't help at all.
Related
I have an openapi file generated through restdocs and converted into this format. They are added to the resources folder, and the properties file is pointing to their format, however, swagger is failing to load and I'm not sure what else I could be missing.
<dependency>
<groupId>org.springframework.restdocs</groupId>
<artifactId>spring-restdocs-mockmvc</artifactId>
<version>${spring-restdocs-mockmvc.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springdoc</groupId>
<artifactId>springdoc-openapi-ui</artifactId>
<version>${springdoc-openapi-ui.version}</version>
</dependency>
<dependency>
<groupId>capital.scalable</groupId>
<artifactId>spring-auto-restdocs-core</artifactId>
<version>${spring-auto-restdocs-core.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.epages</groupId>
<artifactId>restdocs-api-spec</artifactId>
<version>${restdocs-api-spec.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.epages</groupId>
<artifactId>restdocs-api-spec-mockmvc</artifactId>
<version>${restdocs-api-spec.version}</version>
<scope>test</scope>
</dependency>
<plugin>
<groupId>io.github.berkleytechnologyservices</groupId>
<artifactId>restdocs-spec-maven-plugin</artifactId>
<version>${restdocs-spec.version}</version>
<executions>
<execution>
<goals>
<goal>generate</goal>
</goals>
<configuration>
<!--suppress MavenModelInspection -->
<skip>${skipTests}</skip>
<host>localhost:8081</host>
<specification>OPENAPI_V3</specification>
<outputDirectory>${project.build.directory}/classes/static/docs</outputDirectory>
</configuration>
</execution>
</executions>
</plugin>
Thank you very much.
A related question was posted here and this answer solved the issue: https://stackoverflow.com/a/75273492/6654475
In summary, it was related to this line that needed to be removed spring.web.resources.add-mappings=false and the yml file that needed to be moved to the resources/static/ folder.
I am trying out a spark application in java which needs to connect to Azure Key-vault and fetch some secrets and process data after reading from a blob storage and push to a SQL Server table. Here is my pom:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.my.package</groupId>
<artifactId>hdinsightSample</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>hdinsightSample</name>
<url>http://maven.apache.org</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.12</artifactId>
<version>2.4.7</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.12</artifactId>
<version>2.4.7</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-azure</artifactId>
<version>2.10.1</version>
</dependency>
<dependency>
<groupId>com.microsoft.sqlserver</groupId>
<artifactId>mssql-jdbc</artifactId>
<version>6.2.2.jre8</version>
</dependency>
<dependency>
<groupId>com.microsoft.azure</groupId>
<artifactId>azure-keyvault</artifactId>
<version>1.0.0</version>
</dependency>
<dependency>
<groupId>com.microsoft.azure</groupId>
<artifactId>azure</artifactId>
<version>1.3.0</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>com.my.package.hdinsightSample.App</mainClass>
</transformer>
</transformers>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/maven/**</exclude>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<relocations>
<relocation>
<pattern>com.google.common</pattern>
<shadedPattern>repackaged.com.google.common</shadedPattern>
<includes>
<include>com.google.guava:*</include>
</includes>
</relocation>
</relocations>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
After running the jar file or running the code from Eclipse I get the following exception:
Exception in thread "main" java.lang.NoClassDefFoundError: com.google.common.reflect.TypeToken
at java.lang.ClassLoader.defineClassImpl(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:379)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:682)
at java.net.URLClassLoader.access$400(URLClassLoader.java:89)
at java.net.URLClassLoader$ClassFinder.run(URLClassLoader.java:1086)
at java.security.AccessController.doPrivileged(AccessController.java:739)
at java.net.URLClassLoader.findClass(URLClassLoader.java:589)
at java.lang.ClassLoader.loadClassHelper(ClassLoader.java:934)
at java.lang.ClassLoader.loadClass(ClassLoader.java:879)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:862)
at com.microsoft.azure.keyvault.KeyVaultClientImpl.getSecretDelegate(KeyVaultClientImpl.java:2938)
at com.microsoft.azure.keyvault.KeyVaultClientImpl.access$1800(KeyVaultClientImpl.java:92)
at com.microsoft.azure.keyvault.KeyVaultClientImpl$82.call(KeyVaultClientImpl.java:2928)
at com.microsoft.azure.keyvault.KeyVaultClientImpl$82.call(KeyVaultClientImpl.java:2924)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:69)
at retrofit2.adapter.rxjava.CallArbiter.deliverResponse(CallArbiter.java:120)
at retrofit2.adapter.rxjava.CallArbiter.emitResponse(CallArbiter.java:102)
at retrofit2.adapter.rxjava.CallExecuteOnSubscribe.call(CallExecuteOnSubscribe.java:46)
at retrofit2.adapter.rxjava.CallExecuteOnSubscribe.call(CallExecuteOnSubscribe.java:24)
at rx.Observable.unsafeSubscribe(Observable.java:10142)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.subscribe(Observable.java:10238)
at rx.Observable.subscribe(Observable.java:10205)
at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:444)
at rx.observables.BlockingObservable.single(BlockingObservable.java:341)
at com.microsoft.azure.keyvault.KeyVaultClientImpl.getSecret(KeyVaultClientImpl.java:2868)
at com.microsoft.azure.keyvault.KeyVaultClient.getSecret(KeyVaultClient.java:862)
at com.my.package.hdsinsightSample.utils.KeyVaultUtils.getSecret(KeyVaultUtils.java:12)
at com.my.package.hdinsightSample.App.main(App.java:23)
Caused by: java.lang.ClassNotFoundException: com.google.common.reflect.TypeToken
at java.net.URLClassLoader.findClass(URLClassLoader.java:591)
at java.lang.ClassLoader.loadClassHelper(ClassLoader.java:934)
at java.lang.ClassLoader.loadClass(ClassLoader.java:879)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:862)
... 36 more
This is because to fetch the secret from key vault it uses guava jar but guava jar complied by default is 11.0.2 (present in spark-core dependecny) whereas the correct version it should be using is 20 (present in the Azure Key-Vault dependency). I tried shading the jar as shown in the pom above but the issue remains the same.
Kindly help.
I created a small project with the same dependencies and the dependency tree shows the Guava dependency actually comes from org.apache.hadoop:hadoop-azure:jar:2.10.1. Regardless of that, the simplest way to get rid of this problem is to explicitly state which Guava version you want your project to use in your POM (ideally the same as the one in Key Vault in this case):
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>20.0</version>
</dependency>
Additionally, there is a new set of libraries for Azure Key Vault with improved APIs and additional functionality, which also do not rely on Guava, so that is an alternative approach you can take if you also want to update your code base :)
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-security-keyvault-certificates</artifactId>
<version>4.1.3/version>
</dependency>
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-security-keyvault-keys/artifactId>
<version>4.2.3/version>
</dependency>
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-security-keyvault-secrets</artifactId>
<version>4.2.3/version>
</dependency>
I'm building a library for fetching encrypted secrets from cloud storage (in Scala, using the Java clients). I'm using the following google libraries:
"com.google.apis" % "google-api-services-cloudkms" % "v1-rev26-1.23.0" exclude("com.google.guava", "guava-jdk5"),
"com.google.cloud" % "google-cloud-storage" % "1.14.0",
Everything works fine locally, but when I try to run my code in Dataproc I'm getting the following error:
Exception in thread "main" java.lang.NoSuchMethodError: com.google.api.client.googleapis.services.json.AbstractGoogleJsonClient$Builder.setBatchPath(Ljava/lang/String;)Lcom/google/api/client/googleapis/services/AbstractGoogleClient$Builder;
at com.google.api.services.cloudkms.v1.CloudKMS$Builder.setBatchPath(CloudKMS.java:4250)
at com.google.api.services.cloudkms.v1.CloudKMS$Builder.<init>(CloudKMS.java:4229)
at gcp.encryption.EncryptedSecretsUser$class.clients(EncryptedSecretsUser.scala:111)
at gcp.encryption.EncryptedSecretsUser$class.getEncryptedSecrets(EncryptedSecretsUser.scala:62)
The offending line in my code is:
val kms: CloudKMS = new CloudKMS.Builder(credential.getTransport,
credential.getJsonFactory,
credential)
.setApplicationName("Encrypted Secrets User")
.build()
I see in the documentation that some google libraries are available on Dataproc (I'm using a Spark cluster with image version 1.2.15). But as far as I can see the transitive dependency for google-api-client is the same one I'm using locally (1.23.0). So how come the method isn't found?
Should I set up my dependencies differently for running on Dataproc?
EDIT
Finally managed to solve this in another project. Turns out that besides shading all the google dependencies (including the gcs-connector!!), you also have to register your shaded class with the JVM to handle the gs:// file-system.
Below is the maven configuration that works for me, something similar can be achieved with sbt:
Parent POM:
<project xmlns="http://maven.apache.org/POM/4.0.0"...>
...
<properties>
<!-- Spark version -->
<spark.version>[2.2.1]</spark.version>
<!-- Jackson-libs version pulled in by spark -->
<jackson.version>[2.6.5]</jackson.version>
<!-- Avro version pulled in by jackson -->
<avro.version>[1.7.7]</avro.version>
<!-- Kryo-shaded version pulled in by spark -->
<kryo.version>[3.0.3]</kryo.version>
<!-- Apache commons-lang version pulled in by spark -->
<commons.lang.version>2.6</commons.lang.version>
<!-- TODO: need to shade google libs because of version-conflicts on Dataproc. Remove this when Dataproc 1.3/2.0 is released -->
<bigquery-conn.version>[0.10.6-hadoop2]</bigquery-conn.version>
<gcs-conn.version>[1.6.5-hadoop2]</gcs-conn.version>
<google-storage.version>[1.29.0]</google-storage.version>
<!-- The guava version we want to use -->
<guava.version>[23.2-jre]</guava.version>
<!-- The google api version used by the google-cloud-storage lib -->
<api-client.version>[1.23.0]</api-client.version>
<!-- The google-api-services-storage version used by the google-cloud-storage lib -->
<storage-api.version>[v1-rev114-1.23.0]</storage-api.version>
<!-- Picked up by compiler and resource plugins -->
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
...
<build>
<pluginManagement>
<plugins>
...
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.1.1</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<minimizeJar>true</minimizeJar>
<filters>
<filter>
<artifact>com.google.**:*</artifact>
<includes>
<include>**</include>
</includes>
</filter>
<filter>
<artifact>com.google.cloud.bigdataoss:gcs-connector</artifact>
<excludes>
<!-- Register a provider with the shaded name instead-->
<exclude>META-INF/services/org.apache.hadoop.fs.FileSystem</exclude>
</excludes>
</filter>
</filters>
<artifactSet>
<includes>
<include>com.google.*:*</include>
</includes>
<excludes>
<exclude>com.google.code.findbugs:jsr305</exclude>
</excludes>
</artifactSet>
<relocations>
<relocation>
<pattern>com.google</pattern>
<shadedPattern>com.shaded.google</shadedPattern>
</relocation>
</relocations>
</configuration>
</execution>
</executions>
</plugin>
...
</plugins>
</build>
<dependencyManagement>
<dependencies>
<dependency>
...
<groupId>com.google.cloud.bigdataoss</groupId>
<artifactId>gcs-connector</artifactId>
<version>${gcs-conn.version}</version>
<exclusions>
<!-- conflicts with Spark dependencies -->
<exclusion>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
</exclusion>
<!-- conflicts with Spark dependencies -->
<exclusion>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-core</artifactId>
</exclusion>
<exclusion>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<!-- Avoid conflict with the version pulled in by the GCS-connector on Dataproc -->
<groupId>com.google.apis</groupId>
<artifactId>google-api-services-storage</artifactId>
<version>${storage-api.version}</version>
</dependency>
<dependency>
<groupId>commons-lang</groupId>
<artifactId>commons-lang</artifactId>
<version>${commons.lang.version}</version>
</dependency>
<dependency>
<groupId>com.esotericsoftware</groupId>
<artifactId>kryo-shaded</artifactId>
<version>${kryo.version}</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>${jackson.version}</version>
</dependency>
<dependency>
<groupId>com.google.api-client</groupId>
<artifactId>google-api-client</artifactId>
<version>${api-client.version}</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>${guava.version}</version>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-storage</artifactId>
<version>${google-storage.version}</version>
<exclusions>
<!-- conflicts with Spark dependencies -->
<exclusion>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
</dependency>
...
</dependencies>
...
</project>
Child POM:
<dependencies>
<!-- Libraries available on dataproc -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.google.cloud.bigdataoss</groupId>
<artifactId>gcs-connector</artifactId>
</dependency>
<dependency>
<groupId>com.esotericsoftware</groupId>
<artifactId>kryo-shaded</artifactId>
<scope>provided</scope><!-- Pulled in by spark -->
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<scope>provided</scope><!-- Pulled in by spark -->
</dependency>
</dependencies>
And add a file named org.apache.hadoop.fs.FileSystem under path/to/your-project/src/main/resources/META-INF/services, containing the name of your shaded class, e.g:
# WORKAROUND FOR DEPENDENCY CONFLICTS ON DATAPROC
#
# Use the shaded class as a provider for the gs:// file system
#
com.shaded.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem
(Notice that this file was filtered out of the gcs-connector library in the parent POM)
It may not be obvious, but the google-api-client version in the latest stable GCS connector is actually 1.20.0.
The reason is that this was the commit which rolled the api client version forward to 1.23.0, and it was part of a series of commits including this dependency-shading commit with the overall goal of no longer leaking the transitive dependency into the job classpath at all, precisely to avoid version collision issues in the future, at the cost of everyone having to bring their own fat jar containing the full api client dependencies themselves.
However, it turns out that many people have already grown to depend on the GCS-connector-provided api client to be on the classpath, so there are production workloads out there which cannot survive such a change inside of a minor version upgrade; thus, the upgraded GCS connector which uses 1.23.0 but also shades it so that it won't appear in the job classpath anymore is reserved for a future Dataproc 1.3+ or 2.0+ release.
In your case, you could try using a 1.20.0 version of your dependencies (you may also have to downgrade the version of the google-cloud-storage dependency you included, though a 1.22.0 version of that may still work assuming no breaking changes, since setBatchPath was indeed introduced only in 1.23.0), or otherwise you can try to shade all your own dependencies using sbt-assembly.
We can verify that setBatchPath was introduced only in 1.23.0:
$ javap -cp google-api-client-1.22.0.jar com.google.api.client.googleapis.services.AbstractGoogleClient.Builder | grep set
public com.google.api.client.googleapis.services.AbstractGoogleClient$Builder setRootUrl(java.lang.String);
public com.google.api.client.googleapis.services.AbstractGoogleClient$Builder setServicePath(java.lang.String);
public com.google.api.client.googleapis.services.AbstractGoogleClient$Builder setGoogleClientRequestInitializer(com.google.api.client.googleapis.services.GoogleClientRequestInitializer);
public com.google.api.client.googleapis.services.AbstractGoogleClient$Builder setHttpRequestInitializer(com.google.api.client.http.HttpRequestInitializer);
public com.google.api.client.googleapis.services.AbstractGoogleClient$Builder setApplicationName(java.lang.String);
public com.google.api.client.googleapis.services.AbstractGoogleClient$Builder setSuppressPatternChecks(boolean);
public com.google.api.client.googleapis.services.AbstractGoogleClient$Builder setSuppressRequiredParameterChecks(boolean);
public com.google.api.client.googleapis.services.AbstractGoogleClient$Builder setSuppressAllChecks(boolean);
$ javap -cp google-api-client-1.23.0.jar com.google.api.client.googleapis.services.AbstractGoogleClient.Builder | grep set
public com.google.api.client.googleapis.services.AbstractGoogleClient$Builder setRootUrl(java.lang.String);
public com.google.api.client.googleapis.services.AbstractGoogleClient$Builder setServicePath(java.lang.String);
public com.google.api.client.googleapis.services.AbstractGoogleClient$Builder setBatchPath(java.lang.String);
public com.google.api.client.googleapis.services.AbstractGoogleClient$Builder setGoogleClientRequestInitializer(com.google.api.client.googleapis.services.GoogleClientRequestInitializer);
public com.google.api.client.googleapis.services.AbstractGoogleClient$Builder setHttpRequestInitializer(com.google.api.client.http.HttpRequestInitializer);
public com.google.api.client.googleapis.services.AbstractGoogleClient$Builder setApplicationName(java.lang.String);
public com.google.api.client.googleapis.services.AbstractGoogleClient$Builder setSuppressPatternChecks(boolean);
public com.google.api.client.googleapis.services.AbstractGoogleClient$Builder setSuppressRequiredParameterChecks(boolean);
public com.google.api.client.googleapis.services.AbstractGoogleClient$Builder setSuppressAllChecks(boolean);
I am having trouble creating a simple Java AWS Lambda that does not exceed the 50MB size limit and I am not sure where I am going wrong...
My actual Java code is only around 100 lines long. I am using Maven to pull in my dependencies, most of which are the AWS dependencies themselves...
After I do a mvn package or mvn clean install the resulting jar is around 64MB!
What am I doing wrong here? If I do not add the AWS dependencies then I cannot build with maven but if I do then the resulting jar file is huge and I am unable to upload it to AWS lambda because of the size restriction...
My pom file is below:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>lambda-handler</groupId>
<artifactId>lambda-handler</artifactId>
<version>0.0.1-SNAPSHOT</version>
<build>
<sourceDirectory>src</sourceDirectory>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.3</version>
<configuration>
<createDependencyReducedPom>false</createDependencyReducedPom>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
<dependencies>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-lambda-java-core</artifactId>
<version>1.1.0</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-lambda-java-events</artifactId>
<version>1.3.0</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk</artifactId>
<version>1.11.166</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-s3</artifactId>
<version>1.11.166</version>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.5.3</version>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpmime</artifactId>
<version>4.5.3</version>
</dependency>
</dependencies>
</project>
to fix this issue you need to do the following:
Remove the amazon SDK from your pom.
Add the Execution Amazon SDK variable:
AWSEXECUTIONENV -> AWSLambdajava8
Remember to use Java8 in your project. The total size of a simple project will be around 7-10 MB.
If you want to see HOW-TO with github sources and step by step you can see it on my blog:
jmazzetti.com -> Serverless on AWS Lambda using Java
All the best,
Jose.
you need to remove <artifactId>aws-java-sdk-s3</artifactId> and <artifactId>aws-java-sdk</artifactId> dependencies from your maven project and update and rebuild your maven project.
I am attempting to POST test results to TestRails.
Using test suites and test cases, the tests in question were created using SoapUI. What I currently have will POST directly from SoapUI to TestRails. The company I work for wants to initiate the SoapUI tests using Maven, which is possible with a SoapUI plug-in for the Maven pom file. This part of the process works perfectly.
However, POSTing to the TestRails project only works if I initiate the tests via the SoapUI application. When I run the same tests via Maven, all the test steps complete and are visible in the stack trace, but none of the test case results are sent to TestRails.
What I have done is placed a plug-in in the pom file, which is supposed to connect Maven to TestRails and allow the transfer of the test results, but while there are no errors in the program, no POST is done.
Here's how I have it coded in my pom file:
<plugin>
<groupId>com.smartbear.soapui</groupId>
<artifactId>soapui-maven-plugin</artifactId>
<version>5.1.2</version>
<configuration>
<settingsFile>${projectDir}\soapui-settings.xml</settingsFile>
<iface>mobileAdsService</iface>
<exportwAll>true</exportwAll>
<printReport>true</printReport>
<testFailIgnore>true</testFailIgnore>
<junitReport>true</junitReport>
<outputFolder>${projectDir}\TestResults</outputFolder>
<projectFile>${basedir}\Irdeto-v1.xml</projectFile>
<saveAfterRun>true</saveAfterRun>
</configuration>
<dependencies>
<dependency>
<groupId>org.reflections</groupId>
<artifactId>reflections</artifactId>
<version>0.9.9-RC1</version>
</dependency>
***<dependency>
<groupId>com.codepine.api</groupId>
<artifactId>testrail-api-java-client</artifactId>
<version>1.0.0</version>
</dependency>***
<dependency>
<groupId>org.codehaus.groovy</groupId>
<artifactId>groovy-all</artifactId>
<version>2.0.6</version>
</dependency>
<dependency>
<groupId>org.apache.poi</groupId>
<artifactId>poi-ooxml</artifactId>
<version>3.10-FINAL</version>
<exclusions>
<exclusion>
<groupId>org.apache.xmlbeans</groupId>
<artifactId>xmlbeans</artifactId>
</exclusion>
</exclusions>
</dependency>
</dependencies>
<executions>
<execution>
<id>soapui-tests</id>
<phase>test</phase>
<goals>
<goal>test</goal>
</goals>
</execution>
</executions>
</plugin>
This appears to have been due to a bug in the non-licensed version of SoapUI.