We Keep Coding

Apache Beam Java (SDK versions 2.43.0 - 2.44.0) batch loads to BigQuery fail using Storage Write API

I have a batch Apache Beam pipeline that worked successfully until I upgraded from version 2.42 to 2.43 and above. The pipeline uses Storage Write API to write to Biguery and it now fails with the error: "Error message from worker: org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ExecutionError:java.lang.NoSuchMethodError: 'com.google.cloud.biguery.storage.v1.StreamWriter$Builder com.google.cloud.biguery.storage.v1.StreamWriter$Builder.setEnableConnectionPool(bool)'".
When I switched to FileLoads method instead of StorageWriteAPI it works again. But wondering what's wrong with StorageWriteAPI now. Tried checking the release notes but couldn't find anything helpful (https://github.com/apache/beam/releases). Can I please get some help here?

I think you have a dependency conflict with Guava in your Maven pom or Gradle build file. I share with you the Guava version I used in my pom file for Beam 2.43.0 or 2.44.0 version :
<guava.version>31.0.1-jre</guava.version>
If you use the following dependencies, you have to exclude an old version of Guava :
<dependency>
<groupId>com.google.api-client</groupId>
<artifactId>google-api-client</artifactId>
<version>${google-api-client.version}</version>
<exclusions>
<!-- Exclude an old version of guava that is being pulled
in by a transitive dependency of google-api-client -->
<exclusion>
<groupId>com.google.guava</groupId>
<artifactId>guava-jdk5</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.google.apis</groupId>
<artifactId>google-api-services-bigquery</artifactId>
<version>${bigquery.version}</version>
<exclusions>
<!-- Exclude an old version of guava that is being pulled
in by a transitive dependency of google-api-client -->
<exclusion>
<groupId>com.google.guava</groupId>
<artifactId>guava-jdk5</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.google.http-client</groupId>
<artifactId>google-http-client</artifactId>
<exclusions>
<!-- Exclude an old version of guava that is being pulled
in by a transitive dependency of google-api-client -->
<exclusion>
<groupId>com.google.guava</groupId>
<artifactId>guava-jdk5</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.google.apis</groupId>
<artifactId>google-api-services-pubsub</artifactId>
<version>${pubsub.version}</version>
<exclusions>
<!-- Exclude an old version of guava that is being pulled
in by a transitive dependency of google-api-client -->
<exclusion>
<groupId>com.google.guava</groupId>
<artifactId>guava-jdk5</artifactId>
</exclusion>
</exclusions>
</dependency>
I hope it can help.

ERROR JacksonVersion - Version '2.9.4' of package 'jackson-databind' is not supported

i am using azure sdk(azure-storage-file-datalake, azure-core, azure-identity) for writing the avro data to azure datalake gen 2 from kafka(using connectors), but i am getting very weird errors as below
2022-05-12 10:46:10 ERROR JacksonVersion:506 - Version '2.9.4' of package 'jackson-annotations' is not supported (older than earliest supported version - 2.10.0), please upgrade.
2022-05-12 10:46:10 ERROR JacksonVersion:506 - Version '2.9.4' of package 'jackson-core' is not supported (older than earliest supported version - 2.10.0), please upgrade.
2022-05-12 10:46:10 ERROR JacksonVersion:506 - Version '2.9.4' of package 'jackson-databind' is not supported (older than earliest supported version - 2.10.0), please upgrade.
java.lang.LinkageError: Package versions: jackson-annotations=2.9.4, jackson-core=2.9.4, jackson-databind=2.9.4, jackson-dataformat-xml=unknown, jackson-datatype-jsr310=unknown, azure-core=1.28.0, Troubleshooting version conflicts: https://aka.ms/azsdk/java/dependency/troubleshoot\n\tat com.azure.core.implementation
I have tried excluding the jackson dependencies and tried adding explicitly, still i am getting the errors.
please some one help on this
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
<version>2.13.2</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
<version>2.13.2</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.13.2</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.dataformat</groupId>
<artifactId>jackson-dataformat-xml</artifactId>
<version>2.13.2</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.datatype</groupId>
<artifactId>jackson-datatype-jsr310</artifactId>
<version>2.13.2</version>
<scope>runtime</scope>
</dependency>
<dependencies>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>connect-api</artifactId>
<version>2.0.1</version>
<scope>provided</scope>
</dependency>
<!--<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>1.1.0</version>
<scope>provided</scope>
</dependency>-->
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.17</version>
</dependency>
<!-- Apache avro dependency -->
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
<version>1.8.2</version>
</dependency>
<!-- Newer versions of the storage account jar is not working. -->
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-storage-file-datalake</artifactId>
<version>12.9.0</version>
<exclusions>
<exclusion>
<groupId>com.azure</groupId>
<artifactId>azure-core</artifactId>
</exclusion>
</exclusions>
</dependency>
<!-- https://mvnrepository.com/artifact/com.azure/azure-core -->
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-core</artifactId>
<version>1.28.0</version>
<exclusions>
<!-- Collision with Kafka Jackson dependencies -->
<exclusion>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
</exclusion>
<exclusion>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
</exclusion>
<exclusion>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
</exclusion>
<exclusion>
<groupId>com.fasterxml.jackson.dataformat</groupId>
<artifactId>jackson-dataformat-xml</artifactId>
</exclusion>
<exclusion>
<groupId>com.fasterxml.jackson.datatype</groupId>
<artifactId>jackson-datatype-jsr310</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-identity</artifactId>
<version>1.5.1</version>
<exclusions>
<exclusion>
<groupId>com.azure</groupId>
<artifactId>azure-core</artifactId>
</exclusion>
<exclusion>
<groupId>com.microsoft.azure</groupId>
<artifactId>msal4j</artifactId>
</exclusion>
<exclusion>
<groupId>org.linguafranca.pwdb</groupId>
<artifactId>KeePassJava2</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.microsoft.azure</groupId>
<artifactId>msal4j</artifactId>
<version>1.9.1</version> <!-- {x-version-update;com.microsoft.azure:msal4j;external_dependency} -->
<exclusions>
<exclusion>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
</exclusion>
</exclusions>
</dependency>
<!-- Schema registry dependency -->
<dependency>
<groupId>io.confluent</groupId>
<artifactId>kafka-connect-avro-converter</artifactId>
<version>3.3.0</version>
<exclusions>
<exclusion>
<groupId>io.confluent</groupId>
<artifactId>kafka-schema-registry-client</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>io.confluent</groupId>
<artifactId>kafka-schema-registry-client</artifactId>
<version>3.3.0</version>
<exclusions>
<exclusion>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
<version>2.21.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.13.2</version>
<scope>test</scope>
</dependency>
</dependencies>

Dependencies with scope "runtime" are not used for compilation, but only for execution. A dependency with this scope is not put on the compile classpath, so you could remove the runtime scope.
However a better way to overwrite transitive dependency versions that you don't use yourself is to use depependency management like this:
<project>
[...]
<dependencyManagement>
<dependencies>
<dependency>
<groupId>group-a</groupId>
<artifactId>artifact-a</artifactId>
<version>1.0</version>
</dependency>
[...]
</dependencies>
</dependencyManagement>
</project>

This is a challenge with Azure-SDK (here is a similar issue). Azure-SDK has a dependency on Jackson. The specific version of Jackson depends on the azure-sdk version you are using.
The latest release supports Jackson 2.10-2.13. What it means is you may have other dependencies on Jackson, and the SDK will still work. Provided Jackson version fetched is from 2.10-2.13. Azure-SDK is compatible with all these versions.
If you wish to use Jackson 2.9 (or if that's coming as a dependency from other dependency), then there is no way around it. You would necessarily need to upgrade your dependencies so that it fetches jackson 2.10 and above.
In your case, there would be some dependency that is fetching 2.9.x series.
This is what I would do:
Based on all non-azure-sdk dependencies. Decide which jackson version to hold on to. Explicitly add this jackson dependency in the pom.xml
Use azure-sdk BOM
You dont have to specify exclusion. Specifying a version in pom overrides other dependency versions

Hosting environment had those 2.9.4 versioned jackson jars which were loading at run time and causing the issues, i replaced those with newer version jars , its working all fine now.
(we had installed kafka client on the host, which internally had those jars inside kafka/libs)
got the ans from - https://github.com/Azure/azure-sdk-for-java/issues/28902

Maven dependency bcprov issue

I have bouncy castle dependency used in my application and I want this dependency to be excluded in the pom.xml file. Even after removing this dependency from the pom file, it is still appearing in the m2 repository folder. My application is a Spring-MVC 5 framework and running on Tomcat version 9. While deploying there are two different versions of "bcprov". I need to exclude both bcprov-jdk14 and bcprov-jdk15on from the pom.xml file.
Below is the dependency in the pom file.
<dependency>
<groupId>bouncycastle</groupId>
<artifactId>bcprov-jdk14</artifactId>
<version>140</version>
</dependency>
I also tried another link for the exclusion but it did not work - Maven Transitive Dependency issue
Please help how to exclude the above dependencies.

To exclude some transitive dependencies you just have to add <exclusion> tags in your POM file. You need to remove the bcprov-jdk14 artifact from every dependency that uses it (check the dependency tree), e.g.:
<dependency>
<groupId>com.lowagie</groupId>
<artifactId>itext</artifactId>
<version>2.1.7</version>
<exclusions>
<exclusion>
<groupId>bouncycastle</groupId>
<artifactId>bcmail-jdk14</artifactId>
</exclusion>
<exclusion>
<groupId>bouncycastle</groupId>
<artifactId>bcprov-jdk14</artifactId>
</exclusion>
<exclusion>
<groupId>org.bouncycastle</groupId>
<artifactId>bcmail-jdk14</artifactId>
</exclusion>
<exclusion>
<groupId>org.bouncycastle</groupId>
<artifactId>bcprov-jdk14</artifactId>
</exclusion>
<exclusion>
<groupId>org.bouncycastle</groupId>
<artifactId>bctsp-jdk14</artifactId>
</exclusion>
</exclusions>
</dependency>
Remember however to add the correct versions of those dependencies:
<properties>
...
<bc.version>1.69</bc.version>
</properties>
<dependencies>
<dependency>
<groupId>org.bouncycastle</groupId>
<artifactId>bcprov-jdk15on</artifactId>
<version>${bc.version}</version>
</dependency>
<dependency>
<groupId>org.bouncycastle</groupId>
<artifactId>bcmail-jdk15on</artifactId>
<version>${bc.version}</version>
</dependency>
</dependencies>
Alternatively don't use the features of the libraries that depend on BouncyCastle (signature verification/encryption).

How can I remove the old vulnerable Apache commons collection version dependency from my project's maven dependency tree?

My Java app project is being managed by Maven.
My project has a few library dependencies depending again on Apache commons collection 3.2.1 which is vulnerable - e.g. Apache commons configuration, velocity, etc.
(I can see it is being used by running mvn dependency:tree command.)
I did neither write any line of codes using Apache commons collection directly nor defined the dependency of it, but it's being used.
What could I do to remove its dependency and to force to use safe version - 3.2.2, 4.1.
For your information:
JIRA Bug - Arbitrary remote code execution with InvokerTransformer
Here is the part of my pom.xml, and I guess there's nothing remarkable.
...
<dependency>
<groupId>commons-configuration</groupId>
<artifactId>commons-configuration</artifactId>
<version>1.6</version>
</dependency>
<dependency>
<groupId>org.apache.velocity</groupId>
<artifactId>velocity</artifactId>
<version>1.7</version>
</dependency>
...

Unless I am missing something obvious, just specifying dependency in your POM ought to be sufficient:
<dependencies>
<dependency>
<groupId>commons-collections</groupId>
<artifactId>commons-collections</artifactId>
<version>3.2.2</version>
<dependency>
...
</dependencies>
If you specify it a the top of your <dependencies> section, it will override any other transitive inclusion of commons-collections.
Of course, you may wind up with incompatibilities where other dependencies depend on the other version, but that's what unit tests are for, right? ;-)

What you need to do is exclude commons-collections from the affected dependencies and include the desired version in your dependencies directly.
Example pom.xml excerpt assuming commons-configuration uses the vulnerable commons-collections
<dependency>
<groupId>commons-configuration</groupId>
<artifactId>commons-configuration</artifactId>
<version>1.10</version>
<exclusions>
<exclusion>
<artifactId>commons-collections</artifactId>
<groupId>commons-collections</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>commons-collections</groupId>
<artifactId>commons-collections</artifactId>
<version>3.2.2</version>
<scope>runtime</scope>
</dependency>
For simplicity I didn't show configuring this in a root pom.xml in the dependency-management section.
The <scope> should be set to runtime since you mentioned not using the library directly.

I've added these lines in my pom.xml, but still commons-collections3.2 is getting downloaded..
<dependencies>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-collections4</artifactId>
<version>4.1</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>${apachecommonslang.version}</version>
<exclusions>
<exclusion>
<artifactId>commons-collections</artifactId>
<groupId>commons-collections</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>commons-dbcp</groupId>
<artifactId>commons-dbcp</artifactId>
<version>${dbcp.version}</version>
<exclusions>
<exclusion>
<artifactId>commons-collections</artifactId>
<groupId>commons-collections</groupId>
</exclusion>
</exclusions>
</dependency>

Jackson ObjectMapper() constructor throws NoSuchMethod

I'm using Jackson sample code to deserialize a POJO:
ObjectMapper m = new ObjectMapper();
This line throws a NoSuchMethodError:
Exception in thread "main" java.lang.NoSuchMethodError: org.codehaus.jackson.type.JavaType.<init>(Ljava/lang/Class;)V
at org.codehaus.jackson.map.type.TypeBase.<init>(TypeBase.java:15)
at org.codehaus.jackson.map.type.SimpleType.<init>(SimpleType.java:45)
at org.codehaus.jackson.map.type.SimpleType.<init>(SimpleType.java:40)
at org.codehaus.jackson.map.type.TypeBindings.<clinit>(TypeBindings.java:18)
at org.codehaus.jackson.map.type.TypeFactory._fromType(TypeFactory.java:525)
at org.codehaus.jackson.map.type.TypeFactory.type(TypeFactory.java:61)
at org.codehaus.jackson.map.ObjectMapper.<clinit>(ObjectMapper.java:179)
at com.me.util.ctrl.BillingJobStatus.fromJson(BillingJobStatus.java:37)
I don't get it

I'm guessing your Jackson JARs are out of sync. The JavaType class is in the jackson-core JAR, and the ObjectMapper class is in jackson-mapper.
Make sure these are both of the same version.

I had this same problem. The core jar was 1.7.1 while the mapper was 1.8.1. Note: To fix this for maven I added an exclusion and pulled down the proper version.
<exclusions>
<exclusion>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-mapper-asl</artifactId>
</exclusion>
</exclusions>

The trick here is to exclude jackson from the dependencies that use it.
To check which dependencies import it, you can use the following maven command:
mvn dependency:tree -Dincludes=org.codehaus.jackson

In my case it was due to yammer-metrics library including an older version of jackson.
<dependency>
<groupId>com.yammer.metrics</groupId>
<artifactId>metrics-servlet</artifactId>
<version>2.1.2</version>
<exclusions>
<exclusion>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-mapper-asl</artifactId>
</exclusion>
</exclusions>
</dependency>

In my case it was the amazonaws sdk that caused this. Using just the exclusion for jackson-mapper-asl didn't work but using an exclusion for core and mapper did:
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk</artifactId>
<version>1.3.13</version>
<exclusions>
<exclusion>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-core-asl</artifactId>
</exclusion>
<exclusion>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-mapper-asl</artifactId>
</exclusion>
</exclusions>
</dependency>

While working on Hadoop, I eliminated this error by setting the following exclusions
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>1.0.1</version>
<exclusions>
<exclusion>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-mapper-asl</artifactId>
</exclusion>
</exclusions>
</dependency>

it is beacuse of #RequestBody use different jars for different spring versions :
if using spring 4 than have to add different jars :
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
<version>2.4.3</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.4.3</version>
</dependency>
if using spring 3 have to use this jar :
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-mapper-asl</artifactId>
<version>1.5.0</version>
</dependency>

Yes, anyone who is facing this issue, he should definetely investigate the dependencies with
mvn dependency:tree -Dincludes=org.codehaus.jackson
In my case, I had a jackson dependency on Atmosphere-Socket-io :
<dependency>
<groupId>org.atmosphere</groupId>
<artifactId>atmosphere-socketio</artifactId>
<version>2.2.1</version>
<exclusions>
<exclusion>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-mapper-lgpl</artifactId>
</exclusion>
<exclusion>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-core-lgpl</artifactId>
</exclusion>
</exclusions>
</dependency>
It took me several hours to track that down, thank you all guys for the above solutions which inspired me! Keep up the good work!

I have faced this problem when migrating my WebApp from Tomcat 6 to Tomcat 8.
On Tomcat6 and Tomcat7 my WebApp would start just fine, but on Tomcat8 I would get this exception(seems that T6 and T7 loads classes alphabeticaly but T8 doesn't - https://stackoverflow.com/a/26642798/1864614).
The problem was that I was having 2 versions of the class org.codehaus.jackson.map.ObjectMapper
As answered by #wild_nothing I have checked dependency the tree to list all dependencies on org.codehaus.jackson
In my case the problem was that I had several versions of libraries that provided this class:
org.codehaus.jackson:jackson-mapper-lgpl:jar:1.5.0
org.codehaus.jackson:jackson-core-lgpl:jar:1.5.0
org.codehaus.jackson:jackson-mapper-asl:jar:1.8.2
org.codehaus.jackson:jackson-core-asl:jar:1.8.2
My solution was to exclude the older versions(1.5.0) and only leave the 1.8.2 version
<dependency>
<groupId>cfm.foo</groupId>
<artifactId>jive</artifactId>
<exclusions>
<exclusion>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-core-lgpl</artifactId>
</exclusion>
<exclusion>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-mapper-lgpl</artifactId>
</exclusion>
</exclusions>
</dependency>
in this case jive was depending on older 1.5.0 version