We Keep Coding

java.lang.NoClassDefFoundError: Failed to link org/pentaho/di/core/attributes/metastore/EmbeddedMetaStore

I'm building a batch with Pentaho Kettle to process some files. This batch is started by a servlet. However, when I try to start it, I get the following warn which actually block the process:
WARN [org.jboss.modules] (Batch Thread - 2) Failed to define class org.pentaho.di.core.attributes.metastore.EmbeddedMetaStore in Module "deployment.my-program-1.0.0.0-SNAPSHOT.war:main" from Service Module Loader: java.lang.NoClassDefFoundError: Failed to link org/pentaho/di/core/attributes/metastore/EmbeddedMetaStore (Module "deployment.my-program-1.0.0.0-SNAPSHOT.war:main" from Service Module Loader): org/pentaho/metastore/api/BaseMetaStore
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.jboss.modules.ModuleClassLoader.defineClass(ModuleClassLoader.java:446)
at org.jboss.modules.ModuleClassLoader.loadClassLocal(ModuleClassLoader.java:274)
at org.jboss.modules.ModuleClassLoader$1.loadClassLocal(ModuleClassLoader.java:78)
at org.jboss.modules.Module.loadModuleClass(Module.java:605)
at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:190)
at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:363)
at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:351)
at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:93)
at org.pentaho.di.base.AbstractMeta.<init>(AbstractMeta.java:162)
at org.pentaho.di.job.JobMeta.<init>(JobMeta.java:201)
at org.pentaho.di.job.JobMeta.<init>(JobMeta.java:755)
However, when I look in my lib folder, the jar containing this class (kettle-engine-8.2.0.0-342.jar) is there and the class is also in it.
I'm using a JBoss EAP 7.0 with Java 1.8.0.161 on my computer for development. I start it with my IDE (Eclipse Photon). Here are my dependencies in the pom.xml:
<dependencies>
<dependency>
<groupId>javax</groupId>
<artifactId>javaee-api</artifactId>
<version>7.0</version>
</dependency>
<!-- vf2 debug -->
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-vfs2</artifactId>
<version>2.2</version>
</dependency>
<!-- maven-war-plugin-->
<dependency>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-war-plugin</artifactId>
<version>3.2.2</version>
</dependency>
<!-- pentaho kettle -->
<dependency>
<groupId>pentaho-kettle</groupId>
<artifactId>kettle-core</artifactId>
<version>8.2.0.0-342</version>
</dependency>
<dependency>
<groupId>pentaho-kettle</groupId>
<artifactId>kettle-engine</artifactId>
<version>8.2.0.0-342</version>
</dependency>
<dependency>
<groupId>com.googlecode.json-simple</groupId>
<artifactId>json-simple</artifactId>
<version>1.1</version>
</dependency>
<dependency>
<groupId>org.codehaus.janino</groupId>
<artifactId>janino</artifactId>
<version>2.5.16</version>
</dependency>
<dependency>
<groupId>org.mozilla</groupId>
<artifactId>rhino</artifactId>
<version>1.7R3</version>
</dependency>
<!-- this correct an error -->
<dependency>
<groupId>commons-lang</groupId>
<artifactId>commons-lang</artifactId>
<version>2.6</version>
</dependency>
<!-- netty version override to correct netty 3.7 bug / FYI, pentaho don't use netty anyway-->
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty</artifactId>
<version>3.9.0.Final</version>
</dependency>
</dependencies>
Important : this issue does not occur when using previous version of kettle-core and kettle-engine (tested with 7.0).
It's an embedded call so I do not directly call this class in my code.
I do not have multiple versions of this jar in my classpath.

Another person in my company found out so i'll just answer here to close the topic.
I didn't show it here but in my dependencies I had a dependency to some custom framework from work. This framework actually included pentaho, but the 7.0 version. Then, #suresh was right when he talk about multiple version jar.
So the answer is :
Check if any dependency actually contains duplicate dependency you're calling somewhere else. To avoid this issue, use the excluding syntax in the dependency calling the wrong versions :
<exclusions>
<exclusion>
<groupId>pentaho-kettle</groupId>
<artifactId>kettle-engine</artifactId>
</exclusion>
<exclusion>
<groupId>pentaho-kettle</groupId>
<artifactId>kettle-core</artifactId>
</exclusion>
</exclusions>

java.lang.NoClassDefFoundError: io/netty/handler/ssl/SslContextBuilder

I am trying to implement relayrides/pushy, but am getting the following runtime error:
Jun 28, 2017 2:06:58 PM com.turo.pushy.apns.SslUtil getSslProvider
INFO: Native SSL provider not available; will use JDK SSL provider.
Exception in thread "main" java.lang.NoClassDefFoundError: io/netty/handler/ssl/SslContextBuilder
at com.turo.pushy.apns.ApnsClientBuilder.build(ApnsClientBuilder.java:396)
at com.jobs.spring.service.NotificationServiceImpl.sendIOSPushNotification(NotificationServiceImpl.java:122)
Caused by: java.lang.ClassNotFoundException: io.netty.handler.ssl.SslContextBuilder
pom.xml
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.5.2</version>
</dependency>
<!-- Push Notifications -->
<dependency>
<groupId>com.turo</groupId>
<artifactId>pushy</artifactId>
<version>0.10</version>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-tcnative</artifactId>
<version>2.0.5.Final</version>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-handler</artifactId>
<version>4.0.27.Final</version>
</dependency>
<dependency>
<groupId>com.ning</groupId>
<artifactId>async-http-client</artifactId>
<version>1.9.40</version>
</dependency>
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>2.8.1</version>
</dependency>
java
final ApnsClient apnsClient = new ApnsClientBuilder()
.setClientCredentials(new File(PATH_TO_P12_CERT), CERT_PASSWORD )
.build();
I am guessing my mvn dependencies are incorrect. Any help appreciated.
UPDATE
I updated my dependencies to:
<dependency>
<groupId>com.turo</groupId>
<artifactId>pushy</artifactId>
<version>0.10</version>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-all</artifactId>
<version>4.1.11.Final</version>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-tcnative</artifactId>
<version>2.0.1.Final</version>
</dependency>
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>2.6</version>
</dependency>
But now get:
Jun 28, 2017 2:40:18 PM com.turo.pushy.apns.SslUtil getSslProvider
INFO: Native SSL provider not available; will use JDK SSL provider.
Exception in thread "main" java.lang.NoSuchMethodError: io.netty.bootstrap.Bootstrap.config()Lio/netty/bootstrap/BootstrapConfig;
at com.turo.pushy.apns.ApnsClient.<init>(ApnsClient.java:172)
at com.turo.pushy.apns.ApnsClientBuilder.build(ApnsClientBuilder.java:420)
at com.jobs.spring.service.NotificationServiceImpl.sendIOSPushNotification(NotificationServiceImpl.java:121)

seems like you mix netty 4.1 and 4.0. You need to only use 4.1 if you want to use pushy.

Did you clean your classpath in between I created a test project containing only the dependency
<dependency>
<groupId>com.turo</groupId>
<artifactId>pushy</artifactId>
<version>0.10</version>
</dependency>
and a simple main class just creating the client with the snipped you are using above and it works out fine
To me it looks like there might be two versions of BootstrapConfig on the classpath. Try to remove all dependencies besides pushy and clean/refreshing the maven dependencies.

jackson.databind no such method errors

I'm using auth0-java client library to interact with auth0 v2 service, my codes compile and works fine at my development environment but when I deploy that build in another test environment it throws the following exception:
java.lang.NoSuchMethodError:com.fasterxml.jackson.databind.ObjectMapper.readerFor(Lcom/fasterxml/jackson/databind/JavaType;)Lcom/fasterxml/jackson/databind/ObjectReader;
at com.auth0.json.mgmt.users.UsersPageDeserializer.getArrayElements(UsersPageDeserializer.java:52)
at com.auth0.json.mgmt.users.UsersPageDeserializer.deserialize(UsersPageDeserializer.java:30)
at com.auth0.json.mgmt.users.UsersPageDeserializer.deserialize(UsersPageDeserializer.java:15)
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3562)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2597)
at com.auth0.net.CustomRequest.parseResponse(CustomRequest.java:63)
at com.auth0.net.BaseRequest.execute(BaseRequest.java:37)
at com.myapp.security.Auth0Service.getUserByEmail(Auth0Service.java:246)
at com.myapp.security.Auth0Service.checkForExsistingAuth0Account(Auth0Service.java:266)
at com.myapp.security.AdminUILayout.lambda$launchProgressUpdater$0(AdminUILayout.java:293)
at java.lang.Thread.run(Thread.java:745)
I've already gone through several stackover flow questions like this and tried by cleaning .m2/repository/com/fasterxml folder of that test environment, but cloudn't solve the no such method error
here is my pom file and related codes are given below:
code
public User getUserByEmail(String email){
UserFilter filter = new UserFilter();
filter.withQuery(email);
Request<UsersPage> request = mgmt.users().list(filter);
try {
UsersPage response = request.execute();
for (User u:response.getItems()) {
if (u.getEmail().equals(email)) {
return u;
}
}
} catch (APIException exception) {
// api error
System.out.println("APIException:::::"+exception.getDescription());
} catch (Auth0Exception exception) {
// request error
System.out.println("Auth0Exception:::::"+exception.getMessage());
}catch(Exception ex){
System.out.println("Other exception:::::"+ex.getMessage());
}
return null;
}
pom.xml
<properties>
<jackson.version>2.8.5</jackson.version>
</properties>
<dependency>
<groupId>com.fasterxml.jackson.dataformat</groupId>
<artifactId>jackson-dataformat-yaml</artifactId>
<version>${jackson.version}</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>${jackson.version}</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
<version>${jackson.version}</version>
</dependency>
<dependency>
<groupId>com.auth0</groupId>
<artifactId>auth0</artifactId>
<version>1.0.0</version>
</dependency>
Update
I've moved those jackson dependencies into <dependencyManagement> section of my pom.xml file,
<dependencyManagement>
<dependencies>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
<version>2.8.5</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.8.5</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
<version>2.8.0</version>
</dependency>
</dependencies>
</dependencyManagement>
according to maven documentation ,no matter which version of library jackson-core is requested by a dependency, version ${jackson.version} will always be used, still my codes works in local ,but in that test server
the the exception remains same, even if I deploy my war file in that test server code throws the same exception.

most likely you have compiled your code against a different version of the class, than the one you are using when running it.
Please make sure that on the environment where you run the code there is no any other version of jackson-databind dependency on your classpath.

finally solved the issue , I found that glassfish 4.1.1 has a very old version 2.3.2 of jackson-core jackson-annotation and jackson-databind, so we decided to use Payara server that uses fairly latest version of jackson jackson 2.8.5 and tested the no such method error is solved, the code works fine in that Payara server
Update:
glassfish version 5.0 uses updated version jackson 2.8.9 , I think using that will solve this issue too

Probably one of your other artifacts depends on different version of databind. You can try to exlude it explicitly from each of them to guess
<dependency>
...
<exclusions>
<exclusion>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
</exclusion>
</exclusions>
</dependency>

Please check version.
Screenshot:

For me, I encountered the same exception due to lower version of jackson-databind:
From:
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.5.0</version>
</dependency>
Changed to:
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.8.9</version>
</dependency>
I just update my pom.xml file with higher version of jackson-databind.
Then right-click Project --> Update Maven Project and the exception is resolved.

<!-- faster JSON -->
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.12.0</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
<version>2.12.0</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
<version>2.12.0</version>
</dependency>

Embedded Neo4j SPI Class (lucene PostingsFormat) Error

I've embedded Neo4j 3.0.1 into a Java 8 application, but I've been running into SPI issues.
Running from within IntelliJ produces the correct results as expected, but as soon as I build the artifact to a JAR, run it and attempt to write to Neo4j, I get the following exception:
Caused by: org.neo4j.kernel.impl.store.UnderlyingStorageException: java.lang.IllegalArgumentException: An SPI class of type org.apache.lucene.codecs.PostingsFormat with name 'BlockTreeOrds' does not exist. You need to add the corresponding JAR file supporting this SPI to your classpath. The current classpath supports the following names: [Lucene50]
at org.neo4j.kernel.impl.transaction.command.LabelUpdateWork.apply(LabelUpdateWork.java:62)
at org.neo4j.kernel.impl.transaction.command.LabelUpdateWork.apply(LabelUpdateWork.java:33)
at org.neo4j.concurrent.WorkSync.doSynchronizedWork(WorkSync.java:121)
at org.neo4j.concurrent.WorkSync.apply(WorkSync.java:90)
at org.neo4j.kernel.impl.transaction.command.IndexBatchTransactionApplier.close(IndexBatchTransactionApplier.java:105)
at org.neo4j.kernel.impl.api.BatchTransactionApplierFacade.close(BatchTransactionApplierFacade.java:70)
at org.neo4j.kernel.impl.storageengine.impl.recordstorage.RecordStorageEngine.apply(RecordStorageEngine.java:336)
at org.neo4j.kernel.impl.api.TransactionRepresentationCommitProcess.applyToStore(TransactionRepresentationCommitProcess.java:78)
... 25 more
There seems to be no exception starting Neo4j so I'm assuming that certain dependencies are not being resolved with the Maven build.
I have the following in my pom.xml file:
<dependency>
<groupId>com.sparkjava</groupId>
<artifactId>spark-core</artifactId>
<version>2.5</version>
</dependency>
<dependency>
<groupId>org.neo4j</groupId>
<artifactId>neo4j</artifactId>
<version>3.0.1</version>
<type>pom</type>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<version>1.7.21</version>
</dependency>
<dependency>
<groupId>org.apache.lucene</groupId>
<artifactId>lucene-codecs</artifactId>
<version>5.5.0</version>
</dependency>
<dependency>
<groupId>org.neo4j</groupId>
<artifactId>neo4j-slf4j</artifactId>
<version>3.0.0-M02</version>
</dependency>
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>2.6.2</version>
</dependency>
<dependency>
<groupId>com.github.jknack</groupId>
<artifactId>handlebars</artifactId>
<version>4.0.5</version>
</dependency>
<dependency>
<groupId>com.opencsv</groupId>
<artifactId>opencsv</artifactId>
<version>3.7</version>
</dependency>
<dependency>
<groupId>commons-codec</groupId>
<artifactId>commons-codec</artifactId>
<version>1.10</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>${jackson.version}</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
<version>${jackson.version}</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
<version>${jackson.version}</version>
</dependency>
How do I resolve this issue?
UPDATE:
I've re-created this issue with a really simple blank project, source can be found here if you'd like to run it on your end: https://github.com/SeanNieuwoudt/neo4j-spi

I took your project and executed in my eclipse. I have no issues. Here are below console logs after starting:
[Thread-0] INFO org.eclipse.jetty.util.log - Logging initialized #473ms
[Thread-0] INFO spark.embeddedserver.jetty.EmbeddedJettyServer - == Spark has ignited ...
[Thread-0] INFO spark.embeddedserver.jetty.EmbeddedJettyServer - >> Listening on 0.0.0.0:8080
[Thread-0] INFO org.eclipse.jetty.server.Server - jetty-9.3.6.v20151106
[Thread-0] INFO org.eclipse.jetty.server.ServerConnector - Started ServerConnector#5aa07ed2{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
[Thread-0] INFO org.eclipse.jetty.server.Server - Started #772ms
While accessing - http://localhost:8080/
I got output printed as 'Hello World"
Steps i followed:
1) Downloaded your project from your github url
2) Imported as maven projects pointing to java 8
3) Maven install was successful
3) Ran main class
4) Saw the output in the browser.
Am i missing any steps to replicate actual issue?
Or is something to do with your maven. May be check your maven dependencies after running maven install to see if the expected jars are downloaded.
Good luck.

Try to add the following dependency:
<dependency>
<groupId>org.apache.lucene</groupId>
<artifactId>lucene-backward-codecs</artifactId>
<version>5.5.0</version>
</dependency>

This is my solution. https://stackoverflow.com/a/38623295/6579110. You can also check https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#ServicesResourceTransformer for more details.

Hadoop 2.6.x and Amazon AWS SDK Library conflicts http-core conflict

My job writes each record to DynamoDB in the Hadoop's map.
I cannot make it run with Hadoop 2.6 which has httpclient-4.2.5.jar and httpcore-4.2.5.jar.
AWS which I am using was built using httpclient-4.5.2.jar and httpcore-4.4.4.jar.
When I am using classpath to include the new jar files, it gives the following exception.
java.lang.Exception: java.lang.NoSuchFieldError: INSTANCE
at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
Caused by: java.lang.NoSuchFieldError: INSTANCE
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.<clinit>(SSLConnectionSocketFactory.java:144)
at com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.getPreferredSocketFactory(ApacheConnectionManagerFactory.java:87)
at com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.create(ApacheConnectionManagerFactory.java:65)
at com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.create(ApacheConnectionManagerFactory.java:58)
at com.amazonaws.http.apache.client.impl.ApacheHttpClientFactory.create(ApacheHttpClientFactory.java:49)
To me, it looks like Hadoop was built using the old libraries and something has changed in the syntax.
What's the reasonable solution than recompiling older sources of AWS?

As an update, I had to switch to Maven and play around with the versions a bit.
<!-- http://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.6.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-core</artifactId>
<version>2.6.0</version>
</dependency>
<!-- http://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk -->
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.3.4</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-dynamodb</artifactId>
<version>1.9.13</version>
</dependency>
<dependency>
<groupId>org.netpreserve.commons</groupId>
<artifactId>webarchive-commons</artifactId>
<version>1.1.4</version>
</dependency>
Finally, it works