I'm new to the Gobblin and trying to ingest data from the Kafka to HDFS. I was able to flow the Kafka-HDFS Ingestion example successfully. But now I need to add a time-based writer partition option to my job. I did go through the TimeBasedWriterPartitioner google forum and come up with the bellow solution as suggested by Zongjun.
I create a separate Java project for my time-based writer partition class:
import gobblin.writer.partitioner.TimeBasedWriterPartitioner;
public class LogJsonWriterPartitioner extends TimeBasedWriterPartitioner<byte[]> {
public LogJsonWriterPartitioner(gobblin.configuration.State state, int numBranches, int branchId) {
super(state, numBranches, branchId);
}
#Override
public long getRecordTimestamp(byte[] payload) {
return System.currentTimeMillis();
}
}
POM.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.pm.data.gobblin.kafka</groupId>
<artifactId>LogJsonWriterPartitioner </artifactId>
<version>1.0-SNAPSHOT</version>
<dependencies>
<dependency>
<groupId>com.linkedin.gobblin</groupId>
<artifactId>gobblin-api</artifactId>
<version>0.6.2</version>
</dependency>
<dependency>
<groupId>com.linkedin.gobblin</groupId>
<artifactId>gobblin-core</artifactId>
<version>0.6.2</version>
</dependency>
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>2.3</version>
</dependency>
<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<version>6.9.10</version>
<scope>test</scope>
</dependency>
</dependencies>
</project>
Create a Jar from the above project and copy it to gobblin-dist/lib directory
I update gobblin-mapreduce.sh in gobblin-dist/bin directory and add the new jar name under LIBJARS.
Create a Job file as bellow:
job.name=GobblinKafkaQuickStart
job.group=GobblinKafka
job.description=Gobblin quick start job for Kafka
job.lock.enabled=false
fs.uri=file:///
kafka.brokers=localhost:9092
source.class=org.apache.gobblin.source.extractor.extract.kafka.KafkaSimpleSource
extract.namespace=org.apache.gobblin.extract.kafka
writer.builder.class=org.apache.gobblin.writer.SimpleDataWriterBuilder
writer.partitioner.class=com.pm.data.gobblin.kafka.LogJsonWriterPartitioner
writer.partition.granularity=day
writer.partition.pattern=YYYY-MM-dd
writer.partition.timezone=UTC
writer.file.path.type=tablename
writer.destination.type=HDFS
writer.output.format=txt
data.publisher.type=org.apache.gobblin.publisher.BaseDataPublisher
data.publisher.replace.final.dir=false
data.publisher.final.dir=/home/myuser/Desktop/Gobblin
mr.job.max.mappers=1
metrics.reporting.file.enabled=true
metrics.log.dir=${gobblin.cluster.work.dir}/metrics
metrics.reporting.file.suffix=txt
bootstrap.with.offset=earliest
Then I start gobblin as standalone using the gobblin-standalone.sh file in the bin directory.
I got below error on logs/gobblin-current.log
org.apache.gobblin.runtime.fork.Fork 250 - Fork 0 of task task_GobblinKafkaQuickStart_1590391135660_0 failed to process data records. Set throwable in holder org.apache.gobblin.runtime.ForkThrowableHolder#433cf3c0
java.io.IOException: java.lang.ClassNotFoundException: com.pm.data.logging.gobblin.LogJsonWriterPartitioner
at org.apache.gobblin.writer.PartitionedDataWriter.<init>(PartitionedDataWriter.java:135)
at org.apache.gobblin.runtime.fork.Fork.buildWriter(Fork.java:534)
at org.apache.gobblin.runtime.fork.Fork.buildWriterIfNotPresent(Fork.java:542)
at org.apache.gobblin.runtime.fork.Fork.processRecord(Fork.java:502)
at org.apache.gobblin.runtime.fork.AsynchronousFork.processRecord(AsynchronousFork.java:103)
at org.apache.gobblin.runtime.fork.AsynchronousFork.processRecords(AsynchronousFork.java:86)
at org.apache.gobblin.runtime.fork.Fork.run(Fork.java:243)
at org.apache.gobblin.util.executors.MDCPropagatingRunnable.run(MDCPropagatingRunnable.java:39)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: com.pm.data.logging.gobblin.LogJsonWriterPartitioner
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.gobblin.writer.PartitionedDataWriter.<init>(PartitionedDataWriter.java:128)
... 12 more
Whoever when I modify my job file as writer.partitioner.class=LogJsonWriterPartitioner, error is changed as java.lang.NoClassDefFoundError: gobblin/writer/partitioner/TimeBasedWriterPartitioner.
Could some help me to overcome this problem?
For the first problem, make sure that you have correct package statement for LogJsonWriterPartitioner, i would expect it to be package com.pm.data.logging.gobblin
For the second, looks like dependencies in pom.xml are not correct, and that is why TimeBasedWriterPartitioner cannot be loaded. com.linkedin.gobblin was renamed to org.apache.gobblin long time ago, and the version numbers are higher. Recent release was '0.14.0'
Related
I have this error when i run my project :
ERROR StatusLogger Unable to create Lookup for ctx
Here is the whole stacktrace :
ERROR StatusLogger Unable to create Lookup for ctx
java.lang.NoClassDefFoundError: org/apache/logging/log4j/util/ServiceLoaderUtil
at org.apache.logging.log4j.core.impl.ThreadContextDataInjector.getServiceProviders(ThreadContextDataInjector.java:77)
at org.apache.logging.log4j.core.impl.ThreadContextDataInjector.(ThreadContextDataInjector.java:64)
at org.apache.logging.log4j.core.impl.ThreadContextDataInjector$ForCopyOnWriteThreadContextMap.(ThreadContextDataInjector.java:211)
at org.apache.logging.log4j.core.impl.ContextDataInjectorFactory.createDefaultInjector(ContextDataInjectorFactory.java:94)
at org.apache.logging.log4j.core.impl.ContextDataInjectorFactory.createInjector(ContextDataInjectorFactory.java:71)
at org.apache.logging.log4j.core.lookup.ContextMapLookup.(ContextMapLookup.java:34)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at org.apache.logging.log4j.core.util.ReflectionUtil.instantiate(ReflectionUtil.java:189)
at org.apache.logging.log4j.core.lookup.Interpolator.(Interpolator.java:81)
at org.apache.logging.log4j.core.lookup.Interpolator.(Interpolator.java:100)
at org.apache.logging.log4j.core.config.AbstractConfiguration.(AbstractConfiguration.java:135)
at org.apache.logging.log4j.core.config.NullConfiguration.(NullConfiguration.java:32)
at org.apache.logging.log4j.core.LoggerContext.(LoggerContext.java:74)
at org.apache.logging.log4j.core.selector.ClassLoaderContextSelector.createContext(ClassLoaderContextSelector.java:254)
at org.apache.logging.log4j.core.selector.ClassLoaderContextSelector.locateContext(ClassLoaderContextSelector.java:218)
at org.apache.logging.log4j.core.selector.ClassLoaderContextSelector.getContext(ClassLoaderContextSelector.java:136)
at org.apache.logging.log4j.core.selector.ClassLoaderContextSelector.getContext(ClassLoaderContextSelector.java:123)
at org.apache.logging.log4j.core.selector.ClassLoaderContextSelector.getContext(ClassLoaderContextSelector.java:117)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:150)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:47)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:196)
at org.apache.logging.log4j.LogManager.getLogger(LogManager.java:599)
at org.apache.poi.ooxml.POIXMLDocumentPart.(POIXMLDocumentPart.java:56)
at CreateDocumentSimple.main(CreateDocumentSimple.java:88)
Caused by: java.lang.ClassNotFoundException: org.apache.logging.log4j.util.ServiceLoaderUtil
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
... 27 more
Exception in thread "main" java.lang.NoClassDefFoundError: Could not initialize class org.apache.logging.log4j.core.impl.ThreadContextDataInjector
at org.apache.logging.log4j.core.impl.ThreadContextDataInjector$ForCopyOnWriteThreadContextMap.(ThreadContextDataInjector.java:211)
at org.apache.logging.log4j.core.impl.ContextDataInjectorFactory.createDefaultInjector(ContextDataInjectorFactory.java:94)
at org.apache.logging.log4j.core.impl.ContextDataInjectorFactory.createInjector(ContextDataInjectorFactory.java:71)
at org.apache.logging.log4j.core.impl.ReusableLogEventFactory.(ReusableLogEventFactory.java:42)
at org.apache.logging.log4j.core.config.LoggerConfig.(LoggerConfig.java:101)
at org.apache.logging.log4j.core.config.AbstractConfiguration.(AbstractConfiguration.java:138)
at org.apache.logging.log4j.core.config.NullConfiguration.(NullConfiguration.java:32)
at org.apache.logging.log4j.core.LoggerContext.(LoggerContext.java:74)
at org.apache.logging.log4j.core.selector.ClassLoaderContextSelector.createContext(ClassLoaderContextSelector.java:254)
at org.apache.logging.log4j.core.selector.ClassLoaderContextSelector.locateContext(ClassLoaderContextSelector.java:218)
at org.apache.logging.log4j.core.selector.ClassLoaderContextSelector.getContext(ClassLoaderContextSelector.java:136)
at org.apache.logging.log4j.core.selector.ClassLoaderContextSelector.getContext(ClassLoaderContextSelector.java:123)
at org.apache.logging.log4j.core.selector.ClassLoaderContextSelector.getContext(ClassLoaderContextSelector.java:117)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:150)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:47)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:196)
at org.apache.logging.log4j.LogManager.getLogger(LogManager.java:599)
at org.apache.poi.ooxml.POIXMLDocumentPart.(POIXMLDocumentPart.java:56)
at CreateDocumentSimple.main(CreateDocumentSimple.java:88)
I don't know where this come from since i don't user any logger in my whole projet. If you need more information please feel free to ask.
Here is my pom.xml :
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.example</groupId>
<artifactId>TestPoi</artifactId>
<version>1.0-SNAPSHOT</version>
<properties>
<maven.compiler.source>8</maven.compiler.source>
<maven.compiler.target>8</maven.compiler.target>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.poi</groupId>
<artifactId>poi-ooxml</artifactId>
<version>5.2.2</version>
</dependency>
</dependencies>
</project>
Add the library log4j-api.jar to the classpath.
To make your project work you have to remove all jars starting with log4j-1.x.
I made the same mistake, because when I downloaded the log4j dependency package, I didn't know which files I needed, so I added all of them. But by doing so, for whatever reason, now log4j 1 is used instead of log4j 2.
When using org.jboss.resteasy:resteasy-client:4.5.9.Final, I'm getting this exception: Caused by: java.lang.IncompatibleClassChangeError: Expected static method 'java.lang.Object org.jboss.resteasy.spi.ResteasyProviderFactory.getContextData(java.lang.Class)'
However, when I use an earlier version, it seems to work fine. (Or at least it works well enough to fool me.)
Here's my simplified pom:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.example</groupId>
<artifactId>example-project</artifactId>
<version>1.0-SNAPSHOT</version>
<properties>
<maven.compiler.source>15</maven.compiler.source>
<maven.compiler.target>15</maven.compiler.target>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<keycloak.version>12.0.4</keycloak.version>
<!-- Seems to work with this version, but not 4.5.9.Final -->
<resteasy.version>3.6.2.Final</resteasy.version>
</properties>
<dependencies>
<dependency>
<groupId>org.keycloak</groupId>
<artifactId>keycloak-admin-client</artifactId>
<version>${keycloak.version}</version>
</dependency>
<dependency>
<groupId>org.jboss.resteasy</groupId>
<artifactId>resteasy-client</artifactId>
<version>${resteasy.version}</version>
</dependency>
<dependency>
<groupId>org.jboss.resteasy</groupId>
<artifactId>resteasy-jackson2-provider</artifactId>
<version>${resteasy.version}</version>
</dependency>
<dependency>
<groupId>org.jboss.resteasy</groupId>
<artifactId>resteasy-multipart-provider</artifactId>
<version>${resteasy.version}</version>
</dependency>
</dependencies>
</project>
And here's my code:
package org.example.keycloak;
import org.jboss.resteasy.client.jaxrs.ResteasyClientBuilder;
import org.keycloak.OAuth2Constants;
import org.keycloak.admin.client.Keycloak;
import org.keycloak.admin.client.KeycloakBuilder;
public class KeycloakClientAuthExample {
public static void main(String[] args) {
Keycloak keycloak = KeycloakBuilder.builder()
.serverUrl("http://localhost:8080/auth")
.grantType(OAuth2Constants.PASSWORD)
.realm("DungeoneersDemo")
.clientId("dungeoneers-data")
.clientSecret("11111111-2222-3333-4444-555555555555")
.username("user")
.password("pass")
.resteasyClient(
// new ResteasyClientBuilderImpl() // <-- For 4.5.9.Final
new ResteasyClientBuilder()
.connectionPoolSize(10).build()
).build();
keycloak.tokenManager().getAccessToken();
AccessTokenResponse atr =
keycloak.tokenManager().getAccessToken();
System.out.println(atr.getToken());
}
}
Seems to work and I get what looks enough to me like a bearer token: eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICI2bWhhWUQ..etc...
However, when I update my resteasy.version to the a later version✳ (4.5.9.Final), I get an error:
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: RESTEASY003940: Unable to instantiate MessageBodyReader
at org.jboss.resteasy.plugins.providers.RegisterBuiltin.register(RegisterBuiltin.java:78)
at org.jboss.resteasy.plugins.providers.RegisterBuiltin.getClientInitializedResteasyProviderFactory(RegisterBuiltin.java:54)
at org.jboss.resteasy.client.jaxrs.internal.ResteasyClientBuilderImpl.getProviderFactory(ResteasyClientBuilderImpl.java:372)
at org.jboss.resteasy.client.jaxrs.internal.ResteasyClientBuilderImpl.build(ResteasyClientBuilderImpl.java:390)
at org.sandbox.security.openidc.keycloak.KeycloakClientAuthExample.main(KeycloakClientAuthExample.java:22)
Caused by: java.lang.RuntimeException: RESTEASY003940: Unable to instantiate MessageBodyReader
at org.jboss.resteasy.core.providerfactory.CommonProviders.processProviderContracts(CommonProviders.java:93)
at org.jboss.resteasy.core.providerfactory.ClientHelper.processProviderContracts(ClientHelper.java:104)
at org.jboss.resteasy.core.providerfactory.ResteasyProviderFactoryImpl.processProviderContracts(ResteasyProviderFactoryImpl.java:841)
at org.jboss.resteasy.core.providerfactory.ResteasyProviderFactoryImpl.registerProvider(ResteasyProviderFactoryImpl.java:829)
at org.jboss.resteasy.core.providerfactory.ResteasyProviderFactoryImpl.registerProvider(ResteasyProviderFactoryImpl.java:816)
at org.jboss.resteasy.plugins.providers.RegisterBuiltin.registerProviders(RegisterBuiltin.java:109)
at org.jboss.resteasy.plugins.providers.RegisterBuiltin.register(RegisterBuiltin.java:74)
... 4 more
Caused by: java.lang.RuntimeException: RESTEASY003325: Failed to construct public org.jboss.resteasy.plugins.providers.jaxb.JAXBElementProvider()
at org.jboss.resteasy.core.ConstructorInjectorImpl.constructOutsideRequest(ConstructorInjectorImpl.java:250)
at org.jboss.resteasy.core.ConstructorInjectorImpl.construct(ConstructorInjectorImpl.java:209)
at org.jboss.resteasy.core.providerfactory.Utils.createProviderInstance(Utils.java:102)
at org.jboss.resteasy.core.providerfactory.CommonProviders.processProviderContracts(CommonProviders.java:87)
... 10 more
Caused by: java.lang.IncompatibleClassChangeError: Expected static method 'java.lang.Object org.jboss.resteasy.spi.ResteasyProviderFactory.getContextData(java.lang.Class)'
at org.jboss.resteasy.plugins.providers.jaxb.AbstractJAXBProvider.<init>(AbstractJAXBProvider.java:52)
at org.jboss.resteasy.plugins.providers.jaxb.JAXBElementProvider.<init>(JAXBElementProvider.java:46)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:64)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481)
at org.jboss.resteasy.core.ConstructorInjectorImpl.constructOutsideRequest(ConstructorInjectorImpl.java:225)
... 13 more
Process finished with exit code 1
✳ Note: When upgrading to the later version, ResteasyClientBuilder was abstracted and the constructor call needed to be replaced with new ResteasyClientBuilderImpl().
I was typing out my question and, while looking into it one last time before submitting, I found the issue and figured I'd throw it out there just in case someone else has this issue.
It turns out that there is another resteasy dependency that needed to be pulled in because, otherwise, an old version is pulled in. When using the newer version of resteasy-client, the other dependency being pulled in was org.jboss.resteasy:resteasy-jaxb-provider:3.9.1.Final. I added the newer version to my pom (org.jboss.resteasy:resteasy-jaxb-provider:4.5.9.Final) and everything seems to now work.
Not sure why this is the case. I would think if the dependency was being pulled in anyway by resteasy-client, it should've been pulling in the same version.
We were facing the same issue and your reply really helped to find out the mistake. To let other users check the error in detail, further details are:
KC 15.* and KC16.1 uses, in their main pom.xml, resteasy version 3.*
<resteasy.version>3.15.1.Final</resteasy.version>
<resteasy.undertow.version>${resteasy.version}</resteasy.undertow.version>
When you check the version for the container in the server, you can see that, from KC16, the new Widlfly server uses resteasy v4.*
This concept causes a difference in the jars, while it has the behavior to have scope provided (but is not). As you mentioned, the solution to that is the following.
<properties>
<resteasy.version>4.5.9.Final</resteasy.version>
</properties>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.jboss.resteasy</groupId>
<artifactId>resteasy-client</artifactId>
<version>${resteasy.version}</version>
</dependency>
<dependency>
<groupId>org.jboss.resteasy</groupId>
<artifactId>resteasy-jackson2-provider</artifactId>
<version>${resteasy.version}</version>
</dependency>
<dependency>
<groupId>org.jboss.resteasy</groupId>
<artifactId>resteasy-multipart-provider</artifactId>
<version>${resteasy.version}</version>
</dependency>
<dependency>
<groupId>org.jboss.resteasy</groupId>
<artifactId>resteasy-jaxb-provider</artifactId>
<version>${resteasy.version}</version>
</dependency>
</dependencies></dependencyManagement>
This will force 3 main API changes to consider:
final ResteasyClient client = new ResteasyClientBuilder()
.disableTrustManager()
.socketTimeout(60, TimeUnit.SECONDS)
.establishConnectionTimeout(10, TimeUnit.SECONDS)
/// will need to change to the new methods:
final ResteasyClient client = new ResteasyClientBuilderImpl()
.disableTrustManager()
.readTimeout(60, TimeUnit.SECONDS)
.connectTimeout(10, TimeUnit.SECONDS)
And also this change:
HttpServletResponse contextData = ResteasyProviderFactory.getContextData(HttpServletResponse.class);
//Change to
ResteasyProviderFactory resteasyProviderFactory = ResteasyProviderFactoryImpl.getInstance();
HttpServletResponse contextData = resteasyProviderFactory.getContextData(HttpServletResponse.class);
You r previous reply help us to understand under the hood the problem, but, anyway, users shall have more info. See the image bellow as a running KC16 container showing all related libs
I am trying to connect to Azure event hub as described here. But getting below error:
Exception in thread "main" java.lang.NoSuchMethodError: reactor.core.publisher.Flux.retryWhen(Ljava/util/function/Function;)Lreactor/core/publisher/Flux;
at com.azure.core.amqp.implementation.RetryUtil.withRetry(RetryUtil.java:58)
at com.azure.core.amqp.implementation.ReactorConnection.getClaimsBasedSecurityNode(ReactorConnection.java:142)
at com.azure.messaging.eventhubs.implementation.EventHubReactorAmqpConnection.createSession(EventHubReactorAmqpConnection.java:148)
at com.azure.core.amqp.implementation.ReactorConnection.lambda$createSession$9(ReactorConnection.java:203)
at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
at com.azure.core.amqp.implementation.ReactorConnection.lambda$createSession$10(ReactorConnection.java:197)
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:113)
at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onNext(FluxPeekFuseable.java:210)
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1784)
at reactor.core.publisher.MonoCallable.subscribe(MonoCallable.java:61)
at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:157)
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1784)
at com.azure.core.amqp.implementation.AmqpChannelProcessor$ChannelSubscriber.onNext(AmqpChannelProcessor.java:310)
at com.azure.core.amqp.implementation.AmqpChannelProcessor.lambda$onNext$0(AmqpChannelProcessor.java:87)
at java.lang.Iterable.forEach(Iterable.java:75)
at com.azure.core.amqp.implementation.AmqpChannelProcessor.onNext(AmqpChannelProcessor.java:87)
at reactor.core.publisher.FluxRepeatPredicate$RepeatPredicateSubscriber.onNext(FluxRepeatPredicate.java:85)
at reactor.core.publisher.Operators$MonoSubscriber.request(Operators.java:1871)
at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.request(Operators.java:2118)
at com.azure.core.amqp.implementation.AmqpChannelProcessor.requestUpstream(AmqpChannelProcessor.java:257)
at com.azure.core.amqp.implementation.AmqpChannelProcessor.subscribe(AmqpChannelProcessor.java:210)
at reactor.core.publisher.Mono.subscribe(Mono.java:4046)
at reactor.core.publisher.Mono.block(Mono.java:1726)
at com.azure.messaging.eventhubs.EventHubProducerClient.createBatch(EventHubProducerClient.java:127)
at com.sample.eventhub.eventhub.EventhubDemoApplication.main(EventhubDemoApplication.java:63)
The line at which the error is thrown(EventhubDemoApplication.java:63) has "EventDataBatch batch = producer.createBatch();"
My Pom.xml is as below:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.4.1</version>
<relativePath /> <!-- lookup parent from repository -->
</parent>
<groupId>com.sample.eventhub</groupId>
<artifactId>eventhub</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>eventhub-demo</name>
<description>Demo project for eevnt hub</description>
<properties>
<java.version>1.8</java.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-messaging-eventhubs</artifactId>
<version>5.0.1</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>
Tried changing the Spring Boot version(2.3.7), then getting the below error:
Exception in thread "restartedMain" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.devtools.restart.RestartLauncher.run(RestartLauncher.java:49)
Caused by: com.azure.core.amqp.exception.AmqpException: The messaging entity '************************' could not be found. To know more visit https://aka.ms/sbResourceMgrExceptions. TrackingId:"*********", SystemTracker:**********************, Timestamp:2021-01-11T09:40:57, errorContext[NAMESPACE: ******************, PATH: $cbs, REFERENCE_ID: cbs:receiver, LINK_CREDIT: 0]
at com.azure.core.amqp.implementation.ExceptionUtil.distinguishNotFound(ExceptionUtil.java:114)
at com.azure.core.amqp.implementation.ExceptionUtil.amqpResponseCodeToException(ExceptionUtil.java:101)
at com.azure.core.amqp.implementation.RequestResponseChannel.settleMessage(RequestResponseChannel.java:274)
at com.azure.core.amqp.implementation.RequestResponseChannel.lambda$new$0(RequestResponseChannel.java:124)
at reactor.core.publisher.LambdaSubscriber.onNext(LambdaSubscriber.java:160)
at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:114)
at reactor.core.publisher.FluxCreate$BufferAsyncSink.drain(FluxCreate.java:793)
at reactor.core.publisher.FluxCreate$BufferAsyncSink.next(FluxCreate.java:718)
at reactor.core.publisher.FluxCreate$SerializedSink.next(FluxCreate.java:153)
at com.azure.core.amqp.implementation.handler.ReceiveLinkHandler.onDelivery(ReceiveLinkHandler.java:95)
at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:185)
at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
at com.azure.core.amqp.implementation.ReactorExecutor.run(ReactorExecutor.java:82)
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Suppressed: java.lang.Exception: #block terminated with an error
at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:139)
at reactor.core.publisher.Mono.block(Mono.java:1709)
at com.azure.messaging.eventhubs.EventHubProducerClient.createBatch(EventHubProducerClient.java:127)
at com.example.Sender.SenderApplication.main(SenderApplication.java:30)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.devtools.restart.RestartLauncher.run(RestartLauncher.java:49)
Tried looking into the answer here, but no luck.
Also, I am trying only send part mentioned in the Microsoft official docs. Is that causing the issue?
Any input in this regard would be great.
The NoSuchMethodError is a result of conflicting dependencies in which it resolved a version of project reactor that does not match the one used in azure-messaging-eventhubs. The problem is that it is resolving "reactor-core" version 3.4.1 where-as azure-messaging-eventhubs uses version 3.3.0.RELEASE. In between these releases, Flux.retryWhen's methods were changed to use Retry class rather than a function and a publisher.
You can see this dependency conflict if you execute:
.\mvnw.cmd org.apache.maven.plugins:maven-dependency-plugin:2.10:tree -Dverbose=true -Dincludes=*reactor*
One solution is to upgrade your dependency to 5.3.1.
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-messaging-eventhubs</artifactId>
<version>5.3.1</version>
</dependency>
Or, you can downgrade your version of spring-boot to one that works with reactor-core 3.3.0.RELEASE (if you want to keep using azure-messaging-eventhubs 5.0.1). Or explicitly select a version of reactor-core in your pom.xml... Though, you may encounter other NoSuchMethodErrors.
Thank you #Connie for your response.
Found that legacy event hub was used. Following this Microsoft doc helped me in connecting to the event hub.
spring boot version : 2.4.1
spring cloud version : 2020.0.0
My code
#Configuration
public class BaseConfig {
#Bean
public Module sortJacksonModule() {
return new SortJacksonModule();
}
}
my pom.xml dependency
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-commons</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-openfeign-core</artifactId>
</dependency>
my pom.xml plugin
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
When run with IntelliJ IDEA, it work well.
But when run with jar(by mvn clean package), it show
Caused by: java.lang.NoClassDefFoundError: feign/codec/EncodeException
at org.springframework.cloud.openfeign.support.SortJacksonModule.setupModule(SortJacksonModule.java:47) ~[spring-cloud-openfeign-core-3.0.0.jar!/:3.0.0]
at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:819) ~[jackson-databind-2.11.3.jar!/:2.11.3]
at com.fasterxml.jackson.databind.ObjectMapper.registerModules(ObjectMapper.java:1021) ~[jackson-databind-2.11.3.jar!/:2.11.3]
at org.springframework.http.converter.json.Jackson2ObjectMapperBuilder.configure(Jackson2ObjectMapperBuilder.java:712) ~[spring-web-5.3.2.jar!/:5.3.2]
at org.springframework.http.converter.json.Jackson2ObjectMapperBuilder.build(Jackson2ObjectMapperBuilder.java:680) ~[spring-web-5.3.2.jar!/:5.3.2]
at org.springframework.boot.autoconfigure.jackson.JacksonAutoConfiguration$JacksonObjectMapperConfiguration.jacksonObjectMapper(JacksonAutoConfiguration.java:101) ~[spring-boot-autoconfigure-2.4.1.jar!/:2.4.1]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_232]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_232]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_232]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_232]
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154) ~[spring-beans-5.3.2.jar!/:5.3.2]
... 113 common frames omitted
Caused by: java.lang.ClassNotFoundException: feign.codec.EncodeException
at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[na:1.8.0_232]
at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[na:1.8.0_232]
at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:151) ~[demo-spring-core-11-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[na:1.8.0_232]
... 124 common frames omitted
After study the error log, I found that feign.codec.EncodeException is optional dependency in spring-cloud-openfeign-core, so ClassNotFoundException is right behavior(optional dependency not include in final jar).
So my question is: Why IntelliJ IDEA can run without any error? I try both IntelliJ IDEA run and mvn spring-boot:run, both work fine.
update: add example
After more study, I found out this only happen when the class not called.
try {
System.out.println("not important code");
} catch (Exception e) {
throw new EncodeException("not exist class");
}
In this example, the try catch never throw an exception. And the EncodeException class is in an optional dependency.
This code run well in IntelliJ IDEA, but fail when run as java -jar xxx.jar
========== update again with minimal demo
I create a minimal demo to reproduce this issue.
a standalone demo-module
<dependency>
<groupId>io.github.openfeign</groupId>
<artifactId>feign-core</artifactId>
<version>10.10.1</version>
<optional>true</optional>
</dependency>
import feign.codec.EncodeException;
/**
* Hello world!
*/
public class App {
public void testOptional() {
try {
System.out.println("test");
} catch (Exception e) {
throw new EncodeException("never throw this");
}
}
}
demo spring project(create by spring initializr and add a dependency)
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>com.example</groupId>
<artifactId>demo-module</artifactId>
<version>0.0.1-SNAPSHOT</version>
</dependency>
#Component
public class MyMain implements ApplicationRunner {
#Override
public void run(ApplicationArguments args) throws Exception {
new App().testOptional();
}
}
Inspect your project classpath in Idea ( CTRL-Alt-Shift-S ) - I daresay optional jar is somewhere on module compile classpath and it is enough to run your class in IDE - but not in standalone jar. Optional means in maven context that it is present on classpath while compiling, but not packed into resulting artifact.
There is an option called Enable launch optimization in IntelliJ IDEA run config, uncheck it and everything work as expected.
I am using maven dependencies in Eclipse and trying to connect neo4j and java using bolt protocol. I am running application in MacOS.
I am using Eclipse 3 and Java version 8 and jre 1.8. This is my pom.xml
**
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.neo4j</groupId>
<artifactId>neo4j</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>neo4j</name>
<url>http://maven.apache.org</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>org.neo4j.driver</groupId>
<artifactId>neo4j-java-driver</artifactId>
<version>1.2.1</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.neo4j</groupId>
<artifactId>neo4j</artifactId>
<version>3.1.4</version>
<scope>provided</scope>
</dependency>
</dependencies>
</project>
**
This is the method called in .java file
void createDatabase()
{
GraphDatabaseService dfs = new GraphDatabaseFactory().newEmbeddedDatabase(dbPath);
System.out.println("Database created!!");
Config noSSL = Config.build().withEncryptionLevel(Config.EncryptionLevel.NONE).toConfig();
try(Driver dri = GraphDatabase.driver("bolt://localhost:7687", AuthTokens.basic( "neo4j", "hello" ), noSSL))
{
System.out.println("Database connected");
Session session = dri.session();
session.run("CREATE(a:Persona{name:{name}, title:{title}})", parameters("name","Aruna","title","Bhakt"));
System.out.println("\n\t1.CREATED");
StatementResult result =session.run("MATCH(a:Persona) WHERE a.name = {name}" +
"RETURN a.name AS name, a.title AS title",
parameters("name","Aruna"));
System.out.println("\n\t2.Match");
while(result.hasNext())
{
Record record = result.next();
System.out.println(record.get("title").asString() + " " + record.get("name").asString());
}
System.out.println("\n\t3.Found Record");
session.close();
dri.close();
}
}
After running the application, I am getting all this errors
Exception in thread "main" java.lang.RuntimeException: Error starting org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory, /Users/arundhatiwahane/Documents/Neo4j/default.graphdb
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.initFacade(GraphDatabaseFacadeFactory.java:199)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:130)
at org.neo4j.graphdb.factory.GraphDatabaseFactory.newDatabase(GraphDatabaseFactory.java:101)
at org.neo4j.graphdb.factory.GraphDatabaseFactory.lambda$createDatabaseCreator$0(GraphDatabaseFactory.java:89)
at org.neo4j.graphdb.factory.GraphDatabaseFactory$$Lambda$1/1130478920.newDatabase(Unknown Source)
at org.neo4j.graphdb.factory.GraphDatabaseBuilder.newGraphDatabase(GraphDatabaseBuilder.java:183)
at org.neo4j.graphdb.factory.GraphDatabaseFactory.newEmbeddedDatabase(GraphDatabaseFactory.java:66)
at com.sampledb.createDatabase(sampledb.java:41)
at com.sampledb.main(sampledb.java:34)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.internal.StoreLockerLifecycleAdapter#1b15e2a9' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:443)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.initFacade(GraphDatabaseFacadeFactory.java:195)
... 8 more
Caused by: org.neo4j.kernel.StoreLockException: Store and its lock file has been locked by another process: /Users/arundhatiwahane/Documents/Neo4j/default.graphdb/store_lock. Please ensure no other process is using this database, and that the directory is writable (required even for read-only access)
at org.neo4j.kernel.internal.StoreLocker.storeLockException(StoreLocker.java:94)
at org.neo4j.kernel.internal.StoreLocker.checkLock(StoreLocker.java:80)
at org.neo4j.kernel.internal.StoreLockerLifecycleAdapter.start(StoreLockerLifecycleAdapter.java:40)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:433)
... 10 more
Kindly help me to figure all my errors.
Another process is locking, kill it
try
ps aux | grep neo4j
or
ps aux | grep java
read your exception's description
Caused by: org.neo4j.kernel.StoreLockException: Store and its lock file has been locked by another process: /Users/arundhatiwahane/Documents/Neo4j/default.graphdb/store_lock. Please ensure no other process is using this database, and that the directory is writable (required even for read-only access)
if killing the process did not solve you need to add write permission to directory
chmod -R 777 /Users/arundhatiwahane/Documents/Neo4j/