We Keep Coding

Flink cluster - Error when Submitting Job

I have a K8s cluster on EKS with 1 JM, 30 TMs, 3 Zookeepers and we are using the Flinkoperator: https://github.com/lyft/flinkk8soperator
We use zookeeper for High availability mode run.
We are unable to get the flink job i.e our application submitted.
The strange thing is this exact same configuration works in other regions but on this newly setup region we are just unable to submit the Flink Job.
The Flink UI also does not seem accessibile for the new region.
Some observations:
There are some errors in the zookeeper initially but it eventually runs fine and gives Processing... log for the rest.
Job manager is where error seems prominent but we are not sure why this problem is
Flink operator just tells us it could not submit the job.
TM logs don't have too much data.
Would anyone know how to proceed here? Or possibly have a hint as to what is going on? Could it be a resourcing issue as per: https://www.mail-archive.com/user#flink.apache.org/msg21146.html
But then we are using beefy nodes for the pods similar to our deployments to other regions.
Any insights or pointers here would be appreciated.
Here are the logs from the different pods: (Sorry if they are too long)
Flink operator:
{"json":{"app_name":"my-app","ns":"flink-cluster","phase":"SubmittingJob"},"level":"info","msg":"Handle
state skipped for application, lastSeenError UnknownMethod call failed
with status FAILED and message '': Could not find deployments for
service my-app","ts":"2021-09-28T15:35:54Z"}
{"json":{"app_name":"my-app","ns":"flink-cluster","phase":"SubmittingJob"},"level":"info","msg":"Handle
state skipped for application, lastSeenError UnknownMethod call failed
with status FAILED and message '': Could not find deployments for
service my-app","ts":"2021-09-28T15:36:24Z"}
Task manager (1 of them):
2021-09-28 16:13:02,940 ERROR
org.apache.flink.runtime.taskexecutor.TaskExecutor [] -
Registration at ResourceManager failed due to an error
java.util.concurrent.CompletionException:
org.apache.flink.runtime.rpc.exceptions.FencingTokenException: Fencing
token not set: Ignoring message
RemoteFencedMessage(8bb0ba2f8de9e3fa13b269deba364c70,
RemoteRpcInvocation(registerTaskExecutor(TaskExecutorRegistration,
Time))) sent to
akka.tcp://flink#100.117.59.33:35805/user/rpc/resourcemanager_0
because the fencing token is null. at
java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source)
~[?:?] at
java.util.concurrent.CompletableFuture.completeThrowable(Unknown
Source) ~[?:?] at
java.util.concurrent.CompletableFuture$UniAccept.tryFire(Unknown
Source) ~[?:?] at
java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
~[?:?] at
java.util.concurrent.CompletableFuture.completeExceptionally(Unknown
Source) ~[?:?] at
org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$0(AkkaInvocationHandler.java:235)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source)
~[?:?] at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown
Source) ~[?:?] at
java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
~[?:?] at
java.util.concurrent.CompletableFuture.completeExceptionally(Unknown
Source) ~[?:?] at
org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:1044)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.dispatch.OnComplete.internal(Future.scala:263)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.dispatch.OnComplete.internal(Future.scala:261)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.dispatch.japi$CallbackBridge.apply(Future.scala:191)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.dispatch.japi$CallbackBridge.apply(Future.scala:188)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:73)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:284)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:284)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.pattern.PromiseActorRef.$bang(AskSupport.scala:573)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.remote.DefaultMessageDispatcher.dispatch(Endpoint.scala:101)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.remote.EndpointReader$$anonfun$receive$2.applyOrElse(Endpoint.scala:999)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.actor.Actor.aroundReceive(Actor.scala:517)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.actor.Actor.aroundReceive$(Actor.scala:515)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.remote.EndpointActor.aroundReceive(Endpoint.scala:458)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.actor.ActorCell.invoke(ActorCell.scala:561)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.dispatch.Mailbox.run(Mailbox.scala:225)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.dispatch.Mailbox.exec(Mailbox.scala:235)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[flink-dist_2.12-1.12.1.jar:1.12.1] Caused by:
org.apache.flink.runtime.rpc.exceptions.FencingTokenException: Fencing
token not set: Ignoring message
RemoteFencedMessage(8bb0ba2f8de9e3fa13b269deba364c70,
RemoteRpcInvocation(registerTaskExecutor(TaskExecutorRegistration,
Time))) sent to
akka.tcp://flink#100.117.59.33:35805/user/rpc/resourcemanager_0
because the fencing token is null. at
org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:67)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:159)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.actor.Actor.aroundReceive(Actor.scala:517)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.actor.Actor.aroundReceive$(Actor.scala:515)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
~[flink-dist_2.12-1.12.1.jar:1.12.1] ... 9 more 2021-09-28
16:13:02,947 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor
[] - Pausing and re-attempting registration in 10000 ms 2021-09-28
16:13:03,377 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor
[] - Terminating registration attempts towards ResourceManager
akka.tcp://flink#100.117.59.33:35805/user/rpc/resourcemanager_0.
2021-09-28 16:13:03,378 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor [] -
Connecting to ResourceManager
akka.tcp://flink#100.117.59.33:35805/user/rpc/resourcemanager_0(8ebb566d5f310905d0421f7fe9a74498).
2021-09-28 16:13:03,383 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor [] -
Resolved ResourceManager address, beginning registration 2021-09-28
16:13:03,390 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor
[] - Successful registration at resource manager
akka.tcp://flink#100.117.59.33:35805/user/rpc/resourcemanager_0 under
registration id 78d410e0d100d4712171571f087b6476.
Job manager:
2021-09-28 16:13:30,063 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager []
Registering TaskManager with ResourceID 120.117.67.21:34443-d3dd7f (akka.tcp://flink#120.117.67.21:34443/user/rpc/taskmanager_0) at
ResourceManager
2021-09-30 15:36:33,455 WARN akka.remote.Remoting [] - Association to [akka.tcp://flink#120.117.78.156:33767] with
unknown UID is irrecoverably failed. Address cannot be quarantined
without knowing the UID, gating instead for 50 ms.
2021-09-30 15:42:25,636 ERROR akka.remote.Remoting [] - Association to [akka.tcp://flink#120.117.79.241:40783] with UID
[-210531468] irrecoverably failed. Quarantining address.
java.util.concurrent.TimeoutException: Remote system has been silent for too long. (more than 48.0 hours)
at akka.remote.ReliableDeliverySupervisor$$anonfun$idle$1.applyOrElse(Endpoint.scala:387)
~[flink-dist_2.12-1.12.1.jar:1.12.1]
Zookeeper: Please check link as it is too long.
https://pastecode.io/s/as00i6pb

Why is my Springboot application not starting up?

I bootstrapped the springboot application with JHipster but I have not been able to get it start up . It says build success but won't start up on a server . There is an embedded tomcat server added to the POM.XML , what could be wrong ?
Caused by: java.lang.IllegalArgumentException: Unable to resolve Configuration with the provided Issuer of "http://localhost:9080/auth/realms/jhipster"
at org.springframework.security.oauth2.client.registration.ClientRegistrations.getBuilder(ClientRegistrations.java:221)
at org.springframework.security.oauth2.client.registration.ClientRegistrations.fromIssuerLocation(ClientRegistrations.java:145)
at org.springframework.boot.autoconfigure.security.oauth2.client.OAuth2ClientPropertiesRegistrationAdapter.getBuilderFromIssuerIfPossible(OAuth2ClientPropertiesRegistrationAdapter.java:83)
at org.springframework.boot.autoconfigure.security.oauth2.client.OAuth2ClientPropertiesRegistrationAdapter.getClientRegistration(OAuth2ClientPropertiesRegistrationAdapter.java:59)
at org.springframework.boot.autoconfigure.security.oauth2.client.OAuth2ClientPropertiesRegistrationAdapter.lambda$getClientRegistrations$0(OAuth2ClientPropertiesRegistrationAdapter.java:53)
at java.base/java.util.HashMap.forEach(HashMap.java:1425)
at org.springframework.boot.autoconfigure.security.oauth2.client.OAuth2ClientPropertiesRegistrationAdapter.getClientRegistrations(OAuth2ClientPropertiesRegistrationAdapter.java:52)
at org.springframework.boot.autoconfigure.security.oauth2.client.servlet.OAuth2ClientRegistrationRepositoryConfiguration.clientRegistrationRepository(OAuth2ClientRegistrationRepositoryConfiguration.java:49)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
... 118 common frames omitted
Caused by: org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://localhost:9080/auth/realms/jhipster/.well-known/openid-configuration": Connection refused: connect; nested exception is java.net.ConnectException: Connection refused: connect
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:785)
at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:670)
at org.springframework.security.oauth2.client.registration.ClientRegistrations.lambda$oidc$0(ClientRegistrations.java:156)
at org.springframework.security.oauth2.client.registration.ClientRegistrations.getBuilder(ClientRegistrations.java:209)
... 130 common frames omitted
Caused by: java.net.ConnectException: Connection refused: connect
at java.base/sun.nio.ch.Net.connect0(Native Method)
at java.base/sun.nio.ch.Net.connect(Net.java:574)
at java.base/sun.nio.ch.Net.connect(Net.java:563)
at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:588)
at java.base/java.net.Socket.connect(Socket.java:648)
at java.base/java.net.Socket.connect(Socket.java:597)
at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:182)
at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:474)
at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:569)
at java.base/sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:341)
at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:362)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1261)
at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1194)
at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1082)
at java.base/sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1016)
at org.springframework.http.client.SimpleBufferingClientHttpRequest.executeInternal(SimpleBufferingClientHttpRequest.java:76)
at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48)
at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:66)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:776)
... 133 common frames omitted

Ensure that keycloak is started before running the springboot app because jhispter provides ready to use keycloak config. The app tries to reach keycloak at localhost:9080.
If you have docker and docker-compose installed you can use the provided docker-compose configuration and start keycloak via docker-compose -f src/main/docker/keycloak.yml up
If you want to start it manually you need to make sure to setup the jhipster realm in your keycloak instance.

You have to check 9080 port
Is it use by another process
You can do it by this command : netstat -aon
Then kill process by processor: taskkill -f /pid process_id

Example.of() mehtod breaks out by NoClassDefFoundError on Quarkus project

I'm new on Quarkus to build my project.
It shows NoClassDefFoundError while I using org.springframework.data.domain.Example in quarkus-spring-data-jpa and the error message below:
ERROR [io.qua.ver.htt.run.QuarkusErrorHandler] (executor-thread-7) HTTP Request to /ledger/overview-list-15-0 failed, error id: a1af07be-cbf2-4e41-8371-0bff2f671b93-4: org.jboss.resteasy.spi.UnhandledException: java.lang.NoClassDefFoundError: org/springframework/data/domain/TypedExample
at org.jboss.resteasy.core.ExceptionHandler.handleApplicationException(ExceptionHandler.java:106)
at org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:372)
at org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:218)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:519)
at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:261)
at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:161)
at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:364)
at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:164)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:247)
at io.quarkus.resteasy.runtime.standalone.RequestDispatcher.service(RequestDispatcher.java:73)
at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler.dispatch(VertxRequestHandler.java:138)
at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler.access$000(VertxRequestHandler.java:41)
at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler$1.run(VertxRequestHandler.java:93)
at io.quarkus.runtime.CleanableExecutor$CleaningRunnable.run(CleanableExecutor.java:231)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2415)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1452)
at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
at java.base/java.lang.Thread.run(Thread.java:834)
at org.jboss.threads.JBossThread.run(JBossThread.java:501)
Caused by: java.lang.NoClassDefFoundError: org/springframework/data/domain/TypedExample
at org.springframework.data.domain.Example.of(Example.java:39)
at chain.B.ledger.LedgerResource.listOverview(LedgerResource.java:59)
at chain.B.ledger.LedgerResource_ClientProxy.listOverview(LedgerResource_ClientProxy.zig:169)
at jdk.internal.reflect.GeneratedMethodAccessor162.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:170)
at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:130)
at org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:643)
at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:507)
at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$2(ResourceMethodInvoker.java:457)
at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:364)
at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:459)
at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:419)
at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:393)
at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:68)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:492)
... 18 more
Finally found that necessary classes in Example.class missing!!
How can I fix this?

The Query By Example of Spring Data JPA is currently not supported in the quarkus-spring-data-jpa extension.
See https://quarkus.io/guides/spring-data-jpa#what-is-currently-unsupported

java.lang.IllegalStateException: This error handler cannot process 'org.apache.kafka.common.errors.SslAuthenticationException's;

I have a spring boot application where I am trying to connect to the Kafka cluster over SSL.
Error Logs:
[ERROR] 2020-12-21 17:54:06,570 org.springframework.kafka.KafkaListenerEndpointContainer#0-3-C-1 org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer - {} - Consumer exception
java.lang.IllegalStateException: This error handler cannot process 'org.apache.kafka.common.errors.SslAuthenticationException's; no record information is available
at org.springframework.kafka.listener.SeekUtils.seekOrRecover(SeekUtils.java:151) ~[spring-kafka-2.5.5.RELEASE.jar:2.5.5.RELEASE]
at org.springframework.kafka.listener.SeekToCurrentErrorHandler.handle(SeekToCurrentErrorHandler.java:103) ~[spring-kafka-2.5.5.RELEASE.jar:2.5.5.RELEASE]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.handleConsumerException(KafkaMessageListenerContainer.java:1265) ~[spring-kafka-2.5.5.RELEASE.jar:2.5.5.RELEASE]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1022) ~[spring-kafka-2.5.5.RELEASE.jar:2.5.5.RELEASE]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLProtocolException: Unexpected handshake message: server_hello
at sun.security.ssl.Alert.createSSLException(Alert.java:129) ~[?:?]
at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:307) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:263) ~[?:?]
Caused by: org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Please find the below Kafka config Properties:
propsMap.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, kafkaserver)
propsMap.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, true)
propsMap.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, '5000')
propsMap.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, '300000')
propsMap.put(ConsumerConfig.REQUEST_TIMEOUT_MS_CONFIG, '300001')
propsMap.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer")
propsMap.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer")
propsMap.put(ConsumerConfig.GROUP_ID_CONFIG, KAFKA_CONSUMER)
propsMap.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "latest")
propsMap.put(SslConfigs.SSL_PROTOCOL_CONFIG,DTOConstants.SSL)
propsMap.put('security.protocol', DTOConstants.SSL)
propsMap.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, "30000")
propsMap.put(SslConfigs.SSL_TRUSTSTORE_LOCATION_CONFIG, this.trustStoreLocation)
propsMap.put(SslConfigs.SSL_TRUSTSTORE_PASSWORD_CONFIG, this.trustStorePassword)
propsMap.put(SslConfigs.SSL_KEYSTORE_LOCATION_CONFIG, this.keyStoreLocation)
propsMap.put(SslConfigs.SSL_KEYSTORE_PASSWORD_CONFIG, this.keyStorePassword)
propsMap.put(SslConfigs.SSL_KEY_PASSWORD_CONFIG, this.keyPassword)
propsMap.put(SslConfigs.SSL_ENDPOINT_IDENTIFICATION_ALGORITHM_CONFIG, null)

Spring cloud gateway AbstractErrorWebExceptionHandler

I've created global route for all incoming requests and face with AbstractErrorWebExceptionHandler.
My application.yml
spring:
cloud:
gateway:
routes:
- id: global_route
uri: http://localhost:8082
predicates:
- Path=/**
filters:
- RewritePath=/service(?<segment>/?.*), $\{segment}
- id: authorization_route
uri: http://localhost:8082
predicates:
- Path=/key/login
filters:
- JWTFilter=RSA512,HS512
Logs when I run the application:
2020-10-28 09:31:58.606 ERROR 9184 --- [ctor-http-nio-1] a.w.r.e.AbstractErrorWebExceptionHandler : [6d5d3da0] 500 Server Error for HTTP GET "/keyfd"
io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: no further information: localhost/127.0.0.1:8082
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
|_ checkpoint ⇢ org.springframework.cloud.gateway.filter.WeightCalculatorWebFilter [DefaultWebFilterChain]
|_ checkpoint ⇢ HTTP GET "/keyfd" [ExceptionHandlingWebHandler]
Stack trace:
Caused by: java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_221]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[na:1.8.0_221]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330) ~[netty-transport-4.1.52.Final.jar:4.1.52.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) ~[netty-transport-4.1.52.Final.jar:4.1.52.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702) ~[netty-transport-4.1.52.Final.jar:4.1.52.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) ~[netty-transport-4.1.52.Final.jar:4.1.52.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) ~[netty-transport-4.1.52.Final.jar:4.1.52.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[netty-transport-4.1.52.Final.jar:4.1.52.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[netty-common-4.1.52.Final.jar:4.1.52.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.52.Final.jar:4.1.52.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.52.Final.jar:4.1.52.Final]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_221]
Thank you for help.

The error cause is that I specified uri for spring cloud gateway the same as uri I'm running this application. So application started on localhost:8082 and the uri for route was localhost:8082.