We Keep Coding

Why I am getting java.io.FileNotFoundException while running spark job?

I am trying to run a spark job using below gcloud command.
gcloud dataproc jobs submit spark \
--cluster=clusterName \
--class=clazzName \
--jars=gs://abc/def/ghi.jar \
--region=us-central1 \
--files=gs://abc/def/jkl.json \
--properties=spark.driver.extraJavaOptions="-Dconfig.file=application_dev.json",spark.executor.extraJavaOptions="-Dconfig.file=application_dev.json",spark.executor.memory=6G,spark.driver.memory=4G,spark.executor.cores=3,spark.executor.instances=4
I am getting below error
ERROR org.apache.spark.SparkContext: Error initializing SparkContext.
java.io.FileNotFoundException: File not found: gs://temp-bucket/f80f8cb3-0358-445e-8ec2-819e4282bfe4/spark-job-history
Full stack trace
Waiting for job output...
22/09/02 05:30:47 INFO com.polaris.ihub.commons.utils.keymaker.KeymakerApi: ====== Reading App Context ======
22/09/02 05:30:47 INFO com.polaris.ihub.commons.utils.keymaker.KeymakerApi: File to be read from -> gs://abc/def/app_context.txt
22/09/02 05:30:49 INFO org.apache.spark.SparkEnv: Registering MapOutputTracker
22/09/02 05:30:49 INFO org.apache.spark.SparkEnv: Registering BlockManagerMaster
22/09/02 05:30:49 INFO org.apache.spark.SparkEnv: Registering BlockManagerMasterHeartbeat
22/09/02 05:30:49 INFO org.apache.spark.SparkEnv: Registering OutputCommitCoordinator
22/09/02 05:30:49 INFO org.sparkproject.jetty.util.log: Logging initialized #8633ms to org.sparkproject.jetty.util.log.Slf4jLog
22/09/02 05:30:50 INFO org.sparkproject.jetty.server.Server: jetty-9.4.40.v20210413; built: 2021-04-13T20:42:42.668Z; git: someAlphaNumeric1; jvm 1.8.0_322-b06
22/09/02 05:30:50 INFO org.sparkproject.jetty.server.Server: Started #8803ms
22/09/02 05:30:50 INFO org.sparkproject.jetty.server.AbstractConnector: Started ServerConnector#2db33feb{HTTP/1.1, (http/1.1)}{0.0.0.0:38111}
22/09/02 05:30:50 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at clusterName-m/someIp:8032
22/09/02 05:30:51 INFO org.apache.hadoop.yarn.client.AHSProxy: Connecting to Application History server at clusterName-m/someIp:10200
22/09/02 05:30:51 INFO org.apache.hadoop.conf.Configuration: resource-types.xml not found
22/09/02 05:30:51 INFO org.apache.hadoop.yarn.util.resource.ResourceUtils: Unable to find 'resource-types.xml'.
22/09/02 05:30:55 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl: Submitted application application_appID
22/09/02 05:30:56 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at clusterName-m/someIp:8030
22/09/02 05:30:57 ERROR org.apache.spark.SparkContext: Error initializing SparkContext.
java.io.FileNotFoundException: File not found: gs://temp-bucket/f80f8cb3-0358-445e-8ec2-819e4282bfe4/spark-job-history
at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.getFileStatus(GoogleHadoopFileSystemBase.java:958)
at org.apache.spark.deploy.history.EventLogFileWriter.requireLogBaseDirAsDirectory(EventLogFileWriters.scala:77)
at org.apache.spark.deploy.history.SingleEventLogFileWriter.start(EventLogFileWriters.scala:221)
at org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:83)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:612)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2680)
at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:945)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:939)
at claszzName2.spark.adaptor.utils.SparkUtils.newSparkSession(SparkUtils.java:42)
at claszzName2.spark.adaptor.bqtopubsub.BqToPubSubAdaptor.main(BqToPubSubAdaptor.java:30)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1039)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1048)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
22/09/02 05:30:57 INFO org.sparkproject.jetty.server.AbstractConnector: Stopped Spark#2db33feb{HTTP/1.1, (http/1.1)}{0.0.0.0:0}
22/09/02 05:30:58 ERROR clazzName: Spark Batch Application failed : {}
java.io.FileNotFoundException: File not found: gs://temp-bucket/f80f8cb3-0358-445e-8ec2-819e4282bfe4/spark-job-history
at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.getFileStatus(GoogleHadoopFileSystemBase.java:958)
at org.apache.spark.deploy.history.EventLogFileWriter.requireLogBaseDirAsDirectory(EventLogFileWriters.scala:77)
at org.apache.spark.deploy.history.SingleEventLogFileWriter.start(EventLogFileWriters.scala:221)
at org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:83)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:612)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2680)
at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:945)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:939)
at claszzName2.spark.adaptor.utils.SparkUtils.newSparkSession(SparkUtils.java:42)
at claszzName2.spark.adaptor.bqtopubsub.BqToPubSubAdaptor.main(BqToPubSubAdaptor.java:30)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1039)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1048)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Exception in thread "main" java.lang.RuntimeException
at clazzName.main(BqToPubSubAdaptor.java:37)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1039)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1048)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
ERROR: (gcloud.dataproc.jobs.submit.spark) Job [d8c3e7e5e8004e5bba72b921d454bfeb] failed with error:
Google Cloud Dataproc Agent reports job failure. If logs are available, they can be found at:

It seems GCP DataProc/spark is looking for default history server for event logging, which may not be located at specific location.
You can override two properties to update the logs location for history server.
spark.eventLog.dir
spark.history.fs.logDirectory
Either you can use these properties during submit of spark job, just like mentioned in example or you can update your spark-defaults.conf file
with these 2 properties.
Like
gcloud dataproc jobs submit spark \
--cluster=clusterName \
--class=clazzName \
--jars=gs://abc/def/ghi.jar \
--region=us-central1 \
--files=gs://abc/def/jkl.json \
--properties=spark.driver.extraJavaOptions="-Dconfig.file=application_dev.json",spark.executor.extraJavaOptions="-Dconfig.file=application_dev.json",spark.executor.memory=6G,spark.driver.memory=4G,spark.executor.cores=3,spark.executor.instances=4,spark.history.fs.logDirectory=gs://<bucket-name>/<folder-name>,spark.eventLog.dir=gs://<bucket-name>/<folder-name>

Authentication error (code 18) with MongoDB and SpringBoot

I'm writing a spring boot web server which connects to a MongoDB (that runs in Docker). The connection seems to work but when I want to use operations on the DB I fail to connect due to an authentication error (code 18). I don't know why since when I access the DB via shell it works.
Application file
spring.data.mongodb.authentication-database=admin
spring.data.mongodb.username=rootuser
spring.data.mongodb.password=rootpass
spring.data.mongodb.database=databaseBA
spring.data.mongodb.port=27017
spring.data.mongodb.host=localhost
Docker Compose file (.yaml)
version: "3.8"
services:
mongodb:
image: mongo
container_name: mongodb
ports:
- 27017:27017
volumes:
- data:/data
environment:
- MONGO_INITDB_ROOT_USERNAME=rootuser
- MONGO_INITDB_ROOT_PASSWORD=rootpass
mongo-express:
image: mongo-express
container_name: mongo-express
restart: always
ports:
- 8081:8081
environment:
- ME_CONFIG_MONGODB_ADMINUSERNAME=rootuser
- ME_CONFIG_MONGODB_ADMINPASSWORD=rootpass
- ME_CONFIG_MONGODB_SERVER=mongodb
volumes:
data: {}
networks:
default:
name: mongodb_network
Main Class
#SpringBootApplication
public class ServerApplication {
public static void main(String[] args) {
SpringApplication.run(ServerApplication.class, args);
}
#Bean
CommandLineRunner runner(ParticipantRepository repository){
return args -> {
Participant participant = new Participant(
"token",
Gender.MALE,
StudyCourse.BIOPHYSICS
);
repository.insert(participant);
};
}
}
Error Message
2021-11-05 21:45:22.995 INFO 15700 --- [ main] o.s.s.web.DefaultSecurityFilterChain : Will secure any request with [org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter#b46e103, org.springframework.security.web.context.SecurityContextPersistenceFilter#6a9cd0f8, org.springframework.security.web.header.HeaderWriterFilter#6d8796c1, org.springframework.security.web.csrf.CsrfFilter#53c1179a, org.springframework.security.web.authentication.logout.LogoutFilter#6a1d6ef2, org.springframework.security.web.savedrequest.RequestCacheAwareFilter#7be2f29a, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter#77dba4cd, org.springframework.security.web.authentication.AnonymousAuthenticationFilter#2b0e9f30, org.springframework.security.web.session.SessionManagementFilter#3df1a1ac, org.springframework.security.web.access.ExceptionTranslationFilter#3382cf68, org.springframework.security.web.access.intercept.FilterSecurityInterceptor#6198e9b5]
2021-11-05 21:45:23.110 INFO 15700 --- [ main] org.mongodb.driver.cluster : Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms'}
2021-11-05 21:45:23.178 INFO 15700 --- [localhost:27017] org.mongodb.driver.connection : Opened connection [connectionId{localValue:2, serverValue:146}] to localhost:27017
2021-11-05 21:45:23.178 INFO 15700 --- [localhost:27017] org.mongodb.driver.connection : Opened connection [connectionId{localValue:1, serverValue:147}] to localhost:27017
2021-11-05 21:45:23.180 INFO 15700 --- [localhost:27017] org.mongodb.driver.cluster : Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=13, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=23309600}
2021-11-05 21:45:23.725 INFO 15700 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path ''
2021-11-05 21:45:23.735 INFO 15700 --- [ main] s.d.s.application.ServerApplication : Started ServerApplication in 3.595 seconds (JVM running for 4.214)
2021-11-05 21:45:23.838 INFO 15700 --- [ main] org.mongodb.driver.connection : Closed connection [connectionId{localValue:3}] to localhost:27017 because there was a socket exception raised by this connection.
2021-11-05 21:45:23.845 INFO 15700 --- [ main] ConditionEvaluationReportLoggingListener :
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2021-11-05 21:45:23.870 ERROR 15700 --- [ main] o.s.boot.SpringApplication : Application run failed
java.lang.IllegalStateException: Failed to execute CommandLineRunner
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:794) ~[spring-boot-2.5.6.jar:2.5.6]
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:775) ~[spring-boot-2.5.6.jar:2.5.6]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:345) ~[spring-boot-2.5.6.jar:2.5.6]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1343) ~[spring-boot-2.5.6.jar:2.5.6]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1332) ~[spring-boot-2.5.6.jar:2.5.6]
at stachm.dbisbavlasjuk.server.application.ServerApplication.main(ServerApplication.java:16) ~[classes/:na]
Caused by: org.springframework.data.mongodb.UncategorizedMongoDbException: Exception authenticating MongoCredential{mechanism=SCRAM-SHA-1, userName='rootuser', source='admin', password=<hidden>, mechanismProperties=<hidden>}; nested exception is com.mongodb.MongoSecurityException: Exception authenticating MongoCredential{mechanism=SCRAM-SHA-1, userName='rootuser', source='admin', password=<hidden>, mechanismProperties=<hidden>}
at org.springframework.data.mongodb.core.MongoExceptionTranslator.translateExceptionIfPossible(MongoExceptionTranslator.java:140) ~[spring-data-mongodb-3.2.6.jar:3.2.6]
at org.springframework.data.mongodb.core.MongoTemplate.potentiallyConvertRuntimeException(MongoTemplate.java:2899) ~[spring-data-mongodb-3.2.6.jar:3.2.6]
at org.springframework.data.mongodb.core.MongoTemplate.execute(MongoTemplate.java:555) ~[spring-data-mongodb-3.2.6.jar:3.2.6]
at org.springframework.data.mongodb.core.MongoTemplate.insertDocument(MongoTemplate.java:1458) ~[spring-data-mongodb-3.2.6.jar:3.2.6]
at org.springframework.data.mongodb.core.MongoTemplate.doInsert(MongoTemplate.java:1257) ~[spring-data-mongodb-3.2.6.jar:3.2.6]
at org.springframework.data.mongodb.core.MongoTemplate.insert(MongoTemplate.java:1172) ~[spring-data-mongodb-3.2.6.jar:3.2.6]
at org.springframework.data.mongodb.repository.support.SimpleMongoRepository.insert(SimpleMongoRepository.java:283) ~[spring-data-mongodb-3.2.6.jar:3.2.6]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
at org.springframework.data.repository.core.support.RepositoryMethodInvoker$RepositoryFragmentMethodInvoker.lambda$new$0(RepositoryMethodInvoker.java:289) ~[spring-data-commons-2.5.6.jar:2.5.6]
at org.springframework.data.repository.core.support.RepositoryMethodInvoker.doInvoke(RepositoryMethodInvoker.java:137) ~[spring-data-commons-2.5.6.jar:2.5.6]
at org.springframework.data.repository.core.support.RepositoryMethodInvoker.invoke(RepositoryMethodInvoker.java:121) ~[spring-data-commons-2.5.6.jar:2.5.6]
at org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments.invoke(RepositoryComposition.java:529) ~[spring-data-commons-2.5.6.jar:2.5.6]
at org.springframework.data.repository.core.support.RepositoryComposition.invoke(RepositoryComposition.java:285) ~[spring-data-commons-2.5.6.jar:2.5.6]
at org.springframework.data.repository.core.support.RepositoryFactorySupport$ImplementationMethodExecutionInterceptor.invoke(RepositoryFactorySupport.java:599) ~[spring-data-commons-2.5.6.jar:2.5.6]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.12.jar:5.3.12]
at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.doInvoke(QueryExecutorMethodInterceptor.java:163) ~[spring-data-commons-2.5.6.jar:2.5.6]
at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.invoke(QueryExecutorMethodInterceptor.java:138) ~[spring-data-commons-2.5.6.jar:2.5.6]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.12.jar:5.3.12]
at org.springframework.data.projection.DefaultMethodInvokingMethodInterceptor.invoke(DefaultMethodInvokingMethodInterceptor.java:80) ~[spring-data-commons-2.5.6.jar:2.5.6]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.12.jar:5.3.12]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) ~[spring-aop-5.3.12.jar:5.3.12]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.12.jar:5.3.12]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) ~[spring-aop-5.3.12.jar:5.3.12]
at com.sun.proxy.$Proxy71.insert(Unknown Source) ~[na:na]
at stachm.dbisbavlasjuk.server.application.ServerApplication.lambda$runner$0(ServerApplication.java:29) ~[classes/:na]
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:791) ~[spring-boot-2.5.6.jar:2.5.6]
... 5 common frames omitted
Caused by: com.mongodb.MongoSecurityException: Exception authenticating MongoCredential{mechanism=SCRAM-SHA-1, userName='rootuser', source='admin', password=<hidden>, mechanismProperties=<hidden>}
at com.mongodb.internal.connection.SaslAuthenticator.wrapException(SaslAuthenticator.java:271) ~[mongodb-driver-core-4.2.3.jar:na]
at com.mongodb.internal.connection.SaslAuthenticator.getNextSaslResponse(SaslAuthenticator.java:136) ~[mongodb-driver-core-4.2.3.jar:na]
at com.mongodb.internal.connection.SaslAuthenticator.access$100(SaslAuthenticator.java:47) ~[mongodb-driver-core-4.2.3.jar:na]
at com.mongodb.internal.connection.SaslAuthenticator$1.run(SaslAuthenticator.java:62) ~[mongodb-driver-core-4.2.3.jar:na]
at com.mongodb.internal.connection.SaslAuthenticator$1.run(SaslAuthenticator.java:56) ~[mongodb-driver-core-4.2.3.jar:na]
at com.mongodb.internal.connection.SaslAuthenticator.doAsSubject(SaslAuthenticator.java:278) ~[mongodb-driver-core-4.2.3.jar:na]
at com.mongodb.internal.connection.SaslAuthenticator.authenticate(SaslAuthenticator.java:56) ~[mongodb-driver-core-4.2.3.jar:na]
at com.mongodb.internal.connection.DefaultAuthenticator.authenticate(DefaultAuthenticator.java:53) ~[mongodb-driver-core-4.2.3.jar:na]
at com.mongodb.internal.connection.InternalStreamConnectionInitializer.authenticate(InternalStreamConnectionInitializer.java:168) ~[mongodb-driver-core-4.2.3.jar:na]
at com.mongodb.internal.connection.InternalStreamConnectionInitializer.initialize(InternalStreamConnectionInitializer.java:63) ~[mongodb-driver-core-4.2.3.jar:na]
at com.mongodb.internal.connection.InternalStreamConnection.open(InternalStreamConnection.java:144) ~[mongodb-driver-core-4.2.3.jar:na]
at com.mongodb.internal.connection.UsageTrackingInternalConnection.open(UsageTrackingInternalConnection.java:51) ~[mongodb-driver-core-4.2.3.jar:na]
at com.mongodb.internal.connection.DefaultConnectionPool$PooledConnection.open(DefaultConnectionPool.java:431) ~[mongodb-driver-core-4.2.3.jar:na]
at com.mongodb.internal.connection.DefaultConnectionPool.get(DefaultConnectionPool.java:115) ~[mongodb-driver-core-4.2.3.jar:na]
at com.mongodb.internal.connection.DefaultConnectionPool.get(DefaultConnectionPool.java:100) ~[mongodb-driver-core-4.2.3.jar:na]
at com.mongodb.internal.connection.DefaultServer.getConnection(DefaultServer.java:96) ~[mongodb-driver-core-4.2.3.jar:na]
at com.mongodb.internal.binding.ClusterBinding$ClusterBindingConnectionSource.getConnection(ClusterBinding.java:123) ~[mongodb-driver-core-4.2.3.jar:na]
at com.mongodb.client.internal.ClientSessionBinding$SessionBindingConnectionSource.getConnection(ClientSessionBinding.java:135) ~[mongodb-driver-sync-4.2.3.jar:na]
at com.mongodb.internal.operation.OperationHelper.withReleasableConnection(OperationHelper.java:621) ~[mongodb-driver-core-4.2.3.jar:na]
at com.mongodb.internal.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:185) ~[mongodb-driver-core-4.2.3.jar:na]
at com.mongodb.internal.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:76) ~[mongodb-driver-core-4.2.3.jar:na]
at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:187) ~[mongodb-driver-sync-4.2.3.jar:na]
at com.mongodb.client.internal.MongoCollectionImpl.executeSingleWriteRequest(MongoCollectionImpl.java:1009) ~[mongodb-driver-sync-4.2.3.jar:na]
at com.mongodb.client.internal.MongoCollectionImpl.executeInsertOne(MongoCollectionImpl.java:470) ~[mongodb-driver-sync-4.2.3.jar:na]
at com.mongodb.client.internal.MongoCollectionImpl.insertOne(MongoCollectionImpl.java:453) ~[mongodb-driver-sync-4.2.3.jar:na]
at com.mongodb.client.internal.MongoCollectionImpl.insertOne(MongoCollectionImpl.java:447) ~[mongodb-driver-sync-4.2.3.jar:na]
at org.springframework.data.mongodb.core.MongoTemplate.lambda$insertDocument$16(MongoTemplate.java:1464) ~[spring-data-mongodb-3.2.6.jar:3.2.6]
at org.springframework.data.mongodb.core.MongoTemplate.execute(MongoTemplate.java:553) ~[spring-data-mongodb-3.2.6.jar:3.2.6]
... 31 common frames omitted
Caused by: com.mongodb.MongoCommandException: Command failed with error 18 (AuthenticationFailed): 'Authentication failed.' on server localhost:27017. The full response is {"ok": 0.0, "errmsg": "Authentication failed.", "code": 18, "codeName": "AuthenticationFailed"}
at com.mongodb.internal.connection.ProtocolHelper.getCommandFailureException(ProtocolHelper.java:175) ~[mongodb-driver-core-4.2.3.jar:na]
at com.mongodb.internal.connection.InternalStreamConnection.receiveCommandMessageResponse(InternalStreamConnection.java:358) ~[mongodb-driver-core-4.2.3.jar:na]
at com.mongodb.internal.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:279) ~[mongodb-driver-core-4.2.3.jar:na]
at com.mongodb.internal.connection.CommandHelper.sendAndReceive(CommandHelper.java:83) ~[mongodb-driver-core-4.2.3.jar:na]
at com.mongodb.internal.connection.CommandHelper.executeCommand(CommandHelper.java:33) ~[mongodb-driver-core-4.2.3.jar:na]
at com.mongodb.internal.connection.SaslAuthenticator.sendSaslStart(SaslAuthenticator.java:227) ~[mongodb-driver-core-4.2.3.jar:na]
at com.mongodb.internal.connection.SaslAuthenticator.getNextSaslResponse(SaslAuthenticator.java:134) ~[mongodb-driver-core-4.2.3.jar:na]
... 57 common frames omitted
2021-11-05 21:45:24.344 INFO 15700 --- [ main] org.mongodb.driver.connection : Closed connection [connectionId{localValue:4}] to localhost:27017 because there was a socket exception raised by this connection.
Process finished with exit code 1
So as you can see I try to add a document into the database via the command line runner-bean to test if I can use my db correctly. You can also see in the error message that the source and username should be correct but why does the password not match then. The Spring-Boot version is 2.5.6, jdk 11, MongoDB 5.0.3. (If I use mongo admin -u rootuser -p 'rootpass' on the shell it works...) If it's important I also show my model for the database:
Participant
import lombok.Getter;
import lombok.Setter;
import lombok.ToString;
import org.springframework.data.annotation.Id;
import org.springframework.data.mongodb.core.mapping.Document;
#Getter
#Setter
#ToString
#Document("participant")
public class Participant {
#Id
private String subjectPseudonym;
private String tokenID;
private Gender gender;
private StudyCourse studyCourse;
public Participant(String tokenID, Gender gender, StudyCourse studyCourse) {
this.tokenID = tokenID;
this.gender = gender;
this.studyCourse = studyCourse;
}
}
Gender
public enum Gender {
MALE, FEMALE, OTHER;
}
Courses (some German)
public enum StudyCourse {
COGNITIVE_SYSTEMS,
COMMUNICATION_AND_INFORMATION_TECHNOLOGY,
ELEKTROTECHNIK,
INFORMATIK,
INFORMATIONSSYSTEMTECHNIK,
KUENSTLICHE_INTELLIGENZ,
LEHRAMT_GYMNASIUM,
MEDIENINFORMATIK,
PSYCHOLOGIE,
SENSORSYSTEMTECHNIK,
SOFTWARE_ENGINEERING,
AKTUARWISSENSCHAFTEN,
BUSINESS_ANALYTICS,
COMPUTATIONAL_SCIENCE_AND_ENGINEERING,
FINANCE,
MATHEMATICAL_DATA_SCIENCE,
MATHEMATIK,
MATHEMATISCHE_BIOMETRIE,
NACHHALTIGE_UNTERNEHMUNGSFUEHRUNG,
WIRTSCHAFTSMATHEMATIK,
WIRTSCHAFTSWISSENSCHAFTEN,
ADVANCED_ONCOLOGY,
HUMANMEDIZIN,
MOLECULAR_AND_TRANSLATIONAL_NEUROSCIENCES,
MOLEKULARE_MEDIZIN,
ZAHNMEDIZIN,
ADVANCED_MATERIALS,
BIOCHEMIE,
BIOLOGIE,
BIOPHYSICS,
CHEMICAL_ENGINEERING,
CHEMIE,
CHEMIEINGENIEURWESEN,
ENERGY_SCIENCE_AND_TECHNOLOGY,
INDUSTRIELLE_BIOTECHNOLOGIE,
PHARMAZEUTISCHE_BIOTECHNOLOGY,
PHYSIK,
WIRTSCHAFTSCHEMIE,
WIRTSCHAFTSPHYSIK,
ANDERES
}
Repository
import org.springframework.data.mongodb.repository.MongoRepository;
public interface ParticipantRepository extends MongoRepository<Participant, String> {
}

Flink cluster - Error when Submitting Job

I have a K8s cluster on EKS with 1 JM, 30 TMs, 3 Zookeepers and we are using the Flinkoperator: https://github.com/lyft/flinkk8soperator
We use zookeeper for High availability mode run.
We are unable to get the flink job i.e our application submitted.
The strange thing is this exact same configuration works in other regions but on this newly setup region we are just unable to submit the Flink Job.
The Flink UI also does not seem accessibile for the new region.
Some observations:
There are some errors in the zookeeper initially but it eventually runs fine and gives Processing... log for the rest.
Job manager is where error seems prominent but we are not sure why this problem is
Flink operator just tells us it could not submit the job.
TM logs don't have too much data.
Would anyone know how to proceed here? Or possibly have a hint as to what is going on? Could it be a resourcing issue as per: https://www.mail-archive.com/user#flink.apache.org/msg21146.html
But then we are using beefy nodes for the pods similar to our deployments to other regions.
Any insights or pointers here would be appreciated.
Here are the logs from the different pods: (Sorry if they are too long)
Flink operator:
{"json":{"app_name":"my-app","ns":"flink-cluster","phase":"SubmittingJob"},"level":"info","msg":"Handle
state skipped for application, lastSeenError UnknownMethod call failed
with status FAILED and message '': Could not find deployments for
service my-app","ts":"2021-09-28T15:35:54Z"}
{"json":{"app_name":"my-app","ns":"flink-cluster","phase":"SubmittingJob"},"level":"info","msg":"Handle
state skipped for application, lastSeenError UnknownMethod call failed
with status FAILED and message '': Could not find deployments for
service my-app","ts":"2021-09-28T15:36:24Z"}
Task manager (1 of them):
2021-09-28 16:13:02,940 ERROR
org.apache.flink.runtime.taskexecutor.TaskExecutor [] -
Registration at ResourceManager failed due to an error
java.util.concurrent.CompletionException:
org.apache.flink.runtime.rpc.exceptions.FencingTokenException: Fencing
token not set: Ignoring message
RemoteFencedMessage(8bb0ba2f8de9e3fa13b269deba364c70,
RemoteRpcInvocation(registerTaskExecutor(TaskExecutorRegistration,
Time))) sent to
akka.tcp://flink#100.117.59.33:35805/user/rpc/resourcemanager_0
because the fencing token is null. at
java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source)
~[?:?] at
java.util.concurrent.CompletableFuture.completeThrowable(Unknown
Source) ~[?:?] at
java.util.concurrent.CompletableFuture$UniAccept.tryFire(Unknown
Source) ~[?:?] at
java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
~[?:?] at
java.util.concurrent.CompletableFuture.completeExceptionally(Unknown
Source) ~[?:?] at
org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$0(AkkaInvocationHandler.java:235)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source)
~[?:?] at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown
Source) ~[?:?] at
java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
~[?:?] at
java.util.concurrent.CompletableFuture.completeExceptionally(Unknown
Source) ~[?:?] at
org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:1044)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.dispatch.OnComplete.internal(Future.scala:263)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.dispatch.OnComplete.internal(Future.scala:261)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.dispatch.japi$CallbackBridge.apply(Future.scala:191)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.dispatch.japi$CallbackBridge.apply(Future.scala:188)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:73)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:284)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:284)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.pattern.PromiseActorRef.$bang(AskSupport.scala:573)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.remote.DefaultMessageDispatcher.dispatch(Endpoint.scala:101)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.remote.EndpointReader$$anonfun$receive$2.applyOrElse(Endpoint.scala:999)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.actor.Actor.aroundReceive(Actor.scala:517)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.actor.Actor.aroundReceive$(Actor.scala:515)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.remote.EndpointActor.aroundReceive(Endpoint.scala:458)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.actor.ActorCell.invoke(ActorCell.scala:561)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.dispatch.Mailbox.run(Mailbox.scala:225)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.dispatch.Mailbox.exec(Mailbox.scala:235)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[flink-dist_2.12-1.12.1.jar:1.12.1] Caused by:
org.apache.flink.runtime.rpc.exceptions.FencingTokenException: Fencing
token not set: Ignoring message
RemoteFencedMessage(8bb0ba2f8de9e3fa13b269deba364c70,
RemoteRpcInvocation(registerTaskExecutor(TaskExecutorRegistration,
Time))) sent to
akka.tcp://flink#100.117.59.33:35805/user/rpc/resourcemanager_0
because the fencing token is null. at
org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:67)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:159)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.actor.Actor.aroundReceive(Actor.scala:517)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.actor.Actor.aroundReceive$(Actor.scala:515)
~[flink-dist_2.12-1.12.1.jar:1.12.1] at
akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
~[flink-dist_2.12-1.12.1.jar:1.12.1] ... 9 more 2021-09-28
16:13:02,947 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor
[] - Pausing and re-attempting registration in 10000 ms 2021-09-28
16:13:03,377 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor
[] - Terminating registration attempts towards ResourceManager
akka.tcp://flink#100.117.59.33:35805/user/rpc/resourcemanager_0.
2021-09-28 16:13:03,378 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor [] -
Connecting to ResourceManager
akka.tcp://flink#100.117.59.33:35805/user/rpc/resourcemanager_0(8ebb566d5f310905d0421f7fe9a74498).
2021-09-28 16:13:03,383 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor [] -
Resolved ResourceManager address, beginning registration 2021-09-28
16:13:03,390 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor
[] - Successful registration at resource manager
akka.tcp://flink#100.117.59.33:35805/user/rpc/resourcemanager_0 under
registration id 78d410e0d100d4712171571f087b6476.
Job manager:
2021-09-28 16:13:30,063 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager []
Registering TaskManager with ResourceID 120.117.67.21:34443-d3dd7f (akka.tcp://flink#120.117.67.21:34443/user/rpc/taskmanager_0) at
ResourceManager
2021-09-30 15:36:33,455 WARN akka.remote.Remoting [] - Association to [akka.tcp://flink#120.117.78.156:33767] with
unknown UID is irrecoverably failed. Address cannot be quarantined
without knowing the UID, gating instead for 50 ms.
2021-09-30 15:42:25,636 ERROR akka.remote.Remoting [] - Association to [akka.tcp://flink#120.117.79.241:40783] with UID
[-210531468] irrecoverably failed. Quarantining address.
java.util.concurrent.TimeoutException: Remote system has been silent for too long. (more than 48.0 hours)
at akka.remote.ReliableDeliverySupervisor$$anonfun$idle$1.applyOrElse(Endpoint.scala:387)
~[flink-dist_2.12-1.12.1.jar:1.12.1]
Zookeeper: Please check link as it is too long.
https://pastecode.io/s/as00i6pb

Maven build war need to deploy on Weblogic 12.2.x using jenkins

I am trying to deploy war file on weblogic using jenkins. For this I am using Weblogic deployer plugin in jenkins. I have configured everything in the jenkins but while deploying the war it always failing. The only thing I am figure out is that my actual war file name is 'xxx.3.0.0.war' but in weblogic it is taking 'xxx.3.0'.Please find image for jenkins weblogic deployer plugin configuration. Please help
Below are the weblogic deployment logs :
------------------------------------ ARTIFACT UNDEPLOYMENT ------------------------------------------------
weblogic.Deployer invoked with options: -debug -remote -verbose -noexit -name psm-web-3.0 -targets DevServer -adminurl t3://localhost:7001 -user weblogic -undeploy
[WebLogicDeploymentManagerImpl.<init>():119] : Constructing DeploymentManager for J2EE version V1_4 deployments
[WebLogicDeploymentManagerImpl.getNewConnection():162] : Connecting to admin server at localhost:7001, as user weblogic
[ServerConnectionImpl.getEnvironment():295] : setting environment
[ServerConnectionImpl.getEnvironment():298] : getting context using t3://localhost:7001
[ServerConnectionImpl.getMBeanServer():246] : Connecting to MBeanServer at service:jmx:t3://localhost:7001/jndi/weblogic.management.mbeanservers.domainruntime
[ServerConnectionImpl.getMBeanServer():246] : Connecting to MBeanServer at service:jmx:t3://localhost:7001/jndi/weblogic.management.mbeanservers.runtime
[DomainManager.resetDomain():36] : Getting new domain
[DomainManager.resetDomain():39] : Using pending domain: true
[MBeanCache.addNotificationListener():96] : Adding notification listener for weblogic.deploy.api.spi.deploy.mbeans.TargetCache#5fdcaa40
[MBeanCache.addNotificationListener():107] : Disabling mbean caching due to: java.io.InvalidClassException: weblogic.deploy.api.spi.deploy.mbeans.TargetCache; local class incompatible: stream classdesc serialVersionUID = -355364455225633633, local class serialVersionUID = -6063846258165761631
[MBeanCache.addNotificationListener():96] : Adding notification listener for weblogic.deploy.api.spi.deploy.mbeans.ModuleCache#36902638
[MBeanCache.addNotificationListener():103] : Added notification listener for weblogic.deploy.api.spi.deploy.mbeans.ModuleCache#36902638
[ServerConnectionImpl.initialize():178] : Connected to WLS domain: base_domain
[ServerConnectionImpl.init():168] : Initializing ServerConnection : weblogic.deploy.api.spi.deploy.internal.ServerConnectionImpl#f6efaab
[BasicOperation.dumpTmids():740] : Incoming tmids:
[BasicOperation.dumpTmids():742] : {Target=DevServer, WebLogicTargetType=server, Name=psm-web-3.0}, targeted=true
[BasicOperation.deriveAppName():143] : appname established as: psm-web-3.0
<Dec 5, 2019 5:05:52 PM IST> <Info> <J2EE Deployment SPI> <BEA-260121> <Initiating undeploy operation for application, psm-web-3.0 [archive: null], to DevServer .>
[BasicOperation.dumpTmids():740] : Incoming tmids:
[BasicOperation.dumpTmids():742] : {Target=DevServer, WebLogicTargetType=server, Name=psm-web-3.0}, targeted=true
[BasicOperation.loadGeneralOptions():655] : Delete Files:false
Timeout :3600000
Targets:
DevServer
ModuleTargets={}
SubModuleTargets={}
}
Files:
null
Deployment Plan: null
App root: null
App config: null
Deployment Options: {isRetireGracefully=true,isGracefulProductionToAdmin=false,isGracefulIgnoreSessions=false,rmiGracePeriod=-1,retireTimeoutSecs=-1,undeployAllVersions=false,archiveVersion=null,planVersion=null,isLibrary=false,libSpecVersion=null,libImplVersion=null,stageMode=null,clusterTimeout=3600000,altDD=null,altWlsDD=null,name=psm-web-3.0,securityModel=null,securityValidationEnabled=false,versionIdentifier=null,isTestMode=false,forceUndeployTimeout=0,defaultSubmoduleTargets=true,timeout=0,deploymentPrincipalName=null,useExpiredLock=falsespecifiedTargetsOnly=false}
[BasicOperation.execute():472] : Initiating undeploy operation for app, psm-web-3.0, on targets:
[BasicOperation.execute():474] : DevServer
weblogic.management.ManagementException: [Deployer:149001]No application named "psm-web-3.0" exists for operation "undeploy".
at weblogic.deploy.internal.adminserver.operations.OperationHelper.assertAppIsNonNull(OperationHelper.java:968)
at weblogic.deploy.internal.adminserver.operations.RemoveOperation.updateConfiguration(RemoveOperation.java:191)
at weblogic.deploy.internal.adminserver.operations.AbstractOperation.updateConfigurationAndInitializeDeployment(AbstractOperation.java:478)
at weblogic.deploy.internal.adminserver.operations.AbstractOperation.execute(AbstractOperation.java:301)
at weblogic.deploy.internal.adminserver.operations.RemoveOperation.execute(RemoveOperation.java:84)
at weblogic.management.deploy.internal.DeployerRuntimeImpl$2.run(DeployerRuntimeImpl.java:1034)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:326)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:137)
at weblogic.management.deploy.internal.DeployerRuntimeImpl.performDeployerActions(DeployerRuntimeImpl.java:1028)
at weblogic.management.deploy.internal.DeployerRuntimeImpl.undeploy(DeployerRuntimeImpl.java:513)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at weblogic.management.jmx.modelmbean.WLSModelMBean.invoke(WLSModelMBean.java:532)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$21.run(WLSMBeanServerInterceptorBase.java:589)
at java.security.AccessController.doPrivileged(Native Method)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:587)
at weblogic.management.mbeanservers.internal.JMXContextInterceptor.invoke(JMXContextInterceptor.java:249)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$21.run(WLSMBeanServerInterceptorBase.java:589)
at java.security.AccessController.doPrivileged(Native Method)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:587)
at weblogic.management.mbeanservers.internal.SecurityMBeanMgmtOpsInterceptor.invoke(SecurityMBeanMgmtOpsInterceptor.java:65)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$21.run(WLSMBeanServerInterceptorBase.java:589)
at java.security.AccessController.doPrivileged(Native Method)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:587)
at weblogic.management.mbeanservers.internal.SecurityInterceptor.invoke(SecurityInterceptor.java:439)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$21.run(WLSMBeanServerInterceptorBase.java:589)
at java.security.AccessController.doPrivileged(Native Method)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:587)
at weblogic.management.mbeanservers.internal.MBeanCICInterceptor.invoke(MBeanCICInterceptor.java:301)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$21.run(WLSMBeanServerInterceptorBase.java:589)
at java.security.AccessController.doPrivileged(Native Method)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:587)
at weblogic.management.mbeanservers.internal.PartitionJMXInterceptor.invoke(PartitionJMXInterceptor.java:435)
at weblogic.management.mbeanservers.domainruntime.internal.FederatedMBeanServerInterceptor.invoke(FederatedMBeanServerInterceptor.java:355)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$21.run(WLSMBeanServerInterceptorBase.java:589)
at java.security.AccessController.doPrivileged(Native Method)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:587)
at weblogic.management.mbeanservers.internal.CallerPartitionContextInterceptor.invoke(CallerPartitionContextInterceptor.java:217)
at weblogic.management.jmx.mbeanserver.WLSMBeanServer.invoke(WLSMBeanServer.java:327)
at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder$11$1.run(JMXConnectorSubjectForwarder.java:655)
at java.security.AccessController.doPrivileged(Native Method)
at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder$11.run(JMXConnectorSubjectForwarder.java:653)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:368)
at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder.invoke(JMXConnectorSubjectForwarder.java:646)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1471)
at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1312)
at java.security.AccessController.doPrivileged(Native Method)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1411)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:832)
at javax.management.remote.rmi.RMIConnectionImpl_WLSkel.invoke(Unknown Source)
at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:645)
at weblogic.rmi.internal.BasicServerRef$2.run(BasicServerRef.java:534)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:368)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:163)
at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:531)
at weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:138)
at weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:352)
at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:337)
at weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:57)
at weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41)
at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:644)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:415)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:355)
[ServerConnectionImpl.close():341] : Closing DM connection
[ServerConnectionImpl.close():361] : Unregistered all listeners
[ServerConnectionImpl.closeJMX():381] : Closed JMX connection
[ServerConnectionImpl.closeJMX():393] : Closed Runtime JMX connection
[ServerConnectionImpl.closeJMX():405] : Closed Edit JMX connection
Unexpected Error Initializing Deployer: weblogic.management.ManagementException: [Deployer:149001]No application named "psm-web-3.0" exists for operation "undeploy".
------------------------------------ ARTIFACT DEPLOYMENT ------------------------------------------------
weblogic.Deployer invoked with options: -debug -stage -remote -verbose -upload -name psm-web-3.0 -source C:\Users\AA00599156\.jenkins\customwar\psm-web-3.0.0.war -targets DevServer -adminurl t3://localhost:7001 -user weblogic -deploy
[WebLogicDeploymentManagerImpl.<init>():119] : Constructing DeploymentManager for J2EE version V1_4 deployments
[WebLogicDeploymentManagerImpl.getNewConnection():162] : Connecting to admin server at localhost:7001, as user weblogic
[ServerConnectionImpl.getEnvironment():295] : setting environment
[ServerConnectionImpl.getEnvironment():298] : getting context using t3://localhost:7001
[ServerConnectionImpl.getMBeanServer():246] : Connecting to MBeanServer at service:jmx:t3://localhost:7001/jndi/weblogic.management.mbeanservers.domainruntime
[ServerConnectionImpl.getMBeanServer():246] : Connecting to MBeanServer at service:jmx:t3://localhost:7001/jndi/weblogic.management.mbeanservers.runtime
[DomainManager.resetDomain():36] : Getting new domain
[DomainManager.resetDomain():39] : Using pending domain: false
[MBeanCache.addNotificationListener():96] : Adding notification listener for weblogic.deploy.api.spi.deploy.mbeans.TargetCache#5fdcaa40
[MBeanCache.addNotificationListener():107] : Disabling mbean caching due to: java.io.InvalidClassException: weblogic.deploy.api.spi.deploy.mbeans.TargetCache; local class incompatible: stream classdesc serialVersionUID = -355364455225633633, local class serialVersionUID = -6063846258165761631
[MBeanCache.addNotificationListener():96] : Adding notification listener for weblogic.deploy.api.spi.deploy.mbeans.ModuleCache#36902638
[MBeanCache.addNotificationListener():103] : Added notification listener for weblogic.deploy.api.spi.deploy.mbeans.ModuleCache#36902638
[ServerConnectionImpl.initialize():178] : Connected to WLS domain: base_domain
[ServerConnectionImpl.setRemote():489] : Running in remote mode
[ServerConnectionImpl.init():168] : Initializing ServerConnection : weblogic.deploy.api.spi.deploy.internal.ServerConnectionImpl#f6efaab
[BasicOperation.dumpTmids():740] : Incoming tmids:
[BasicOperation.dumpTmids():742] : {Target=DevServer, WebLogicTargetType=server, Name=psm-web-3.0}, targeted=true
[BasicOperation.deriveAppName():143] : appname established as: psm-web-3.0
<Dec 5, 2019 5:06:02 PM IST> <Info> <J2EE Deployment SPI> <BEA-260121> <Initiating deploy operation for application, psm-web-3.0 [archive: C:\Users\AA00599156\.jenkins\customwar\psm-web-3.0.0.war], to DevServer .>
[BasicOperation.dumpTmids():740] : Incoming tmids:
[BasicOperation.dumpTmids():742] : {Target=DevServer, WebLogicTargetType=server, Name=psm-web-3.0}, targeted=true
[BasicOperation.loadGeneralOptions():655] : Delete Files:false
Timeout :3600000
Targets:
DevServer
ModuleTargets={}
SubModuleTargets={}
}
Files:
null
Deployment Plan: null
App root: C:\Users\AA0059~1\AppData\Local\Temp\AA00599156\.\config\deployments\psm-web-3.0
App config: C:\Users\AA0059~1\AppData\Local\Temp\AA00599156\.\config\deployments\psm-web-3.0\plan
Deployment Options: {isRetireGracefully=true,isGracefulProductionToAdmin=false,isGracefulIgnoreSessions=false,rmiGracePeriod=-1,retireTimeoutSecs=-1,undeployAllVersions=false,archiveVersion=null,planVersion=null,isLibrary=false,libSpecVersion=null,libImplVersion=null,stageMode=stage,clusterTimeout=3600000,altDD=null,altWlsDD=null,name=psm-web-3.0,securityModel=null,securityValidationEnabled=false,versionIdentifier=null,isTestMode=false,forceUndeployTimeout=0,defaultSubmoduleTargets=true,timeout=0,deploymentPrincipalName=null,useExpiredLock=falsespecifiedTargetsOnly=false}
[ServerConnectionImpl.upload():862] : Uploaded app to D:\VHA\Software\Oracle2\user_projects\domains\base_domain\.\servers\DevServer\upload\psm-web-3.0
[BasicOperation.execute():472] : Initiating deploy operation for app, psm-web-3.0, on targets:
[BasicOperation.execute():474] : DevServer
Task 5 initiated: [Deployer:149026]deploy application psm-web-3.0 on DevServer.
dumping Exception stack
Task 5 failed: [Deployer:149026]deploy application psm-web-3.0 on DevServer.
Target state: deploy failed on Server DevServer
java.lang.ClassNotFoundException: com.mysema.query.dml.UpdateClause
at weblogic.utils.classloaders.GenericClassLoader.findLocalClass(GenericClassLoader.java:1029)
at weblogic.utils.classloaders.GenericClassLoader.findClass(GenericClassLoader.java:990)
at weblogic.utils.classloaders.ChangeAwareClassLoader.findClass(ChangeAwareClassLoader.java:104)
at weblogic.utils.classloaders.GenericClassLoader.doFindClass(GenericClassLoader.java:611)
at weblogic.utils.classloaders.GenericClassLoader.loadClass(GenericClassLoader.java:543)
at weblogic.utils.classloaders.GenericClassLoader.loadClass(GenericClassLoader.java:496)
at weblogic.utils.classloaders.ChangeAwareClassLoader.loadClass(ChangeAwareClassLoader.java:89)
at weblogic.utils.classloaders.ChangeAwareClassLoader.loadClass(ChangeAwareClassLoader.java:53)
at java.lang.Class.getDeclaredFields0(Native Method)
at java.lang.Class.privateGetDeclaredFields(Class.java:2583)
at java.lang.Class.getDeclaredFields(Class.java:1916)
at weblogic.j2ee.dd.xml.BaseJ2eeAnnotationProcessor.getFields(BaseJ2eeAnnotationProcessor.java:1047)
at weblogic.j2ee.dd.xml.BaseJ2eeAnnotationProcessor.getFields(BaseJ2eeAnnotationProcessor.java:1041)
at weblogic.j2ee.dd.xml.BaseJ2eeAnnotationProcessor.processJ2eeAnnotations(BaseJ2eeAnnotationProcessor.java:145)
at weblogic.j2ee.dd.xml.J2eeAnnotationProcessor.processJ2eeAnnotations(J2eeAnnotationProcessor.java:47)
at weblogic.j2ee.dd.xml.BaseJ2eeAnnotationProcessor.processJ2eeAnnotations(BaseJ2eeAnnotationProcessor.java:120)
at weblogic.j2ee.dd.xml.PojoAnnotationProcessorImpl.processJ2eeAnnotations(PojoAnnotationProcessorImpl.java:93)
at weblogic.application.internal.flow.PojoAnnotationProcessingFlow.processAnnotations(PojoAnnotationProcessingFlow.java:304)
at weblogic.application.internal.flow.PojoAnnotationProcessingFlow.processPOJOsInModuleScopes(PojoAnnotationProcessingFlow.java:229)
at weblogic.application.internal.flow.PojoAnnotationProcessingFlow.prepare(PojoAnnotationProcessingFlow.java:73)
at weblogic.application.internal.BaseDeployment$1.next(BaseDeployment.java:729)
at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:45)
at weblogic.application.internal.BaseDeployment.prepare(BaseDeployment.java:241)
at weblogic.application.internal.SingleModuleDeployment.prepare(SingleModuleDeployment.java:52)
at weblogic.application.internal.DeploymentStateChecker.prepare(DeploymentStateChecker.java:158)
at weblogic.deploy.internal.targetserver.AppContainerInvoker.prepare(AppContainerInvoker.java:65)
at weblogic.deploy.internal.targetserver.operations.ActivateOperation.createAndPrepareContainer(ActivateOperation.java:229)
at weblogic.deploy.internal.targetserver.operations.ActivateOperation.doPrepare(ActivateOperation.java:103)
at weblogic.deploy.internal.targetserver.operations.AbstractOperation.prepare(AbstractOperation.java:242)
at weblogic.deploy.internal.targetserver.DeploymentManager.handleDeploymentPrepare(DeploymentManager.java:800)
at weblogic.deploy.internal.targetserver.DeploymentManager.prepareDeploymentList(DeploymentManager.java:1352)
at weblogic.deploy.internal.targetserver.DeploymentManager.handlePrepare(DeploymentManager.java:270)
at weblogic.deploy.internal.targetserver.DeploymentServiceDispatcher.prepare(DeploymentServiceDispatcher.java:177)
at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.doPrepareCallback(DeploymentReceiverCallbackDeliverer.java:186)
at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.access$000(DeploymentReceiverCallbackDeliverer.java:14)
at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer$1.run(DeploymentReceiverCallbackDeliverer.java:47)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:670)
at weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:352)
at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:337)
at weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:57)
at weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41)
at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:644)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:415)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:355)
Target Assignments:
+ psm-web-3.0 DevServer
[ServerConnectionImpl.close():341] : Closing DM connection
[ServerConnectionImpl.close():361] : Unregistered all listeners
[ServerConnectionImpl.closeJMX():381] : Closed JMX connection
[ServerConnectionImpl.closeJMX():393] : Closed Runtime JMX connection
[ServerConnectionImpl.closeJMX():405] : Closed Edit JMX connection

Spark error on ALS trainImplicit: assertion failed: lapack.dppsv returned 1

I am getting below error when training ALS (implicit) using hadoop 2.6.1 and spark 1.5.2 on ubuntu 14
16/06/16 06:26:41 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS
16/06/16 06:26:41 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS
16/06/16 06:26:45 WARN TaskSetManager: Lost task 0.0 in stage 57.0 (TID 39, 172.17.0.10): java.lang.AssertionError: assertion failed: lapack.dppsv returned 1.
at scala.Predef$.assert(Predef.scala:179)
at org.apache.spark.ml.recommendation.ALS$CholeskySolver.solve(ALS.scala:393)
at org.apache.spark.ml.recommendation.ALS$$anonfun$org$apache$spark$ml$recommendation$ALS$$computeFactors$1.apply(ALS.scala:1170)
at org.apache.spark.ml.recommendation.ALS$$anonfun$org$apache$spark$ml$recommendation$ALS$$computeFactors$1.apply(ALS.scala:1131)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$mapValues$1$$anonfun$apply$41$$anonfun$apply$42.apply(PairRDDFunctions.scala:700)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$mapValues$1$$anonfun$apply$41$$anonfun$apply$42.apply(PairRDDFunctions.scala:700)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at org.apache.spark.storage.MemoryStore.unrollSafely(MemoryStore.scala:278)
at org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:171)
at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:78)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:262)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:88)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
------------------------------------------------
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1283)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1271)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1270)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1270)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:697)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1496)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1458)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1447)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:567)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1824)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1944)
at org.apache.spark.rdd.RDD$$anonfun$aggregate$1.apply(RDD.scala:1082)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:310)
at org.apache.spark.rdd.RDD.aggregate(RDD.scala:1075)
at org.apache.spark.ml.recommendation.ALS$.computeYtY(ALS.scala:1182)
at org.apache.spark.ml.recommendation.ALS$.org$apache$spark$ml$recommendation$ALS$$computeFactors(ALS.scala:1123)
at org.apache.spark.ml.recommendation.ALS$$anonfun$train$3.apply(ALS.scala:578)
at org.apache.spark.ml.recommendation.ALS$$anonfun$train$3.apply(ALS.scala:575)
at scala.collection.immutable.Range.foreach(Range.scala:141)
at org.apache.spark.ml.recommendation.ALS$.train(ALS.scala:575)
at org.apache.spark.mllib.recommendation.ALS.run(ALS.scala:239)
at org.apache.spark.mllib.recommendation.ALS$.trainImplicit(ALS.scala:417)
------------------------------
Can anyone let me know why BLAS is not getting loaded?