We Keep Coding

In dockerized environment spark worker not able to resolve application host

We are running Spark as Docker microservice. My spark application able to submit the tasks to worker node, but worker node is not able to connect to application, throwing UnknowHostException. Basically worker node trying to communicate with application with containerId(658e5d214a60) which is not getting resolved to container IP.
It is working when I run these services with docker-compose yaml files in local Linux machine but failing in AWS EC2 container.
Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1713)
at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:64)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:188)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:293)
at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
Caused by: org.apache.spark.SparkException: Exception thrown in awaitResult:
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:205)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:101)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:201)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$2.run(SparkHadoopUtil.scala:65)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$2.run(SparkHadoopUtil.scala:64)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
... 4 more
Caused by: java.io.IOException: Failed to connect to 658e5d214a60:36335
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:245)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:187)
at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:198)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:194)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:190)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.UnknownHostException: 658e5d214a60
at java.net.InetAddress.getAllByName0(InetAddress.java:1259)
at java.net.InetAddress.getAllByName(InetAddress.java:1171)
at java.net.InetAddress.getAllByName(InetAddress.java:1105)
at java.net.InetAddress.getByName(InetAddress.java:1055)
at io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:146)
at io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:143)
at java.security.AccessController.doPrivileged(Native Method)
at io.netty.util.internal.SocketUtils.addressByName(SocketUtils.java:143)
at io.netty.resolver.DefaultNameResolver.doResolve(DefaultNameResolver.java:43)
at io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:63)
at io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:55)
at io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:57)
at io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:32)
at io.netty.resolver.AbstractAddressResolver.resolve(AbstractAddressResolver.java:108)
at io.netty.bootstrap.Bootstrap.doResolveAndConnect0(Bootstrap.java:208)
at io.netty.bootstrap.Bootstrap.access$000(Bootstrap.java:49)
at io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:188)
at io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:174)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetSuccess(AbstractChannel.java:978)
at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:512)
at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:423)
at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:482)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
... 1 more
Spark version 2.3.0
Docker version 1.12.6
Since the container port is dynamic we cant map it to host then use.

ClassNotFoundException when trying to add configuration to a CompositeConfiguration using commons-configuration2

I'm using org.apache.commons:commons-configuration2 to load and merge configuration from different sources:
val root = CompositeConfiguration()
root.addConfiguration(SystemConfiguration())
val configurations = Configurations()
root.addConfiguration(configurations.properties("defaults.properties"))
My problem is that whenever I run this code I get an exception:
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/commons/beanutils/BeanIntrospector at
java.lang.Class.forName0(Native Method) at
java.lang.Class.forName(Class.java:264) at
com.sun.proxy.$Proxy0.(Unknown Source) at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.reflect.Proxy.newProxyInstance(Proxy.java:739) at
org.apache.commons.configuration2.builder.fluent.Parameters.createParametersProxy(Parameters.java:294) at
org.apache.commons.configuration2.builder.fluent.Parameters.fileBased(Parameters.java:185)
at
org.apache.commons.configuration2.builder.fluent.Configurations.fileParams(Configurations.java:602)
at
org.apache.commons.configuration2.builder.fluent.Configurations.fileParams(Configurations.java:638)
at
org.apache.commons.configuration2.builder.fluent.Configurations.fileBasedBuilder(Configurations.java:164)
at
org.apache.commons.configuration2.builder.fluent.Configurations.propertiesBuilder(Configurations.java:264)
at
org.apache.commons.configuration2.builder.fluent.Configurations.properties(Configurations.java:318)
at io.sspinc.datahub.sparkapp.MainKt.main(main.kt:18) Caused by:
java.lang.ClassNotFoundException:
org.apache.commons.beanutils.BeanIntrospector at
java.net.URLClassLoader.findClass(URLClassLoader.java:381) at
java.lang.ClassLoader.loadClass(ClassLoader.java:424) at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338) at
java.lang.ClassLoader.loadClass(ClassLoader.java:357)
It says that org.apache.commons.beanutils.BeanIntrospector is not found. If I add commons-beanutils:commons-beanutils:1.9.3 to the classpath the problem is solved but the main problem is that I'm using Apache Spark 2.3.0 in a managed environment (Databricks) and it comes with commons-beanutils:1.7.0 so I can't do anything about the beanutils version.
How can I solve this problem?

You may want to try using commons-configuration:commons-configuration:1.10 if you only want the composite configuration feature form commons-configuration. It works like this:
val root = CompositeConfiguration()
root.addConfiguration(SystemConfiguration())
root.addConfiguration(PropertiesConfiguration("defaults.properties"))
Configurations 2 won't work with 1.7.0 bean utils according to their documentation.

Intellij can't run Wildfly server

I'm trying to get Intellij IDEA to work with a Wildfly 10 server. I installed Intellij following this and Wildfly thanks to this script.
The startup script used by Intellij is /opt/wildfly-10.0.0.Final/bin/standalone.sh, which I tried to run manually, and I got the following error :
=========================================================================
JBoss Bootstrap Environment
JBOSS_HOME: /opt/wildfly
JAVA: /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java
JAVA_OPTS: -server -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
=========================================================================
java.lang.IllegalArgumentException: Failed to instantiate class "org.jboss.logmanager.handlers.PeriodicRotatingFileHandler" for handler "FILE"
at org.jboss.logmanager.config.AbstractPropertyConfiguration$ConstructAction.validate(AbstractPropertyConfiguration.java:116)
at org.jboss.logmanager.config.LogContextConfigurationImpl.doPrepare(LogContextConfigurationImpl.java:335)
at org.jboss.logmanager.config.LogContextConfigurationImpl.prepare(LogContextConfigurationImpl.java:288)
at org.jboss.logmanager.config.LogContextConfigurationImpl.commit(LogContextConfigurationImpl.java:297)
at org.jboss.logmanager.PropertyConfigurator.configure(PropertyConfigurator.java:546)
at org.jboss.logmanager.PropertyConfigurator.configure(PropertyConfigurator.java:97)
at org.jboss.logmanager.LogManager.readConfiguration(LogManager.java:514)
at org.jboss.logmanager.LogManager.readConfiguration(LogManager.java:476)
at java.util.logging.LogManager$3.run(LogManager.java:399)
at java.util.logging.LogManager$3.run(LogManager.java:396)
at java.security.AccessController.doPrivileged(Native Method)
at java.util.logging.LogManager.readPrimordialConfiguration(LogManager.java:396)
at java.util.logging.LogManager.access$800(LogManager.java:145)
at java.util.logging.LogManager$2.run(LogManager.java:345)
at java.security.AccessController.doPrivileged(Native Method)
at java.util.logging.LogManager.ensureLogManagerInitialized(LogManager.java:338)
at java.util.logging.LogManager.getLogManager(LogManager.java:378)
at org.jboss.modules.Main.main(Main.java:482)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.jboss.logmanager.config.AbstractPropertyConfiguration$ConstructAction.validate(AbstractPropertyConfiguration.java:114)
... 17 more
Caused by: java.io.FileNotFoundException: /opt/wildfly/standalone/log/server.log (Permission non accordée)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at org.jboss.logmanager.handlers.FileHandler.setFile(FileHandler.java:151)
at org.jboss.logmanager.handlers.PeriodicRotatingFileHandler.setFile(PeriodicRotatingFileHandler.java:102)
at org.jboss.logmanager.handlers.FileHandler.setFileName(FileHandler.java:189)
at org.jboss.logmanager.handlers.FileHandler.<init>(FileHandler.java:119)
at org.jboss.logmanager.handlers.PeriodicRotatingFileHandler.<init>(PeriodicRotatingFileHandler.java:70)
... 22 more
java.util.concurrent.ExecutionException: Operation failed
at org.jboss.threads.AsyncFutureTask.operationFailed(AsyncFutureTask.java:74)
at org.jboss.threads.AsyncFutureTask.get(AsyncFutureTask.java:268)
at org.jboss.as.server.Main.main(Main.java:103)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.jboss.modules.Module.run(Module.java:329)
at org.jboss.modules.Main.main(Main.java:507)
Caused by: org.jboss.msc.service.StartException in service jboss.as: Failed to start service
at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1904)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: WFLYDR0006: Directory /opt/wildfly/standalone/data/content is not writable
at org.jboss.as.repository.ContentRepository$Factory$ContentRepositoryImpl.<init>(ContentRepository.java:188)
at org.jboss.as.repository.ContentRepository$Factory.addService(ContentRepository.java:154)
at org.jboss.as.server.ApplicationServerService.start(ApplicationServerService.java:146)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881)
... 3 more
This is exactly the same I get as output in Intellij.
I found thanks to this topic that it may be a matter of user permissions.
However, when I try to add a user to wildfly with the add-user.sh script, I get this error : ./add-user.sh: 1: eval: /usr/lib/jvm/jdk1.8.0_60/bin/java: not found
It is looking for a wrong JDK path. I tried to change it following different solutions but none of them work.
My JAVA_HOME is set to /usr/lib/jvm/java-8-oracle .
Is any of you having an idea of what to do ? Thank you in advance :)

Forget the add-user.sh script. This is for adding users to wildfly. Your issue is with your Linux users.
The directory has to be writable by whatever user wildfly is running as.
If you are running it as a user named wildfly then you have to change the ownership of those directories to that user. According to your question it looks like you are running as some other user that does not have permissions to those directories.
If you want a quick easy fix and you aren't worried about other users on the system you could just change permissions like:
sudo chmod -R 766 /opt/wildfly/standalone/
This will give the owner all permissions, and other users read/write permissions to those directories.
This is not best practice. Best practice is to give that directory ownership to user 'wildfly' with permissions of 600. Then you should run wildfly as the 'wildfly' user on linux. Any startup script you can find will likely do this for you.

Play Framework - JBoss Channel Exception

I'm starting with Play Framework and when I'm sending the following commands, it throws a exception.
# play
In Play Console:
[play-java] $ run
It throws:
org.jboss.netty.channel.ChannelException: Failed to bind to: /0.0.0.0:9000
at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272)
at play.core.server.NettyServer$$anonfun$10.apply(NettyServer.scala:171)
at play.core.server.NettyServer$$anonfun$10.apply(NettyServer.scala:168)
at scala.Option.map(Option.scala:145)
at play.core.server.NettyServer.<init>(NettyServer.scala:168)
at play.core.server.NettyServer$$anonfun$mainDev$1.apply(NettyServer.scala:347)
at play.core.server.NettyServer$$anonfun$mainDev$1.apply(NettyServer.scala:345)
at play.utils.Threads$.withContextClassLoader(Threads.scala:18)
at play.core.server.NettyServer$.mainDev(NettyServer.scala:344)
at play.core.server.NettyServer$.mainDevHttpMode(NettyServer.scala:340)
at play.core.server.NettyServer.mainDevHttpMode(NettyServer.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at play.PlayRun$$anonfun$playRunTask$1$$anonfun$apply$1.apply(PlayRun.scala:233)
at play.PlayRun$$anonfun$playRunTask$1$$anonfun$apply$1.apply(PlayRun.scala:91)
at scala.Function7$$anonfun$tupled$1.apply(Function7.scala:35)
at scala.Function7$$anonfun$tupled$1.apply(Function7.scala:34)
at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:463)
at sun.nio.ch.Net.bind(Net.java:455)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.jboss.netty.channel.socket.nio.NioServerBoss$RegisterTask.run(NioServerBoss.java:193)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:366)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:290)
at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
I'm trying using my local user, root user, changing ports (ie: run 8080), start instead of run, and it always throw the same exception.
By the way, port 9000 is not in use by another application.
Anyone has any clue?
I'm using Ubuntu 12.04 LTS, and Play Framework 2.2.6.
Thanks.

The error is that your port is being used by another process.
In a UNIX machine the command lsof -i:9000 will give the processes that are using the port.
As you have noted correctly processes with root permissions can be hidden if you use the terminal not as root

Knox error : java.lang.reflect.InvocationTargetException

I am getting following error while installing and setup of Knox on Hortonworks Hadoop 2.1.5.
I have already installed java jdk-7u65-linux-x64.tar.gz
SGSCAI0081:~ # su -l knox -c '/usr/lib/knox/bin/gateway.sh setup'
***************************************************************************************************
You have indicated that you would like to persist the master secret for this service instance.
Be aware that this is less secure than manually entering the secret on startup.
The persisted file will be encrypted and primarily protected through OS permissions.
***************************************************************************************************
Enter master secret:
Enter master secret again:
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at org.apache.hadoop.gateway.launcher.Invoker.invokeMainMethod(Invoker.java:70)
at org.apache.hadoop.gateway.launcher.Invoker.invoke(Invoker.java:39)
at org.apache.hadoop.gateway.launcher.Command.run(Command.java:101)
at org.apache.hadoop.gateway.launcher.Launcher.run(Launcher.java:69)
at org.apache.hadoop.gateway.launcher.Launcher.main(Launcher.java:46)
Caused by: java.lang.NoClassDefFoundError: sun.security.x509.X509CertImpl
at java.lang.J9VMInternals.verifyImpl(Native Method)
at java.lang.J9VMInternals.verify(J9VMInternals.java:72)
at java.lang.J9VMInternals.verify(J9VMInternals.java:70)
at java.lang.J9VMInternals.initialize(J9VMInternals.java:134)
at org.apache.hadoop.gateway.services.DefaultGatewayServices.init(DefaultGatewayServices.java:59)
at org.apache.hadoop.gateway.GatewayServer.main(GatewayServer.java:110)
... 9 more
Caused by: java.lang.ClassNotFoundException: sun.security.x509.X509CertImpl
at java.net.URLClassLoader.findClass(URLClassLoader.java:434)
at java.lang.ClassLoader.loadClass(ClassLoader.java:660)
at java.lang.ClassLoader.loadClass(ClassLoader.java:626)
... 15 more

This is likely due to using Java 8 rather than Java 7. Even though you have installed Java 7 - perhaps you still have JAVA_HOME set to Java 8?