Apache Spark cannot start due to a NoSuchMethodError - java

I have brew installed apache-spark 1.6.1 on OSX El Capitan. Upon spark-shell, I receive:
Exception in thread "main" java.lang.NoSuchMethodError: org.slf4j.spi.LocationAwareLogger.log(Lorg/slf4j/Marker;Ljava/lang/String;ILjava/lang/String;[Ljava/lang/Object;Ljava/lang/Throwable;)V
at org.apache.commons.logging.impl.SLF4JLocationAwareLog.debug(SLF4JLocationAwareLog.java:133)
at org.apache.hadoop.metrics2.lib.MutableMetricsFactory.newForField(MutableMetricsFactory.java:42)
at org.apache.hadoop.metrics2.lib.MetricsSourceBuilder.add(MetricsSourceBuilder.java:133)
at org.apache.hadoop.metrics2.lib.MetricsSourceBuilder.<init>(MetricsSourceBuilder.java:59)
at org.apache.hadoop.metrics2.lib.MetricsAnnotations.newSourceBuilder(MetricsAnnotations.java:42)
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:223)
at org.apache.hadoop.metrics2.MetricsSystem.register(MetricsSystem.java:60)
at org.apache.hadoop.security.UserGroupInformation$UgiMetrics.create(UserGroupInformation.java:108)
at org.apache.hadoop.security.UserGroupInformation.<clinit>(UserGroupInformation.java:224)
at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2160)
at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2160)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2160)
at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:214)
at org.apache.spark.repl.SparkIMain.<init>(SparkIMain.scala:118)
at org.apache.spark.repl.SparkILoop$SparkILoopInterpreter.<init>(SparkILoop.scala:187)
at org.apache.spark.repl.SparkILoop.createInterpreter(SparkILoop.scala:217)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:949)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)
at org.apache.spark.repl.Main$.main(Main.scala:31)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
14:17:50.398 [Thread-0] INFO o.a.spark.util.ShutdownHookManager - Shutdown hook called
14:17:50.401 [Thread-0] INFO o.a.spark.util.ShutdownHookManager - Deleting directory /private/var/folders/pc/wx3xk3sx7_35n6v1qj9knv9c0000gq/T/spark-0a979df4-f61f-4805-a942-8e62350331db
I installed Jave SE already, and I suspect this same problem is preventing my pyspark (with a similar error).

Related

UnsupportedClassVersionError while deploying TCPIP connector in IBM Websphere

I'm trying to deploy a TCPIP connector in the form of an ear in IBM Websphere. But I have no idea what is going wrong here.
WAS version : 8.5
As WAS version uses java 6 by default I tried changing it to java 7 as the jar files my ear uses have been compiled using JAVA 7 - 1.7
But still It didn't work. While starting the app I get the following error:
[3/11/21 16:45:11:905 IST] 0000006b RALifeCycleMa E J2CA0143E: Embedded ResourceAdapter initialization failed due to Exception: com.ibm.ws.exception.RuntimeWarning: J2CA0128E: An Exception occurred while trying to start ResourceAdapter custtcpip.custtcpip. The exception is: java.lang.UnsupportedClassVersionError: com/my_corp/ci/fiusb/tcpip/FIUsbTCPIPServer
at com.ibm.ejs.j2c.RALifeCycleManagerImpl.resourceProviderEvent(RALifeCycleManagerImpl.java:907)
at com.ibm.ws.runtime.component.ResourceMgrImpl$ResourceEventListenerFilterProxy.resourceProviderEvent(ResourceMgrImpl.java:2208)
at com.ibm.ws.runtime.component.ResourceMgrImpl.notifyResourceEventListeners(ResourceMgrImpl.java:1622)
at com.ibm.ws.runtime.component.ResourceMgrImpl.installResourceProvider(ResourceMgrImpl.java:1518)
at com.ibm.ws.runtime.component.ResourceMgrImpl.installResourceProvider(ResourceMgrImpl.java:1462)
at com.ibm.ejs.j2c.RALifeCycleManagerImpl.start(RALifeCycleManagerImpl.java:686)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:1177)
at com.ibm.ws.runtime.component.DeployedApplicationImpl.fireDeployedObjectStart(DeployedApplicationImpl.java:1382)
at com.ibm.ws.runtime.component.DeployedModuleImpl.start(DeployedModuleImpl.java:639)
at com.ibm.ws.runtime.component.DeployedApplicationImpl.start(DeployedApplicationImpl.java:971)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:776)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplicationDynamically(ApplicationMgrImpl.java:1379)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:2189)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:435)
at com.ibm.ws.runtime.component.CompositionUnitImpl.start(CompositionUnitImpl.java:123)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:378)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.access$500(CompositionUnitMgrImpl.java:126)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl$1.run(CompositionUnitMgrImpl.java:653)
at com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:5477)
at com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem(ContextManagerImpl.java:5603)
at com.ibm.ws.security.core.SecurityContext.runAsSystem(SecurityContext.java:255)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.startCompositionUnit(CompositionUnitMgrImpl.java:667)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.startCompositionUnit(CompositionUnitMgrImpl.java:611)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:1269)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
at sun.reflect.GeneratedMethodAccessor33.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
at javax.management.modelmbean.RequiredModelMBean$4.run(RequiredModelMBean.java:1249)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:77)
at javax.management.modelmbean.RequiredModelMBean.invokeMethod(RequiredModelMBean.java:1243)
at javax.management.modelmbean.RequiredModelMBean.invoke(RequiredModelMBean.java:1081)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
at com.ibm.ws.management.AdminServiceImpl$1.run(AdminServiceImpl.java:1350)
at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:118)
at com.ibm.ws.management.AdminServiceImpl.invoke(AdminServiceImpl.java:1243)
at com.ibm.ws.management.connector.AdminServiceDelegator.invoke(AdminServiceDelegator.java:181)
at com.ibm.ws.management.connector.ipc.CallRouter.route(CallRouter.java:247)
at com.ibm.ws.management.connector.ipc.IPCConnectorInboundLink.doWork(IPCConnectorInboundLink.java:360)
at com.ibm.ws.management.connector.ipc.IPCConnectorInboundLink$IPCConnectorReadCallback.complete(IPCConnectorInboundLink.java:602)
at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$QueuedWork.run(SSLReadServiceContext.java:1981)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1881)
I have been banging my head around this for a while.

java.lang.NoClassDefFoundError: com/sun/istack/Pool

When running a script i have encountered this error :
Exception in thread "main" java.lang.NoClassDefFoundError: com/sun/istack/Pool
at com.sun.xml.bind.v2.runtime.JAXBContextImpl$JAXBContextBuilder.build(JAXBContextImpl.java:1126)
at com.sun.xml.bind.v2.ContextFactory.createContext(ContextFactory.java:135)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.xml.bind.ContextFinder.newInstance(ContextFinder.java:247)
at javax.xml.bind.ContextFinder.newInstance(ContextFinder.java:234)
at javax.xml.bind.ContextFinder.find(ContextFinder.java:441)
at javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:641)
at javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:584)
at org.wso2.carbon.datasource.utils.DataSourceUtils.loadJAXBConfiguration(DataSourceUtils.java:344)
at org.wso2.carbon.datasource.core.DataSourceManager.initDataSource(DataSourceManager.java:159)
at org.wso2.carbon.datasource.core.DataSourceManager.initDataSources(DataSourceManager.java:137)
at org.wso2.carbon.datasource.core.DataSourceManager.initDataSources(DataSourceManager.java:109)
at org.wso2.carbon.privacy.forgetme.sql.instructions.DatasourceProcessorConfigReader.readProcessorConfig(DatasourceProcessorConfigReader.java:45)
at org.wso2.carbon.privacy.forgetme.sql.instructions.DatasourceProcessorConfigReader.readProcessorConfig(DatasourceProcessorConfigReader.java:32)
at org.wso2.carbon.privacy.forgetme.ConfigReader.loadExtensions(ConfigReader.java:213)
at org.wso2.carbon.privacy.forgetme.ConfigReader.readSystemConfig(ConfigReader.java:112)
at org.wso2.carbon.privacy.forgetme.ForgetMeTool.process(ForgetMeTool.java:201)
at org.wso2.carbon.privacy.forgetme.ForgetMeTool.main(ForgetMeTool.java:141)
jaxb-core and jaxb-impl jars are properly added in the project.

The jaxb-core is depending on the istack-commons-runtime, which can be found in the pom of the jaxb-core. (https://mvnrepository.com/artifact/com.sun.istack/istack-commons-runtime)

Java 7 to migration compatibility issue

After migrating the jobs from java 7 to 8 getting following error,even though compilation is successful.
Same version of java i.e java 8 was used in compilation and the same version is being used while runtime. So typically there should be any mismatch in the version of java.Even scala version are same.
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.JavaMain], main() threw exception, java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
org.apache.oozie.action.hadoop.JavaMainException: java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:59)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:51)
at org.apache.oozie.action.hadoop.JavaMain.main(JavaMain.java:35)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:242)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
at com.twitter.scalding.RichXHandler$.<init>(XHandler.scala:38)
at com.twitter.scalding.RichXHandler$.<clinit>(XHandler.scala)
at com.twitter.scalding.Tool$.main(Tool.scala:152)
at com.twitter.scalding.Tool.main(Tool.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:56)

Cannot connect to local spark cluster from Java application

I'm trying to run a java application that connects to a local standalone spark cluster. I start the cluster with the default configuration, using start-all.sh. When I go to the web page for the cluster, it is started ok. I can connect to this cluster with SparkR, but when I use the same master URL to connect from within Java, I get an error message.
I'm using Spark 1.5.
Here is my Java code:
SparkConf conf = new SparkConf();
conf.setAppName("test");
conf.setMaster("spark://Ellens-MacBook-Pro.local:7077");
conf.setSparkHome("/Applications/spark-1.5.0-bin-hadoop2.6");
SparkContext sc = new SparkContext(conf);
Here is a snippet of the error message:
ReliableDeliverySupervisor: Association with remote system
[akka.tcp://sparkMaster#Ellens-MacBook-Pro.local:7077] has failed,
address is now gated for [5000] ms. Reason: [Disassociated] 15/10/09
17:31:41 INFO AppClient$ClientEndpoint: Connecting to master
spark://Ellens-MacBook-Pro.local:7077... 15/10/09 17:31:41 WARN
ReliableDeliverySupervisor: Association with remote system
[akka.tcp://sparkMaster#Ellens-MacBook-Pro.local:7077] has failed,
address is now gated for [5000] ms. Reason: [Disassociated] 15/10/09
17:32:01 INFO AppClient$ClientEndpoint: Connecting to master
spark://Ellens-MacBook-Pro.local:7077... 15/10/09 17:32:01 ERROR
SparkUncaughtExceptionHandler: Uncaught exception in thread
Thread[appclient-registration-retry-thread,5,main]
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.FutureTask#54e2b678 rejected from
java.util.concurrent.ThreadPoolExecutor#5d9f3e0d[Running, pool size =
1, active threads = 1, queued tasks = 0, completed tasks = 2]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
Here is the output in the Spark log:
Spark Command: /Library/Java/JavaVirtualMachines/jdk1.8.0_60.jdk/Contents/Home/jre/bin/java -cp /Applications/spark-1.5.0-bin-hadoop2.6/sbin/../conf/:/Applications/spark-1.5.0-bin-hadoop2.6/lib/spark-assembly-1.5.0-hadoop2.6.0.jar:/Applications/spark-1.5.0-bin-hadoop2.6/lib/datanucleus-api-jdo-3.2.6.jar:/Applications/spark-1.5.0-bin-hadoop2.6/lib/datanucleus-core-3.2.10.jar:/Applications/spark-1.5.0-bin-hadoop2.6/lib/datanucleus-rdbms-3.2.9.jar -Xms1g -Xmx1g org.apache.spark.deploy.master.Master --ip Ellens-MacBook-Pro.local --port 7077 --webui-port 8080
========================================
15/10/12 17:56:43 INFO Master: Registered signal handlers for [TERM, HUP, INT]
15/10/12 17:56:43 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/10/12 17:56:43 INFO SecurityManager: Changing view acls to: ellenk
15/10/12 17:56:43 INFO SecurityManager: Changing modify acls to: ellenk
15/10/12 17:56:43 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(ellenk); users with modify permissions: Set(ellenk)
15/10/12 17:56:44 INFO Slf4jLogger: Slf4jLogger started
15/10/12 17:56:44 INFO Remoting: Starting remoting
15/10/12 17:56:44 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkMaster#Ellens-MacBook-Pro.local:7077]
15/10/12 17:56:44 INFO Utils: Successfully started service 'sparkMaster' on port 7077.
15/10/12 17:56:44 INFO Master: Starting Spark master at spark://Ellens-MacBook-Pro.local:7077
15/10/12 17:56:44 INFO Master: Running Spark version 1.5.0
15/10/12 17:56:44 INFO Utils: Successfully started service 'MasterUI' on port 8080.
15/10/12 17:56:44 INFO MasterWebUI: Started MasterWebUI at http://192.168.1.3:8080
15/10/12 17:56:44 INFO Utils: Successfully started service on port 6066.
15/10/12 17:56:44 INFO StandaloneRestServer: Started REST server for submitting applications on port 6066
15/10/12 17:56:45 INFO Master: I have been elected leader! New state: ALIVE
15/10/12 17:56:50 INFO Master: Registering worker 192.168.1.3:57180 with 8 cores, 15.0 GB RAM
15/10/12 17:57:23 WARN ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkDriver#192.168.1.3:57238] has failed, address is now gated for [5000] ms. Reason: [null]
15/10/12 17:57:23 INFO Master: 192.168.1.3:57238 got disassociated, removing it.
15/10/12 17:57:43 ERROR Remoting:
java.io.OptionalDataException
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1371)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at scala.collection.immutable.$colon$colon.readObject(List.scala:366)
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1900)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at scala.collection.immutable.$colon$colon.readObject(List.scala:366)
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1900)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at scala.collection.immutable.$colon$colon.readObject(List.scala:366)
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1900)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at scala.collection.immutable.$colon$colon.readObject(List.scala:366)
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1900)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at scala.collection.immutable.$colon$colon.readObject(List.scala:366)
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1900)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at scala.collection.immutable.$colon$colon.readObject(List.scala:366)
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1900)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at scala.collection.immutable.$colon$colon.readObject(List.scala:366)
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1900)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at scala.collection.immutable.$colon$colon.readObject(List.scala:366)
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1900)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at scala.collection.immutable.$colon$colon.readObject(List.scala:366)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1900)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at scala.collection.immutable.$colon$colon.readObject(List.scala:366)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1900)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at scala.collection.immutable.$colon$colon.readObject(List.scala:366)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1900)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at scala.collection.immutable.$colon$colon.readObject(List.scala:366)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1900)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at akka.serialization.JavaSerializer$$anonfun$1.apply(Serializer.scala:136)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
at akka.serialization.JavaSerializer.fromBinary(Serializer.scala:136)
at akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104)
at scala.util.Try$.apply(Try.scala:161)
at akka.serialization.Serialization.deserialize(Serialization.scala:98)
at akka.remote.MessageSerializer$.deserialize(MessageSerializer.scala:23)
at akka.remote.DefaultMessageDispatcher.payload$lzycompute$1(Endpoint.scala:58)
at akka.remote.DefaultMessageDispatcher.payload$1(Endpoint.scala:58)
at akka.remote.DefaultMessageDispatcher.dispatch(Endpoint.scala:76)
at akka.remote.EndpointReader$$anonfun$receive$2.applyOrElse(Endpoint.scala:935)
at akka.actor.Actor$class.aroundReceive(Actor.scala:467)
at akka.remote.EndpointActor.aroundReceive(Endpoint.scala:411)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
15/10/12 17:57:43 INFO Master: 192.168.1.3:57238 got disassociated, removing it.
15/10/12 17:58:03 INFO Master: 192.168.1.3:57238 got disassociated, removing it.
15/10/12 17:58:03 WARN ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkDriver#192.168.1.3:57238] has failed, address is now gated for [5000] ms. Reason: [Disassociated]
15/10/12 17:58:03 INFO Master: 192.168.1.3:57238 got disassociated, removing it.
15/10/12 17:58:03 INFO Master: 192.168.1.3:57238 got disassociated, removing it.
15/10/12 17:58:03 INFO Master: 192.168.1.3:57238 got disassociated, removing it.

Most likely this is due to Spark version mismatch. Make sure that you are using the same versions (including Scala version that should be 2.10 if you are using one of the provided binary builds) of spark libraries both on the client and server.
Also if you are using akka in your client application you should make sure that it is compatible with the one used by Spark (2.3.4+).

Spark can be connected to locally using the JavaSparkContext. For instance, here is one of the constructors:
JavaSparkContext(String master, String appName, String sparkHome, String[] jars)
JavaSparkContext was introduced as of 1.4:
https://spark.apache.org/docs/1.4.0/api/java/org/apache/spark/api/java/JavaSparkContext.html
Here is an example similar to what I have working. I'm using Maven. so the application jar is created under the target directory:
public JavaSparkContext getJavaSparkContext(String myAppName ) {
// Based on example above:
String sparkHome="/Applications/spark-1.5.0-bin-hadoop2.6";
JavaSparkContext sc = new JavaSparkContext("local", myAppName,
sparkHome, new String[]{"target/sparkapp-0.0.1-SNAPSHOT.jar"});
return sc;
}
The argument myAppName is just a string, for example: "My App"

I am not sure if this would help; I was getting error of not able to connect to spark://master:7077; whereas when I started the master/slave it showed me the spark master at spark://127.0.1.1:7077.
So I mapped "master" to 127.0.1.1 in /etc/hosts, and voila it worked like charm; it could find sparkContext instance easily.
Jag

LinkageError after upgrading dropwizard to 0.8.1

I've upgraded dropwizard from 0.7.1 to 0.8.1 and keep getting this exception:
Error:Groovyc: A groovyc error occurred while trying to load one of the classes in project dependencies, please ensure it's present. See the message and the stack trace below for reference
java.lang.LinkageError: ClassCastException: attempting to castjar:file:/home/.gradle/caches/modules-2/files-2.1/javax.ws.rs/jsr311-api/1.1.1/59033da2a1afd56af1ac576750a8d0b1830d59e6/jsr311-api-1.1.1.jar!/javax/ws/rs/ext/RuntimeDelegate.classtojar:file:/home/.gradle/caches/modules-2/files-2.1/javax.ws.rs/jsr311-api/1.1.1/59033da2a1afd56af1ac576750a8d0b1830d59e6/jsr311-api-1.1.1.jar!/javax/ws/rs/ext/RuntimeDelegate.class
at javax.ws.rs.ext.RuntimeDelegate.findDelegate(RuntimeDelegate.java:116)
at javax.ws.rs.ext.RuntimeDelegate.getInstance(RuntimeDelegate.java:91)
at javax.ws.rs.core.MediaType.<clinit>(MediaType.java:44)
at sun.misc.Unsafe.ensureClassInitialized(Native Method)
at sun.reflect.UnsafeFieldAccessorFactory.newFieldAccessor(UnsafeFieldAccessorFactory.java:43)
at sun.reflect.ReflectionFactory.newFieldAccessor(ReflectionFactory.java:142)
at java.lang.reflect.Field.acquireFieldAccessor(Field.java:1088)
at java.lang.reflect.Field.getFieldAccessor(Field.java:1069)
at java.lang.reflect.Field.get(Field.java:393)
at org.codehaus.groovy.classgen.AnnotationVisitor.transformInlineConstants(AnnotationVisitor.java:131)
at org.codehaus.groovy.classgen.AnnotationVisitor.visit(AnnotationVisitor.java:76)
at org.codehaus.groovy.classgen.ExtendedVerifier.visitAnnotation(ExtendedVerifier.java:246)
at org.codehaus.groovy.classgen.ExtendedVerifier.visitAnnotations(ExtendedVerifier.java:137)
at org.codehaus.groovy.classgen.ExtendedVerifier.visitClass(ExtendedVerifier.java:66)
at org.codehaus.groovy.control.CompilationUnit$16.call(CompilationUnit.java:789)
at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1052)
at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:588)
at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:566)
at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:543)
at org.jetbrains.groovy.compiler.rt.GroovyCompilerWrapper.compile(GroovyCompilerWrapper.java:54)
at org.jetbrains.groovy.compiler.rt.DependentGroovycRunner.runGroovyc(DependentGroovycRunner.java:87)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.jetbrains.groovy.compiler.rt.GroovycRunner.main(GroovycRunner.java:100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.intellij.rt.execution.CommandLineWrapper.main(CommandLineWrapper.java:130)
I have two RuntimeDelegate instances in: jersey-core-1.18.3.jar & jsr311-api-1.1.1.jar.
Did anyone encouter this?

Categories