Need help. I'm trying this job given in the documentation.
I have the following versions installed in my system:
Hadoop:3.2.0
Scala: 2.13.1
Spark: 3.0.0
Java: 1.8.0
Hive: 3.1.2
I'm submitting the spark job to run data quality using griffin. I'm getting below exception can you pls help me in this regard.
2020-04-04 08:37:35,456 INFO cluster.YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after waiting maxRegisteredResourcesWaitingTime: 30000000000(ns)
Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps;
at org.apache.griffin.measure.datasource.connector.batch.HiveBatchDataConnector.<init>(HiveBatchDataConnector.scala:47)
at org.apache.griffin.measure.datasource.connector.DataConnectorFactory$$anonfun$getDataConnector$1.apply(DataConnectorFactory.scala:63)
at org.apache.griffin.measure.datasource.connector.DataConnectorFactory$$anonfun$getDataConnector$1.apply(DataConnectorFactory.scala:62)
at scala.util.Try$.apply(Try.scala:213)
at org.apache.griffin.measure.datasource.connector.DataConnectorFactory$.getDataConnector(DataConnectorFactory.scala:61)
at org.apache.griffin.measure.datasource.DataSourceFactory$$anonfun$1.apply(DataSourceFactory.scala:58)
at org.apache.griffin.measure.datasource.DataSourceFactory$$anonfun$1.apply(DataSourceFactory.scala:57)
at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245)
at scala.collection.immutable.List.foreach(List.scala:392)
at scala.collection.TraversableLike.flatMap(TraversableLike.scala:245)
at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:242)
at scala.collection.immutable.List.flatMap(List.scala:355)
at org.apache.griffin.measure.datasource.DataSourceFactory$.org$apache$griffin$measure$datasource$DataSourceFactory$$getDataSource(DataSourceFactory.scala:57)
at org.apache.griffin.measure.datasource.DataSourceFactory$$anonfun$getDataSources$1.apply(DataSourceFactory.scala:40)
at org.apache.griffin.measure.datasource.DataSourceFactory$$anonfun$getDataSources$1.apply(DataSourceFactory.scala:38)
at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245)
at scala.collection.immutable.List.foreach(List.scala:392)
at scala.collection.TraversableLike.flatMap(TraversableLike.scala:245)
at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:242)
at scala.collection.immutable.List.flatMap(List.scala:355)
at org.apache.griffin.measure.datasource.DataSourceFactory$.getDataSources(DataSourceFactory.scala:38)
at org.apache.griffin.measure.launch.batch.BatchDQApp$$anonfun$run$1.apply$mcZ$sp(BatchDQApp.scala:75)
at org.apache.griffin.measure.launch.batch.BatchDQApp$$anonfun$run$1.apply(BatchDQApp.scala:67)
at org.apache.griffin.measure.launch.batch.BatchDQApp$$anonfun$run$1.apply(BatchDQApp.scala:67)
at scala.util.Try$.apply(Try.scala:213)
at org.apache.griffin.measure.launch.batch.BatchDQApp.run(BatchDQApp.scala:67)
at org.apache.griffin.measure.Application$.main(Application.scala:88)
at org.apache.griffin.measure.Application.main(Application.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:928)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps;
It definitely means that you have conflicts in the Scala version somewhere in your application.
There is a very similar question here. Try to follow the answer provided there (downgrade Scala version to 2.11.x).
I don't think any of the Spark versions have been compiled/released for Scala 2.13
Related
I am trying to profile my java application running on OpenJDK 10 using VisualVM 1.4.2 under windows 8.1 but when opening a connection to app in VisualVM, exception is thrown
java.lang.IllegalArgumentException: Unexpected composite type for ThreadInfo
at sun.management.ThreadInfoCompositeData.validateCompositeData(ThreadInfoCompositeData.java:372)
at sun.management.ThreadInfoCompositeData.getInstance(ThreadInfoCompositeData.java:68)
at java.lang.management.ThreadInfo.<init>(ThreadInfo.java:263)
at java.lang.management.ThreadInfo.from(ThreadInfo.java:794)
Caused: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
at com.sun.jmx.mbeanserver.DefaultMXBeanMappingFactory$CompositeBuilderViaFrom.fromCompositeData(DefaultMXBeanMappingFactory.java:1018)
Caused: java.io.InvalidObjectException: Failed to invoke from(CompositeData)
at com.sun.jmx.mbeanserver.DefaultMXBeanMappingFactory.invalidObjectException(DefaultMXBeanMappingFactory.java:1457)
at com.sun.jmx.mbeanserver.DefaultMXBeanMappingFactory$CompositeBuilderViaFrom.fromCompositeData(DefaultMXBeanMappingFactory.java:1021)
at com.sun.jmx.mbeanserver.DefaultMXBeanMappingFactory$CompositeMapping.fromNonNullOpenValue(DefaultMXBeanMappingFactory.java:919)
at com.sun.jmx.mbeanserver.DefaultMXBeanMappingFactory$NonNullMXBeanMapping.fromOpenValue(DefaultMXBeanMappingFactory.java:133)
at com.sun.jmx.mbeanserver.ConvertingMethod.fromOpenReturnValue(ConvertingMethod.java:131)
at com.sun.jmx.mbeanserver.MXBeanProxy.invoke(MXBeanProxy.java:168)
at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:258)
Caused: java.lang.reflect.UndeclaredThrowableException
at com.sun.proxy.$Proxy13.getThreadInfo(Unknown Source)
at com.sun.tools.visualvm.jmx.impl.JmxSupport.isReadOnlyConnection(JmxSupport.java:126)
at com.sun.tools.visualvm.jmx.impl.JmxModelImpl.isTakeThreadDumpSupported(JmxModelImpl.java:369)
at com.sun.tools.visualvm.application.views.threads.ApplicationThreadsViewProvider.resolveThreads(ApplicationThreadsViewProvider.java:85)
at com.sun.tools.visualvm.application.views.threads.ApplicationThreadsViewProvider.supportsViewFor(ApplicationThreadsViewProvider.java:49)
at com.sun.tools.visualvm.application.views.threads.ApplicationThreadsViewProvider.supportsViewFor(ApplicationThreadsViewProvider.java:45)
at com.sun.tools.visualvm.core.ui.DataSourceViewsManager.getViews(DataSourceViewsManager.java:136)
at com.sun.tools.visualvm.core.ui.DataSourceWindowManager.openWindowAndAddView(DataSourceWindowManager.java:206)
at com.sun.tools.visualvm.core.ui.DataSourceWindowManager.access$000(DataSourceWindowManager.java:50)
at com.sun.tools.visualvm.core.ui.DataSourceWindowManager$2.run(DataSourceWindowManager.java:117)
at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:1443)
at org.netbeans.modules.openide.util.GlobalLookup.execute(GlobalLookup.java:68)
at org.openide.util.lookup.Lookups.executeWith(Lookups.java:303)
[catch] at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:2058)
the issue seems very similar to Cannot open some applications in VisualVM but the accepted workaround doesn't work for me. Any suggestions, please?
Previous lower java versions worked, btw.
EDIT:
OracleJDK 11 does not work as well
It looks like you are hitting JDK bug JDK-8165005. Please upgrade JDK 8 to JDK 8u152 or newer.
I am trying to run the CountVectorizerDemo program provided here:
https://github.com/apache/spark/blob/master/examples/src/main/java/org/apache/spark/examples/ml/JavaCountVectorizerExample.java
I'm getting the following error and don't know what the problem is.
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.ml.util.SchemaUtils$.checkColumnType$default$4()Ljava/lang/String;
at org.apache.spark.ml.feature.CountVectorizerParams$class.validateAndTransformSchema(CountVectorizer.scala:71)
at org.apache.spark.ml.feature.CountVectorizer.validateAndTransformSchema(CountVectorizer.scala:107)
at org.apache.spark.ml.feature.CountVectorizer.transformSchema(CountVectorizer.scala:168)
at org.apache.spark.ml.PipelineStage.transformSchema(Pipeline.scala:59)
at org.apache.spark.ml.feature.CountVectorizer.fit(CountVectorizer.scala:130)
at com.bah.ossem.spark.topic.CountVectorizerDemo.main(CountVectorizerDemo.java:42)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
The problem was that my cluster was using Spark core 1.4 but my application was using Spark core 1.5.1 and MLlib 1.5.1. I updated my AWS cluster to Spark 1.5.1 and that fixed the problem.
I have problem with my tomcat application, after changing the server and installing the last version of tomcat7 my application won't read/load jpg files..
I installed imageio and jai on the server, try to change java version but every time I have the same error..
Anybody have an idea?
Error: One factory fails for the operation "jpeg"
Occurs in: javax.media.jai.ThreadSafeOperationRegistry
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at javax.media.jai.FactoryCache.invoke(FactoryCache.java:122)
at javax.media.jai.OperationRegistry.invokeFactory(OperationRegistry.java:1674)
at javax.media.jai.ThreadSafeOperationRegistry.invokeFactory(ThreadSafeOperationRegistry.java:473)
at javax.media.jai.registry.RIFRegistry.create(RIFRegistry.java:332)
at com.sun.media.jai.opimage.StreamRIF.create(StreamRIF.java:102)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at javax.media.jai.FactoryCache.invoke(FactoryCache.java:122)
at javax.media.jai.OperationRegistry.invokeFactory(OperationRegistry.java:1674)
at javax.media.jai.ThreadSafeOperationRegistry.invokeFactory(ThreadSafeOperationRegistry.java:473)
at javax.media.jai.registry.RIFRegistry.create(RIFRegistry.java:332)
at javax.media.jai.RenderedOp.createInstance(RenderedOp.java:819)
at javax.media.jai.RenderedOp.createRendering(RenderedOp.java:867)
at javax.media.jai.RenderedOp.getWidth(RenderedOp.java:2179)
The whole error log can be found here -> http://paste.ubuntu.com/7653452/.
Update: The problem is related to grails plugin called ImageTools
If you look at the code for JPEGImageDecoder you'll see it depends on com.sun.image.codec.jpeg.ImageFormatException in its imports.
However, com.sun.image.codec.jpeg was removed from Java 7 onwards.
So likely the problem is that JAI is simply out of date, and you would have to use a Java 6 runtime to use it.
I am running graphbuilder with hadoop (single-node). I have followed the tutorial "https://01.org/graphbuilder/documentation/how-run-demo-application". However, when I run the command
bin/hadoop jar /usr/local/graphbuilder/target/graphbuilder-0.0.1-SNAPSHOT-hadoop-job.jar com.intel.hadoop.graphbuilder.demoapps.wikipedia.docwordgraph.TFIDFGraphEnd2End 1 /user/hduser/wiki-input /user/hduser/en-wiki-articles-output ingressCode*
I am getting error
INFO docwordgraph.CreateWordCountGraph: ========== Creating Graph ================
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.filecache.DistributedCache.addFileToClassPath(Lorg/apache/hadoop/fs/Path;Lorg/apache/hadoop/conf/Configuration;Lorg/apache/hadoop/fs/FileSystem;)V
at com.intel.hadoop.graphbuilder.util.FsUtil.distributedTempClassToClassPath(FsUtil.java:46)
at com.intel.hadoop.graphbuilder.job.AbstractPreprocessJob.run(AbstractPreprocessJob.java:111)
at com.intel.hadoop.graphbuilder.demoapps.wikipedia.docwordgraph.CreateWordCountGraph.main(CreateWordCountGraph.java:77)
at com.intel.hadoop.graphbuilder.demoapps.wikipedia.docwordgraph.TFIDFGraphEnd2End.main(TFIDFGraphEnd2End.java:69)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)"
Can you please guide me how to solve this issue?
Thanks
Looks like you're using a back level version of hadoop. Check the version of hadoop that your version of graph builder needs and make sure that's the version you're running.
fed up with raspberry pi.....
i have created a javaFX application from windows 7 using netbeans 7.3.1 , jdk 1.7 ..
application has included some external jar including javafx-dialogs-0.0.3,jaybird etc..
its all works fine with windows 7,ubuntu and centOS
Then i tried to run the same application jar from command line in raspberry pi..
but it gives Exception like "noSuchMethoderror " and exiting the application..
what may be the issue ???
am using same jar during comple and run time..
here is full stack
Exception in Application start method
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.sun.javafx.application.LauncherImpl.launchApplicationWithArgs(LauncherImpl.java:367)
at com.sun.javafx.application.LauncherImpl.launchApplication(LauncherImpl.java:305)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at sun.launcher.LauncherHelper$FXHelper.main(LauncherHelper.java:767)
Caused by: java.lang.RuntimeException: Exception in Application start method
at com.sun.javafx.application.LauncherImpl.launchApplication1(LauncherImpl.java:894)
at com.sun.javafx.application.LauncherImpl.access$000(LauncherImpl.java:56)
at com.sun.javafx.application.LauncherImpl$1.run(LauncherImpl.java:158)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.NoSuchMethodError: com.sun.javafx.css.StyleManager.getPseudoclassMask(Ljava/lang/String;)J
at javafx.scene.control.Dialogs$FXDialog$RootPane.<clinit>(Dialogs.java:1310)
at javafx.scene.control.Dialogs$FXDialog.<init>(Dialogs.java:1131)
at javafx.scene.control.Dialogs$FXDialog.<init>(Dialogs.java:1088)
at javafx.scene.control.Dialogs$DialogTemplate.<init>(Dialogs.java:616)
at javafx.scene.control.Dialogs.showSimpleContentDialog(Dialogs.java:494)
at javafx.scene.control.Dialogs.showInformationDialog(Dialogs.java:266)
at displayshelftry.Utilities.propertySettings(Utilities.java:325)
at displayshelftry.DisplayShelfTry.run(DisplayShelfTry.java:356)
at displayshelftry.DisplayShelfTry.start(DisplayShelfTry.java:1150)
at com.sun.javafx.application.LauncherImpl$8.run(LauncherImpl.java:837)
at com.sun.javafx.application.PlatformImpl$7.run(PlatformImpl.java:331)
at com.sun.javafx.application.PlatformImpl$6$1.run(PlatformImpl.java:297)
at com.sun.javafx.application.PlatformImpl$6$1.run(PlatformImpl.java:294)
at java.security.AccessController.doPrivileged(Native Method)
at com.sun.javafx.application.PlatformImpl$6.run(PlatformImpl.java:294)
at com.sun.glass.ui.lens.LensApplication$RunnableEvent.dispatch(LensApplication.java:182)
at com.sun.glass.ui.lens.LensApplication._runLoop(LensApplication.java:861)
at com.sun.glass.ui.lens.LensApplication.access$1700(LensApplication.java:58)
at com.sun.glass.ui.lens.LensApplication$4.run(LensApplication.java:914)
... 1 more
thanks...
Try building you app with jdk8. jdk8 usually should run jdk7 built jars as well, but it's not yet released and rasberry pi was supported only since jdk8.
Also official pi vs javafx page claims that not all demos will work on pi (note it's not yet released product): https://wiki.openjdk.java.net/display/OpenJFX/OpenJFX+on+the+Raspberry+Pi
So you may want to try BrickBreaker instead of DisplayShelf :)