Scala module requiring specific version of data bind for Spark - java

I am having issues trying to get Spark to load, read and query a parquet file. The infrastructure seems to be set up (Spark standalone 3.0) and can be seen and will pick up jobs.
The issue I am having is when this line is called
Dataset<Row> parquetFileDF = sparkSession.read().parquet(parquePath);
the following error is thrown
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Scala module 2.10.0 requires Jackson Databind version >= 2.10.0 and < 2.11.0
at com.fasterxml.jackson.module.scala.JacksonModule.setupModule(JacksonModule.scala:61)
I looked into JacksonModule.setupModule and when it gets to context.getMapperVersion the version that is being passed is 2.9.10. It appears to me that the DefaultScalaModule is pulling some older version.
I'm using Gradle to build and have the dependencies set up as such
implementation 'com.fasterxml.jackson.core:jackson-core:2.10.0'
implementation 'com.fasterxml.jackson.core:jackson-databind:2.10.0'
implementation 'org.apache.spark:spark-core_2.12:3.0.0'
implementation 'org.apache.spark:spark-sql_2.12:3.0.0'
implementation 'org.apache.spark:spark-launcher_2.12:3.0.0'
implementation 'org.apache.spark:spark-catalyst_2.12:3.0.0'
implementation 'org.apache.spark:spark-streaming_2.12:3.0.0'
That didn't work, so I tried forcing databind
implementation ('com.fasterxml.jackson.core:jackson-databind') {
version {
strictly '2.10.0'
}
}
I've tried a few different versions and still keep hitting this issue. Maybe I'm missing something super simple, but right now, I can't seem to get past this error.
Any help would be appreciated.

I was able to figure out the issue. I was pulling in jar file from another project. The functionality in the jar file wasn't being used at all, so it wasn't suspect. Unfortunately, that project hadn't been updated and there were some older Spark libraries that were some how being picked up by my current running app. Once I removed that, the error went away. What's interesting is the dependency graph didn't show anything about the libraries the other jar file was using.
I suppose if you run into a similar issue, double check any jar files being imported.

Related

Does Java classpath different order give "No Method found error"

I have a Scala Akka Application which connects to HBase (currently CDP earlier HDP) deployed on rancher; Never faced any trouble when connecting to HDP hbase; Since recent HDP to CDP change, with the same image we are getting no method found on one of the dependency's class in one of the container, where as another container of same image connects to hbase properly.; even though the jar exists in the same image and classpath also.
one of the noticeable difference is change in the order of classpath.
Does change in the classpath order will effect the jars availability.
Does java libraries/classes would load in different order when they would hit a faster CPU cycle at startup.
What could be the reason for such "no class method found".
It certainly can, if the same class file is present in different classpath entries. For example, if your classpath is: java -cp a.jar:b.jar com.foo.App, and:
a.jar:
pkg/SomeClass.class
b.jar:
pkg/SomeClass.class
Then this can happen - usually because one of the jars on your classpath is an older version than the other, or the same but more complicated: one of the jars of your classpath contains a whole heap of different libraries all squished together and one of those components is an older version.
There are some basic hygiene rules to observe:
Don't squish jars together. If you have 500 deps, put 500 entries on your classpath. We have tools to manage this stuff, use them. Don't make striped jars, uber jars, etc.
Use dependency trackers to check if there are version difference in your dependency chain. If your app depends on, say, 'hibernate' and 'jersey', and they both depend on google's guava libraries, but hibernate imports v26 and jersey imports v29, that's problematic. Be aware of it and ensure that you explicitly decide which version ends up making it. Presumably, you'd want to explicitly pick v29 and perhaps check that hibernate also runs on v29*. If it doesn't, you have bigger problems. They are fixable (with modular classloaders), but not easily.
*) Neither hibernate nor jersey actually depend on guava, I'm just using them as hypothetical examples.
For example, if you use maven, check out the enforcer plugin. (groupId: org.apache.maven.plugins, artifactId: maven-enforcer-plugin).
My bet is that there is another version of the jar somewhere in CDP, and occasionally it is loaded before the version that you ship with your project, causing the error.
So, when your container starts, try logging from which location the conflicting class is loaded. This question might help you: Determine which JAR file a class is from

javax.xml.bind.JAXBException with Java 11

I have an older widget app, it is using JAXB and now being migrated to Java 11.
I am aware that java.xml.bind/JAXB has been removed in this version, so we are trying to replace it with Jakarta.
We have jakarta.activation.jar and jakarta.xml.bind-api.jar and it works fine for compilation, but not for runtime
when the app starts I am getting this:
javax.xml.bind.JAXBException: Implementation of JAXB-API has not been found on module path or classpath.
- with linked exception:
[java.lang.ClassNotFoundException: com.sun.xml.internal.bind.v2.ContextFactory]
at java.xml.bind/javax.xml.bind.ContextFinder.newInstance(ContextFinder.java:232)
... ...
The com.sun.xml.internal.* classes were part of rt.jar, which i believe was removed in Java 9,
but the latest jakarta source still refers to them... how is that supposed to work?
I saw some posts with a similar problem, and typical solution is Add Maven dependency.
I'm not sure about details, but in any case we don't use Maven or Gradle and don't have pom.xml.
Is there anything i can do to make it work?
Turned out it is not as bad as I thought.
rt. jar was broken down to multiple modules and I started looking for the ones I needed.
jaxb-runtime.jar looked right to me and I ran strings / grep on it - it indeed had all the com.sun.xml.bind classes in it!
When I added this jar to the ones I had the original error changed to "some-other-class not found" and I had to add more jars until the app was happy.
In the end I have this:
jakarta.activation.jar, jakarta.xml.bind-api.jar, jaxb-runtime.jar,
istack-commons-runtime.jar, stax-ex.jar, FastInfoset.jar, txw2.jar
No changes to manifest, makefile or anything else related to build, purely packaging issue.
Big sigh...

Scala unsatisfiable cyclic dependency in "table-layout" library (Toolkit class)

When I try to compile with sbt some code containing an instance of a Table from this library I get this error:
java.lang.AssertionError: assertion failed: unsatisfiable cyclic dependency in 'class Toolkit'
It seems to work with Java so I don't understand why it fails in Scala.
Here is the toolkit class: http://code.google.com/p/table-layout/source/browse/branches/v1/tablelayout/src/com/esotericsoftware/tablelayout/Toolkit.java
As long as I get this error I'm totally stopped in my project :(.
Edit: It works with Scala 2.10.0 every Milestone, But this version of scala doesn't work with Android (or at least not yet ... [or at least not with libgdx ...]). So I still need a way to solve this problem even if it's a bit constraignant.
it will compile if you force scalac to load dependencies in correct order like this:
classOf[com.esotericsoftware.tablelayout.Toolkit[_,_,_]]
println(new com.badlogic.gdx.scenes.scene2d.ui.Table toString)
must be a bug which was accidentally fixed in 2.10

ClassCastException in weblogic when using prefer-web-inf-classes

I had to add
<prefer-web-inf-classes>true</prefer-web-inf-classes>
to weblogic.xml to resolve a Hibernate antr compatibility issue with Weblogic. after adding that I was getting all different kind of classCastException related to XML parsers.
I understood from reading other threads that weblogic is trying to use a different class that what the application is expecting.
I spend all day researching and tried different solutions like removing "xml-apis......." jar files. but everytime I get ClassCastException. The cast "from" class changes when I remove jar files, but I always get
ClassCastException: "some xml parser related class" can not be cast to javax.xml.parsers.DocumentBuilderFactory
is there a way to know which xml parser jars are really causing the issue.
I m using Maven 2 to manage dependencies
Answering my own question:
I removed all jars that contains classes from javax.xml.* package by doing a java search and searching Package and check search in "application Libraries". then I had to remove sax..jar file. everything worked as expected after that.
Good answer to your own question. I also found this method (if you use maven), which generates a really nice dependency chart. I had this same problem, and using the dependency chart I was able to determine that the offending library, xml-apis*.jar, was loaded by maven as a dependency of jasperreports. Adding the exclusion element to the pom.xml for the jasperreports module fixed the problem.
http://maven.40175.n5.nabble.com/where-is-xml-apis-1-0-b2-jar-coming-from-td88057.html

Using MessagePack with Android

Has someone tried to use MessagePack with an Android app?
Is it possible? I have tried to use the Jar from msgpack-java and received the following Exception:
Caused by: java.lang.ExceptionInInitializerError
at org.msgpack.Packer.pack(Packer.java:532)
at org.msgpack.MessagePack.pack(MessagePack.java:31)
... 15 more
Caused by: java.lang.ExceptionInInitializerError
at org.msgpack.template.TemplateRegistry.<clinit>(TemplateRegistry.java:38)
... 17 more
Caused by: java.lang.VerifyError: org.msgpack.template.BeansFieldEntryReader
at org.msgpack.template.builder.BeansTemplateBuilder.<init (BeansTemplateBuilder.java:42)
at org.msgpack.template.builder.BuilderSelectorRegistry.initForJava(BuilderSelectorRegistry.java:73)
at org.msgpack.template.builder.BuilderSelectorRegistry.<clinit>(BuilderSelectorRegistry.java:38)
... 18 more
The code that I use is very simple
PrintWriter out = new PrintWriter(socket.getOutputStream());
Message msg = new Message();
msg.body = "asdasdasd";
msg.from = "qwe";
msg.to = "ttt";
byte[] bytes = MessagePack.pack(msg);
out.print(bytes);
out.flush();
I have javassist.jar, msgpack-0.5.2.jar, slf4j-api-1.6.2.jar and slf4j-jdk14-1.6.2.jar in my lib directory.
In my server application this code works fine with the same libraries.
(Hopefully) FINAL UPDATE
msgpack : 0.6.8 works on Android without any problems
msgpack-rpc : 0.7.0 works on Android with one caveat.
Specifically, you need to add the following to onCreate for API Level 8 (Android 2.2.1), and possibly lower:
java.lang.System.setProperty("java.net.preferIPv4Stack", "true");
java.lang.System.setProperty("java.net.preferIPv6Addresses", "false");
due to this bug.
If you want to see a simple example, here's a pair of projects set up for this purpose:
https://github.com/mikkoz/msgpack-android-test-server/tree/master/msgpack-android-test-server
https://github.com/mikkoz/msgpack-android-test-client/tree/master/msgpack-android-test-client
Previous Versions
UPDATE: as of 0.6.7 msgpack should be compatible with Android (there is a small dependency exclusion issue). Check the text below for msgpack-rpc (which also might be adapted in the future).
NOTE: If you're also using msgpack-rpc, you need to do the following steps:
Download the msgpack-rpc source from git://github.com/msgpack/msgpack-rpc.git (specifically, the "java" folder).
Change the main msgpack artifact version to the one you've built.
In org.msgpack.rpc.loop.netty.NettyEventLoop, change the NioClientSocketChannelFactory to OioClientSocketChannelFactory(getWorkerExecutor()).
Build the MessagePack-RPC in the same way as in the case of the main MessagePack JAR (see Step 11 above).
The NettyEventLoop replacement is due to this issue:
http://markmail.org/message/ypa3nrr64kzsyfsa .
Important: I've only tested synchronous communication. Asynchronous might not work.
And here's the reason for msgpack not working with Android prior to 0.6.7:
The reason for the error is that MessagePack uses several java.beans classes that are not included in the Android SDK. You're probably using the MessagePackBeans annotation.
This is a similar problem to the one described here, for which the general solution is outlined here. Unfortunately, in our case it requires a rebuild of msgpack. Here's what I did (you can almost certainly skip Steps 5 and 8, but I haven't tried it that way) :
Download the MessagePack source from https://github.com/msgpack/msgpack-java.git.
Import the MessagePack source as a project in your IDE.
Download the Apache Harmony source for the relevant packages from http://svn.apache.org/repos/asf/harmony/enhanced/java/trunk/classlib/modules/beans/src/main/java .
Copy these packages into your MessagePack project's src/main/java folder:
java.beans
java.beans.beancontext
org.apache.harmony.beans
org.apache.harmony.beans.internal.nls
In your MessagePack project, remove the following classes:
PropertyChangeListener
IndexedPropertyChangeEvent
PropertyChangeEvent
PropertyChangeListenerProxy
PropertyChangeSupport
Rename the java.beans packages to something different, e.g. custom.beans .
Change all java.beans references to the renamed ID, so again e.g. custom.beans. This applies especially to BeansFieldEntryReader (this class is the reason for the original error).
Change the custom.beans references for the five classes you removed in Step 5 back to java.beans.
In the org.apache.harmony.beans.internal.nls.Messages class, comment out the method setLocale, and remove the imports associated with it.
Remove all classes that still have errors, except Encoder. In that class, comment out all references to the classes you've removed. You should now have an error-free project.
Build the MessagePack JAR:
If you're using Maven, change the version in the pom.xml to something unique, run Maven build with the install goal, then add the dependency in your Android project with that version.
If you're not using Maven, you have to run the jar goal for Ant with the included build.xml. Replace the msgpack JAR in your Android project with this one.
If you're publishing your app, remember to include the relevant legal notice for Apache Harmony. It's an Apache License, just like MessagePack.
That should do it. Using your example code, and my own data class, I was successfully able to pack and unpack data.
The entire renaming ritual is due to the fact that the DEX compiler complains about java.* package naming.
There is a critical msgpack bug saying data packed with msgpack will get corrupted on the Dalvik VM. http://jira.msgpack.org/browse/MSGPACK-51
There is an ongoing effort by #TheTerribleSwiftTomato and the MessagePack core team to get MessagePack working on Android, please see the related GitHub issue. The fix mentioned in #TheTerribleSwiftTomato's answer is to be found here.
Update
I've managed to get it at least running on Android by (painstakingly) adding all the necessary javassist Classes which are currently required for the build to succeed. An extra 600KB gain in size, yet at least it seems to work. All in all, it appears to be working to some extent on Android, eventually check out the lesser-known resources about Message Pack such as its User Group and its Wiki for more information.
On a side-note, be sure to use a HTTP Request Library (such as LoopJ's Android Async HTTP or Apache's HttpClient) which can handle binary data.
Last but not least you can ping me if there is interest in this jar which makes MessagePack seemingly work on Android – credits go out of course to #TheTerribleSwiftTomato who supplied the fix above!
I suggest you write this in the main proguard-rules file-
-dontwarn org.msgpack.**
-keep class org.msgpack.** { *; }

Categories