I have a Scala Akka Application which connects to HBase (currently CDP earlier HDP) deployed on rancher; Never faced any trouble when connecting to HDP hbase; Since recent HDP to CDP change, with the same image we are getting no method found on one of the dependency's class in one of the container, where as another container of same image connects to hbase properly.; even though the jar exists in the same image and classpath also.
one of the noticeable difference is change in the order of classpath.
Does change in the classpath order will effect the jars availability.
Does java libraries/classes would load in different order when they would hit a faster CPU cycle at startup.
What could be the reason for such "no class method found".
It certainly can, if the same class file is present in different classpath entries. For example, if your classpath is: java -cp a.jar:b.jar com.foo.App, and:
a.jar:
pkg/SomeClass.class
b.jar:
pkg/SomeClass.class
Then this can happen - usually because one of the jars on your classpath is an older version than the other, or the same but more complicated: one of the jars of your classpath contains a whole heap of different libraries all squished together and one of those components is an older version.
There are some basic hygiene rules to observe:
Don't squish jars together. If you have 500 deps, put 500 entries on your classpath. We have tools to manage this stuff, use them. Don't make striped jars, uber jars, etc.
Use dependency trackers to check if there are version difference in your dependency chain. If your app depends on, say, 'hibernate' and 'jersey', and they both depend on google's guava libraries, but hibernate imports v26 and jersey imports v29, that's problematic. Be aware of it and ensure that you explicitly decide which version ends up making it. Presumably, you'd want to explicitly pick v29 and perhaps check that hibernate also runs on v29*. If it doesn't, you have bigger problems. They are fixable (with modular classloaders), but not easily.
*) Neither hibernate nor jersey actually depend on guava, I'm just using them as hypothetical examples.
For example, if you use maven, check out the enforcer plugin. (groupId: org.apache.maven.plugins, artifactId: maven-enforcer-plugin).
My bet is that there is another version of the jar somewhere in CDP, and occasionally it is loaded before the version that you ship with your project, causing the error.
So, when your container starts, try logging from which location the conflicting class is loaded. This question might help you: Determine which JAR file a class is from
Related
Recently I've been working on report generation with Jasper. I have created a simple program to test it and when running it via IDE it did work fine.
Then I moved the (very short) class to WildFly sever application and despite having the exact same code and library generation fails with cannot find symbol. Those symbols it cannot find are JREvaluator, JRFillVariable as well as packages such as net.sf.jasperreports.engine
In so far I have confirmed that:
Project builds (meaning those classes are visible for javac, but not jvm)
jasperreports-6.13.0.jar is added to war (it's present in /WEB-INF/lib folder alongside other libraries, like gson and hibernate
jasperreports-6.13.0.jar contains the missing classes
It looks to me like the problem doesn't lie in library not being loaded or missing classes (because in testing environment it works), but like something was preventing JBoss class loader from loading those classes
Attempted (and failed) solutions
Clean and Build
Adding -Djava.awt.headless=true to VM options - this did not changed anything
Adding -Djava.awt.headless=false to VM options - also didn't change a thing, but once caused NullPointerException inside jasperreport library. For testing program - worked in both cases
Adding commons-beanutils-1.9.4.jar, commons-digester-2.1.jar, commons-collections4-4.4.jar and commons-loggin-1.2.jar - with no changes
Adding jasper-compiler-jdt-5.5.23.jar - this caused a different error, namely NoSuchMethodError for org.eclipse.jdt.internal.compiler.ICompilerRequestor and few others. This library however should not be necessary as - from what I understand - jasperreport-6.13.0.jar already contains it's compiler and separate library for compiler is not required since a long time.
What has not been attempted:
Forcing the classes to load (http://www.java2s.com/Code/Java/Reflection/Forcethegivenclasstobeloadedfully.htm)
Dynamically loading jar during Runtime or using custom class loader
Update: after looking at this answer and applying the suggestion the missing class was different. Which suggests that the dependencies inside jasperreport.jar are not being loaded properly
I have figured it out
For some reason in server project libraries used by jasperreport.jar were not loaded, but in the testing project they were (might be due to WildFly, might be due to differences between IntelliJ and NetBeans)
Here is the list of libraries, based on pom.xml file in jasperreport.jar that I have added. Some might not be necessary and the list might not be exhaustive (I basically stopped adding libraries once report started generating) but it's good enough base if someone else runs into this problem:
commons-beanutils-1.9.4.jar
itext-2.1.7.jar
poi-ooxml-4.1.1.jar
commons-collections4-4.4.jar
jcommon-1.0.23.jar
xalan-2.7.2.jar
commons-digester-2.1.jar
jfreechart-1.0.19.jar
xmpcore-5.1.3.jar
commons-logging-1.2.jar
poi-4.1.1.jar
I know at least 2 ways to get my dependencies into a Spark EMR job. One is to create a fat jar and another is to specify which packages you want in spark submit using the --packages option.
The fat jar takes quite a long time to zip up. Is that normal? ~10 minutes. Is it possible that we have it incorrectly configured?
The command line option is fine, but error prone.
Are there any alternatives? I'd like it if there (already existed) a way to include the dependency list in the jar with gradle, then have it download them. Is this possible? Are there other alternatives?
Update: I'm posting a partial answer. One thing I didn't make clear in my original question was that I also care about when you have dependency conflicts because you have the same jar with different versions.
Update
Thank you for the responses relating to cutting back the number of dependencies or using provided where possible. For the sake of this question, lets say we have the minimal number of dependencies necessary to run the jar.
Spark launcher can used if spark job has to be launched through some application with the help of Spark launcher you can configure your jar patah and no need to create fat.jar for runing application.
With a fat-jar you have to have Java installed and launching the Spark application requires executing java -jar [your-fat-jar-here]. It's hard to automate it if you want to, say, launch the application from a web application.
With SparkLauncher you're given the option of launching a Spark application from another application, e.g. the web application above. It is just much easier.
import org.apache.spark.launcher.SparkLauncher
SparkLauncher extends App {
val spark = new SparkLauncher()
.setSparkHome("/home/knoldus/spark-1.4.0-bin-hadoop2.6")
.setAppResource("/home/knoldus/spark_launcher-assembly-1.0.jar")
.setMainClass("SparkApp")
.setMaster("local[*]")
.launch();
spark.waitFor();
}
Code:
https://github.com/phalodi/Spark-launcher
Here
setSparkHome(“/home/knoldus/spark-1.4.0-bin-hadoop2.6”) is use to set spark home which is use internally to call spark submit.
.setAppResource(“/home/knoldus/spark_launcher-assembly-1.0.jar”) is use to specify jar of our spark application.
.setMainClass(“SparkApp”) the entry point of the spark program i.e driver program.
.setMaster(“local[*]”) set the address of master where its start here now we run it on loacal machine.
.launch() is simply start our spark application
What are the benefits of SparkLauncher vs java -jar fat-jar?
https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-SparkLauncher.html
https://spark.apache.org/docs/2.0.0/api/java/org/apache/spark/launcher/SparkLauncher.html
http://henningpetersen.com/post/22/running-apache-spark-jobs-from-applications
For example on Cloudera's clusters, there is some set of libraries already available on all nodes which will be available on classpath for drivers, executors.
Those are e.g. spark-core, spark-hive, hadoop etc
Versions are grouped by Cloudera, so e.g. you have spark-core-cdh5.9.0 where cdh5.9.0 suffix means that all libraries with that suffix were actually verified by Cloudera to be working together properly.
Only thing you should do is to use libraries with the same group suffix and you shouldn't have any classpath conflicts.
What that allows is:
set dependencies configured in an app as Maven's scope provided, so they will not be part of fat jar, but resolved from classpath on nodes.
You dind't write what kind of cluster you have, but maybe you you can use similliar approach.
maven shade plugin may be used to create fat jar which additionally allows to set libraries you want to include ina jar, and those not in a list are not included.
I think something similiar is described in this answer Spark, Alternative to Fat Jar but using S3 as dependency storage.
HubSpot has a (partial) solution: SlimFast. You can find an explanation here http://product.hubspot.com/blog/the-fault-in-our-jars-why-we-stopped-building-fat-jars and you can find the code here https://github.com/HubSpot/SlimFast
Effectively it stores all the jars it'll ever need on s3, so when it builds it does it without packaging the jars, but when it needs to run it gets them from s3. So you're builds are quick, and downloads don't take long.
I think if this also had the ability to shade the jar's paths on upload, in order to avoid conflicts, then it would be a perfect solution.
The fat jar indeed take a lot of time to create. I was able to optimize a little bit by removing the dependencies which were not required at runtime. But it is really a pain.
Assuming there are two jars of different library versions on a classpath, e.g.
java -cp A-2.1.jar:A-2.2.jar ...
The package and class names in the first and second jars are the same, but class implementation is different. Is it specified whether root jvm classloader will try to find a class in A-2.1 before A-2.2?
The problem is that AWS EMR adds hadoop jars to a classpath and some of its dependencies are of older versions. However, our application needs to use new versions of the same libraries, so will prepending the classpath with newer versions of libraries be enough or is shading a recommended practice in this case? http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-hadoop-config_hadoop-user-env.sh.html
From the Setting the Class Path documentation:
The order in which you specify multiple class path entries is
important. The Java interpreter will look for classes in the
directories in the order they appear in the class path variable.
That said, overriding the dependency JARs of another library will always be risky since the library provider might not have tested that combination, so you'll either need to ask them for reassurance, do your own testing, or shade/repackage the classes as you suggested.
We are developing a fairly large project and have many dependencies. Recently, we ran into an issue with a conflict between two of them, agileAPI.jar and axis.jar. Both are 3rd party libraries.
The code in question depends directly on agileAPI.jar. If I build it with just that in the build path, everything that depends on it works correctly.
As soon as I add axis.jar to the build path (just adding it, not writing code that depends on it), everything goes wrong. Some of the code that depended on the first library is now throwing exceptions from the 2nd library. It is as if the first library is picking and choosing methods to call from the 2nd library, instead of whereever it was calling them from prior.
I have code in the project that needs axis.jar directly, so I can't just remove it from the build path. I need to find a way to have these two exist in the same build path, but ignore each other.
It should be noted that both libraries coexisted prior to a recent upgrade with agile. I have been working with Oracle's support team to try and resolve this. After two weeks, though, I am looking for other sources of help.
Our environment is Windows and Eclipse, although in testing this, it also occurs when running java from a command line. Our JDK is 1.5.0_22.
Any help would be appreciated.
Thank you,
David
EDIT:
As requested, here are the stack traces that we see. The first stack trace is printed in the code beyond my control:
java.lang.NoSuchMethodError: org.apache.axis.description.OperationDesc.setStyle(Lorg/apache/axis/constants/Style;)V
at com.agile.webfs.components.fileserver.client.FileServerSoapBindingStub._initOperationDesc1(FileServerSoapBindingStub.java:37)
at com.agile.webfs.components.fileserver.client.FileServerSoapBindingStub.<clinit>(FileServerSoapBindingStub.java:20)
at com.agile.webfs.components.fileserver.client.FileServerWSServiceLocator.getFileServer(FileServerWSServiceLocator.java:43)
at com.agile.webfs.client.IFSLocator.getRemoteFileServer(IFSLocator.java:128)
at com.agile.webfs.client.IFSLocator.getConnection(IFSLocator.java:101)
at com.agile.api.pc.EJBLookup.createFileSession(EJBLookup.java:444)
at com.agile.api.pc.EJBLookup.getFileSession(EJBLookup.java:432)
at com.agile.api.pc.attachment.IFSOutputStream.getFileSession(IFSOutputStream.java:133)
at com.agile.api.pc.attachment.IFSOutputStream.copyFrom(IFSOutputStream.java:87)
at com.agile.api.pc.attachment.IFSOutputStream.copyFrom(IFSOutputStream.java:115)
at com.agile.api.pc.TableAttachment.uploadFile(TableAttachment.java:886)
at com.agile.api.pc.TableAttachment$AddFiles2Action.doSdkAction(TableAttachment.java:724)
at com.agile.api.common.SDKAction.run(SDKAction.java:23)
at com.agile.api.common.OracleAuthenticator.doAs(OracleAuthenticator.java:131)
at com.agile.api.common.Security.doAs(Security.java:54)
at com.agile.api.common.Security.doAs(Security.java:109)
at com.agile.api.pc.TableAttachment.addFiles2(TableAttachment.java:483)
at com.agile.api.pc.TableAttachment.createNewBlob2(TableAttachment.java:459)
at com.agile.api.pc.TableAttachment.doCreateServerRowWithParam(TableAttachment.java:363)
at com.agile.api.pc.Table.createTableRow(Table.java:238)
at com.agile.api.pc.TableAttachment.createTableRow(TableAttachment.java:169)
at com.agile.api.pc.Table.createRow(Table.java:202)
at com.[snip].updateAttachments(VaultImportService.java:3068)
at com.[snip].processIncorporatedFile(VaultImportService.java:926)
at com.[snip].processPdxFile(VaultImportService.java:532)
at com.[snip].processPdxRequest(VaultImportService.java:388)
at com.[snip].<init>(VaultImportService.java:299)
at com.[snip].main(VaultImportService.java:3660)
After the exception bubbles up and we catch it, the stacktrace that we print looks like:
at com.agile.api.pc.Session.createError(Session.java:1772)
at com.agile.api.pc.EJBLookup.createFileSession(EJBLookup.java:454)
at com.agile.api.pc.EJBLookup.getFileSession(EJBLookup.java:432)
at com.agile.api.pc.attachment.IFSOutputStream.getFileSession(IFSOutputStream.java:133)
at com.agile.api.pc.attachment.IFSOutputStream.copyFrom(IFSOutputStream.java:87)
at com.agile.api.pc.attachment.IFSOutputStream.copyFrom(IFSOutputStream.java:115)
at com.agile.api.pc.TableAttachment.uploadFile(TableAttachment.java:886)
at com.agile.api.pc.TableAttachment$AddFiles2Action.doSdkAction(TableAttachment.java:724)
at com.agile.api.common.SDKAction.run(SDKAction.java:23)
at com.agile.api.common.OracleAuthenticator.doAs(OracleAuthenticator.java:131)
at com.agile.api.common.Security.doAs(Security.java:54)
at com.agile.api.common.Security.doAs(Security.java:109)
at com.agile.api.pc.TableAttachment.addFiles2(TableAttachment.java:483)
at com.agile.api.pc.TableAttachment.createNewBlob2(TableAttachment.java:459)
at com.agile.api.pc.TableAttachment.doCreateServerRowWithParam(TableAttachment.java:363)
at com.agile.api.pc.Table.createTableRow(Table.java:238)
at com.agile.api.pc.TableAttachment.createTableRow(TableAttachment.java:169)
at com.agile.api.pc.Table.createRow(Table.java:202)
at com.[snip].updateAttachments(VaultImportService.java:3068)
at com.[snip].processIncorporatedFile(VaultImportService.java:926)
at com.[snip].processPdxFile(VaultImportService.java:532)
at com.[snip].processPdxRequest(VaultImportService.java:388)
at com.[snip].<init>(VaultImportService.java:299)
at com.[snip].main(VaultImportService.java:3660)
In both cases, the line "at com.agile.api.pc.Table.createRow(Table.java:202)" is the agileAPI call that I am making. I have removed our package structure, as it identifies the company that I work for. They value privacy and security.
I'd advise you to check these two things first:
Open the axis.jar file with some zip utility, like 7-Zip or WinRar. See if there's a folder called "services" in the META-INF folder in the jar. If there is, it's possible that the axis.jar file specifies implementations for specific interfaces that somehow don't interoperate with agileAPI. Also do the same for agileAPI.jar, since it might itself declare an interface implementation that axis doesn't like.
Open both agileAPI.jar and axis.jar with a zip utility, then check if there's packages with the same name. If there's none, it won't be a naming conflict. If there's one or more, open the corresponding folders and do the same check recursively. If you end up with at least one class with the same name in the same package across the two jars, it's probably a naming conflict.
That should catch the most obvious issues. If none of this is the case, we'll need to look deeper.
A way to solve such classpath issues is to use a module system such as OSGi or the NetBeans Platform module system where each module has its own classloader.
I have a project which I want to add plugins. I have all the interfaces/factories/etc. setup (my gateway interface is called ApplicationMonitorFactory), I just need to make a way to locate/activate the plugin. My configuration file is a java properties file.
I think what I need to do is:
find a good way to specify a set of one or more plugins
for each plugin, run it
1. find a good way to specify a set of one or more plugins
something like:
application.plugins=foo-monitor.jar,bar-monitor.jar
I think maybe it's just best to specify a list of jar files; for each jar file specified, the implication is that it contains one or more classes which implement ApplicationMonitorFactory, and these are the ones that will be instantiated. (I might also add an annotation #ApplicationMonitorPlugin so that a .jar file can have a test ApplicationMonitorFactory that does not get instantiated)
Does this sound reasonable?
2. for each plugin, run it
I did this once a while back, and if I remember right I think I need to use a custom classloader to add the appropriate .jar file to the classpath dynamically. Or is there an easier way?
Could I suggest using OSGI instead? If it's a serverside project, something like Apache Karaf gives you quite a lot out of the box in terms of plugin deployment and specification.
To answer the questions based on what you have at the moment:
1. find a good way to specify a set of one or more plugins
The properties file approach is fine. You may want to just be able to drop plugins into a folder that you monitor if you want hot deploy. Just having 1 jar file for a plugin does limit plugin developers to packaging all of their dependencies into a single jar file (maven shade plugin is useful for this). The annotation approach should work (the approach that Servlet 3.0 uses). Using OSGI, you'd have a manifest file with a Bundle-Activator property that would reference the plugin class that should be instantiated.
2. for each plugin, run it
Yes, you would need to fire up a class loader for the Jar files. This is where things get a bit hairier. It's easy enough to do but Class loading has all sorts of gotcha's. This is where OSGI would really help, even though it is a bit of an upfront cost.