We Keep Coding

Hadoop Jar FileSystemNotFoundException

I am running a hadoop cluster and attempting to run a mapreduce job using a jar file I send to the cluster. The problem is that on certain nodes it succeeds and others it fails. The error I am seeing on certain nodes is:
Exception in thread "main" java.nio.file.FileSystemNotFoundException
at com.sun.nio.zipfs.ZipFileSystemProvider.getFileSystem(ZipFileSystemProvider.java:183)
at com.sun.nio.zipfs.ZipFileSystemProvider.getPath(ZipFileSystemProvider.java:169)
at java.nio.file.Paths.get(Paths.java:154)
at customLib.readFile(CustomClass2.java:254)
at customLib.access$000(CustomClass2.java:210)
at customLib.getFieldOrder(CustomClass2.java:591)
at customLib.run(CustomClass1.java:177)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at customLib.main(CustomClass1.java:136)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:508)
at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
I see that others have had similar problems such as this questions with solutions and links to other questions and soltuions: java.nio.file.FileSystemNotFoundException when getting file from resources folder. However, none of these solutions could explain why it would work on one node and not another... this must mean there is some config I have wrong somewhere.
I have checked all the hadoop xml config files as well as the $CLASSPATH, $PATH, and java runtime version and everything seems to be the same. If anyone has any other ideas on what I could check, that would be greatly appreciated - thanks.
EDIT: To clarify, the file it is trying to get is a resource that is packaged within the JAR.

This answer helped me out: https://stackoverflow.com/a/14607953/7969573. It looks like the nodes that were failing had a copy of the jar with the class I was running in the #HADOOP_HOME/share/hadoop/mapreduce/ folder which was in my CLASSPATH. Deleting the jar from the nodes fixed the issue as I run the jar from a shell invoked by my application. I could have just updated that jar and it would have worked as well, but it made more sense to delete it under my current workflow's architecture.

Jar hell for missing classes folder when running ESRestTestCase

So I'm running into a Jar hell problem when trying to run individual integration tests, using -Dtest=, that runs as a ESRestTestCase (ESTestCase). The issue here seems to be that some elasticsearch classpath validation class requires target/classes to exist. However, this project is only for testing so that requirement doesn't make sense.
This happened with Elasticsearch 7.0.0 and Java 1.8.0.251. Not sure if this is a problem with later versions.
java.lang.RuntimeException: found jar hell in test classpath
at org.elasticsearch.bootstrap.BootstrapForTesting.<clinit>(BootstrapForTesting.java:98)
at org.elasticsearch.test.ESTestCase.<clinit>(ESTestCase.java:229)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:623)
Caused by: java.nio.file.NoSuchFileException: <MY PROJECT FOLDER PATH HERE>/target/classes
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
at java.nio.file.Files.readAttributes(Files.java:1737)
at java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219)
at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:322)
at java.nio.file.Files.walkFileTree(Files.java:2662)
at java.nio.file.Files.walkFileTree(Files.java:2742)
at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:199)
at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:86)
at org.elasticsearch.bootstrap.BootstrapForTesting.<clinit>(BootstrapForTesting.java:96)
... 4 more

At first I tried various methods for making sure the classes folder was still created. I ran into new problems that seemed fixed in later versions so I upgraded to Elasticsearch 7.6.2 (And lucene 8.4.0). Upgrading seemed to remove the need for the folder to exist but triggered other issues (similar to this). I worked around such issues with -Dtests.security.manager=false. Then I got thread leak issues which I "solved" by setting #ThreadLeakScope(ThreadLeakScope.Scope.NONE) on the test class.
Then finally it seemed to work when running maven in command line for individual tests. But when trying to run the maven task with debugger in Intellij the debugger does not seem to attach.
So I try to run it as JUnit templated test in Intellij. Here I get jar hell when running it but it works to debug. The jar hell was later solved by setting idea.no.launcher=true in idea.properties.
Alright, good enough for now. These workarounds are far from ideal but it works for my current purposes. A lot of things will change before moving into mainline anyway. And maybe this will help someone else. Hopefully I can post a better solution later.

Why is my IDE looking in the wrong dependency folder for a needed class?

I'm trying to run a script that was working fine until I changed the home folder.
Once I changed the home folder I ran into a few issues but I was able to resolve them. The only thing I have not been able to resolve is the following error.
Exception in thread "main" java.lang.NoClassDefFoundError: org/openqa/selenium/remote/internal/JsonToWebElementConverter
at BrowserStackAndroid.main(BrowserStackAndroid.java:43)
Caused by: java.lang.ClassNotFoundException: org.openqa.selenium.remote.internal.JsonToWebElementConverter
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
... 1 more
This is an interesting error that, after some research, I have not been able to wrap my head around.
The class that it is searching for is in the org.selenium.remote dependency. Except that it's not in the internal folder but the remote folder.
So the class that it is searching for is JsonToWebElementConverter.
This class is located in the following path
org/openqa/selenium/remote/JsonToWebElementConverter
The folder that the IDE (or Maven?) is looking into is
org/openqa/selenium/remote/internal
Which only has the class WebElementToJsonConverter
I've tried reinstalling the library via maven in several different ways, using different dependencies, etc but haven't been able to fix this issue.
I've never come across an issue like this before so I'm really excited to see what the solution is going to be.
Notes:
The error is being thrown because of the following line of code
AndroidDriver<AndroidElement> driver = new AndroidDriver<AndroidElement>(new URL("https://" + userName + ":" + accessKey + "#hub-cloud.browserstack.com/wd/hub"), caps);
This project worked perfectly fine before but as soon I changed the directory of the project stopped working.

This is happening as you have same class or library with in 2 dependencies. Simply removing selenium based dependencies will resolve the issue as all selenium dependencies already comes with Appium java client. For classpath you can check at java docs
So the way to set class path via cmd (Navigate to project root folder)
set CLASSPATH= target/org/openqa/selenium/remote/internal/JsonToWebElementConverter
Note: for maven project all compiled java files should be in target folder. If you are facing same issue you can add missing class inclasspath destination should be target folder as explained in java docs

java.lang.NoClassDefFoundError when trying to instantiate class from jar

I did found quite a lot about this error, but somehow none of the suggested solutions resolved the problem.
I am trying to use JNA bindings for libgphoto2 under Ubuntu in Eclipse (moderate experience with Java on Eclipse, none whatsoever on Ubuntu, I'm afraid). The bindings in question I want to use are here:
http://angryelectron.com/projects/libgphoto2-jna/
I followed the steps described on that page, and made a simple test client that failed with the above error. So I reduced the test client until the only thing I tried to do was to instantiate a GPhoto2 object, which still produced the error. The test client looks like this:
import com.angryelectron.gphoto2.*;
public class test_class
{
public static void main(String[] args)
{
GPhoto2 cam = new GPhoto2();
}
}
The errors I get take up considerably more space:
Exception in thread "main" java.lang.NoClassDefFoundError: com/sun/jna/Structure
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at test_class.main(test_class.java:12)
Caused by: java.lang.ClassNotFoundException: com.sun.jna.Structure
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 13 more
libgphoto2 itself is installed, it runs from the command line, I even have the development headers and am able to call GPhoto2 functions from python, so the problem can't be located there.
When looking at the .class files in Eclipse, however, they didn't have any definitions. So I figured that might be the problem, especially since there was an error when building the whole thing with ant (although the .jar was succesfully exported, from what I could make out the error concerned only the generation of documentation).
So I loaded the source into eclipse and built the .jar myself. At this occasion Eclipse stated there were warnings during the build (though no errors), but didn't show me the actual warnings. If anyone could tell me where the hell the build log went, that might already help something. I searched for it everywhere without success, and if I click on "details" in eclipse it merely tells me where the warnings occured, not what they were.
Be that as it may, a warning isn't necessarily devastating, so I imported the resulting Jar into the above client. I checked the .class files, this time they contained all the code. But I still get the exact same list of errors (yes, I have made very sure that the old library was removed from the classpath and the new ones added. I repeated the process several times, just in case).
Since I don't have experience with building jars, I made a small helloworld jar, just to see if I could call that from another program or if I'd be getting similar errors. It worked without a hitch. I even tried to reproduce the problem deliberately by exporting it with various options, but it still worked. I tried re-exporting the library I actully need with the settings that had worked during my experiment, but they still wouldn't run. I'm pretty much stuck by now. Any hints that help me resolve the problem would be greatly appreciated.

In addition to what #Paul Whelan has said. You might have better luck by just get the missing jar directly.
Get the missing library here, set the classpath and then re-run the application again and see whether it will run fine or not.

What version of java are you using com/sun/jna/Structure may only work with certain JVMs.
In general, packages such as sun., that are outside of the Java platform, can be different across OS platforms (Solaris, Windows, Linux, Macintosh, etc.) and can change at any time without notice with SDK versions (1.2, 1.2.1, 1.2.3, etc). Programs that contain direct calls to the sun. packages are not 100% Pure Java.
More details here

Your jar needs a MANIFEST.MF which tells your application where the library is found. Create the file in you project root-directory in eclipse and add the following lines:
Manifest-Version: 1.0
Class-Path: <PATH_TO_LIB__CAN_BE_RELATIVE>.jar // e.g Class-Path: ../test.jar
<empty line>
Right-click your project in eclipse, go to Export->next->next->next->Use existing manifest from workspace, select it and click on finish. This should work.
Another solution is to compile the classes into the jar itself with Maven.

javax.mail.NoSuchProviderException: No provider for smtps

I'm trying to set up my Java project to be able to send e-mail (via g-mail, if it matters) and am getting "javax.mail.NoSuchProviderException: No provider for smtps" every time I try to run the following line (which is copy/paste from their example).
Transport transport = session.getTransport("smtps");
I've looked around and found that this is generally thrown because you don't have the mail.jar included in your classpath, but I do in fact have the mail.jar included. Since I am running JDK 1.6 I do not need to include the activation.jar according to the FAQ here (http://www.oracle.com/technetwork/java/javamail/faq-135477.html#classpath). Further, the activation.jar does not seem to be present in version 1.4.7 of javamail.
Just in case something got corrupted, I re-downloaded the entire zip from oracle's website, extracted it and added the jar fresh (after deleting the old jar) and I am still getting the same error. Any thoughts as to what the issue could be at this point?
EDIT: Here is the full stack trace that is being printed:
javax.mail.NoSuchProviderException: No provider for smtps
at javax.mail.Session.getProvider(Session.java:433)
at javax.mail.Session.getTransport(Session.java:627)
at javax.mail.Session.getTransport(Session.java:608)
... my code that calls getTransport() ...
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
... more of my code ...
at java.lang.Thread.run(Thread.java:662)

Just in case anyone makes the same mistake as I have: you have to use lowercase letters for the protocol resolving to work. If you type SMTP as a protocol name instead of smtp you will get the NoSuchProviderException. It most likely works the same way for all other providers.

When you use Mail API make sure that which protocol are you expecting to work with?
In this case you are missing smtp.jar with your eclipse project.
there are several jars for different protocols are available in mail api.
EX: dsn.jar , gimap.jar , imap.jar ,mailapi.jar ,pop3.jar ,smtp.jar

So it turns out that the issue was that an outdated version of mail.jar was included in a project that I was referencing and, upon updating that copy of the mail.jar, the issue was resolved.
For future reference, is there any way to log or provide visibility on such jar conflicts?
https://confluence.atlassian.com/confkb/cannot-send-email-due-to-javax-mail-nosuchproviderexception-smtp-error-154079.html

Make sure that you have the javax.mail.jar in you build path. If you are using eclipse you may have to refresh or right click your project in the files explorer, select configure build path, add external JAR and then add it to the build path.
Send email using java gives working code (I've tested it) in case you just want to look over yours.
If that is not the problem, or you are not using eclipse, a stack trace would be nice

JavaMail asks the ClassLoader for the configuration file that configures the protocol providers. If the ClassLoader doesn't work correctly, JavaMail won't be able to find the configuration file. There's an incompatibility between the way the OSGi ClassLoaders work and what JavaMail expects, which can cause this problem. If you're running your application in Eclipse itself, that might explain this problem. Another common cause of this problem is importing the mail.jar file into your project in such a way that the class files are extracted from the jar file and included in your application, but the configuration files are left behind.
Try running your program from the command line using the "java" command and with the mail.jar file in your CLASSPATH.

for the benefit of others with a similar problem as I had.
make sure you remember to set
properties.put("mail.transport.protocol", "smtp"));
instead of
properties.put("mail.transport.protocol", "SMTP"));

I had mail.jar and activation.jar in tomcat's lib directory and mailapi.jar in application's lib directory. Application was reading mailapi.jar during runtime, since mailapi.jar is light weight version of mailing api and it needs smtp.jar that why application was throwing smtp exception. So, if you want to get rid of this exception,
Please resolve conflict between mail.jar and mailapi.jar by:
Removing mailapi.jar (if it is there in the classpath).
OR just keep
one pair of mailapi.jar and smtp.jar in the classpath and remove
mail.jar.
OR just keep one pair of mail.jar and activation.jar in
the classpath (clean approach).
(FYI: I searched file system to find out mail related jar files and got to know that I have conflicting jar file in the following path (added by gradle in classpath) C:\workspace.metadata.plugins\org.eclipse.wst.server.core\tmp0\wtpwebapps\testapp\WEB-INF\lib)

I got the same issue. i resolved it by adding the correct dependency in maven.
group id - javax.mail
artifactory - mail
version 1.4.7
if you are doing, without maven build, add the correct library for javax.mail

try
Transport transport = session.getTransport("smtp");