We Keep Coding

javassist throws ClassNotFoundException when loading external classes from jar

I'm trying load an external jar file with javassist & call its main method at runtime, however when i try to do this with the below code:
File file = new File("C:\\Users\\MainPC\\Desktop\\test.jar");
ClassPool cp = ClassPool.getDefault();
cp.insertClassPath(file.getAbsolutePath());
Class<?> MainClass = cp.get("TestPackage.MainClass").toClass();
MainClass.getMethod("main", String[].class).invoke(null, new Object[] {args});
It throws the following exception:
Exception in thread "main" java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at ReflectionTests.main(ReflectionTests.java:99)
Caused by: java.lang.NoClassDefFoundError: TestPackage/OtherClass
at TestPackage.MainClass.main(Unknown Source)
... 5 more
Caused by: java.lang.ClassNotFoundException: TestPackage.OtherClass
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:602)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
... 6 more
When i attempt the same thing only using the built in java reflection api it works with no problems:
File file = new File("C:\\Users\\MainPC\\Desktop\\test.jar");
URLClassLoader cl = new URLClassLoader(new URL[] {new URL("jar:file:"+file.getAbsoluteFile()+"!/")});
Class<?> clazz = cl.loadClass("TestPackage.MainClass");
clazz.getMethod("main", String[].class).invoke(null, new Object[] {args});
(the above throws no exceptions & calls the main method of the jar file as expected)
This leaves me to believe i'm doing something wrong in javassist (specifically with the loading of the jar file classes). Can someone explain to me what it is?
I should mention the jar file only contains 2 classes: MainClass.class & OtherClass.Class. Both reside in a package called TestPackage. It seems the error has something to do with the MainClass class not being able to find the OtherClass class, when javassist loads it.

The problem was that when i call ct.toClass() it only exposes the class itself to my runtimes classloader (not the entire classpool's classpath). When i then later attempt to invoke the main method of this class, my runtimes classloader tries to execute the part that loads the other class which it obviously doesn't know about and so throws a ClassNotFoundException.
The solution is to use the javassist provided classloader (javassist.Loader) which takes a classpool as argument in the constructor and then is able to load & resolve classes from the classpools classpath properly.
Here's a working code example of what i was trying to achieve:
File file = new File("C:\\Users\\MainPC\\Desktop\\test.jar");
ClassPool cp = ClassPool.getDefault();
cp.insertClassPath(file.getAbsolutePath());
Loader loader = new Loader(cp);
Class<?> MainClass = loader.loadClass("TestPackage.MainClass");
MainClass.getMethod("main", String[].class).invoke(null, new Object[] {args});

java.lang.NoClassDefFoundError when trying to load class from JAR

I am working on a project that is supposed to parse texts from PDF files.
Having multiple dependencies I have decided to build a combined JAR with all the dependencies and the classes.
However, when I build JAR including dependencies via Intellij IDEA even though the JAR file is added properly and I can import the class the program throws NoClassDefFoundError (Please refer to the screenshot).
Firstly, I thought the jar wasn't in the classpath. However, even if I add -cp TessaractPDF.jar through VM Options the class still get undetected.
I think it is worth to mention that, everything works smoothly if I build JAR without dependencies and add the dependencies manually.
What should I do?
Exception in thread "main" java.lang.NoClassDefFoundError: me/afifaniks/parsers/TessPDFParser
at Test.main(Test.java:20)
Caused by: java.lang.ClassNotFoundException: me.afifaniks.parsers.TessPDFParser
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 1 more
Code Snippet:
import me.afifaniks.parsers.TessPDFParser;
import java.io.IOException;
import java.util.HashMap;
public class Test {
public static void main(String[] args) throws IOException {
System.out.println(System.getProperty("java.classpath"));
HashMap<String, Object> arguments = new HashMap<>();
arguments.put("imageMode", "binary");
arguments.put("toFile", false);
arguments.put("tessDataPath", "/home/afif/Desktop/PDFParser/tessdata");
TessPDFParser pdfParser = new TessPDFParser("hiers15.pdf", arguments);
String text = (String) pdfParser.convert();
System.out.println(text);
}
}

cmu sphinx4 java - Runtime exception caused by FileNotFoundException

I have recently made a Java project with Sphinx4. I found this code online, and I slimmed it down to this to test if Sphinx4 was working:
public class App
{
private static final String ACOUSTIC_MODEL =
"resource:/edu/cmu/sphinx/models/en-us/en-us";
private static final String DICTIONARY_PATH =
"resource:/edu/cmu/sphinx/models/en-us/cmudict-en-us.dict";
public static void main(String[] args) throws Exception {
Configuration configuration = new Configuration();
configuration.setAcousticModelPath(ACOUSTIC_MODEL);
configuration.setDictionaryPath(DICTIONARY_PATH);
configuration.setGrammarName("dialog");
LiveSpeechRecognizer jsgfRecognizer =
new LiveSpeechRecognizer(configuration);
jsgfRecognizer.startRecognition(true);
while (true) {
String utterance = jsgfRecognizer.getResult().getHypothesis();
if (utterance.startsWith("hello")) {
System.out.println("Hello back!");
}
else if (utterance.startsWith("exit")) {
break;
}
}
jsgfRecognizer.stopRecognition();
}
}
However, it gave me this error:
Exception in thread "main" java.lang.RuntimeException: Allocation of search manager resources failed
at edu.cmu.sphinx.decoder.search.WordPruningBreadthFirstSearchManager.allocate(WordPruningBreadthFirstSearchManager.java:247)
at edu.cmu.sphinx.decoder.AbstractDecoder.allocate(AbstractDecoder.java:103)
at edu.cmu.sphinx.recognizer.Recognizer.allocate(Recognizer.java:164)
at edu.cmu.sphinx.api.LiveSpeechRecognizer.startRecognition(LiveSpeechRecognizer.java:47)
at com.weebly.controllingyourcomputer.bartimaeus.App.main(App.java:27)
Caused by: java.io.FileNotFoundException:
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at java.io.FileInputStream.<init>(FileInputStream.java:93)
at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)
at sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188)
at java.net.URL.openStream(URL.java:1038)
at edu.cmu.sphinx.linguist.language.ngram.SimpleNGramModel.open(SimpleNGramModel.java:403)
at edu.cmu.sphinx.linguist.language.ngram.SimpleNGramModel.load(SimpleNGramModel.java:277)
at edu.cmu.sphinx.linguist.language.ngram.SimpleNGramModel.allocate(SimpleNGramModel.java:114)
at edu.cmu.sphinx.linguist.lextree.LexTreeLinguist.allocate(LexTreeLinguist.java:334)
at edu.cmu.sphinx.decoder.search.WordPruningBreadthFirstSearchManager.allocate(WordPruningBreadthFirstSearchManager.java:243)
... 4 more
I thought it might be something about it not being able to find the paths for ACOUSTIC_MODEL or DICTIONARY_PATH, so I changed the resource: strings to things like %HOME%\\Downloads\\sphinx4-5prealpha-src\\sphinx4-5prealpha-src\\sphinx4-data\\src\\main\\resources\\edu\\cmu\\sphinx\\models\\en-us or paths with forward slashes or with C:\Users\Username\... but none of the paths worked. I know the paths exist because I copy and pasted them from the properties window of the actual resources.
So my question is: is it some of the code that I deleted from the original source code that is causing this error, is it something wrong with the paths, or is it entirely different?
EDIT
By the way, I am using Maven to build my project. I added the dependencies specified on the Sphinx4 website to my pom.xml, but it didn't work (it didn't recognize imports such as edu.com.sphinx.xxx) so I downloaded the JARs from the website they said to download them from and added them to my projects "Libraries" in my Java Build Path in Eclipse.

is it some of the code that I deleted from the original source code that
is causing this error
Yes, you deleted too much.
To recognize with grammar you need to make three calls:
configuration.setGrammarPath(GRAMMAR_PATH);
configuration.setGrammarName(GRAMMAR_NAME);
configuration.setUseGrammar(true);

How to run a java class from a jar file dynamically

I'm working on a java project that needs a third-party java program running as a server to work.
Normally, I'd do:
java -cp jarfile1.jar:jarfile2.jar className arg1 arg2
And then I'd run my java code. This way it works.
I'd like to know if there is any way to, including the two .jars required into my project, run the class directly from my code instead of having to manually start it.
I've tried to use URLClassLoader as I saw in some examples, but either I'm doing it wrong or none cover this specific use case.
URLClassLoader classLoader = URLClassLoader.newInstance(new URL[]{new URL("file:///tmp/jarfile1.jar"),new URL("file:///tmp/jarfile2.jar")});
Class<?> cls = classLoader.loadClass("className");
Method method = cls.getDeclaredMethod ("main");
Object instance = cls.newInstance();
Object result = method.invoke (instance);
yields
Exception in thread "main" java.lang.NoClassDefFoundError: alice/tuprolog/lib/InvalidObjectIdException
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2615)
at java.lang.Class.getDeclaredMethod(Class.java:2007)
at pkg1.MainClass.main(MainClass.java:54)
Caused by: java.lang.ClassNotFoundException: alice.tuprolog.lib.InvalidObjectIdException
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 4 more
Please note that I copied the .jars to /tmp to isolate the failure cause. The files exist and are accessible.
How can I make that run the class as specified above within java code?
Thanks!

If the class exists in a different ClassLoader, you need to use reflection to get to it:
ClassLoader classLoader = new URLClassLoader(
new URL[] { firstJarURL, secondJarURL });
String[] args = { arg1, arg2 };
try {
Class<?> mainClass = classLoader.loadClass("com.somepackage.ClassName");
mainClass.getMethod("main", String[].class).invoke(null, args);
} catch (ReflectiveOperationException e) {
throw new RuntimeException(e);
}

I finally fixed it! Things done:
I forgot to add the second jar to the classpath of the project (duh!)
Since everything was on the project classpath, I just reused the current classloader
Final working code:
ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
Class<?> cls = classLoader.loadClass("className");
Method method = cls.getDeclaredMethod("main", String[].class);
Object instance = cls.newInstance();
Object result = method.invoke(null, (Object)args);
Thanks everyone and specially to VGR and Joop Eggen in the comments for pointing out the error with the second jar!
EDIT: As JB Nizet pointed out in the comments, calling the class's main() method directly is simpler:
className.main(args);
And you're done

parsing json input in hadoop java

My input data is in hdfs. I am simply trying to do wordcount but there is slight difference.
The data is in json format.
So each line of data is:
{"author":"foo", "text": "hello"}
{"author":"foo123", "text": "hello world"}
{"author":"foo234", "text": "hello this world"}
I only want to do wordcount of words in "text" part.
How do I do this?
I tried the following variant so far:
public static class TokenCounterMapper
extends Mapper<Object, Text, Text, IntWritable> {
private static final Log log = LogFactory.getLog(TokenCounterMapper.class);
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(Object key, Text value, Context context)
throws IOException, InterruptedException {
try {
JSONObject jsn = new JSONObject(value.toString());
//StringTokenizer itr = new StringTokenizer(value.toString());
String text = (String) jsn.get("text");
log.info("Logging data");
log.info(text);
StringTokenizer itr = new StringTokenizer(text);
while (itr.hasMoreTokens()) {
word.set(itr.nextToken());
context.write(word, one);
}
} catch (JSONException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
But I am getting this error:
Error: java.lang.ClassNotFoundException: org.json.JSONException
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:820)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:865)
at org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:199)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:719)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.mapred.Child.main(Child.java:249)

Seems you forgot to embed the JSon library in your Hadoop job jar.
You can have a look there to see how you can build your job with the library:
http://tikalk.com/build-your-first-hadoop-project-maven

There are several ways to use external jars with your map reduce code:
Include the referenced JAR in the lib subdirectory of the submittable JAR: The job will unpack the JAR from this lib subdirectory into the jobcache on the respective TaskTracker nodes and point your tasks to this directory to make the JAR available to your code. If the JARs are small, change often, and are job-specific this is the preferred method. This is what #clement suggested in his answer.
Install the JAR on the cluster nodes. The easiest way is to place the JAR into $HADOOP_HOME/lib directory as everything from this directory is included when a Hadoop daemon starts. Note that a start stop will be needed to make this effective.
TaskTrackers will be using the external JAR, so you can provide it by modifying HADOOP_TASKTRACKER_OPTS option in the hadoop-env.sh configuration file and make it point to the jar. The jar needs to be present at the same path on all the nodes where task-tracker runs.
Include the JAR in the “-libjars” command line option of the hadoop jar … command. The jar will be placed in distributed cache and will be made available to all of the job’s task attempts. Your map-reduce code must use GenericOptionsParser. For more details read this blog post.
Comparison:
1 is a legacy method but discouraged because it has a large negative performance cost.
2 and #3 are good for private clusters but pretty lame practice as you cannot expect end users to do that.
4 is the most recommended option.
Read the main post from Cloudera).