We Keep Coding

Display Java console output in R using rJava

I have created the following Java class and saved it as Test.java, then compiled into Test.class on the command line using javac Test.java:
public class Test {
public Test() {
}
public double power(double number) {
System.out.println("calculating...");
return number * number;
}
}
Furthermore, I have created the following R script and saved it as test.R:
library("rJava")
.jinit(classpath = getwd())
test <- .jnew("Test")
.jcall(test, "D", "power", 3)
When I execute it, for example using R CMD BATCH test.R on the command line, I get the following output, which is what I want:
calculating...
[1] 9
However, when I wrap this script in a Markdown document and compile it using knitr, I lose the message that is printed about the calculation taking place. For example, I save the following script in test.Rmd and compile it using RStudio:
```{r echo=TRUE, warning=TRUE, results='show', message=TRUE}
library("rJava")
.jinit(classpath = getwd())
test <- .jnew("Test")
.jcall(test, "D", "power", 3)
```
This only returns the following output, without the message:
## [1] 9
I read somewhere that the reason is that System.out.println in Java writes to stdout, and whether this is shown in the R console or not depends on the interpreter. For example, the output is shown on Unix systems but not on Windows or in knitr.
My questions:
Is the above interpretation correct?
How can I reliably capture or display the output of System.out.println in R, irrespective of operating system or interpreter?
If that's not possible, what is a better way of designing status messages about the current calculations and progress in Java, such that R can display these messages?
Thanks!

I'll take a stab at answering my own question... Looks like the RJava folks actually offer a built-in solution (thanks Simon Urbanek if you read this). On the side of the Java code, there is the LGPL-licensed JRI.jar, which is delivered with rJava (look at the jri sub-directory in the rJava package directory in your local R library path) and which can be copied/extracted into the Java library path. It's only 82kb, so fairly light-weight.
JRI offers a replacement of the default print stream in Java. Essentially, you redirect the system output into an RConsoleOutputStream provided by JRI. The code in my question above can be modified as follows to print to the R console instead of stdout.
import java.io.PrintStream;
import org.rosuda.JRI.RConsoleOutputStream;
import org.rosuda.JRI.Rengine;
public class Test {
public Test() {
Rengine r = new Rengine();
RConsoleOutputStream rs = new RConsoleOutputStream(r, 0);
System.setOut(new PrintStream(rs));
}
public double power(double number) {
System.out.println("calculating...");
return number * number;
}
}

Hadoop Hive UDF with external library

I'm trying to write a UDF for Hadoop Hive, that parses User Agents. Following code works fine on my local machine, but on Hadoop I'm getting:
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to execute method public java.lang.String MyUDF .evaluate(java.lang.String) throws org.apache.hadoop.hive.ql.metadata.HiveException on object MyUDF#64ca8bfb of class MyUDF with arguments {All Occupations:java.lang.String} of size 1',
Code:
import java.io.IOException;
import org.apache.hadoop.hive.ql.exec.UDF;
import org.apache.hadoop.hive.ql.metadata.HiveException;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.*;
import com.decibel.uasparser.OnlineUpdater;
import com.decibel.uasparser.UASparser;
import com.decibel.uasparser.UserAgentInfo;
public class MyUDF extends UDF {
public String evaluate(String i) {
UASparser parser = null;
parser = new UASparser();
String key = "";
OnlineUpdater update = new OnlineUpdater(parser, key);
UserAgentInfo info = null;
info = parser.parse(i);
return info.getDeviceType();
}
}
Facts that come to my mind I should mention:
I'm compiling with Eclipse with "export runnable jar file" and extract required libraries into generated jar option
I'm uploading this "fat jar" file with Hue
Minimum working example I managed to run:
public String evaluate(String i) {
return "hello" + i.toString()";
}
I guess the problem lies somewhere around that library (downloaded from https://udger.com) I'm using, but I have no idea where.
Any suggestions?
Thanks, Michal

It could be a few things. Best thing is to check the logs, but here's a list of a few quick things you can check in a minute.
jar does not contain all dependencies. I am not sure how eclipse builds a runnable jar, but it may not include all dependencies. You can do
jar tf your-udf-jar.jar
to see what was included. You should see stuff from com.decibel.uasparser. If not, you have to build the jar with the appropriate dependencies (usually you do that using maven).
Different version of the JVM. If you compile with jdk8 and the cluster runs jdk7, it would also fail
Hive version. Sometimes the Hive APIs change slightly, enough to be incompatible. Probably not the case here, but make sure to compile the UDF against the same version of hadoop and hive that you have in the cluster
You should always check if info is null after the call to parse()
looks like the library uses a key, meaning that actually gets data from an online service (udger.com), so it may not work without an actual key. Even more important, the library updates online, contacting the online service for each record. This means, looking at the code, that it will create one update thread per record. You should change the code to do that only once in the constructor like the following:
Here's how to change it:
public class MyUDF extends UDF {
UASparser parser = new UASparser();
public MyUDF() {
super()
String key = "PUT YOUR KEY HERE";
// update only once, when the UDF is instantiated
OnlineUpdater update = new OnlineUpdater(parser, key);
}
public String evaluate(String i) {
UserAgentInfo info = parser.parse(i);
if(info!=null) return info.getDeviceType();
// you want it to return null if it's unparseable
// otherwise one bad record will stop your processing
// with an exception
else return null;
}
}
But to know for sure, you have to look at the logs...yarn logs, but also you can look at the hive logs on the machine you're submitting the job on ( probably in /var/log/hive but it depends on your installation).

such a problem probably can be solved by steps:
overide the method UDF.getRequiredJars(), make it returning a hdfs file path list which values are determined by where you put the following xxx_lib folder into your hdfs. Note that , the list mist exactly contains each jar's full hdfs path strings ,such as hdfs://yourcluster/some_path/xxx_lib/some.jar
export your udf code by following "Runnable jar file exporting wizard" (chose "copy required libraries into a sub folder next to the generated jar". This steps will result in a xxx.jar and a lib folder xxx_lib next to xxx.jar
put xxx.jar and the folders xxx_lib to your hdfs filesystem according to your code in step 0.
create a udf using: add jar ${the-xxx.jar-hdfs-path}; create function your-function as $}qualified name of udf class};
Try it. I test this and it works

Using JunitCore for tests in different projects

Background:
I am currently working on a project in eclipse that programatically executes JUnit tests that are pushed to a server.
So far everything works but I would like to know the results of the tests (specifically any failures) so I can push them out to an email. Right now the tests just output to the console but that doesn't seem to give me much output to actually use.
Right now I use the Runtime class to call the tests but that doesn't seem to have the functionality I need for getting results.
I have looked into the JUnitCore class but can't call any tests outside of the current java project.
So my main question would be how can I use JUnitCore to run junit tests in a specific JAR file? Or is there an easier way to approach this problem using a different class?
This is the only thing I've been able to get to work:
RunTests()
{
junitCore = new JUnitCore();
junitCore.run(AllTests.class);
}
But I would like to do something along the lines of this:
RunTests()
{
junitCore = new JUnitCore();
junitCore.run("C:\\$batch\\test\\hil research\\201507071307\\CommsTestRunner\\plugins\\TestSuite\\US35644.class");
}
I would appreciate any suggestions to this problem I am having. I'm an EE and was just introduced to java last month so this has been quite the challenge for me.

JUnitCore expects to read loaded classes, not class files in a JAR. Your real question is likely how to load the JAR (or directory of .class files) so it can be run.
Poke around with URLClassLoader; once you've amended the classpath appropriately, you can get a Class out of findClass and pass it into the JUnitCore methods you've found.

Since the tests might have classes that are also used by your server (but not necessarily at the same version) I would suggest not having your server directly run the tests. Instead, you can have your server start a new JVM that runs the tests. This is how IDEs like Eclipse run tests. You simply need to write a main class that has JUnit run the tests, and serializes the results on disk.
Your main class would look something like this:
public class MyRunner {
public static void main(String... args) throws IOException {
String path = System.getProperty("resultPath");
if (path == null) {
throw new NullPointerException("must specify resultPath property");
}
// Possibly install a security manager to prevent calls to System.exit()
Result result = new JUnitCore().runMain(new RealSystem(), args);
try (ObjectOutputStream out = new ObjectOutputStream(new FileOutputStream(path)) {
out.writeObject(result);
}
System.exit(result.wasSuccessful() ? 0 : 1);
}
}
Then your server simply needs to construct a java command line with the jars that include the tests, the JUnit jar file, and a jar that contains MyRunner.

Empty benchmarks when running JMH samples in Intellij IDEA

I have some problem with using JMH.
So, I create an empty project in Intellij Idea, then in project structure add jmh-core jar file. Finally, try to run samples, for example
import org.openjdk.jmh.annotations.GenerateMicroBenchmark;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.RunnerException;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;
public class JMHSample_01_HelloWorld {
#GenerateMicroBenchmark
public void wellHelloThere() {
// this method was intentionally left blank.
}
public static void main(String[] args) throws RunnerException {
Options opt = new OptionsBuilder()
.include(".*" + JMHSample_01_HelloWorld.class.getSimpleName() + ".*")
.forks(1)
.build();
new Runner(opt).run();
}
}
but result is
No matching benchmarks. Miss-spelled regexp?
Use EXTRA verbose mode to debug the pattern matching.
Process finished with exit code 0
with using verbosity(VerboseMode.EXTRA)
No matching benchmarks. Miss-spelled regexp?
Benchmarks:
Process finished with exit code 0
I tried to change output path to projectFolder\target\classes but nothing was changed.
Then I looked at the source code in the debug mode and see that resource = "/META-INF/MicroBenchmarks",urls.hasMoreElements() is false and therefore benchmarks is empty. Then I saw at the samples jar file which has MicroBenchmarks file with information about tests and work well.
So, the question is what do I do wrong?
Do I have to write information about test manually?

Please follow the instructions on JMH page to set up the benchmark project, namely:
"Make sure you tried these things before getting support:
- Archetypes provide the golden build configuration. Try to generate the clean JMH benchmark project and transplant the benchmark
there. This is important to try when upgrading to the newer JMH
versions, since the minute differences in the build configurations may
attribute to the failures you are seeing."
If you'd follow that, you will notice that you also need to add jmh-generator-annprocess as the dependency, and make sure it runs before you run any test.

Drools accessing generated java files

I am executing Drools rule through Mockito test. The rule fails at run time reporting the error with a line number of a java file having some long arbitrary name. It seems that Drools generates java files on the fly and injects into JVM. But when I search those files on my disc I don't find any. Is there a way I could store them on my disc?

Got the solution:
You can dump the Drools generated java files in two ways.
1) Through command line:
-Ddrools.dump.dir="target/dumpDir"
e.g. I use maven command to execute the rule so it would be
mvn -Ddrools.dump.dir="target/dumpDir" -Dtest=DroolsRuleTest test
2) Through the API
public class FileKnowledgeBaseFactory implements KnowledgeBaseFactory {
private Log log = LogFactory.getLog(FileKnowledgeBaseFactory.class);
public KnowledgeBase load(String drlFullFilename) {
KnowledgeBuilderConfiguration config = KnowledgeBuilderFactory.newKnowledgeBuilderConfiguration();
config.setOption(DumpDirOption.get(new File("target/dumpDir")));
KnowledgeBuilder knowledgeBuilder = KnowledgeBuilderFactory.newKnowledgeBuilder(config);
....
....
}
}