We Keep Coding

Allow Evosuite to write files during test generation

Im currently trying to get Evosuite to work with JNA. Consider the following basic example:
import com.sun.jna.ptr.IntByReference;
public class Example {
public static int foo(int x) {
IntByReference c = new IntByReference(x);
if (c.getValue() == 100) {
return 100;
} else {
return 0;
}
}
}
Im running Evosuite from the command line with these options:
java32 -jar evosuite.jar -projectCP "src;E:\evosuite\test\lib\jna-5.2.0.jar" -class Example -criterion branch
Evosuite wont reach 100% branch coverage (only the trivial 33%), but notifies me with this message after the timeout:
* Permissions denied during test execution:
- java.io.FilePermission:
write C:\Users\PC\AppData\Local\Temp\jna--2025216854: 1
I found out that JNA needs to write some temp files in order to work, but Evosuite will block any atempt of file writing during test generation. I understand that this is a reasonable policy in most cases because you dont want Evosuite to write random files to your disk while generating tests for a saveFile() function, but in my case this shouldn't be a problem.
Is there a way to tell Evosuite to allow file writing during test generation or a different approach to generate tests for java programms using the JNA library?

I figured out how to run JNA without the need of writing temporary files thanks to cubrr.
copy the system specific jnidispatch.[dll, ...] file to a folder
add -Djna.boot.library.path=folder -Djna.nounpack=true to the command
Note: the jna.boot.library.path should only point to the containig folder, do not write folder/jnidispatch.
Solution to the initial question:
Setting the evosuite option -Dsandbox=false will remove most restrictions for test generation and finally allowed me to generate my tests!

java.lang.ClassNotFoundException when running program on spark cluster

I have a spark scala program which loads a jar I wrote in java. From that jar a static function is called, which tried to read a serialized object from a file (Pattern.class), but throws a java.lang.ClassNotFoundException.
Running the spark program locally works, but on the cluster workers it doesn't. It's especially weird because before I try to read from the file, I instantiate a Pattern object and there are no problems.
I am sure that the Pattern objects I wrote in the file are the same as the Pattern objects I am trying to read.
I've checked the jar in the slave machine and the Pattern class is there.
Does anyone have any idea what the problem might be ? I can add more detail if it's needed.
This is the Pattern class
public class Pattern implements Serializable {
private static final long serialVersionUID = 588249593084959064L;
public static enum RelationPatternType {NONE, LEFT, RIGHT, BOTH};
RelationPatternType type;
String entity;
String pattern;
List<Token> tokens;
Relation relation = null;
public Pattern(RelationPatternType type, String entity, List<Token> tokens, Relation relation) {
this.type = type;
this.entity = entity;
this.tokens = tokens;
this.relation = relation;
if (this.tokens != null)
this.pattern = StringUtils.join(" ", this.tokens.toString());
}
}
I am reading the file from S3 the following way:
AmazonS3 s3Client = new AmazonS3Client(credentials);
S3Object confidentPatternsObject = s3Client.getObject(new GetObjectRequest("xxx","confidentPatterns"));
objectData = confidentPatternsObject.getObjectContent();
ois = new ObjectInputStream(objectData);
confidentPatterns = (Map<Pattern, Tuple2<Integer, Integer>>) ois.readObject();
LE: I checked the classpath at runtime and the path to the jar was not there. I added it for the executors but I still have the same problem. I don't think that was it, as I have the Pattern class inside the jar that is calling the readObject function.

Would suggest this adding this kind method to find out the classpath resources before call, to make sure that everything is fine from caller's point of view
public static void printClassPathResources() {
final ClassLoader cl = ClassLoader.getSystemClassLoader();
final URL[] urls = ((URLClassLoader) cl).getURLs();
LOG.info("Print All Class path resources under currently running class");
for (final URL url : urls) {
LOG.info(url.getFile());
}
}
This is sample configuration spark 1.5
--conf "spark.driver.extraLibrayPath=$HADOOP_HOME/*:$HBASE_HOME/*:$HADOOP_HOME/lib/*:$HBASE_HOME/lib/htrace-core-3.1.0-incubating.jar:$HDFS_PATH/*:$SOLR_HOME/*:$SOLR_HOME/lib/*" \
--conf "spark.executor.extraLibraryPath=$HADOOP_HOME/*" \
--conf "spark.executor.extraClassPath=$(echo /your directory of jars/*.jar | tr ' ' ',')
As described by this Trouble shooting guide :Class Not Found: Classpath Issues
Another common issue is seeing class not defined when compiling Spark programs this is a slightly confusing topic because spark is actually running several JVM’s when it executes your process and the path must be correct for each of them. Usually this comes down to correctly passing around dependencies to the executors. Make sure that when running you include a fat Jar containing all of your dependencies, (I recommend using sbt assembly) in the SparkConf object used to make your Spark Context. You should end up writing a line like this in your spark application:
val conf = new SparkConf().setAppName(appName).setJars(Seq(System.getProperty("user.dir") + "/target/scala-2.10/sparktest.jar"))
This should fix the vast majority of class not found problems. Another option is to place your dependencies on the default classpath on all of the worker nodes in the cluster. This way you won’t have to pass around a large jar.
The only other major issue with class not found issues stems from different versions of the libraries in use. For example if you don’t use identical versions of the common libraries in your application and in the spark server you will end up with classpath issues. This can occur when you compile against one version of a library (like Spark 1.1.0) and then attempt to run against a cluster with a different or out of date version (like Spark 0.9.2). Make sure that you are matching your library versions to whatever is being loaded onto executor classpaths. A common example of this would be compiling against an alpha build of the Spark Cassandra Connector then attempting to run using classpath references to an older version.

Hadoop Hive UDF with external library

I'm trying to write a UDF for Hadoop Hive, that parses User Agents. Following code works fine on my local machine, but on Hadoop I'm getting:
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to execute method public java.lang.String MyUDF .evaluate(java.lang.String) throws org.apache.hadoop.hive.ql.metadata.HiveException on object MyUDF#64ca8bfb of class MyUDF with arguments {All Occupations:java.lang.String} of size 1',
Code:
import java.io.IOException;
import org.apache.hadoop.hive.ql.exec.UDF;
import org.apache.hadoop.hive.ql.metadata.HiveException;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.*;
import com.decibel.uasparser.OnlineUpdater;
import com.decibel.uasparser.UASparser;
import com.decibel.uasparser.UserAgentInfo;
public class MyUDF extends UDF {
public String evaluate(String i) {
UASparser parser = null;
parser = new UASparser();
String key = "";
OnlineUpdater update = new OnlineUpdater(parser, key);
UserAgentInfo info = null;
info = parser.parse(i);
return info.getDeviceType();
}
}
Facts that come to my mind I should mention:
I'm compiling with Eclipse with "export runnable jar file" and extract required libraries into generated jar option
I'm uploading this "fat jar" file with Hue
Minimum working example I managed to run:
public String evaluate(String i) {
return "hello" + i.toString()";
}
I guess the problem lies somewhere around that library (downloaded from https://udger.com) I'm using, but I have no idea where.
Any suggestions?
Thanks, Michal

It could be a few things. Best thing is to check the logs, but here's a list of a few quick things you can check in a minute.
jar does not contain all dependencies. I am not sure how eclipse builds a runnable jar, but it may not include all dependencies. You can do
jar tf your-udf-jar.jar
to see what was included. You should see stuff from com.decibel.uasparser. If not, you have to build the jar with the appropriate dependencies (usually you do that using maven).
Different version of the JVM. If you compile with jdk8 and the cluster runs jdk7, it would also fail
Hive version. Sometimes the Hive APIs change slightly, enough to be incompatible. Probably not the case here, but make sure to compile the UDF against the same version of hadoop and hive that you have in the cluster
You should always check if info is null after the call to parse()
looks like the library uses a key, meaning that actually gets data from an online service (udger.com), so it may not work without an actual key. Even more important, the library updates online, contacting the online service for each record. This means, looking at the code, that it will create one update thread per record. You should change the code to do that only once in the constructor like the following:
Here's how to change it:
public class MyUDF extends UDF {
UASparser parser = new UASparser();
public MyUDF() {
super()
String key = "PUT YOUR KEY HERE";
// update only once, when the UDF is instantiated
OnlineUpdater update = new OnlineUpdater(parser, key);
}
public String evaluate(String i) {
UserAgentInfo info = parser.parse(i);
if(info!=null) return info.getDeviceType();
// you want it to return null if it's unparseable
// otherwise one bad record will stop your processing
// with an exception
else return null;
}
}
But to know for sure, you have to look at the logs...yarn logs, but also you can look at the hive logs on the machine you're submitting the job on ( probably in /var/log/hive but it depends on your installation).

such a problem probably can be solved by steps:
overide the method UDF.getRequiredJars(), make it returning a hdfs file path list which values are determined by where you put the following xxx_lib folder into your hdfs. Note that , the list mist exactly contains each jar's full hdfs path strings ,such as hdfs://yourcluster/some_path/xxx_lib/some.jar
export your udf code by following "Runnable jar file exporting wizard" (chose "copy required libraries into a sub folder next to the generated jar". This steps will result in a xxx.jar and a lib folder xxx_lib next to xxx.jar
put xxx.jar and the folders xxx_lib to your hdfs filesystem according to your code in step 0.
create a udf using: add jar ${the-xxx.jar-hdfs-path}; create function your-function as $}qualified name of udf class};
Try it. I test this and it works

Using JunitCore for tests in different projects

Background:
I am currently working on a project in eclipse that programatically executes JUnit tests that are pushed to a server.
So far everything works but I would like to know the results of the tests (specifically any failures) so I can push them out to an email. Right now the tests just output to the console but that doesn't seem to give me much output to actually use.
Right now I use the Runtime class to call the tests but that doesn't seem to have the functionality I need for getting results.
I have looked into the JUnitCore class but can't call any tests outside of the current java project.
So my main question would be how can I use JUnitCore to run junit tests in a specific JAR file? Or is there an easier way to approach this problem using a different class?
This is the only thing I've been able to get to work:
RunTests()
{
junitCore = new JUnitCore();
junitCore.run(AllTests.class);
}
But I would like to do something along the lines of this:
RunTests()
{
junitCore = new JUnitCore();
junitCore.run("C:\\$batch\\test\\hil research\\201507071307\\CommsTestRunner\\plugins\\TestSuite\\US35644.class");
}
I would appreciate any suggestions to this problem I am having. I'm an EE and was just introduced to java last month so this has been quite the challenge for me.

JUnitCore expects to read loaded classes, not class files in a JAR. Your real question is likely how to load the JAR (or directory of .class files) so it can be run.
Poke around with URLClassLoader; once you've amended the classpath appropriately, you can get a Class out of findClass and pass it into the JUnitCore methods you've found.

Since the tests might have classes that are also used by your server (but not necessarily at the same version) I would suggest not having your server directly run the tests. Instead, you can have your server start a new JVM that runs the tests. This is how IDEs like Eclipse run tests. You simply need to write a main class that has JUnit run the tests, and serializes the results on disk.
Your main class would look something like this:
public class MyRunner {
public static void main(String... args) throws IOException {
String path = System.getProperty("resultPath");
if (path == null) {
throw new NullPointerException("must specify resultPath property");
}
// Possibly install a security manager to prevent calls to System.exit()
Result result = new JUnitCore().runMain(new RealSystem(), args);
try (ObjectOutputStream out = new ObjectOutputStream(new FileOutputStream(path)) {
out.writeObject(result);
}
System.exit(result.wasSuccessful() ? 0 : 1);
}
}
Then your server simply needs to construct a java command line with the jars that include the tests, the JUnit jar file, and a jar that contains MyRunner.

how to create ant listener for specific task

We have around 80 jars in our applications. All are created using javac task and jar task in ant.
I would like to introduce findbug checks. One option was to create single findbug check ant project. This has all jars , all source paths defined in it. This works -- require lot of space. Analysis of result too not very straight forward. There are thousands of warnings to start with.
One option I am considering is to run ant with special listener on javac task ant , extract source and class location, call findbug task with source and class file information. Any other way introduce findbug to a large project.

tweaked taskFinished()... Fine for my usage.
public class JavacListener implements BuildListener
public void taskFinished(BuildEvent be) {
if ( be.getTask() instanceof UnknownElement ) {
UnknownElement ue= (UnknownElement) be.getTask();
ue.maybeConfigure();
if ( ue.getTask() instanceof Javac ) {
Javac task = (Javac)ue.getTask();
final Path sourcepath = task.getSrcdir();
FindBugsTask fbtask = new FindBugsTask();
System.out.println ("Trying FindBugs");
fbtask.setSourcePath(sourcepath);
fbtask.setAuxClasspath(task.getClasspath());
Path destPath = new Path( task.getProject() );
destPath.setPath(task.getDestdir().getAbsolutePath());
fbtask.setAuxAnalyzepath(destPath);
fbtask.setOutputFile(getFileName(task.getProject()));
fbtask.setProject(task.getProject());
fbtask.setHome(new File("C:\\apps\\findbugs-1.3.0"));
fbtask.execute();
}
} else {
System.out.println(be.getTask().getClass().getName());
System.out.println(be.getTask().getTaskName());
}
}
..