We Keep Coding

How to import LibSVM into my Java code

In Java programming, we should firstly add weka.jar into our classpath, thus we can call all classify or cluster algorithms in WEKA in the form of the following codes,
import weka.classifiers.trees.RandomForest;
...
RandomForest rf = new RandomForest(); // RandomForest object
But unfortunately, we can not use this way to import LibSVM algorithm, because there is not such class in weka.jar.
So, my question is How to import LibSVM into my Java code? Any help will be grateful :)

Firstly, I'd like to say there are so many methods to solve the problem. The solution I mentioned is quite simple, but other answers from StackOverflow are not detailed descripted, with waste my too much time to verify. So I'm happy to share it with all WEKA beginners :)
a) Download the LibSVM.jar from Maven Repository Center. Note that this LibSVM.jar is different from the libsvm.jar developed by Chih-Chung Chang and Chih-Jen Lin;
b) Add the LibSVM.jar to the classpath of our Java project;
c) Call the classifier LibSVM when you need, see the following Java code.
import weka.classifiers.functions.LibSVM; // contained in LibSVM.jar
String path = "file/train.arff";
Instances train = DataSource.read(path); // load the dataset
train.setClassIndex(train.numAttribute()-1); // set class index
LibSVM svm = new LibSVM(); // load the svm classifier
svm.buildClassifier(train);
Evaluation eval = new Evaluation(train);
eval.crossValidateModel(svm, train, 10, new Random(1)); // 10-fold cross-validation

See: https://weka.wikispaces.com/LibSVM
Use Weka's package manager to install the LibSVM. Suppose "weka.jar" is in your current folder, than run this:
java -cp weka.jar weka.core.WekaPackageManager -install-package LibSVM
During the installation, it shows:
[DefaultPackageManager] Tmp file: /tmp/LibSVM1.0.107382715397815864641.zip
[DefaultPackageManager] Installing: Description.props
[DefaultPackageManager] Installing: LibSVM.jar
[DefaultPackageManager] Installing: build_package.xml
...
You can see that "LibSVM.jar" is installed somewhere. In my case, it is at:
/home/john/wekafiles/packages/LibSVM/LibSVM.jar

Hadoop Hive UDF with external library

I'm trying to write a UDF for Hadoop Hive, that parses User Agents. Following code works fine on my local machine, but on Hadoop I'm getting:
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to execute method public java.lang.String MyUDF .evaluate(java.lang.String) throws org.apache.hadoop.hive.ql.metadata.HiveException on object MyUDF#64ca8bfb of class MyUDF with arguments {All Occupations:java.lang.String} of size 1',
Code:
import java.io.IOException;
import org.apache.hadoop.hive.ql.exec.UDF;
import org.apache.hadoop.hive.ql.metadata.HiveException;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.*;
import com.decibel.uasparser.OnlineUpdater;
import com.decibel.uasparser.UASparser;
import com.decibel.uasparser.UserAgentInfo;
public class MyUDF extends UDF {
public String evaluate(String i) {
UASparser parser = null;
parser = new UASparser();
String key = "";
OnlineUpdater update = new OnlineUpdater(parser, key);
UserAgentInfo info = null;
info = parser.parse(i);
return info.getDeviceType();
}
}
Facts that come to my mind I should mention:
I'm compiling with Eclipse with "export runnable jar file" and extract required libraries into generated jar option
I'm uploading this "fat jar" file with Hue
Minimum working example I managed to run:
public String evaluate(String i) {
return "hello" + i.toString()";
}
I guess the problem lies somewhere around that library (downloaded from https://udger.com) I'm using, but I have no idea where.
Any suggestions?
Thanks, Michal

It could be a few things. Best thing is to check the logs, but here's a list of a few quick things you can check in a minute.
jar does not contain all dependencies. I am not sure how eclipse builds a runnable jar, but it may not include all dependencies. You can do
jar tf your-udf-jar.jar
to see what was included. You should see stuff from com.decibel.uasparser. If not, you have to build the jar with the appropriate dependencies (usually you do that using maven).
Different version of the JVM. If you compile with jdk8 and the cluster runs jdk7, it would also fail
Hive version. Sometimes the Hive APIs change slightly, enough to be incompatible. Probably not the case here, but make sure to compile the UDF against the same version of hadoop and hive that you have in the cluster
You should always check if info is null after the call to parse()
looks like the library uses a key, meaning that actually gets data from an online service (udger.com), so it may not work without an actual key. Even more important, the library updates online, contacting the online service for each record. This means, looking at the code, that it will create one update thread per record. You should change the code to do that only once in the constructor like the following:
Here's how to change it:
public class MyUDF extends UDF {
UASparser parser = new UASparser();
public MyUDF() {
super()
String key = "PUT YOUR KEY HERE";
// update only once, when the UDF is instantiated
OnlineUpdater update = new OnlineUpdater(parser, key);
}
public String evaluate(String i) {
UserAgentInfo info = parser.parse(i);
if(info!=null) return info.getDeviceType();
// you want it to return null if it's unparseable
// otherwise one bad record will stop your processing
// with an exception
else return null;
}
}
But to know for sure, you have to look at the logs...yarn logs, but also you can look at the hive logs on the machine you're submitting the job on ( probably in /var/log/hive but it depends on your installation).

such a problem probably can be solved by steps:
overide the method UDF.getRequiredJars(), make it returning a hdfs file path list which values are determined by where you put the following xxx_lib folder into your hdfs. Note that , the list mist exactly contains each jar's full hdfs path strings ,such as hdfs://yourcluster/some_path/xxx_lib/some.jar
export your udf code by following "Runnable jar file exporting wizard" (chose "copy required libraries into a sub folder next to the generated jar". This steps will result in a xxx.jar and a lib folder xxx_lib next to xxx.jar
put xxx.jar and the folders xxx_lib to your hdfs filesystem according to your code in step 0.
create a udf using: add jar ${the-xxx.jar-hdfs-path}; create function your-function as $}qualified name of udf class};
Try it. I test this and it works

Error reading image from scala playn

I researched and looked into the PlayN game framework and I liked it a lot. I program in Scala and actually don't know Java but it's not usually a problem since they work together great.
I've set up a basic project in eclipse and imported all the libraries and dependencies. I even translated over the base maven project code. Here's the two files:
Zeitgeist.scala
package iris.zeit.core
import playn.core.PlayN._
import playn.core.Game
import playn.core.Image
import playn.core.ImageLayer
class Zeitgeist extends Game {
override def init (){
var bgImage: Image = assets().getImage("images/bg.png")
var bgLayer: ImageLayer = graphics().createImageLayer(bgImage)
graphics().rootLayer().add(bgLayer)
}
override def paint (alpha: Float){
//painting stuffs
}
override def update(delta: Float){
}
override def updateRate(): Int = {
25
}
}
Main.scala
package iris.zeit.desktop
import playn.core.PlayN
import playn.java.JavaPlatform
import iris.zeit.core.Zeitgeist
object Main {
def main(args: Array[String]){
var platform: JavaPlatform = JavaPlatform.register()
platform.assets().setPathPrefix("resources")
PlayN.run(new Zeitgeist())
}
}
The cool thing is it works! A window comes up perfectly. The only problem is I can't seem to load images. With the above line, "assets().getImage("images/bg.png")" it pops out
Could not load image: resources/images/bg.png [error=java.io.FileNotFoundException: resources/images/bg.png]
I've played around with the location of my resources file to no avail. I was even able to find bg.png myself with java.io.File. Am I doing something wrong? Is there something I'm forgetting?

Looking at the code of JavaAssetsManager, it looks like it is trying to load a resource and not a file. So you should check that your images are actually in the classpath and at the path you give ("resources/images/bp.png")
Alternatively, you can use getRemoteImage and pass a File URL. As you succeeded in using a java.io.File, you can just get the URL with method toUri of File (toUrl is deprecated).

This almost certainly doesn't work because you're doing this:
platform.assets().setPathPrefix("resources")
That means you're saying your source folder looks like this:
src/main/resources/resources/images/bg.png
src/main/resources/resources/images/pea.png
src/main/resources/resources/images
I imagine it actually looks like one of these:
src/main/resources/assets/images/bg.png <-- 'assets' the default prefix
src/main/resources/assets/images/pea.png
src/main/resources/assets/images
or:
src/main/resources/images/bg.png <-- You have failed to put a subfolder prefix in
src/main/resources/images/pea.png
src/main/resources/images
You can either do this, if you have no prefix:
plat.assets().setPathPrefix("")
Or just put your files in the assets sub-folder inside the resources folder.
It's worth noting that the current implementation calls:
getClass().getClassLoader().getResource(...)
Not:
getClass().getResource(...)
The difference is covered elsewhere, but the tldr is that plat.assets.getImage("images/pea.png") will work, but plat.assets.getImage("/images/pea.png") will not.

can I load user packages into eclipse to run at start up and how?

I am new to java and to the eclipse IDE.
I am running Eclipse
Eclipse SDK
Version: 3.7.1
Build id: M20110909-1335
On a windows Vista machine.
I am trying to learn from the book Thinking in Java vol4.
The author uses his own packages to reduce typing. However the author did not use Eclipse and this is where the problem commes in..
This is an example of the code in the book.
import java.util.*;
import static net.mindview.util.print.*;
public class HelloWorld {
public static void main(String[] args) {
System.out.println("hello world");
print("this does not work");
}
this is the contents of print.Java
//: net/mindview/util/Print.java
// Print methods that can be used without
// qualifiers, using Java SE5 static imports:
package net.mindview.util;
import java.io.*;
public class Print {
// Print with a newline:
public static void print(Object obj) {
System.out.println(obj);
}
// Print a newline by itself:
public static void print() {
System.out.println();
}
// Print with no line break:
public static void printnb(Object obj) {
System.out.print(obj);
}
// The new Java SE5 printf() (from C):
public static PrintStream
printf(String format, Object... args) {
return System.out.printf(format, args);
}
} ///:~
The error I get the most is in the statement.
Import static net.mindview.util.print.*;
On this staement the Eclipse IDE says it cannot resolve net
also on the
print("this does not work");
The Eclipse IDE says that the class print() does not exist for the class HelloWorld.
I have been trying to get these to work, but with only limited success, The autor uses another 32 of these packages through the rest of the book.
I have tried to add the directory to the classpath, but that seems to only work if you are using the JDK compiler. I have tried to add them as libraries and i have tried importing them into a package in a source file in the project. I have tried a few other things but cant remember them all now.
I have been able to make one of the files work, the print.java file I gave the listing for in this message. I did that by creating a new source folder then making a new package in that foldeer then importing the print.java file into the package.
But the next time I try the same thing it does not work for me.
What I need is a way to have eclipse load all these .java files at start up so when I need them for the exercises in the book they will be there and work for me, or just an easy way to make them work everytime.
I know I am not the only one that has had this problem I have seen other questions about it on google searches and they were also asking about the Thinking In Java book.
I have searched this site and others and am just not having any luck.
Any help with this or sugestions are welcome and very appreciated.
thank you
Ok I have tried to get this working as you said, I have started a new project and I removed the static from the import statement, I then created a new source folder, then I created a new package in the source folder. Then I imported the file system and selected the the net.mindview.util folder.
Now the immport statement no longer gives me an error. But the the print statement does, the only way to make the print statement work is to use its fully qualified name. Here is the code.
import net.mindview.util.*;
public class Hello2 {
public static void main(String[] args) {
Hello2 test = new Hello2();
System.out.println();
print("this dooes not work");
net.mindview.util.Print.print("this stinks");
}
}
The Error on the print statement is:
The method print(String) is undefined for the type Hello2
and if I try to run it the error I get is:
Exception in thread "main" java.lang.Error: Unresolved compilation problem:
The method print(String) is undefined for the type Hello2
at Hello2.main(Hello2.java:6)
The Statement::::: net.mindview.util.Print.print("this stinks") is the fully qualified print statement and it does not throw an error but it does totally defeat the purpose of the print.java file..
If you have any questions please ask Ill get back to you as soon as I can.

I've had similar issues. I solved it by following the steps below:
Click File->New->Java Project. Fill in UtilBuild for the ProjectName. Chose the option "Use project folder as root and click 'Finish'.
Right-click on UtilBuild in the PackageExplorer window and click New->package. For the Package Name, fill in net.mindview.util
Navigate within the unzipped Thinking In Java (TIJ) folder to TIJ->net\mindview\util. Here you will find all the source code (.java) files for util.
Select all the files in the net\mindview\util folder and drag them to the net.mindview.util package under UtilBuild in Eclipse. Chose the 'Copy Files' option and hit 'OK'.
You will probably already have the 'Build Automatically' option checked. If not, go to Project and click 'Build Automatically'. This will create the .class files from the .java source files.
In Eclipse, right-click on the project you were working on (the one where you couldn't get that blasted print() method to work!) Click Properties and Java Build Path->Libraries. Click 'Add Class Folder...' check the box for UtilBuild (the default location for the .class files).
I think the confusion here arises due to CLASSPATH. If you use Eclipse to build and run your code then Eclipse manages your CLASSPATH. (You don't have to manually edit CLASSPATH in the 'Environment Variables' part of your computer properties, and doing so changes nothing as far as Eclipse Build and Run are concerned.)
In order to call code that exists outside your current project (I will name this 'outside code' for convenience) you need to satisfy three things:
A. You need to have the .class files for that code (as .class files or inside a JAR)
B. You need to indicate in your source code where to look for the 'outside code'
C. You need to indicate where to start looking for the 'outside code'
In order to satisfy these requirements, in this example we:
A. Build the project UtilBuild which creates the .class files we need.
B. Add the statement import static net.mindview.util.Print.*; in our code
C. Add the Class Folder library in Eclipse (Java Build Path->Libraries).
You can investigate the effect of Step C by examining the .classpath file that lives directly in your project folder. If you open it in notepad you will see a line similar to the following:
<classpathentry kind="lib" path="/UtilBuild>
You should combine this with your import statement to understand where the compiler will look for the .class file. Combining path="/UtilBuild" and import static net.mindview.util.Print.*; tells us that the compiler will look for the class file in:
UtilBuild/net/mindview/util
and that it will take every class that we built from the Print.java file (Print.*).
NOTE:
There is no problem with the keyword static in the statement
import static net.mindview.util.Print.*;
static here just means that you don't have to give specify the class name from Print.java, just the methods that you want to call. If we omit the keyword static from the import statement, then we would need to qualify that print() method with the class it belongs to:
import net.mindview.util.Print.*;
//...
Print.print("Hello");
which is slightly more verbose than what is achieved with the static import.
OPINION:
I think most people new to Java will use Eclipse at least initially. The Thinking in Java book seems to assume you will do things via command line (hence it's guidance to edit environment variables in order to update CLASSPATH). This combined with using the util folder code from very early in the book I think is a source of confusion to new learners of the language. I would love to see all the source code organised into an Eclipse project and available for download. Short of that, it would be a nice touch to include the .class files in just the 'net/mindview/util' folder so that things would be a little easier.

U should import package static net.mindview.util not static net.mindview.util.Print
and you should extend the class Print to use its method.......

You should remove the static keyword from your import decleration, this: import static net.mindview.util.print.*; becomes this: import net.mindview.util.print.*;
If that also does not work, I am assuming you did the following:
Create your own project;
Start copying code directly from the book.
The problem seems to be that this: package net.mindview.util; must match your folder structure in your src folder. So, if your src folder you create a new package and name it net.mindview.util and in it you place your Print class, you should be able to get it working.
For future reference, you should always make sure that your package decleration, which is at the top of your Java class, matches the package in which it resides.
EDIT:
I have seen your edit, and the problem seems to have a simple solution. You declare a static method named print(). In java, static methods are accessed through the use of ClassName.methodName(). This: print("this dooes not work"); will not work because you do not have a method named print which takes a string argument in your Hello2 class. In java, when you write something of the sort methodName(arg1...), the JVM will look for methods with that signature (method name + parameters) in the class in which you are making the call and any other classes that your calling class might extend.
However, as you correctly noted, this will work net.mindview.util.Print.print("this stinks");. This is because you are accessing the static method in the proper way, meaning ClassName.methodName();.
So in short, to solve your problem, you need to either:
Create a method named print which takes a string argument in your Hello2 class;
Call your print method like so: Print.print("this stinks");
Either of these two solutions should work for you.

In my case I've dowloaded and decompressed the file TIJ4Example-master.zip. in eclipse workspace folder. The three packages : net.mindview.atunit, net.mindview.simple and net.mindview.util are in this point of the project :
and java programs runs with no problems (on the right an example of /TIJ4Example/src/exercises/E07_CoinFlipping.java)

Working with file in Eclipse

I have instruction to run program in command line, for example:
java SetTest < alice30.txt
I wonder how to do this in Eclipse. I tried to put this in Run Configuration like this:
Another thing I don't know is where to put this file (alice30.txt). Is this in root of project or in src folder where source files are located?
I know these are beginner questions but I am stuck and need help.
EDIT:
As #Kane suggested I passed File and opened stream.
Instead of:
Scanner in = new Scanner(System.in);
I now use:
Scanner in = new Scanner(new File("alice30.txt"));

You can pass full file path in arguments (e.g. c:/.../alice30.txt))

The eclipse root directory is the base directory of the project (i.e., not the src/ directory, directly under the project.)
It's generally good style to have a 'resources' folder for txt, graphics, etc.
Rather than trying to pass a stream you could just pass the filename and open the stream yourself.
The reason what you're doing in Eclipse isn't working is because your command prompt/shell/dos/bash/whatever is handling creating the input stream out of the file for you. Eclipse doesn't do this. So, from the command line: < alice.txt means "run this program with no arguments, and create a stream to system.in", while doing that in Eclipse means "run this program with two arguments '<' and 'alice.txt'

you need do like this:
add:
import java.io.IOException;
import java.nio.file.Paths;
then:
replace"Scanner in = new Scanner(System.in);"to"Scanner in =new Scanner(Paths.get("alice30.txt"));" .
and you also need do this : "public static void main(String args[]) throws IOException "

With information from this link/page and several tries, I figure out a way to pass argument and file using the local route in eclipse Run -> Run Configurations.. , though it is not recommended as Kane said.
For my case: I need to do " $java someClass tinyW.txt < tinyT.txt " (This is an example from Algorithms book by Robert Sedgewick)
In my case, " tinyW.txt " is a argument, so in the eclipse environment, you can set in Run -> Run Configurations -> Arguments -> Program arguments: /local address/tinyW.txt. For my Ubuntu: /home/****/tinyW.txt
" < tinyT.txt " is a file that pipe to the main arguments, so you can set the route and file in " Run -> RUn Configurations -> Common ", click the "Input File", use the File System icon and select the file from local compute. See the figure. So in Input File: /local_address/tinyT.txt. My case is: /home/***/tinyT.txt. Hope it also works for you.