We Keep Coding

How to use other python modules with jep?

I've set up a jep project within IntelliJ-IDEA, and keep getting this error when I run my code:
Exception in thread "main" jep.JepException: <class 'ModuleNotFoundError'>: No module named 'spacy'
at src/main/python\nlq_wrapper.<module>(nlq_wrapper.py:2)
at <string>.<module>(<string>:1)
at jep.Jep.exec(Native Method)
at jep.Jep.exec(Jep.java:478)
at com.siemens.nlqwrapper.NLQWrapper.load(NLQWrapper.java:37)
at com.siemens.nlqwrapper.Main.main(Main.java:9)
Even though spacy is included in my interpreter SDK packages.
Another weird thing is that the Python terminal within IntelliJ-IDEA can find and use spacy.
But when I try and run the program from the terminal or from my system's CLI, it can't find the modules and I get the same error.
Is there some extra configuration for jep that I need to do to be able to use other python modules with it? or is jep just not compatible with other modules?
EDIT
For further clarification here are the run configurations for Java and Python.
Java run configuration:
Python run configuration:

I fixed it thru this code you call before JEP interpreter:
PyConfig pyConfig = new PyConfig();
pyConfig.setPythonHome("/home/user/[NEW_PYTHON_HOME]/");
try {
MainInterpreter.setInitParams(pyConfig);
} catch (JepException e) {
e.printStackTrace();
}
Ali

Building Go from source fails tests when run using Java's Runtime.getRuntime().exec()

This question is related to a previous question, here: Java Runtime.getRuntime().exec() appears to be overwriting $PATH
I am trying to build Go from source from inside a Java program. I can build it properly using Terminal, but Java's Runtime.getRuntime().exec() gets interesting results. I tried using ProcessBuilder, but could not get it to properly make Go. Using my current setup with exec(), it makes properly, but then fails two tests. Code snippet:
String[] envp = new String[4];
envp[0] = "CC=/usr/bin/clang";
envp[1] = "GOROOT_BOOTSTRAP=/usr/local/go";
envp[2] = "CGO_ENABLED=0";
envp[3] = "PATH=" + System.getenv().get("PATH");
Runtime.getRuntime().exec("./all.bash", envp, "$HOME/Desktop/go/src");
It runs properly and compiles properly, but when it gets to running the test suite, I get two errors:
--- FAIL: TestCurrent (0.00s)
user_test.go:24: Current: user: Current not implemented on darwin/amd64 (got &user.User{Uid:"502", Gid:"20", Username:"", Name:"", HomeDir:""})
FAIL
FAIL os/user 0.009s
and a much longer one that I won't paste here due to absurd length, but it comes down to:
panic: test timed out after 3m0s
...
FAIL runtime 180.056s
I haven't any idea why the former is failing, but for the runtime when I build from the Terminal, it says:
ok runtime 19.096s
So something is causing that to take absurd amounts of time. I did some googling, and heard that it might be fixed if I use ARM=5 as an environment variable, but that didn't change anything. Does anyone have any idea why these tests are failing when I build from Java as opposed to the Terminal?

Looking at the source code for the os/user package, it looks like the native user handling depends on having cgo enabled. If cgo=0 (your case), it will fall back to the USER and HOME environment variables.
source code in question
Try putting USER=whatever in your exec environment.
I'm afraid I'd need more information to diagnose the runtime issue.

How to install apache mahout on mac?

I am new to Mahout. I want to install it and try it out. So far I have Maven3 and Java 1.6 installed and configured on my Mac. My question is:
Do I have to install Hadoop firstly before installing Mahout?
Some tutorials include installing Hadoop and some not which confuse me. I know Mahout is built on top of Hadoop. But not all of Mahout depends on Hadoop.
Can someone provide some useful detailed resources about installation?

http://chimpler.wordpress.com/2013/02/20/playing-with-the-mahout-recommendation-engine-on-a-hadoop-cluster/
http://chimpler.wordpress.com/2013/03/13/using-the-mahout-naive-bayes-classifier-to-automatically-classify-twitter-messages/
these 2 links helped me get up and running on OSX. It's not strictly necessary to use hadoop with mahout, however almost certainly it would be useful to gain experience with both as you go, if you are planning to use in a scalable system ...

Giving another answer to this question now that it's two years later and I finally got an itemsimilarity command to run on a mac after a lot of cursing and some blood spilled... Hope this saves someone some time and misery. Except my coworkers! Your weakness disgusts me! Anyway...
First for the "do I need $FINICKY_BIG_DATA_PLATFORM" question, see:
http://mahout.apache.org/users/basics/algorithms.html
Hadoop and/or spark are not hard requirements, some algorithms run on a single machine. But, the algorithm you may be interested in may only run on hadoop and/or spark. The docs on recommendations also steer you pretty strongly toward running the spark based algorithms. They also encourage you to use the black box command line commands, which can have different arguments between the single machine and spark versions (itemsimilarity, for example). So you don't NEED it, but you'll probably still need it.
I tried brew installs of hadoop, apache-spark and mahout. If you use the absolute latest versions (mahout 0.11.0, apache-spark 1.4.1, hadoop 2.7.1), you may have some of these problems:
" Got error Cannot find Spark class path. Is 'SPARK_HOME' set? " To fix this, not only do you need to have that environment variable set (mine is set to "/usr/local/Cellar/apache-spark/1.4.1/libexec"), you also need the apparently now deprecated compute-classpath.sh script in ${SPARK_HOME}/bin/ . I had a 1.2.0 spark installation handy, so I lifted one from there.
Bonus gotcha, in that 1.2.0 install there are two compute-classpath.sh scripts, one is just a one-liner invoking the other. You will probably be happier if you copy over the "real" one, so use less to check.
" java.lang.UnsatisfiedLinkError: no snappyjava in java.library.path " To fix this, the Internet will tell you to get a copy of libsnappyjava.jnilib , put it in /usr/lib/java and rename it libsnappyjava.dylib . I did "brew install snappy," which installed version 1.1.3 and included symlinks named libsnappy.dylib and libsnappy.jnilib. Note that these are just symlinks and that the names aren't quite right... So after copying and renaming the main lib file I at least got a new error, which brings us to...
" Exception in thread "main" java.lang.UnsatisfiedLinkError: org.xerial.snappy.SnappyNative.maxCompressedLength(I)I " The Internet was less forthcoming with suggestions. I did see one post saying that version 1.0.xxx didn't have whatever magic pony code but version 1.1.1.3 did. I went to http://central.maven.org/maven2/org/xerial/snappy/snappy-java/ , downloaded snappy-java-1.1.1.3.jar and dropped that as-is into /usr/lib/java , no name changes. This made the snappy errors go away and I could run a "mahout spark-itemsimilarity" command to completion, YMMV, this advice is provided as-is with no warranty.
Please note that snappy error induced despair may drive you to download the spark .tgz and build it from scratch. The build process will take up ~2 hours of your life that you will never get back and you will still get snappy errors at the end. Ultimately I could run the same command with this hand-built version as with the brew installed version, the snappy jar ended up being the main thing.

You don't need hadoop at all to try out mahout. Below is a sample code which take model as input from a file and will print recommendations.
package com.ml.recommend;
import java.io.File;
import java.io.IOException;
import java.util.List;
import org.apache.mahout.cf.taste.common.TasteException;
import org.apache.mahout.cf.taste.impl.model.file.FileDataModel;
import org.apache.mahout.cf.taste.impl.neighborhood.NearestNUserNeighborhood;
import org.apache.mahout.cf.taste.impl.recommender.CachingRecommender;
import org.apache.mahout.cf.taste.impl.recommender.GenericUserBasedRecommender;
import org.apache.mahout.cf.taste.impl.similarity.PearsonCorrelationSimilarity;
import org.apache.mahout.cf.taste.model.DataModel;
import org.apache.mahout.cf.taste.neighborhood.UserNeighborhood;
import org.apache.mahout.cf.taste.recommender.RecommendedItem;
import org.apache.mahout.cf.taste.recommender.Recommender;
import org.apache.mahout.cf.taste.similarity.UserSimilarity;
public class App {
public static void main(String[] args) throws IOException, TasteException {
DataModel model = new FileDataModel(new File("data.txt"));
UserSimilarity userSimilarity = new PearsonCorrelationSimilarity(model);
UserNeighborhood neighborhood = new NearestNUserNeighborhood(3,
userSimilarity, model);
Recommender recommender = new GenericUserBasedRecommender(model,
neighborhood, userSimilarity);
Recommender cachingRecommender = new CachingRecommender(recommender);
List<RecommendedItem> recommendations = cachingRecommender.recommend(
1000000000000006075L, 10);
System.out.println(recommendations);
}
}

Error message when trying to write a xlsx.-file with R

I try to save some R-dataframes into .xlsx-files using the write.xlsx function of the xlsx package like this
write.xlsx(tab,file="test",sheetName="testsheet",col.names=TRUE,row.names=FALSE,append=FALSE)
whereas the object tab is a data frame, as prooved here
> class(tab)
[1] "data.frame"
When I run the code I get the following error message
> write.xlsx(tab,file="test.xlsx",sheetName="testsheet",col.names=TRUE,row.names=FALSE,append=FALSE)
Fehler in .jcall("RJavaTools", "Z", "hasField", .jcast(x, "java/lang/Object"), :
RcallMethod: cannot determine object class
and I have no particular idea what the problem could be.
PS: I'm running R 2.14.1 in the StatET 2.0 plugin in Eclipse 3.7 on a 64bit machine.

When you work in Eclipse, you can start R using either rj - a Java terminal, or RTerm - the native R terminal.
If you are using the rj terminal and something doesn't work, try the same thing with RTerm.
I have never tried to figure out why, but a few things don't work properly in rj. This includes all use of RCOM as well as printing of the return value of system().
I use rj by default because I like the way it deals with help (amongst other benefits).
But if things don't work, I try it in RTerm. One day I'll have some spare time and I'll take it up with the author.
PS. I want to stress that I absolutely love StatET in Eclipse. These oddities or rj are very minor inconveniences in the grand scheme of things.

From my experience these kind of errors are produced when the standard rj package is installed instead of the one supplied bij the StatET developer.
Check the installation guide here:
http://www.walware.de/goto/statet
If you would happen to be using Debian or Ubuntu, you can also use the repository from OpenAnalytics to install StatET and the correct rj packages in one go.
http://deb.openanalytics.eu/howto.html

I had same problem. Two codes work with my problem:
FIRST) Convert vector to dataframe:
library(xlsx)
data <- data.frame(c(1,2,3))
write.xlsx(data, file = "C:/Users/Name/Downloads/data.xlsx")
SECOND) Use another library:
`# Using openxlsx package
library(openxlsx)
dataD1 <- data.frame(c(1,2,3))
write.xlsx(dataD1, "C:/Users/Name/Downloads/dataD1.xlsx")
I hope you have solved your problem.

How to use Sqoop in Java Program?

I know how to use sqoop through command line.
But dont know how to call sqoop command using java programs .
Can anyone give some code view?

You can run sqoop from inside your java code by including the sqoop jar in your classpath and calling the Sqoop.runTool() method. You would have to create the required parameters to sqoop programmatically as if it were the command line (e.g. --connect etc.).
Please pay attention to the following:
Make sure that the sqoop tool name (e.g. import/export etc.) is the first parameter.
Pay attention to classpath ordering - The execution might fail because sqoop requires version X of a library and you use a different version. Ensure that the libraries that sqoop requires are not overshadowed by your own dependencies. I've encountered such a problem with commons-io (sqoop requires v1.4) and had a NoSuchMethod exception since I was using commons-io v1.2.
Each argument needs to be on a separate array element. For example, "--connect jdbc:mysql:..." should be passed as two separate elements in the array, not one.
The sqoop parser knows how to accept double-quoted parameters, so use double quotes if you need to (I suggest always). The only exception is the fields-delimited-by parameter which expects a single char, so don't double-quote it.
I'd suggest splitting the command-line-arguments creation logic and the actual execution so your logic can be tested properly without actually running the tool.
It would be better to use the --hadoop-home parameter, in order to prevent dependency on the environment.
The advantage of Sqoop.runTool() as opposed to Sqoop.Main() is the fact that runTool() return the error code of the execution.
Hope that helps.
final int ret = Sqoop.runTool(new String[] { ... });
if (ret != 0) {
throw new RuntimeException("Sqoop failed - return code " + Integer.toString(ret));
}
RL

Find below a sample code for using sqoop in Java Program for importing data from MySQL to HDFS/HBase. Make sure you have sqoop jar in your classpath:
SqoopOptions options = new SqoopOptions();
options.setConnectString("jdbc:mysql://HOSTNAME:PORT/DATABASE_NAME");
//options.setTableName("TABLE_NAME");
//options.setWhereClause("id>10"); // this where clause works when importing whole table, ie when setTableName() is used
options.setUsername("USERNAME");
options.setPassword("PASSWORD");
//options.setDirectMode(true); // Make sure the direct mode is off when importing data to HBase
options.setNumMappers(8); // Default value is 4
options.setSqlQuery("SELECT * FROM user_logs WHERE $CONDITIONS limit 10");
options.setSplitByCol("log_id");
// HBase options
options.setHBaseTable("HBASE_TABLE_NAME");
options.setHBaseColFamily("colFamily");
options.setCreateHBaseTable(true); // Create HBase table, if it does not exist
options.setHBaseRowKeyColumn("log_id");
int ret = new ImportTool().run(options);
As suggested by Harel, we can use the output of the run() method for error handling. Hoping this helps.

There is a trick which worked out for me pretty well. Via ssh, you can execute the Sqoop command directly. Just you have to use is an SSH Java Library
This is independent of Java. You just need to include any SSH library and sqoop installed in the remote system you want to perform the import. Now connect to the system via ssh and execute the commands which will export data from MySQL to hive.
You have to follow this step.
Download sshxcute java library: https://code.google.com/p/sshxcute/
and Add it to the build path of your java project which contains the following Java code
import net.neoremind.sshxcute.core.SSHExec;
import net.neoremind.sshxcute.core.ConnBean;
import net.neoremind.sshxcute.task.CustomTask;
import net.neoremind.sshxcute.task.impl.ExecCommand;
public class TestSSH {
public static void main(String args[]) throws Exception{
// Initialize a ConnBean object, the parameter list is IP, username, password
ConnBean cb = new ConnBean("192.168.56.102", "root","hadoop");
// Put the ConnBean instance as parameter for SSHExec static method getInstance(ConnBean) to retrieve a singleton SSHExec instance
SSHExec ssh = SSHExec.getInstance(cb);
// Connect to server
ssh.connect();
CustomTask sampleTask1 = new ExecCommand("echo $SSH_CLIENT"); // Print Your Client IP By which you connected to ssh server on Horton Sandbox
System.out.println(ssh.exec(sampleTask1));
CustomTask sampleTask2 = new ExecCommand("sqoop import --connect jdbc:mysql://192.168.56.101:3316/mysql_db_name --username=mysql_user --password=mysql_pwd --table mysql_table_name --hive-import -m 1 -- --schema default");
ssh.exec(sampleTask2);
ssh.disconnect();
}
}

If you know the location of the executable and the command line arguments you can use a ProcessBuilder, this can then be run a separate Process that Java can monitor for completion and return code.

Please follow the code given by vikas it worked for me and include these jar files in classpath and import these packages
import com.cloudera.sqoop.SqoopOptions;
import com.cloudera.sqoop.tool.ImportTool;
Ref Libraries
Sqoop-1.4.4 jar /sqoop
ojdbc6.jar /sqoop/lib (for oracle)
commons-logging-1.1.1.jar hadoop/lib
hadoop-core-1.2.1.jar /hadoop
commons-cli-1.2.jar hadoop/lib
commmons-io.2.1.jar hadoop/lib
commons-configuration-1.6.jar hadoop/lib
commons-lang-2.4.jar hadoop/lib
jackson-core-asl-1.8.8.jar hadoop/lib
jackson-mapper-asl-1.8.8.jar hadoop/lib
commons-httpclient-3.0.1.jar hadoop/lib
JRE system library
1.resources.jar jdk/jre/lib
2.rt.jar jdk/jre/lib
3. jsse.jar jdk/jre/lib
4. jce.jar jdk/jre/lib
5. charsets,jar jdk/jre/lib
6. jfr.jar jdk/jre/lib
7. dnsns.jar jdk/jre/lib/ext
8. sunec.jar jdk/jre/lib/ext
9. zipfs.jar jdk/jre/lib/ext
10. sunpkcs11.jar jdk/jre/lib/ext
11. localedata.jar jdk/jre/lib/ext
12. sunjce_provider.jar jdk/jre/lib/ext
Sometimes u get error if your eclipse project is using JDK1.6 and the libraries you add are JDK1.7 for this case configure JRE while creating project in eclipse.
Vikas if i want to put the imported files into hive should i use options.parameter ("--hive-import") ?