We Keep Coding

What is the proper value of HADOOP_HOME and PIG_CLASSPATH for APACHE HADOOP version 2.8.0?

I have the problem executing the Hadoop command from the PIG command line. The command and the error stack is below
My instructor suspects that it is because HADDOP_HOME and PIG_CLASSPATH are incorrect. I am on the HADOOP version 2.8.0.
So, originally I had HADOOP_HOME as
HADOOP_HOME=<CELLAR_DIRECTORY>/hadoop/2.8.0/
Then I switched the following setup:
HADOOP_HOME=<CELLAR_DIRECTORY>/hadoop/2.8.0/libexec/etc/hadoop
PIG_CLASSPATH is defined as $HADOOP_HOME
Commands I used in pig:
A = LOAD '/Users/anarinsky/Downloads/loaddata1.txt';
B = MAPREDUCE '/Users/anarinsky/workspace/wordcount/target/wordcount-1.jar' STORE A INTO '/Users/anarinsky/Downloads/tempwrite2' LOAD '/Users/anarinsky/Downloads/tempwrite2' AS (word:chararray, count:int) `com.systemskills.hadoop.wordcount.WordCountDriver /wordcountdata /Users/anarinsky/Downloads/pigoptdir`;
Pig Stack Trace
ERROR 2025: Expected leaf of reduce plan to always be POStore. Found PONative
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias B
at org.apache.pig.PigServer.openIterator(PigServer.java:1019)
at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:747)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:376)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:231)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:206)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:66)
at org.apache.pig.Main.run(Main.java:564)
at org.apache.pig.Main.main(Main.java:176)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:234)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias B
at org.apache.pig.PigServer.storeEx(PigServer.java:1122)
at org.apache.pig.PigServer.store(PigServer.java:1081)
at org.apache.pig.PigServer.openIterator(PigServer.java:994)
... 13 more
Caused by: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompilerException: ERROR 2025: Expected leaf of reduce plan to always be POStore. Found PONative
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:321)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.compile(MapReduceLauncher.java:629)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:152)
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:308)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1474)
at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1459)
at org.apache.pig.PigServer.storeEx(PigServer.java:1118)
... 15 more

Alex!
Unfortunately, it's not related to Pig paths (tried it on my configured hadoop cluster) with same result. The error you get refers to the fact that Physical plan compiler has a bug in compile method. So in order to make your attempt work you have two possibilities
Run native MR job using hadoop and after it finishes process it's results in pig
Edit pig source code and compile your own version. You'll need to edit
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler#compile method and replace
for (PhysicalOperator op : leaves) {
if (!(op instanceof POStore)) {
int errCode = 2025;
String msg = "Expected leaf of reduce plan to " +
"always be POStore. Found " + op.getClass().getSimpleName();
throw new MRCompilerException(msg, errCode, PigException.BUG);
}
}
with
for (PhysicalOperator op : leaves) {
if (!(op instanceof POStore) && !(op instanceof PONative)) {
int errCode = 2025;
String msg = "Expected leaf of reduce plan to " +
"always be POStore. Found " + op.getClass().getSimpleName();
throw new MRCompilerException(msg, errCode, PigException.BUG);
}
}

Run java file using 'java filename' command [duplicate]

This question already has an answer here:
Error when trying to execute a command in java [duplicate]
(1 answer)
Closed 9 years ago.
I want to run java file using 'java filename' command when you're not in the file's directly.
In the terminal we use :
cd filepath
java filename
but in Eclipse you cannot change the directory using 'cd' so how can I run the file although I can't change the directory
I am using this method to run a command using Java
try {
String line;
Process p = Runtime.getRuntime().exec( "cd /Users/apple/Documents/Documents/workspace/UserTesting/src" );
Process p2 = Runtime.getRuntime().exec( "java NewFile" );
BufferedReader in = new BufferedReader(
new InputStreamReader(p2.getInputStream()) );
while ((line = in.readLine()) != null) {
System.out.println(line);
}
in.close();
}
catch (Exception e) {
// ...
}
Here are several trials
Apples-MacBook-Pro:~ apple$ cd
/Users/apple/Documents/Documents/workspace/UserTesting/src
Apples-MacBook-Pro:src apple$ java NewFile 5 90 35 45 150 3
Reichweite---- nach blase art
3 5 35 45 90 150
Apples-MacBook-Pro:src apple$ java /Users/apple/Documents/Documents/workspace/UserTesting/src/NewFile
Exception in thread "main" java.lang.NoClassDefFoundError:
/Users/apple/Documents/Documents/workspace/UserTesting/src/NewFile
Caused by: java.lang.ClassNotFoundException:
.Users.apple.Documents.Documents.workspace.UserTesting.src.NewFile at
java.net.URLClassLoader$1.run(URLClassLoader.java:202) at
java.security.AccessController.doPrivileged(Native Method) at
java.net.URLClassLoader.findClass(URLClassLoader.java:190) at
java.lang.ClassLoader.loadClass(ClassLoader.java:306) at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at
java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Apples-MacBook-Pro:src apple$ java
/Users/apple/Documents/Documents/workspace/UserTesting/src/NewFile.java
Exception in thread "main" java.lang.NoClassDefFoundError:
/Users/apple/Documents/Documents/workspace/UserTesting/src/NewFile/java
Caused by: java.lang.ClassNotFoundException:
.Users.apple.Documents.Documents.workspace.UserTesting.src.NewFile.java
at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at
java.security.AccessController.doPrivileged(Native Method) at
java.net.URLClassLoader.findClass(URLClassLoader.java:190) at
java.lang.ClassLoader.loadClass(ClassLoader.java:306) at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at
java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Apples-MacBook-Pro:src apple$
Blockquote

This will not work.
For one, cd is a shell command. And then you create another process to run the java command.
You need a ProcessBuilder. You can set up the initial directory, environment etc:
final File wantedCwd = new File(...);
final ProcessBuilder pb = new ProcessBuilder("java", "thefile");
// Change directory
pb.directory(wantedCwd);
You can even change the stdin, stdout, stderr etc of the command.
DO NOT use Runtime.exec(). It won't (exec()) in many situations.

Java ProcessBuilder Debugger Netbeans

I am having trouble with the deeper layers of the JVM and its debugging functionality.
What I am trying to do is start a separate java program using ProcessBuilder and let it communicate with my main process. All works fine unless I add the command
"-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=1044",
to the ProcessBuilder.
Class toExecute = ExampleSimulationController.class;
String javaHome = System.getProperty("java.home");
String javaBin = javaHome
+ File.separator + "bin"
+ File.separator + "java";
String classpath = System.getProperty("java.class.path");
String className = toExecute.getCanonicalName();
ProcessBuilder builder = new ProcessBuilder(javaBin, "-cp",
"-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=1044",
classpath, className);
builder.redirectErrorStream(true);
In adding this line to the ProcessBuilder (with the intention to add debugging functionality to the subprocess as described, for example, here: What are Java command line options to set to allow JVM to be remotely debugged?
I get an exception when trying to read as follows:
BufferedReader mainProcessConsoleOutput = new BufferedReader(new InputStreamReader(mainSimulation.getInputStream()));
and further down:
if(!(line = mainProcessConsoleOutput.readLine()).equals("someText"))
The exception is as follows:
Main Process: Exception in thread "main" java.lang.NoClassDefFoundError: /Users/...[path].../build/classes
Main Process: Caused by: java.lang.ClassNotFoundException: .Users.[same_Path].build.classes
Main Process: at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
Main Process: at java.security.AccessController.doPrivileged(Native Method)
Main Process: at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
Main Process: at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
Main Process: at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
Main Process: at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Exception in thread "main" java.lang.NullPointerException
at [PacketStructure].SimulationController.main(SimulationController.java:66)
I am using Netbeans as IDE and know about "Attaching a Debugger" and giving it the same port as specified in the command I pass to the ProcessBuilder. However, I don't know when I have to do this - before I specify a breakpoint in the subprocess? Afterwards? So far I did not find any indication that my subprocess is communicating with a debugger in any way.
What seems suspicious to me as well is the fact that the exception is thrown when I try to read from subprocess' stream - and not someplace earlier.
I do use ObjectInputStream and ObjectOutputStream to pass serialized data from one process to the other, but since I cannot debug the subprocess I don't know if that is a potential source of the problem.
I use MacOs.
Since the solution of this problem lies beyond my knowledge of the Java Magic, please help me in solving this riddle.
Thanks,
M

The classpath value must immediately follow the classpath argument.

After a lot of time spent trying to solve the problem I finally did:
ProcessBuilder builder = new ProcessBuilder(javaBin,"-agentlib:jdwp=transport=dt_socket,address=localhost:8000,server=n,suspend=y","-cp", classpath, className);
By adding the "-cp" command after the "-agentlib" command apparently the classpath is matched correctly with the className.
Should I have known that the order in which commands are passed is important?
Thx

Not able to execute an exe file from java using ProcessBuilder

I am making a project to run C, C++ and Java, from within Java code itself. It works absolutely fine for Java, and the problem is faced when compiling and executing C and C++ files.
I got my compilation right with this code and I can get the executable file generated in my specified path. But now when I run the executable binary from ProcessBuilder I get an error saying that 'file was not found'. Please see to the code and tell me what is going wrong and where??
public void processCode(String path,String lang)throws IOException
{
String cmd="",s=null,out=null,file="";
totalTime=0;
ProcessBuilder process=new ProcessBuilder();
process.directory(new File(path));
if(lang.equals("c")||lang.equals("cpp"))
{
cmd=threadNum+".exe";
process.command(cmd);
}
else if(lang.equals("java"))
{
cmd="java";
file="Main"+threadNum;
process.command(new String[]{cmd,file});
}
process.redirectInput(new File(PATH+"Input\\" + prob + ".txt"));
process.redirectOutput(new File(PATH+"Output.txt"));
Process p=process.start();
long start=System.currentTimeMillis();
while (true)
{
try{
if(p.exitValue()==0)
{
totalTime=(int)(System.currentTimeMillis()-start);
break;
}
}
catch (Exception e)
{
}
if(System.currentTimeMillis()-start>2000)
{
res=1;
p.destroy();
break;
}
}
if(res!=1)
{
compareFile();
}
}
The method is called from here
And the error generated is :
Exception in thread "main" java.io.IOException: Cannot run program "19.exe" (in directory "C:\wamp\www\usercodes\lokesh"): CreateProcess error=2, The system cannot find the file specified
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
at Contest.processCode(Main.java:202)
at Contest.compileCode(Main.java:180)
at Contest.makeFile(Main.java:157)
at Contest.main(Main.java:53)
at Main.main(Main.java:15)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
Caused by: java.io.IOException: CreateProcess error=2, The system cannot find the file specified
at java.lang.ProcessImpl.create(Native Method)
at java.lang.ProcessImpl.<init>(ProcessImpl.java:188)
at java.lang.ProcessImpl.start(ProcessImpl.java:132)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1021)
... 10 more

Setting the directory of a ProcessBuilder does not have any effect on where the system will look for the executable when it tries to start a process. It merely sets the current working directory of the newly-created process to this directory, should it be able to launch a process successfully. Your program 19.exe may well exist in C:\wamp\www\usercodes\lokesh, but unless this folder is on the PATH, the system will not be able to start your process.
Try running the process using the full path of the executable instead of just 19.exe.
It does have to be said that the error message is somewhat misleading. It says that it couldn't find your executable, and then it says 'in directory ...', which implies that that was where it was looking for it.

Calling system() or IPC::Run3 commands from perl don't seem to pass environment variable ($ENV{JAVA_HOME})

I've been struggling with launching a java process from perl. The root of the problem is that the java process is missing the JAVA_HOME environment variable causing a ClassNotFoundException.
I started by using IPC::Run3 because of its relatively elegant redirection of STDIN/STDOUT.
Assuming IPC::Run3 would use %ENV, I tried adding $ENV{JAVA_HOME}.
When that didn't work I tried doing system(). That didn't work, so finally, I got it to work using system("JAVA_HOME=/path/to/java && /path/to/java_program");
My test program is below. Naturally I'd uncomment the proper block to test the appropriate invocation.
#!/usr/bin/perl -w
use strict;
use IPC::Run3;
use vars qw(%Config $nutch_stdout $nutch_stderr);
%Config = (
'nutch_binary' => q[/home/crawl/nutch/runtime/local/bin/nutch],
'nutch_crawl_dir' => q[/home/crawl/nutch-crawl/crawl/crawldb/current/part-00000],
'nutch_seed_dir' => q[/home/crawl/urls],
'solr_url' => q[http://localhost:8080/solr],
);
my #nutch_command = ("$Config{nutch_binary}",
"crawl $Config{nutch_seed_dir}",
"-solr $Config{solr_url}",
"-d $Config{nutch_crawl_dir}",
"-threads 1",
"-depth 1");
$ENV{JAVA_HOME} = '/usr/lib/jvm/java-1.6.0';
while ((my $key,my $value) = each %ENV) {
print "$key=$value\n";
}
print "Running #nutch_command\n";
# My original code. Next few lines are shown in first batch of output below.
#run3 \#nutch_command, undef, \$nutch_stdout, \$nutch_stderr;
#print "Output from Nutch:\n";
#print $nutch_stdout;
#print "Errors from Nutch:\n";
#print $nutch_stderr;
# Second try. The next line's output is the second batch of output.
#system(#nutch_command);
# Third try. Despite setting and displaying %ENV, this is the only thing I tried that worked
system("JAVA_HOME=/usr/lib/jvm/java-1.6.0 && #nutch_command");
Here's the output of running the run3:
-bash-3.2$ ./test.pl
... [snip] ...
JAVA_HOME=/usr/lib/jvm/java-1.6.0
... [snip] ...
Running /home/crawl/nutch/runtime/local/bin/nutch crawl /home/crawl/urls -solr http://localhost:8080/solr -d /home/crawl/nutch-crawl/crawl/crawldb/current/part-00000 -threads 1 -depth 1
Output from Nutch:
Errors from Nutch:
Exception in thread "main" java.lang.NoClassDefFoundError: crawl
Caused by: java.lang.ClassNotFoundException: crawl
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
Could not find the main class: crawl. Program will exit.
And the output of the first system() call:
-bash-3.2$ ./test.pl
... [snip] ...
JAVA_HOME=/usr/lib/jvm/java-1.6.0
... [snip] ...
Running /home/crawl/nutch/runtime/local/bin/nutch crawl /home/crawl/urls -solr http://localhost:8080/solr -d /home/crawl/nutch-crawl/crawl/crawldb/current/part-00000 -threads 1 -depth 1
Exception in thread "main" java.lang.NoClassDefFoundError: crawl
Caused by: java.lang.ClassNotFoundException: crawl
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
Could not find the main class: crawl. Program will exit.
Finally, the third system call-- the only one that worked!-- with the environment variable set inline:
-bash-3.2$ ./test.pl
... [snip] ...
JAVA_HOME=/usr/lib/jvm/java-1.6.0
... [snip] ...
Running /home/crawl/nutch/runtime/local/bin/nutch crawl /home/crawl/urls -solr http://localhost:8080/solr -d /home/crawl/nutch-crawl/crawl/crawldb/current/part-00000 -threads 1 -depth 1
crawl started in: crawl-20120216133832
... continue success stdout output
Finally to the question: Aside from having to set the environment in-line with the system() call, what's the appropriate way to pass an environment var to a IPC::Run3 or a system() call?
(Note: output of %ENV is truncated to only relevant lines... lines like PATH, SHELL, _, etc. not relevant to the question omitted)
In case it's relevant:
-bash-3.2$ uname -a
Linux hostname 2.6.18-238.el5xen #1 SMP Thu Jan 13 16:41:45 EST 2011 x86_64 x86_64 x86_64 GNU/Linux
-bash-3.2$ perl --version
This is perl, v5.8.8 built for x86_64-linux-thread-multi

The root of the problem is that the java process is missing the JAVA_HOME environment variable causing a ClassNotFoundException.
REVISED
That is not the root of the problem. In fact, Java itself does not require JAVA_HOME to be set.
The immediate cause of the problem is one of the following:
The wrapper is not setting the classpath correctly for the application that you are trying to execute.
The wrapper using the wrong class name. The class name "nutch" is unusual and suspicious - there's no package name.
It seems likely that the real root cause is that you are assembling the argument list incorrectly. Each of those arguments with a space inside them should really be two arguments; i.e.
my #nutch_command = ("$Config{nutch_binary}",
"crawl", "$Config{nutch_seed_dir}",
"-solr", "$Config{solr_url}",
"-d", "$Config{nutch_crawl_dir}",
"-threads", "1",
"-depth", "1");
I suspect that this has confused the nutch wrapper script, and caused it to use the wrong classname (among other things). When you pass the entire command as one string and let the shell parse it, the problem (naturally) goes away.