Error in Udf of pig - java

I am new to pig. I wrote a UDF in pig and used it in my pig script. But it gives following error
ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1070: Could not resolve UserDefined.PartsOfSpeech using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]
Here is my UDF code
public String exec(Tuple input) throws IOException {
//my code here
}
Here is my pig script
REGISTER /home/bigdata/NetBeansProjects/UserDefined/dist/UserDefined.jar
a = load '/user/bigdata/json' using TextLoader() as (input:chararray);
b = foreach a GENERATE UserDefined.PartsOfSpeech(input);
In the above code UserDefined is my package name and PartsOfSpeech is my class name

The error message says that Pig cannot find UserDefined.PartsOfSpeech.
What package declaration does PartsOfSpeech.java have at the top of the file?
If the package declaration is package com.my.company; try this instead:
REGISTER /home/bigdata/NetBeansProjects/UserDefined/dist/UserDefined.jar
a = load '/user/bigdata/json' using TextLoader() as (input:chararray);
b = foreach a GENERATE com.my.company.PartsOfSpeech(input);
That is, replace UserDefined.PartsOfSpeech(input) with com.my.company.PartsOfSpeech(input) since the UDF is located in the package com.my.company.
Also, consider using the DEFINE keyword in your Pig script so you don't need to repeat com.my.company every time you use PartsOfSpeech.
DEFINE PartsOfSpeech UserDefined.dist.PartsOfSpeech();
REGISTER /home/bigdata/NetBeansProjects/UserDefined/dist/UserDefined.jar
a = load '/user/bigdata/json' using TextLoader() as (input:chararray);
b = foreach a GENERATE PartsOfSpeech(input);
There is more information about DEFINE in Chapter 5 of Alan Gates' Programming Pig: http://chimera.labs.oreilly.com/books/1234000001811/ch05.html#udf_define.
Here is an example of DEFINE from Gates' book:
--define.pig
register 'your_path_to_piggybank/piggybank.jar';
define reverse org.apache.pig.piggybank.evaluation.string.Reverse();
divs = load 'NYSE_dividends' as (exchange:chararray, symbol:chararray,
date:chararray, dividends:float);
backwards = foreach divs generate reverse(symbol);

Before compiling your UDF(java class) make sure you have mentioned package name properly. for example if you have mentioned package name-
package com.pig.udf;
It means you need to take care of directory in your linux box as well.
you can follow below mentioned steps to create jar -
Create directory using
mkdir -p com/pig/udf
Create your java class with package com.pig.udf
Compile your java source code using command
javac -cp /usr/lib/pig-0.12.0.2.0.6.0-76.jar YourClass.java
Then go to the directory where you want to create jar for now -
cd ../../..
Now create jar using below command
jar -cvf yourJarName.jar com/
Register the jar in your script using keyword "register" followed by path of the jar
Now use your jar with keyword com.pig.udf.YourJavaClassName
for your scenerio -
REGISTER /home/bigdata/NetBeansProjects/UserDefined/dist/UserDefined.jar
a = load '/user/bigdata/json' using TextLoader() as (input:chararray);
b = foreach a GENERATE com.pig.udf.PartsOfSpeech(input);

Related

Executing the single code java file using java Vs javac using jdk11

I am using JDK11. Below is my sample class -
public class SayHi {
public static void main(String[] args) {
System.out.println("Hi There");
}
}
I executed the above class with command "java filename.java" for below scenarios
ColumnA -> Class declared as public?
ColumnB -> File name same as class name?
ColumnA ColumnB Result
Yes Yes Yes
No Yes Yes
*Yes No Yes
No No Yes
For all the scenarios, the command executed successfully and I got the result. I get compile-time error for the "Yes-No" case, if I run the "javac" command on the file name.
Why I am not getting the compile-time error when I am executing "java" command on the file name?
I have multiple public classes in a single code file. I am able to execute the file using "java filename.java" command. What I am missing with the compile-time issues when running the file with "java" command. Please help me on this.
The answers to all your questions can be found in JEP 330. I believe the following excerpts provide answers to your questions.
the first class found in the source file is executed
The source file should contain one or more top-level classes, the first of which is taken as the class to be executed
The compiler does not enforce the optional restriction defined at the end of JLS §7.6, that a type in a named package should exist in a file whose name is composed from the type name followed by the .java extension
In other words, when you compile a java source code file with javac, the source code file must contain a single, "public" class whose name matches the name of the file. But when you run a java source code file using the java command, the above restriction does not apply.
The class to be executed is the first top-level class found in the source file. It must contain a declaration of the standard public static void main(String[]) method.

Run single junit test with cli

I need to run a single test case run through the cli. I created a runner class
public class Runner {
public static void main(String ... args) throws ClassNotFoundException {
String[] classAndMethod = args[0].split("#");
Request request = Request.method(Class.forName(classAndMethod[0]),
classAndMethod[1]);
Result result = new JUnitCore().run(request);
System.exit(result.wasSuccessful() ? 0 : 1);
}
}
using this answer
Run single test from a JUnit class using command-line.
I downloaded from github a project (https://github.com/apache/incubator-dubbo) on which I want to run the single method, I positioned myself from terminal on the directory containing the Runner class I created and I launched the following command:
java -cp /path/incubator-dubbo/dubbo-cluster /usr/share/java/junit4-4.12.jar /pathClassRunner/src/com/company/Runner org.apache.dubbo.rpc.cluster.StickyTest#testHeartbeat
but I got the following error:
Error: Could not find or load main class pathClassRunner.src.com.company.Runner
can someone help me?
thanks
The command expects you to include any paths in the classpath with NO spaces. A space means that the arguments for the classpath has ended. It there is a space in one of your paths you may include double quotes:
java -cp path1:path2:path3 classname argument
java -cp "path1:path2:path3" classname argument
If a path starts with / it's a full path. If not it's relative to where you run your command. The classname must be the name of the class, not the file, so that means no .class. If your class specifies any package, then the classname is required to contain the package name too, and the path is to the root of the package, not the class itself.
Check that your user has proper access to the files, then try:
java -cp /path/incubator-dubbo/dubbo-cluster:/usr/share/java/junit4-4.12.jar:/pathClassRunner/src/com/company Runner org.apache.dubbo.rpc.cluster.StickyTest#testHeartbeat
I'm assuming your Runner is in the default package. In case it's in the com.company package, you'll want to run this instead:
java -cp /path/incubator-dubbo/dubbo-cluster:/usr/share/java/junit4-4.12.jar:/pathClassRunner/src com.company.Runner org.apache.dubbo.rpc.cluster.StickyTest#testHeartbeat
Another assumption is that your Runner.class is in your /pathClassRunner/src... directory.

Execute Java class using groovy code

I am trying to execute a java .class file using groovy script :
def sout = new StringBuffer(), serr = new StringBuffer()
def proc = "java ${fileName}".execute()
proc.consumeProcessOutput(sout, serr)
proc.waitFor()
println "out> $sout err> $serr"
It gives error: err> Error: Could not find or load main class cle47d1d78d99a44a8ba01f0bc7612ad16 (class name generated using uuid). But when I execute it in bash it gives proper output. I checked filename, pwd and ensured that the .class file exist in the folder where groovy tries to execute command.
I am able to run other commands and also compile .java file successfully using:
def proc = "javac ${file.path}".execute()
Please suggest.
What is the value of ${fileName} . I think it contains value like "ABC.java" . Where as you must use command "java ${className}".
Use:
java ${className}
ex:
If com.abc.ABC is the class containing the main function
java com.abc.ABC
Package is also necessary

ProcessBuilder not executing Java class file properly

In a java file I am calling command line statement to execute another java file. This is what I am doing:
List<String> paramsExecute = new ArrayList<String>();
paramsExecute.add("java");
paramsExecute.add("-cp");
paramsExecute.add("input\programs\User_K Program1");
paramsExecute.add("1 2 3");
ProcessBuilder builderExecute = new ProcessBuilder(paramsExecute);
builderExecute.redirectOutput(new File(a.txt));
builderExecute.redirectError(new File(b.txt));
Execution of one of the Java files is producing b.txt as:
Error: Could not find or load main class 1 2 3
Another java file is producing b.txt as:
Usage: java [-options] class [args...] ...
But, when I am running these statements directly from the command line, it is executing correctly. The folder input\programs\ is in the same path as the src folder. The src folder contains the Java file containing the ProcessBuilder program. I have verified that the .class file is getting created correctly and in the right folder. I am running the program in windows.
Any help appreciated!!
This paramsExecute.add("input\programs\User_K Program1"); is been treated as a single command/parameter, saying that the class path should be equal to input\programs\User_K Program1
I think you want to use something more like...
paramsExecute.add("input\programs\User_K");
paramsExecute.add("Program1");
You should specify the classpath after '-cp' like
List<String> params = new ArrayList<String>();
params.add("java"); /* name of program to run, java */
params.add("-cp"); /* -cp */
params.add(System.getProperty("java.class.path")); /* class path information */
params.add("pkg.to.yourclass.ClassToRun"); /* full quailified class name */
params.add("1"); params.add("2"); params.add("3"); /* this is parameter to main */
"input\programs\User_K Program1" in your code is treated as a classpath information, not class to run because it follows '-cp', and "1 2 3" as a class name, not arguments passed to the main method.
It is not easy to retrieve classpath from the scatch.
If you want to create a process using an class located in the sample src folder, It is good to use System.getProperty("java.class.path"); to inherite classpath, or You should type the path info manually.

Command of start of.class of the file from command line

I created the project in NetBeans (according to the book P. Noutona, G. Shildta "Java2. The fullest management"), having specified as an Example2 project name.
/*
* To change this template, choose Tools | Templates
* and open the template in the editor.
*/
package example2;
/**
*
* #author Asus
*/
/*
* Другой короткий пример.
* Файл "Example2.java"
*/
public class Example2 {
/**
* #param args the command line arguments
*/
public static void main(String[] args) {
// TODO code application logic here
int num; //объявляет переменную с именем num
num=100; //присваивает num значение 100
System.out.println("Значение num: "+num);
num*=2;
System.out.print("Удвоенное значение num: ");
System.out.println(num);
}
}
In case of creation of the project the class example2.Example2 with the D:\ExamplesFromBook\Example2\Example2 project folder (and project layout according to D:\ExamplesFromBook\Example2) was created. I created byte code by means of a command in command line
D:\ExamplesFromBook\Example2\Example2\src\example2>javac Example2.java
therefore the Example2.class file was created. Further I tried to launch various methods.class the file from command line among which, for example, I was such:
D:\ExamplesFromBook\Example2\Example2>java example2.Example2
therefore received an error
Error: Could not find or load main class example2.Example2
What command needs to be used and from what folder to launch?
Your problem lies within the package names. The class you are compiling is inside the package example2 and is called Example2, therefor it's full path is example2.Example2. So far so good. The problem is how java will find the class. Java will transform the package declaration in the command into a file path, in this case example2/Example2.class. Since your class is placed at Example2.class and not at example2/Example2.class, it will not be found and cause an error.
To fix this, create a subdirectory called example2 and place the class file in there.
go to directory D:\ExamplesFromBook\Example2\Example2\src\
then type java -cp D:\ExamplesFromBook\Example2\Example2\src example2.Example2
The main directory where all your class files will lie should be in classpath ( which is D:\ExamplesFromBook\Example2\Example2\src in your case ). Alternatively you can set classpath using your windows Environment variables also. Once that is done; you can run any of your class file using
java <full qualified class name>
for example
java example2.Example2
You are here
D:\ExamplesFromBook\Example2\Example2\src\example2
When you compile
javac Example2.java
the classes will appear in
D:\ExamplesFromBook\Example2\Example2\src\example2
Try dir and list them
Dir D:\ExamplesFromBook\Example2\Example2\src\example2
Now to run a class, it must be in the class path. The lookup will add fully qualified name of class (with dot replaced with / ) and search in every path. In your case
java -cp D:\ExamplesFromBook\Example2\Example2\src example2.Example
should work.
The java program will look for example2/Example2.class under all classpaths you give. See more details at wiki
Change your java compile command
To compile use
javac -d . ClassName.java
to run java class file with package name use
java packageName/className
OR
java packageName.className
Have you tried to compile your .java file to another folder, for xample to "build" as it is done usually?
When Netbeans compiles files automatically it creates the following structure:
"proj_dir"/src/"package"/"code".java
"proj_dir"/build/classes/"package"/"code".class
Try doing it like this, but manually. Then go to "build/classes" dir from command line and type:
java "package"."code"
This is general way to run compiled java-code.
In your case it has to look like:
D:\ExamplesFromBook\Example2\Example2\src\example2\Example2.java
D:\ExamplesFromBook\Example2\Example2\build\classes\example2>Example2.class
CMD commands to do it:
cd D:\ExamplesFromBook\Example2\Example2\src\example2
javac -d ..\..\build\classes Example2.java
cd D:\ExamplesFromBook\Example2\Example2\build\classes
java example2.Example2
It works with simple programs.

Categories