We Keep Coding

What is the proper value of HADOOP_HOME and PIG_CLASSPATH for APACHE HADOOP version 2.8.0?

I have the problem executing the Hadoop command from the PIG command line. The command and the error stack is below
My instructor suspects that it is because HADDOP_HOME and PIG_CLASSPATH are incorrect. I am on the HADOOP version 2.8.0.
So, originally I had HADOOP_HOME as
HADOOP_HOME=<CELLAR_DIRECTORY>/hadoop/2.8.0/
Then I switched the following setup:
HADOOP_HOME=<CELLAR_DIRECTORY>/hadoop/2.8.0/libexec/etc/hadoop
PIG_CLASSPATH is defined as $HADOOP_HOME
Commands I used in pig:
A = LOAD '/Users/anarinsky/Downloads/loaddata1.txt';
B = MAPREDUCE '/Users/anarinsky/workspace/wordcount/target/wordcount-1.jar' STORE A INTO '/Users/anarinsky/Downloads/tempwrite2' LOAD '/Users/anarinsky/Downloads/tempwrite2' AS (word:chararray, count:int) `com.systemskills.hadoop.wordcount.WordCountDriver /wordcountdata /Users/anarinsky/Downloads/pigoptdir`;
Pig Stack Trace
ERROR 2025: Expected leaf of reduce plan to always be POStore. Found PONative
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias B
at org.apache.pig.PigServer.openIterator(PigServer.java:1019)
at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:747)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:376)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:231)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:206)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:66)
at org.apache.pig.Main.run(Main.java:564)
at org.apache.pig.Main.main(Main.java:176)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:234)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias B
at org.apache.pig.PigServer.storeEx(PigServer.java:1122)
at org.apache.pig.PigServer.store(PigServer.java:1081)
at org.apache.pig.PigServer.openIterator(PigServer.java:994)
... 13 more
Caused by: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompilerException: ERROR 2025: Expected leaf of reduce plan to always be POStore. Found PONative
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:321)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.compile(MapReduceLauncher.java:629)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:152)
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:308)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1474)
at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1459)
at org.apache.pig.PigServer.storeEx(PigServer.java:1118)
... 15 more

Alex!
Unfortunately, it's not related to Pig paths (tried it on my configured hadoop cluster) with same result. The error you get refers to the fact that Physical plan compiler has a bug in compile method. So in order to make your attempt work you have two possibilities
Run native MR job using hadoop and after it finishes process it's results in pig
Edit pig source code and compile your own version. You'll need to edit
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler#compile method and replace
for (PhysicalOperator op : leaves) {
if (!(op instanceof POStore)) {
int errCode = 2025;
String msg = "Expected leaf of reduce plan to " +
"always be POStore. Found " + op.getClass().getSimpleName();
throw new MRCompilerException(msg, errCode, PigException.BUG);
}
}
with
for (PhysicalOperator op : leaves) {
if (!(op instanceof POStore) && !(op instanceof PONative)) {
int errCode = 2025;
String msg = "Expected leaf of reduce plan to " +
"always be POStore. Found " + op.getClass().getSimpleName();
throw new MRCompilerException(msg, errCode, PigException.BUG);
}
}

Trying to read *.xlsx file using POI API

I am trying to read an *.xlsx file and i receive quite a lot of errors in the console. I am a beginner at this and please note that if i try to read *.xls files i have no problems there.
This is the code that i use:
public class XlsxTest {
public static void main(String[] args) throws IOException {
FileInputStream fis = new FileInputStream(new File("autotest2.xlsx"));
XSSFWorkbook wb = new XSSFWorkbook(fis);
XSSFSheet sheet = wb.getSheetAt(0);
FormulaEvaluator formulaEvaluator = wb.getCreationHelper().createFormulaEvaluator();
for (Row row : sheet) {
for (Cell cell : row) {
switch (formulaEvaluator.evaluateInCell(cell).getCellType()) {
case Cell.CELL_TYPE_NUMERIC:
System.out.println(cell.getNumericCellValue() + "\t\t");
break;
case Cell.CELL_TYPE_STRING:
System.out.println(cell.getStringCellValue() + "\t\t");
break;
}
}
System.out.println();
}
}
}
When i try to run this i receive the following errors:
Exception in thread "main" org.apache.poi.POIXMLException: java.lang.reflect.InvocationTargetException
at org.apache.poi.xssf.usermodel.XSSFFactory.createDocumentPart(XSSFFactory.java:62)
at org.apache.poi.POIXMLDocumentPart.read(POIXMLDocumentPart.java:465)
at org.apache.poi.POIXMLDocument.load(POIXMLDocument.java:173)
at org.apache.poi.xssf.usermodel.XSSFWorkbook.<init>(XSSFWorkbook.java:278)
at XlsxTest.main(XlsxTest.java:18)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at org.apache.poi.xssf.usermodel.XSSFFactory.createDocumentPart(XSSFFactory.java:60)
... 4 more
Caused by: java.lang.ExceptionInInitializerError
at org.apache.xmlbeans.impl.schema.BuiltinSchemaTypeSystem.fillInType(BuiltinSchemaTypeSystem.java:1025)
at org.apache.xmlbeans.impl.schema.BuiltinSchemaTypeSystem.<clinit>(BuiltinSchemaTypeSystem.java:223)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.xmlbeans.XmlBeans.getNoType(XmlBeans.java:856)
at org.apache.xmlbeans.XmlBeans.<clinit>(XmlBeans.java:881)
at org.openxmlformats.schemas.drawingml.x2006.main.ThemeDocument$Factory.parse(Unknown Source)
at org.apache.poi.xssf.model.ThemesTable.<init>(ThemesTable.java:83)
... 9 more
Caused by: java.lang.RuntimeException: Installation Problem??? Couldn't load messages: Can't find bundle for base name org.apache.xmlbeans.impl.regex.message, locale en_US
at org.apache.xmlbeans.impl.regex.RegexParser.setLocale(RegexParser.java:88)
at org.apache.xmlbeans.impl.regex.RegexParser.<init>(RegexParser.java:78)
at org.apache.xmlbeans.impl.regex.ParserForXMLSchema.<init>(ParserForXMLSchema.java:28)
at org.apache.xmlbeans.impl.regex.RegularExpression.setPattern(RegularExpression.java:2996)
at org.apache.xmlbeans.impl.regex.RegularExpression.setPattern(RegularExpression.java:3009)
at org.apache.xmlbeans.impl.regex.RegularExpression.<init>(RegularExpression.java:2975)
at org.apache.xmlbeans.impl.regex.SchemaRegularExpression.<init>(SchemaRegularExpression.java:27)
at org.apache.xmlbeans.impl.regex.SchemaRegularExpression.<init>(SchemaRegularExpression.java:23)
at org.apache.xmlbeans.impl.regex.SchemaRegularExpression$1.<init>(SchemaRegularExpression.java:44)
at org.apache.xmlbeans.impl.regex.SchemaRegularExpression.buildKnownPatternMap(SchemaRegularExpression.java:43)
at org.apache.xmlbeans.impl.regex.SchemaRegularExpression.<clinit>(SchemaRegularExpression.java:38)
... 19 more
Can anyone help me with this issue? I hope that i provided the necessary information.
Thank you!

Rename jar of lib in zip or simply open org/apache/xmlbeans/impl/regex of loaded lib.
Find file message.properties (and message_xx.properties if necessary).
Put file(s) in used library by org/apache/xmlbeans/impl/regex path.
Profit!

I experienced the very similar exception when using Apache POI: a java.lang.ExceptionInInitializerError, caused by "java.lang.RuntimeException: Installation Problem??? Couldn't load messages: Can't find bundle for base name org.apache.xmlbeans.impl.regex.message" in XMLBeans.
Looked completely weird. Code worked on one machine but failed this way on another for no apparent reason.
After LOTS of debugging hours, I found out the reason was COMPLETELY unrelated to Excel, POI or XMLBeans: my project was located in the folder ending with "!". The solution was simply to rename the folder.
I guess the author had something similar happening, and indeed the linked post about Libre Office has nothing to do with this problem. Instead, the linked post should be the following: Unable to open resources in directories which end with an exclamation mark (!)

Got this error ONLY in junit test. Application runs perfectly fine. So check your case as well.
For test, following Sarck's suggestion worked. #4 is little confusing. So below is simple process.. works fine for 2.6 version of xmlbeans library.
In your project where this is failing, create a file named message.properties under src/test/resources/org/apache/xmlbeans/impl/regex folder. (src/test/resources is typical maven root resource folder). Add following content in that file
parser.parse.1=Wrong character.
parser.parse.2=Invalid reference number.
parser.next.1=A character is required after \\.
parser.next.2='?' is not expected. '(?:' or '(?=' or '(?!' or '(?<' or '(?#' or '(?>'?
parser.next.3='(?<=' or '(?<!' is expected.
parser.next.4=A comment is not terminated.
parser.factor.1=')' is expected.
parser.factor.2=Unexpected end of the pattern in a modifier group.
parser.factor.3=':' is expected.
parser.factor.4=Unexpected end of the pattern in a conditional group.
parser.factor.5=A back reference or an anchor or a lookahead or a lookbehind is expected in a conditional pattern.
parser.factor.6=There are more than three choices in a conditional group.
parser.atom.1=A character in U+0040-U+005f must follow \\c.
parser.atom.2=A '{' is required before a character category.
parser.atom.3=A property name is not closed by '}'.
parser.atom.4=Unexpected meta character.
parser.atom.5=Unknown property.
parser.cc.1=A POSIX character class must be closed by ':]'.
parser.cc.2=Unexpected end of the pattern in a character class.
parser.cc.3=Unknown name for a POSIX character class.
parser.cc.4='-' is invalid here.
parser.cc.5=']' is expected.
parser.cc.6='[' is invalid in a character class. Write '\\['.
parser.cc.7=']' is invalid in a character class. Write '\\]'.
parser.cc.8='-' is an invalid character range. Write '\\-'.
parser.ope.1='[' is expected.
parser.ope.2=')' or '-[' or '+[' or '&[' is expected.
parser.ope.3=The range end code point is less than the start code point.
parser.descape.1=Invalid Unicode hex notation.
parser.descape.2=Overflow in a hex notation.
parser.descape.3='\\x{' must be closed by '}'.
parser.descape.4=Invalid Unicode code point.
parser.descape.5=An anchor must not be here.
parser.process.1=This expression is not supported in the current option setting.
parser.quantifier.1=Invalid quantifier. A digit is expected.
parser.quantifier.2=Invalid quantifier. Invalid quantity or a '}' is missing.
parser.quantifier.3=Invalid quantifier. A digit or '}' is expected.
parser.quantifier.4=Invalid quantifier. A min quantity must be <= a max quantity.
parser.quantifier.5=Invalid quantifier. A quantity value overflow.
Duplicate this file as message_en.properties at same location.
If you are using newer version of xmlbeans jar then get content of this file from Apache source - replace version in url

Unexpected result with fax4j: cannot send fax

I'm trying to send a fax programatically with java by using the fax4j library (http://fax4j.sourceforge.net) and I am unfortunately unable to do so.
I used the sample code provided in the javadoc to try it and ended up modifying it to try and get a feel of the library.
This code WORKS, as in, the program complete its execution without throwing any exception, but it doesn't really send the fax, which is what I'm trying to achieve.
package com.myclasses;
import java.io.*;
import java.util.Properties;
import org.fax4j.FaxClient;
import org.fax4j.FaxClientFactory;
import org.fax4j.FaxJobStatus;
import org.fax4j.FaxJob;
import org.fax4j.FaxJob.FaxJobPriority;
public class MyTestClass
{
public static void main(String[] args)
throws InterruptedException, IOException
{
File file = new File("Drive:\\Directory\\MyFile.pdf");
System.out.println("File Opened");
FaxClient faxClient = FaxClientFactory.createFaxClient();
System.out.println("Client created");
System.out.println(faxClient.getProvider().getName());
System.out.println(faxClient.getProvider().getDescription());
FaxJob faxJob = faxClient.createFaxJob();
System.out.println("Job Created");
faxJob.setFile(file);
System.out.println("file added");
faxJob.setPriority(FaxJobPriority.HIGH_PRIORITY);
System.out.println("priority set");
faxJob.setTargetAddress(recipient);
System.out.println("recipient set");
faxClient.submitFaxJob(faxJob);
System.out.println("Fax Job Sumbitted: " + faxJob.getID());
while (faxClient.getFaxJobStatus(faxJob) == FaxJobStatus.PENDING)
{
System.out.println("STATUS: "
+ faxClient.getFaxJobStatus(faxJob).toString());
for (int i = 0; i < 100; i++)
{
Thread.sleep(100);
System.out.print(".");
}
}
System.out.println("STATUS: "
+ faxClient.getFaxJobStatus(faxJob).toString());
}
}
It does more or less exactly the same than the sample code except that it provides output to the console to mark the points.
I also added the loop at the end to periodically check on the job's status and keep the program running until the job is not "pending" any more. Since that doesn't happen, what I get is an endless loop.
Although quite complete, the javadoc is a bit obscure and difficult to wade through since I do not know exactly what I'm looking for. I couldn't find no tutorial online so any help would be greatly appreciated.
Or if someone knows of a fax sending app using fax4j of which the sources are available (even if it's just consultation only) that would be amazing because that way I could see how it is properly used.
EDIT
When I wrote the question first, I forgot to add a detail which, in retrospect may have some importance.
When the faxClient.sumbitFaxJob(faxJob) instruction is executed, Adobe Reader is opened but no file is loaded. But looking at the file history in Adobe Reader, I get that is is probably located in %HOMEDIR%\AppData\Local\Temp\long-randomly-generated-name.pdf (can't get the full directory with Adobe Reader and file is non-existant when I try to reload it).
Similarly, when I change the source file to a non-pdf, I get an exception within Java about a null file at %HOMEDIR%\AppData\Local\Temp called `fax4j_random-suite-of-digit.vbs :
Exception in thread "main" org.fax4j.FaxException: Error while invoking VBS script, error found at line 145: faxJobID=faxDocumentInstance.Send, script output:
Microsoft (R) Windows Script Host Version 5.8
Copyright (C) Microsoft Corporation 1996-2001. Tous droits réservés.
[fax4j-vbs][DEBUG] Sending fax
Script error:
C:\Users\8809898B\AppData\Local\Temp\fax4j_1590168650663566246.vbs(145, 1) (null): Données non valides.
at org.fax4j.spi.vbs.VBSFaxClientSpi.invokeScript(VBSFaxClientSpi.java:722)
at org.fax4j.spi.vbs.VBSFaxClientSpi.submitFaxJobImpl(VBSFaxClientSpi.java:416)
at org.fax4j.spi.AbstractFaxClientSpi.submitFaxJob(AbstractFaxClientSpi.java:204)
at org.fax4j.spi.adapter.AdapterFaxClientSpi.submitFaxJobImpl(AdapterFaxClientSpi.java:505)
at org.fax4j.spi.AbstractFaxClientSpi.submitFaxJob(AbstractFaxClientSpi.java:204)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.fax4j.spi.AbstractFaxClientSpiProxy.invoke(AbstractFaxClientSpiProxy.java:202)
at $Proxy0.submitFaxJob(Unknown Source)
at org.fax4j.FaxClient.submitFaxJob(FaxClient.java:211)
at com.myclasses.MyTestClass.main(MyTestClass.java:43)

i see you use windows so i suggest to use the windows SPI.
basically you just need to make sure the fax4j.dll and fax4j.exe are on the native path (either set via java.library.path or windows PATH)
also acrobat is not a good third party when it comes to printing and has many issues.
go to the official fax4j forum and you will see recommendation to other pdf readers that support better printing. with acrobat things won't work good.

Exception in thread "main" java.lang.NoClassDefFoundError: =

I created a new maven project in Eclipse and on runtime I get this error:
Exception in thread "main" java.lang.NoClassDefFoundError: =
Caused by: java.lang.ClassNotFoundException: =
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
Could not find the main class: =. Program will exit.
In other threads the class is mentioned where the problem occurs but here it simply says nothing.
The code is also used in a different project (with slight tweaks in terms of calling a method) but the rest of it is same.
If anyone can help me resolve this issue..it will be highly appreciated.

It looks like something is passing in = as the class name. It doesn't say nothing - it says =.
For example, when I run:
java =
I get:
Error: Could not find or load main class =
There's no colon, but it's otherwise the same.
Look at where you're trying to specify the class name, and see whether there's a stray = around. For example, suppose you had:
java -Dfoo = bar ClassName
instead of
java -Dfoo=bar ClassName
You'd see the same thing. I'm not familiar with Maven, but if you ever specify a set of arguments in it, I'd look at that part of the configuration file.

Deleting the workspace worked for me.

java.lang.NoClassDefFoundError: Could not initialize class java.net.ProxySelector

I'm running into a problem with a java app constantly throwing:
java.lang.NoClassDefFoundError: Could not initialize class java.net.ProxySelector.
I am running Suse Linux 10.3 and running java 1.6.0. My CLASSPATH is set to
/usr/lib/jvm/jre-1.6.0-openjdk/lib.
No other users seem to be having this error so I'm assuming its my setup. For those wondering the app is yamj (http://code.google.com/p/moviejukebox/)
Any ideas as to what maybe missing or what I maybe doing wrong?
Edit the full trace of the error is as follows:
java.lang.NoClassDefFoundError: Could not initialize class java.net.ProxySelector
at sun.net.www.protocol.http.HttpURLConnection$5.run(HttpURLConnection.java:736)
at java.security.AccessController.doPrivileged(Native Method)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:732)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:672)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:997)
at com.moviejukebox.thetvdb.tools.XMLHelper.getEventReader(XMLHelper.java:19)
at com.moviejukebox.thetvdb.model.Mirrors.(Mirrors.java:30)
at com.moviejukebox.thetvdb.TheTVDB.(TheTVDB.java:37)
at com.moviejukebox.plugin.TheTvDBPlugin.(TheTvDBPlugin.java:57)
at sun.reflect.GeneratedConstructorAccessor2.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
at java.lang.Class.newInstance0(Class.java:372)
at java.lang.Class.newInstance(Class.java:325)
at com.moviejukebox.plugin.DatabasePluginController.getMovieDatabasePlugin(DatabasePluginController.java:96)
at com.moviejukebox.plugin.DatabasePluginController.access$000(DatabasePluginController.java:30)
at com.moviejukebox.plugin.DatabasePluginController$1.initialValue(DatabasePluginController.java:44)
at com.moviejukebox.plugin.DatabasePluginController$1.initialValue(DatabasePluginController.java:39)
at java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:160)
at java.lang.ThreadLocal.get(ThreadLocal.java:150)
at com.moviejukebox.plugin.DatabasePluginController.scan(DatabasePluginController.java:70)
at com.moviejukebox.MovieJukebox.updateMovieData(MovieJukebox.java:1051)
at com.moviejukebox.MovieJukebox.access$100(MovieJukebox.java:80)
at com.moviejukebox.MovieJukebox$4.call(MovieJukebox.java:613)
at com.moviejukebox.MovieJukebox$4.call(MovieJukebox.java:600)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java

ProxySelector is an abstract class. Are you trying to instantiate it directly?

My CLASSPATH is set to
/usr/lib/jvm/jre-1.6.0-openjdk/lib.
don't think that should be in your CLASSPATH
try clearing the CLASSPATH and running it

Firstly, you shouldn't have to put "/usr/lib/jvm/jre-1.6.0-openjdk/lib" on your classpath. The "java" command should put all of the standard J2SE libraries on the bootclasspath without you doing anything.
Second, it would help if you gave us the full stacktrace, not just the exception message. I suspect that the real problem is that java.net.ProxySelector (or something it depends on) is failing during static initialization. But only a stacktrace would confirm that.

Since it's second result in google search on this error, I want to post this piece of code I found at some forum that helped me with the same exception. Cannot explain in details - it was just a quick test project for me, so I didn't have time for deeper investigation.
static {
try {
Class c = Class.forName("sun.net.spi.DefaultProxySelector");
if (c != null && ProxySelector.class.isAssignableFrom(c)) {
theProxySelector = (ProxySelector) c.newInstance();
}
} catch (Exception e) {
theProxySelector = null;
}
}