Distributedfilesystem class use local instead of distributed classes - java

I have this line in my code:
DistributedFileSystem.get(conf).delete(new Path(new URI(otherArgs[1])), true);
otherArgs[1] has this value: hdfs://master:54310/input/results
I receive this exception:
Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS:hdfs://master:54310/input/results, expected: file:///
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:354)
at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:55)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:367)
at org.apache.hadoop.fs.ChecksumFileSystem.delete(ChecksumFileSystem.java:430)
at <package>.<classname>.main(Degree.java:137)
Note: I tried to use new Path(otherArgs[1]) without URI but got the exact same error !
Thanks,
-K

It looks like you have not set fs.default.name in core-site.xml.
check this link
If you have already set that, make sure the config files are in classpath
You can also set the fs.default.name property from your driver
conf.set("fs.default.name", "hdfs://yourserver:port");

It turns out that I was running my jar using "hadoop -jar " instead of "hadoop jar ". All conf files are correct and in place.
Problem solved but i still have no idea why using "-jar" made it run as a local (pseudo distributed) !

Related

Getting FlywayException: SP2-0310: unable to open file - when sqlpus # is used with file path

I am using flyway pro with oracleSqlplus enabled. I created a folder structure to organize scripts based on objects. Trying to use # (with path) in the flyway version scripts, which is not working.
folder structure,
topFolder
-folderA
-AScript.sql
-folderB
-BScript.sql
-folderCommon
-AScript.sql
-V1__ASCRIPT.sql
-V2__BSCRIPT.sql
-V3__ASCRIPT.sql
Content of:
V1__ASCRIPT.sql
#AScript.sql -> Which correctly executes topFolder/folderA/Ascript.sql
V2__BScript.sql
#BScript.sql -> Which correctly executes topFolder/folderB/Bscript.sql
V3__ASCRIPT.sql
#topFolder/folderCommon/AScript.sql -> Which is throwing me below error,
org.flywaydb.core.api.FlywayException: SP2-0310: unable to open file "topFolder/folderCommon/AScript.sql"
I tried almost all possible combinations,
with absolute path,
with "#../../topFolder/folderCommon/AScript.sql"
tried setting SQLPATH varible
with ##
which didnt work.
Just giving the file name alone in #, works. But I want to specify the path, so that file names can be re-used and it is less error prone.
I expect the relative path should work with flyway + oraclesqlplus option.
Thanks in advance.
looks like you start the path from the point where fly flyway.locations=filesystem: ends is where your path begins.
Example:
flyway.locations=filesystem:/opt/app/sql
where you scripts are /opt/app/sql/appadb/script.sql
inside masterscript would be
##appadb/script.sql

Docker+Java -VSTS No such file or directory

I'm trying to do continuous integration using VSTS for Java rest api.
I've added following lines for building Docker image:
FROM openjdk:8
ADD $(build.artifactstagingdirectory)/docker-spring.jar docker-spring.jar
EXPOSE 8085
ENTRYPOINT ["java","-jar","docker-spring.jar"]
I'm getting following error:
ADD failed: stat /var/lib/docker/tmp/docker-builder946930284/SpringRest/target/docker-spring.jar: no such file or directory
/usr/local/bin/docker failed with return code: 1
I could see lot few references, still not able to correlate the issue exactly.
This happens when you haven't supplied the path right. Make sure that your variable that has the path has the . for current directory, or / and the full path to the resources:
$(build.artifactstagingdirectory) = ./correct/relative/path/docker-spring.jar
or
$(build.artifactstagingdirectory) = /correct/full/path/path/docker-spring.jar<
Looking at your output I'd assume you're using "path/to/docker-spring.jar" which doesn't work.
EDIT: In your particular case you want to change the following line in the docker file:
What you have:
COPY /SpringRest/target/docker-spring.jar /usr/local/app/docker-spring.jar
What it should be:
COPY ./target/docker-spring.jar /usr/local/app/docker-spring.jar
Notice the little dot. This way it should work.

Error running a java program that uses log4j

So i am trying to run a java file launch.bat sdoc(launch is a small script that have the jar used, but if necessary i can post it here) and i get this error.
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: C:Users
ennasDesktopwebstorelogsvolume-server.sdoc.log
My config file is:
log4j.appender.dest.File=C:\Users\nennas\Desktop\webstore\logs\volume-server.sdoc.log
log4j.appender.dest.layout=org.apache.log4j.PatternLayout
log4j.appender.dest=org.apache.log4j.RollingFileAppender
log4j.rootCategory=INFO, dest
log4j.appender.dest.layout.ConversionPattern=%d - %m%n
I already try some solution like changing \ to / and running as admin, the path is the absolute path. The folder exist but the strange thing is that in the error the url appears all together, by the way i am using windows 7. Ty for the help
Change this
log4j.appender.dest.File=C:\Users\nennas\Desktop\webstore\logs\volume-server.sdoc.log
by
log4j.appender.dest.File=C://Users//nennas//Desktop//webstore//logs//volume-server.sdoc.log
You can try also this
log4j.appender.dest.File=C:\\Users\\nennas\\Desktop\\webstore\\logs\\volume-server.sdoc.log
There is a lot of time since I donĀ“t work with windows, but i remember similar problems with windows path and log4j

Exception Error in creating an instance of RandomDataImpl

I am getting this exception error at
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/commons/math3/random/RandomDataImpl
at this line of code:
randomData_ = new RandomDataImpl();
and this is how the class is imported in the code :
import org.apache.commons.math3.random.RandomDataImpl;
i have added all of the math common classes to the classpath while running the code. any ideas whats going on ?
It looks like you did not set the appropriate classpath.
You can set it as follows:
java -classpath lib/*.jar:. mypackage.MyClass
Try that and I am sure it will work.
If you are using maven, then read this article
You can also check your current classpath with:
System.getProperty("java.class.path")
Just write simple app printing this property out and run it just as you run an app with which you are having problems.

ZMQ - libzmq.so.3: cannot open shared object file: No such file or directory

I'm trying to embed zeroMQ in my app, I followed this guideline to install ZMQ, so till here everything works fine.
I have this line of code in my app:
ZMQ.Context m_context = ZMQ.context(1);
but above line of code raise below exception:
Exception in thread "main" java.lang.UnsatisfiedLinkError: /tmp/libjzmq-812339378390536247.lib: libzmq.so.3: cannot open shared object file: No such file or directory
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary1(ClassLoader.java:1939)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1864)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1825)
at java.lang.Runtime.load0(Runtime.java:792)
at java.lang.System.load(System.java:1059)
at org.zeromq.EmbeddedLibraryTools.loadEmbeddedLibrary(EmbeddedLibraryTools.java:136)
at org.zeromq.EmbeddedLibraryTools.<clinit>(EmbeddedLibraryTools.java:22)
at org.zeromq.ZMQ.<clinit>(ZMQ.java:38)
at com.castaclip.verticals.Messenger.<init>(Messenger.java:125)
at com.castaclip.verticals.PushMessenger.<init>(PushMessenger.java:30)
at com.castaclip.verticals.pushserver.App.setup(App.java:60)
at com.castaclip.verticals.pushserver.App.main(App.java:41)
The error is exactly pointing to this line.
P.S: its a little bit difficult to fully explain this question.. if you have any question plz let me know. thanks.
If you've successfully built libzmq and jzmq in that order, I would run:
$ sudo ldconfig
to update the system library cache. Then I would check to see if LD_LIBRARY_PATH is defined like Raffian mentioned, or set your library path explicitly to something like:
$ java -Djava.library.path=/usr/lib:/usr/local/lib
Finally I tried to figure out the problem.
I was using zeromq-2.1.10 and this was part of the problem.
So I installed zeromq-3.2.3 from the source and problem resolved.
I encountered a mystifying instance of this message when I:
# java -Djava.library.path=/usr/hf/zmq/lib/ -cp '/usr/hf/lib/*:.' com.zmqtest.MA
Exception in thread "main"
java.lang.UnsatisfiedLinkError: /usr/hf/zmq/lib/libjzmq.so:
libzmq.so.3: cannot open shared object file: No such file or directory
which was fixed with a solution that makes no sense at all to me:
# LD_LIBRARY_PATH=/usr/hf/zmq/lib/ java -Djava.library.path=/usr/hf/zmq/lib/ -cp '/usr/hf/lib/*:.' com.zmqtest.MA
wierd.

Categories