We Keep Coding

Getting java.nio.file.AccessDeniedException , when trying to create zip file using java NIO 2 on MAC

import java.io.IOException;
import java.net.URI;
import java.nio.file.FileSystem;
import java.nio.file.FileSystems;
import java.util.HashMap;
import java.util.Map;
public class ZipFileSystem {
public static void main(String[] args) throws IOException {
URI uri = URI.create("jar:file:///sample.zip");
Map<String,String> options = new HashMap<>();
options.put("create","true");
FileSystem fileSystem = FileSystems.newFileSystem(uri, options);
}
}
I have this java simple code , when i try to run this on MAC, getting below exception, am I missing anything?
Exception in thread "main" java.nio.file.AccessDeniedException: /sample.zip
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434)
at java.nio.file.Files.newOutputStream(Files.java:216)
at com.sun.nio.zipfs.ZipFileSystem.<init>(ZipFileSystem.java:116)
at com.sun.nio.zipfs.ZipFileSystemProvider.newFileSystem(ZipFileSystemProvider.java:117)
at java.nio.file.FileSystems.newFileSystem(FileSystems.java:326)
at java.nio.file.FileSystems.newFileSystem(FileSystems.java:276)
at ZipFileSystem.main(ZipFileSystem.java:14)

Typo in word "hdfs" gives me: "java.io.IOException: No FileSystem for scheme: hdfs". Using FileSystem lib over hadoop 2.7.7

While using FileSystem.get(URI.create("hdfs://localhost:9000/"), configuration) I'm getting the error "Typo in word hdfs", when I tried to run the code gives me the IOException:
java.io.IOException: No FileSystem for scheme: hdfs
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2658)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2665)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:93)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2701)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2683)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:372)
at com.oracle.hadoop.client.Test.main(Test.java:53)
I already tried to use in different ways to use the call to hdfs, I'm using the libraries for hadoop 2.7.7
Here is my current code:
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.*;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;
import org.apache.log4j.BasicConfigurator;
import java.io.IOException;
import java.io.InputStream;
import java.net.URI;
public class Test {
public static void main(String []args) {
Configuration conf = new Configuration();
InputStream in = null;
try {
FileSystem fs = FileSystem.get(URI.create("hdfs://localhost:9000/"), conf);
in = fs.open(new Path(uri));
IOUtils.copyBytes(in, System.out, 4096, false);
} catch (IOException e) {
e.printStackTrace();
} finally {
IOUtils.closeStream(in);
}
}

Actually, I just added this maven dependency: http://mvnrepository.com/artifact/org.apache.hadoop/hadoop-hdfs/2.7.7
to maven pom.xml and problem solved.

MapReduce set input and output

I have a file
import java.io.IOException;
import java.nio.file.Paths;
import java.util.*;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.fs.*;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapreduce.*;
import org.apache.hadoop.mapreduce.lib.input.*;
import org.apache.hadoop.mapreduce.lib.output.*;
import org.apache.hadoop.util.*;
public class ViewCount extends Configured implements Tool {
public static void main(String args[]) throws Exception {
int res = ToolRunner.run(new ViewCount(), args);
System.exit(res);
}
public int run(String[] args) throws Exception {
//Path inputPath = new Path(args[0]);
Path inputPath = Paths.get("C:/WorkSpace/input.txt");
Path outputPath = Paths.get("C:/WorkSpace/output.txt");
Configuration conf = getConf();
Job job = new Job(conf, this.getClass().toString());
I try to run a the app in windows. How can I set inputPath and outputPath? The method I use now doesn't work. Before I had
Path inputPath = new Path(args[0]);
Path outputPath = new Path(args[1]);
and I had to go to the command line. Now I want to run the app from the IDE.
I'm getting
Required:
org.apache.hadoop.fs.Path
Found:
java.nio.file.Path

For Eclipse, you could set arguments :
Run -> run configuration -> arguments.
It should be the same in Intellij.

The error tells you that it is expecting a org.apache.hadoop.fs.Path, but instead it receives a java.nio.file.Paths.
This means that you should change the second import of your code to
org.apache.hadoop.fs.Path. IDEs import suggestions can be wrong some times ;)
Change the import and then use the method that you already had to add the input and output path. Those arguments are given in Eclipse with right-clicking the project -> Run as -> Run configurations -> arguments. The two paths should be white-space separated. Apply and run!
For the next executions, just run the project.

Exceptions in accessing HDFS file system in Java

I tried accessing HDFS file with Java API as the following code:
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.FSDataInputStream;
public static void main(args[]) {
Configuration conf = new Configuration();
conf.addResource(new Path("/etc/hadoop/conf/core-site.xml"));
conf.addResource(new Path("/etc/hadoop/conf/hdfs-site.xml"));
try {
Path path = new Path("hdfs://mycluster/user/mock/test.txt");
FileSystem fs = FileSystem.get(path.toUri(), conf);
if (fs.exists(path)) {
FSDataInputStream inputStream = fs.open(path);
// Process input stream ...
}
else
System.out.println("File does not exist");
} catch (IOException e) {
System.out.println(e.getMessage());
An exception occurred at FileSystem.get(path.toUri(), conf) saying that Couldn't create proxy provider class org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider which is caused by java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.security.Credentials.
I did not find much information about the error. Is the issue due to the wrong API (org.apache.hadoop.hdfs instead of org.apache.hadoop.fs)?

1)Do you have the hadoop-hdfs-.jar available in your classpath?
2)How are you downloading the dependencies? Maven/Manual/Other
3)Could you please provide the stacktrace?

how to create directory on hadoop machine running on vm using hadoop api

I have a java client program that creates directory, but when execute the program its creating directory on my local machine even i have configured fs.defaultFS to vm url that matches core-site.xml.
here is the sample program that creates directory.
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
public class Mkdir {
public static void main(String ar[]) throws IOException
{
Configuration conf = new Configuration();
conf.set("fs.defaultFS", "hdfs://testing:8020");
FileSystem fileSystem = FileSystem.get(conf);
Path path = new Path("/user/newuser");
fileSystem.mkdirs(path) ;
fileSystem.close();
}
}

add this two file in your code
Configuration conf = new Configuration();
conf.addResource(new Path("/home/user17/BigData/hadoop/core-site.xml"));
conf.addResource(new Path("/home/user17/BigData/hadoop/hdfs-site.xml"));
FileSystem fileSystem = FileSystem.get(conf);
give path according to your system