We Keep Coding

Typo in word "hdfs" gives me: "java.io.IOException: No FileSystem for scheme: hdfs". Using FileSystem lib over hadoop 2.7.7

While using FileSystem.get(URI.create("hdfs://localhost:9000/"), configuration) I'm getting the error "Typo in word hdfs", when I tried to run the code gives me the IOException:
java.io.IOException: No FileSystem for scheme: hdfs
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2658)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2665)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:93)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2701)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2683)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:372)
at com.oracle.hadoop.client.Test.main(Test.java:53)
I already tried to use in different ways to use the call to hdfs, I'm using the libraries for hadoop 2.7.7
Here is my current code:
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.*;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;
import org.apache.log4j.BasicConfigurator;
import java.io.IOException;
import java.io.InputStream;
import java.net.URI;
public class Test {
public static void main(String []args) {
Configuration conf = new Configuration();
InputStream in = null;
try {
FileSystem fs = FileSystem.get(URI.create("hdfs://localhost:9000/"), conf);
in = fs.open(new Path(uri));
IOUtils.copyBytes(in, System.out, 4096, false);
} catch (IOException e) {
e.printStackTrace();
} finally {
IOUtils.closeStream(in);
}
}

Actually, I just added this maven dependency: http://mvnrepository.com/artifact/org.apache.hadoop/hadoop-hdfs/2.7.7
to maven pom.xml and problem solved.

How to Invoke "Powershell script file" from Java in Linux operating system

Class:-
=====================
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
public class TestPowershell {
public static void main(String[] args) throws IOException
{
Runtime runtime = Runtime.getRuntime();
Process proc = runtime.exec("cmd powershell \"\\Test\\Powershell\\powershell.ps1\" ");
proc.getOutputStream().close();
InputStream is = proc.getInputStream();
InputStreamReader isr = new InputStreamReader(is);
BufferedReader reader = new BufferedReader(isr);
String line;
while ((line = reader.readLine()) != null)
{
System.out.println(line);
}
reader.close();
proc.getOutputStream().close();
}
}
I am trying to execute a powershell file by using java in linux environment , i am getting exceptions (above i attached class and exceptions), kindly provide me a test class which can execute powershell script file in linux. Thanks in advance

1st download the FreeSSHD http://www.freesshd.com/?ctt=download in your windows(server). make sure run it as Administrator.
for setup FreeSSHD follow this URL http://www.techrepublic.com/blog/tr-dojo/set-up-a-free-ssh-server-on-windows-7-with-freesshd/ after setup you can ssh that windows system from linux or using putty.
to execute powershell script from linux to remote windows system using java
package com.sysvana.router.config;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import ch.ethz.ssh2.Connection;
import ch.ethz.ssh2.Session;
import ch.ethz.ssh2.StreamGobbler;
public class Test {
static String hostname = "10.1.10.60";
static String username = "administrator";
static String password = "P#ssw0rd";
public static void main(String[] args) throws IOException {
Connection conn = new Connection(hostname);
conn.connect();
boolean isAuthenticated = conn.authenticateWithPassword (username, password);
if (isAuthenticated == false){
System.out.println("authentication failed");
}
System.out.println(isAuthenticated);
Session sess = conn.openSession ();
sess.execCommand ("powershell C:/Users/Administrator/Desktop/test.ps1");
InputStream stdout = new StreamGobbler (sess.getStdout ());
BufferedReader br = new BufferedReader (new InputStreamReader (stdout));
while (true)
{
String line = br.readLine ();
if (line == null) break;
System.out.println (line);
}
System.out.println ("Exit code" + sess.getExitStatus ());
sess.close ();
conn.close ();
}
}
use Ganymed SSH-2 jar http://www.ganymed.ethz.ch/ssh2/

Your problem is that you're not actually trying to run PowerShell. You're using this:
runtime.exec("cmd powershell \"\\Test\\Powershell\\powershell.ps1\" ")
which means it's trying to run a file called cmd, which is not part of Linux.
Also, in another part of the code, you're trying to run powershell.exe. Typically, Linux programs don't have a .exe extension. You may have ported your Java application from Windows, but you need to be aware that Linux is different. It doesn't come with Powershell as standard, and doesn't use .exe file extension.
You can install PowerShell at https://github.com/PowerShell/PowerShell.
On Linux, PowerShell (once installed from the above) is invoked with the pwsh command. You will need to change your Java program accordingly.

Thanks for all your answers.
Finally, I got to know while working with PowerShell we should run the script in windows OS only because Microsoft is the owner for PowerShell and they give more features in Windows OS.
what I did is, I ran the script in Windows OS & generated a CSV file and kept in SFTP folder, by using java I loaded my file and processed my next process.

Copy Json Flat file from local to HDFS

package com.Main;
import java.io.BufferedInputStream;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.URI;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;
public class Main {
public static void main(String[] args) throws IOException {
//Source file in the local file system
String localSrc = args[0];
//Destination file in HDFS
String dst = args[1];
//Input stream for the file in local file system to be written to HDFS
InputStream in = new BufferedInputStream(new FileInputStream(localSrc));
//Get configimport org.apache.commons.configuration.Configuration;uration of Hadoop system
Configuration conf = new Configuration();
System.out.println("Connecting to -- "+conf.get("fs.defaultFS"));
//Destination file in HDFS
FileSystem fs = FileSystem.get(URI.create(dst), conf);
OutputStream out = fs.create(new Path(dst));
//Copy file from local to HDFS
IOUtils.copyBytes(in, out, 4096, true);
System.out.println(dst + " copied to HDFS");
}
}
AM getting following error message "Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 0
at com.Main.Main.main(Main.java:22)"
I have Json file in my local , have to move that in HDFS
Ex:
{"Del":"Ef77xvP","time":1509073785106},
{"Del":"2YXsF7r","time":1509073795109}

Specify command line arguments to your program. You code snippet expects first argument to be source and next arguments to be destination.
For more details refer to What is "String args[]"? parameter in main method Java

How to run NPM Command in Java using Process Builder

import java.io.BufferedOutputStream;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStreamReader;
import java.util.ArrayList;
import java.util.List;
import java.util.*;
public class TestUnZip {
public static void main(String[] args) throws IOException, InterruptedException{
String destFolder="E:\\TestScript";
/*
* Location where the Nodejs Project is Present
*/
System.out.println(destFolder);
String cmdPrompt="cmd";
String path="/c";
String npmUpdate="npm update";
String npm="npm";
String update="update";
File jsFile=new File(destFolder);
List<String> updateCommand=new ArrayList<String>();
updateCommand.add(cmdPrompt);
updateCommand.add(path);
updateCommand.add(npmUpdate);
runExecution(updateCommand,jsFile);
}
public static void runExecution(List<String> command, File navigatePath) throws IOException, InterruptedException{
System.out.println(command);
ProcessBuilder executeProcess=new ProcessBuilder(command);
executeProcess.directory(navigatePath);
Process resultExecution=executeProcess.start();
BufferedReader br=new BufferedReader(new InputStreamReader(resultExecution.getInputStream()));
StringBuffer sb=new StringBuffer();
String line;
while((line=br.readLine())!=null){
sb.append(line+System.getProperty("line.separator"));
}
br.close();
int resultStatust=resultExecution.waitFor();
System.out.println("Result of Execution"+(resultStatust==0?"\tSuccess":"\tFailure"));
}
}
The Above Program works fine, but this program is depend on Windows Machine, I want to run the same program in other Machine as well.
1) NPM is a Command comes as a bundle of NodeJS. (I run NodeJS as a service, I have defined the Environment Variable, so I can run npm update command from any folder)
2) I can't find a work around to run the npm update command without using the "cmd", "/c". If I do I get following error
Exception in thread "main" java.io.IOException: Cannot run program "npm update" (in directory "E:\TestScript"): CreateProcess error=2, The system cannot find the file specified
at java.lang.ProcessBuilder.start(Unknown Source)
3) Do we have option of Running the npm update command as a parameter of Node.exe. If so can anyone provide me the proper work around.
4) Same as I like, I use mocha framework to run the test script and result generates the .xml file.
5) I want mocha command also being invoked using process builder.

The problem is that ProcessBuilder does not respect the PATHEXT variable on Windows.
It's true there is no npm binary on Windows, there's a npm.cmd. My best solution is to check the platform. Something like this:
static boolean isWindows() {
return System.getProperty("os.name").toLowerCase().contains("win");
}
static String npm = isWindows() ? "npm.cmd" : "npm";
static void run() {
Process process = new ProcessBuilder(npm, "update")
.directory(navigatePath)
.start()
}

In Unix or Linux os , the PathBuilder takes the default environment path , so we have to change the environment path and run the npm command through the bash.
import java.io.File;
import java.util.Map;
public class CommandExecutor {
public void exceuteCommand(String commandString,String
directoryToExecuteCommand) {
try {
ProcessBuilder processBuilder = new ProcessBuilder(new String{"bash", "-c",commandString});
Map<String, String> env = processBuilder.environment();
processBuilder.directory(new File(directoryToExecuteCommand));
String envPath="/home/admin123/.nvm/versions/node/v10.15.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin";
env.put("PATH",envPath);
processBuilder.start();
} catch (Exception e) {
e.printStackTrace();
}
}
public static void main(String args[]) {
CommandExecutor commandExecutor=new CommandExecutor();
commandExecutor.exceuteCommand("npm install", "/home/admin123/Desktop");
}
}

Write an executable .sh file with Java for OSX

So I am trying to write an .sh file that will be executable, this is how I'm currently writing it:
Writer output = null;
try {
output = new BufferedWriter(new FileWriter(file2));
output.write(shellScriptContent);
output.close();
} catch (IOException ex) {
Logger.getLogger(PunchGUI.class.getName()).log(Level.SEVERE, null, ex);
}
So that writes the file just fine, but it is not executable. Is there a way to change the executable status when I write it?
Edit: To further clarify, I am trying to make it execute by default, so that for instance, if you double clicked the generated file, it would automatically execute.

You can call File.setExecutable() to set the owner's executable bit for the file, which might be sufficient for your case. Or you can just chmod it yourself with a system call with Process.
Alas, full-powered programmatic alteration of file permissions isn't available until Java 7. It'll be part of the New IO feature set, which you can read more about here.

You'd need to chmod it, and you can probably do it by exec'ing a system command like such:
Really all you'd need is to fire off something like this:
Runtime.getRuntime().exec("chmod u+x "+FILENAME);
But if you want to keep track of it more explicitly can capture stdin / stderr then something more like:
Process p = Runtime.getRuntime().exec("chmod u+x "+FILENAME);
BufferedReader stdInput = new BufferedReader(new InputStreamReader(p.getInputStream()));
BufferedReader stdError = new BufferedReader(new InputStreamReader(p.getErrorStream()));
Which I got from here:
http://www.devdaily.com/java/edu/pj/pj010016/pj010016.shtml
Update:
Test program:
package junk;
import java.io.BufferedWriter;
import java.io.FileWriter;
import java.io.IOException;
import java.io.Writer;
public class Main{
private String scriptContent = '#!/bin/bash \n echo "yeah toast!" > /tmp/toast.txt';
public void doIt(){
try{
Writer output = new BufferedWriter(new FileWriter("/tmp/toast.sh"));
output.write(scriptContent);
output.close();
Runtime.getRuntime().exec("chmod u+x /tmp/toast.sh");
}catch (IOException ex){}
}
public static void main(String[] args){
Main m = new Main();
m.doIt();
}
}
On linux if you open up a file browser and double click on /tmp/toast.sh and choose to run it, it should generate a text file /tmp/toast.txt with the words 'yeah toast'. I assume Mac would do the same since it's BSD under the hood.

In Java 7 you can call Files.setPosixFilePermissions. Here is an example:
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.attribute.PosixFilePermission;
import java.util.Set;
class FilePermissionExample {
public static void main(String[] args) throws IOException {
final Path filepath = Paths.get("path", "to", "file.txt");
final Set<PosixFilePermission> permissions = Files.getPosixFilePermissions(filepath);
permissions.add(PosixFilePermission.OWNER_EXECUTE);
Files.setPosixFilePermissions(filepath, permissions);
}
}

On Mac OS X, besides chmod +x, you have to give a .command extension to your shell script if you want to launch it with a double-click.

This answer I wrote for the question how do I programmatically change file permissions shows a chmod example via a native call using jna, which should work on Mac OS X.