I need to grep some text inside a list of files(file count is huge) in unix server and then list the file name in a web gui. So I decided best way to achieve this is by writing a unix command executer using Runtime.getRuntime().
Works fine for most of the unix command but facing this strange grep issue.
First of all code:
public class UnixCommandExecutor {
private static StringBuffer output = new StringBuffer();
public static String exec(String command) throws Exception{
Process process = null;
process = Runtime.getRuntime().exec(command);
BufferedReader stdErr = getBufferedReader(process.getErrorStream());
BufferedReader stdIn = getBufferedReader(process.getInputStream());
StringBuffer data = extractData(stdErr);
if (data.length() >= 1) {
System.out.println("Error: " +data.toString());
throw new Exception(data.toString());
}
data = extractData(stdIn);
if (data.length() >= 1) {
output = data;
System.out.println("Output: " +data.toString());
}
return output.toString();
}
private static BufferedReader getBufferedReader(InputStream stream) {
InputStreamReader inReader = new InputStreamReader(stream);
BufferedReader buffReader = new BufferedReader(inReader);
return buffReader;
}
private static StringBuffer extractData(BufferedReader reader)
throws IOException {
StringBuffer data = new StringBuffer();
String s = "";
while ((s = reader.readLine()) != null) {
data.append(s + "\n");
}
return data;
}
public StringBuffer getOutput() {
return output;
}
}
Now the call would be something like this:
String output = exec("find . -name blah");
This works fine and the result is perfect. Or any other unix command executes and provides the result properly.
But when grep command is used it gives a strange error:
String output = exec("grep -l executor *");
Error: grep: *: No such file or directory
This is strange, since if I run this command directly on unix it gives the desired result.
Also tried giving the file path something like,
String output = exec("grep -l executor /file/path/*");
even then the error is:
Error: grep: /file/path/*: No such file or directory
Any ideas? or any other better way to solve this?
Any suggestion is welcome.
The * is interpreted by the shell, not grep itself. So it doesn't work if you start grep from your Java program.
You can use grep with -r to make it search recursively, then use . for the current directory:
grep -lr executor .
To simulate the behaviour of the shell, just replace * with a space separated list of all files in the folder you want to search in. You can get the list of all files with:
StringBuilder str = new StringBuilder();
for (File f : new File("your folder").listFiles()) {
if (f.isFile()) {
str.append(' ').append(f.getPath());
}
}
Now just replace the * with the result of str.toString().
Related
String str;
Process p;
try {
String command = "wmctrl -l|awk '{$1=\"\"; $2=\"\"; $3=\"\"; print}'";
p = Runtime.getRuntime().exec(command);
BufferedReader br = new BufferedReader(
new InputStreamReader(p.getInputStream()));
while ((str = br.readLine()) != null) {
activeWindowtitles.add(str);
System.out.println(str);
}
p.waitFor();
p.destroy();
} catch (Exception ex) {
}
I am writing a java code to get all applications name in Linux system. I found a command to achieve this. I ran this command in Terminal and it works fine. But it is not working in Java code as i want only applications name instead of other details. The command is "wmctrl -l | awk '{$1=""; $2=""; $3=""; print}'"
I am getting full output after executing this in java code.
Please tell me how to write this command properly..
Thanks
Personally I would put the wmctrl command in a script and do something like this:
public static List<String> getRunningApps(String executablePath) throws IOException, InterruptedException {
final String ERR_LOG_PATH = "stderr.log";
List<String> result = new ArrayList<>();
ProcessBuilder pb = new ProcessBuilder(executablePath);
pb.redirectError(new File(ERR_LOG_PATH));
Process p = pb.start();
int exitCode = p.waitFor();
if (exitCode != 0) {
throw new RuntimeException(String.format("Error get apps. Check error log %s%n", ERR_LOG_PATH));
}
try (Scanner s = new Scanner(p.getInputStream())) {
while (s.hasNextLine()) {
result.add(s.nextLine().trim());
}
}
return result;
}
That way you can tweak it more easily and keep your code cleaner. The script I used was:
#!/bin/bash
wmctrl -l | awk '{$1=""; $2=""; $3=""; print}'
I am running sed command in java program when i execute my java file in linux.
when i ran the command alone in linux, this will work -> sed '1d;$d' /home/sample/testdata.txt > /home/output/testupdate.txt
however when i ran my java program in linux, its returning exit(1) error . i have read through and use as an array but its still failing .
public static void main(String[] args) {
String[] cmd = {"sed", "sed '1d;$d' /home/sample/testdata.txt > /home/output/testupdate.txt"};
String s;
Process p;
try {
p = Runtime.getRuntime().exec(cmd);
BufferedReader br = new BufferedReader(
new InputStreamReader(p.getInputStream()));
while ((s = br.readLine()) != null) {
System.out.println("line: " + s);
}
p.waitFor();
System.out.println("exit: " + p.exitValue());
p.destroy();
} catch (Exception e) {
}
}
Java is not a shell so does not handle redirects - your original command only works via your Linux shell which understands ">".
You can make Java launch the shell to execute the command, say if you are using bash:
String[] cmd = {"/bin/bash", "-c"
, "sed '1d;$d' /home/sample/testdata.txt > /home/output/testupdate.txt"};
Alternatively see the other helpful comments which link to other pages on dealing with the redirects within the Java code, or to do it inside Java itself (such as with Files.lines / streams).
You don’t need sed. Java can do everything sed can do.
In your case, it appears you are using sed to remove the first and last line from a file. Here’s how you would do that in Java:
Path source = Path.of("/home/sample/testdata.txt");
Path destination = Path.of("/home/output/testupdate.txt");
long lineCount;
try (Stream<?> lines = Files.lines(source)) {
lineCount = lines.count();
}
long lastLine = lineCount - 1;
try (BufferedReader in = Files.newBufferedReader(source);
BufferedReader out = Files.newBufferedWriter(destination)) {
// Skip first line
String line = in.readLine();
long counter = 0;
while ((line = in.readLine()) != null) {
if (++counter < lastLine) {
out.write(line);
out.newLine();
}
}
}
If for some reason you absolutely need to use an external process, there are many other questions and answers which cover your mistake: > is not understood by sed. Normally, it is the shell (like bash) which interprets file redirection.
The proper way to run your command would have been:
ProcessBuilder builder =
new ProcessBuilder("sed", "1d;$d", "/home/sample/testdata.txt");
builder.redirectOutput(new File("/home/output/testupdate.txt"));
builder.redirectError(ProcessBuilder.Redirect.INHERIT);
Process p = builder.start();
does anyone know how to use linux grep with the java ProcessBuilder? Why does this code return an empty string when it should return "sing" ?
import java.io.*;
import java.util.*;
public class Test2 {
public static void main(String[] args) throws InterruptedException,IOException{
String line;
// Initiate grep process.
ProcessBuilder pb = new ProcessBuilder("grep", "\"sing\"", "<<<\"sing\"");
Process p = pb.start();
p.waitFor();
// Get grep output:
BufferedReader reader = new BufferedReader(new InputStreamReader(p.getInputStream()));
StringBuilder builder = new StringBuilder();
line = null;
while ( (line = reader.readLine()) != null) {
builder.append(line);
builder.append(System.getProperty("line.separator"));
}
String result = builder.toString();
System.out.println(result);
}
}
I also try to echo what I execute with this code:
ProcessBuilder pb = new ProcessBuilder("echo","grep", "\"sing\"", "<<<\"sing\"");
and get the correct result:
grep "sing" <<<"sing"
I finally try to execute the command at the shell and get:
sing
although it is in red font for some reason. So what am I doing wrong?
what am I doing wrong?
Something which is pretty obvious.
Do you expect, say, execve(), to understand shell constructs? No.
Well, you shouldn't be expecting ProcessBuilder to understand those either. Although it is not as low level as execve(), it is low level enough that the arguments to a command are "raw". Therefore, in your command, <<<"sing" is passed as is as an argument to grep; which means grep views it as a file to read from.
Get that in your head: what you type in the shell is interpreted by the shell; a ProcessBuilder WILL NOT use a shell to execute its processes, nor will execve(). Which, in turn, means that you cannot use shell constructs.
If you want to grep you'll have to feed your process' input with the text you want. But why use grep when Java has a builtin regex engine is another question, of course.
As to:
although it is in red font for some reason
it is simply text decoration from the grep command (well, GNU grep at least); see its manpage and the --color option. In short, in your case, it has detected that your tty had the capabilities to change the color of text and it uses that to decorate the matched text.
Try and:
echo foobar | grep foo
It will echo foobar with foo in red.
You can actually run the same command using ProcessBuilder but you have to make sure it is execute by bash. I prefer this utility method:
public static int runCmd(final String command) {
Process process=null;
int ret = 0;
String[] finalCommand = new String[] { "bash", "-c", command };
try {
final ProcessBuilder processBuilder = new ProcessBuilder(finalCommand);
processBuilder.redirectErrorStream(true);
process = processBuilder.start();
ret = process.waitFor();
// stdout+stderr
InputStreamReader isr = new InputStreamReader( process.getInputStream() );
BufferedReader br = new BufferedReader(isr);
String line;
while ((line = br.readLine()) != null) {
System.out.println(line);
}
//System.out.println("Program terminated!");
process.destroy();
br.close();
isr.close();
}
catch (IOException|InterruptedException e) {
e.printStackTrace();
}
return ret;
}
Then call it as:
runCmd("grep -o \"sing\" <<<\"icansing\"");
And it gives me this output:
sing
I am using the following code to start a process builder.I want to know how can I redirect its output to a String.
ProcessBuilder pb = new ProcessBuilder(
System.getProperty("user.dir") + "/src/generate_list.sh", filename);
Process p = pb.start();
I tried using ByteArrayOutputStream but it didn't seem to work.
Read from the InputStream. You can append the output to a StringBuilder:
BufferedReader reader =
new BufferedReader(new InputStreamReader(process.getInputStream()));
StringBuilder builder = new StringBuilder();
String line = null;
while ( (line = reader.readLine()) != null) {
builder.append(line);
builder.append(System.getProperty("line.separator"));
}
String result = builder.toString();
Using Apache Commons IOUtils you can do it in one line:
ProcessBuilder pb = new ProcessBuilder("pwd");
String output = IOUtils.toString(pb.start().getInputStream(), StandardCharsets.UTF_8);
As of Java 9, we finally have a one liner:
ProcessBuilder pb = new ProcessBuilder("pwd");
Process process = pb.start();
String result = new String(process.getInputStream().readAllBytes());
Java 8 example:
public static String runCommandForOutput(List<String> params) {
ProcessBuilder pb = new ProcessBuilder(params);
Process p;
String result = "";
try {
p = pb.start();
final BufferedReader reader = new BufferedReader(new InputStreamReader(p.getInputStream()));
StringJoiner sj = new StringJoiner(System.getProperty("line.separator"));
reader.lines().iterator().forEachRemaining(sj::add);
result = sj.toString();
p.waitFor();
p.destroy();
} catch (Exception e) {
e.printStackTrace();
}
return result;
}
Usage:
List<String> params = Arrays.asList("/bin/sh", "-c", "cat /proc/cpuinfo");
String result = runCommandForOutput(params);
I use this exact code and it works well for single or multiple line results. You could add an error stream handler as well.
You might do something like this:
private static BufferedReader getOutput(Process p) {
return new BufferedReader(new InputStreamReader(p.getInputStream()));
}
private static BufferedReader getError(Process p) {
return new BufferedReader(new InputStreamReader(p.getErrorStream()));
}
...
Process p = Runtime.getRuntime().exec(commande);
BufferedReader output = getOutput(p);
BufferedReader error = getError(p);
String ligne = "";
while ((ligne = output.readLine()) != null) {
System.out.println(ligne);
}
while ((ligne = error.readLine()) != null) {
System.out.println(ligne);
}
Just add .inheritIO(); to the process builder line.
IE:
ProcessBuilder pb = new ProcessBuilder(script.sh).inheritIO();
For Java 7 and 8 this should work:
private String getInputAsString(InputStream is)
{
try(java.util.Scanner s = new java.util.Scanner(is))
{
return s.useDelimiter("\\A").hasNext() ? s.next() : "";
}
}
Then in your code, do this:
String stdOut = getInputAsString(p.getInputStream());
String stdErr = getInputAsString(p.getErrorStream());
I shamelessly stole that from: How to redirect Process Builder's output to a string?
In java 8 there is a nice lines() stream you can combine with String.join and System.lineSeparator():
try (BufferedReader outReader = new BufferedReader(new InputStreamReader(p.getInputStream()))
{
return String.join(System.lineSeparator(), outReader.lines().collect(toList()));
\\ OR using jOOλ if you like reduced verbosity
return Seq.seq(outReader.lines()).toString(System.lineSeparator())
}
After trying to handle different cases (handling both stderr and stdout and not blocking any of these, terminate process after timeout, properly escaping slashes, quotation marks, special characters, spaces, .... ) I gave up and found Apache Commons Exec https://commons.apache.org/proper/commons-exec/tutorial.html that seems to be doing all these things pretty well.
I do recommend everyone who needs to invoke external process in java to use Apache Commons Exec library instead of reinventing it again.
Another solution for Java 8:
BufferedReader stdOut = new BufferedReader(new InputStreamReader(process.getInputStream()));
String stdOutStr = stdOut.lines()
.collect(Collectors.joining(System.lineSeparator()));
This is for Kotlin users ending up here:
val myCommand = "java -version"
val process = ProcessBuilder()
.command(myCommand.split(" "))
// .directory(File("./")) // Working directory
.redirectOutput(ProcessBuilder.Redirect.INHERIT)
.redirectError(ProcessBuilder.Redirect.INHERIT)
.start()
process.waitFor(60, TimeUnit.SECONDS)
val result = process.inputStream.reader().readText()
println(result)
Solutions
This code is a running example for the general solution to your question:
How to redirect Process Builder's output to a string?
Credits go to Greg T, after trying multiple solutions to run various commands and capture their outputs Greg T's answer contained the essence of the particular solution. I hope the general example is of use for someone combining multiple requirements whilst capturing the output.
To obtain your particular solution you can uncomment ProcessBuilder pb = new ProcessBuilder(System.getProperty("user.dir")+"/src/generate_list.sh", filename);, uncomment the line, and comment out: ProcessBuilder processBuilder = new ProcessBuilder(commands);.
Functionality
It is a working example that executes command echo 1 and returns the output as a String.
I also added setting a working path and an environment variable, which is not required for your particular example so you can delete it.
Usage & verification
You can copy paste this code as a class, compile it to jar and run it.
It is verified in WSL Ubuntu 16.04.
Setting the workdirectory is verified by setting binaryCommand[0]="touch";and binaryCommand[1]="1";, re-compiling and running the .jar file.
Limitations
If the pipe is full (due to a "too large" output), the code hangs.
Code
import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.PrintWriter;
import java.util.Arrays;
import java.util.Map;
import java.util.StringJoiner;
public class GenerateOutput {
/**
* This code can execute a command and print the output accompanying that command.
* compile this project into a .jar and run it with for example:
* java -jar readOutputOfCommand.jar
*
* #param args
* #throws Exception
*/
public static void main(String[] args) throws Exception {
boolean answerYes = false; // no yes answer to any command prompts is needed.
// to execute a command with spaces in it in terminal, put them in an array of Strings.
String[] binaryCommand = new String[2];
// write a command that gives a binary output:
binaryCommand[0] = "echo";
binaryCommand[1] = "1";
// pass the commands to a method that executes them
System.out.println("The output of the echo command = "+executeCommands(binaryCommand,answerYes));
}
/**
* This executes the commands in terminal.
* Additionally it sets an environment variable (not necessary for your particular solution)
* Additionally it sets a working path (not necessary for your particular solution)
* #param commandData
* #param ansYes
* #throws Exception
*/
public static String executeCommands(String[] commands,Boolean ansYes) throws Exception {
String capturedCommandOutput = null;
System.out.println("Incoming commandData = "+Arrays.deepToString(commands));
File workingDirectory = new File("/mnt/c/testfolder b/");
// create a ProcessBuilder to execute the commands in
ProcessBuilder processBuilder = new ProcessBuilder(commands);
//ProcessBuilder processBuilder = new ProcessBuilder(System.getProperty("user.dir")+"/src/generate_list.sh", "a");
// this is not necessary but can be used to set an environment variable for the command
processBuilder = setEnvironmentVariable(processBuilder);
// this is not necessary but can be used to set the working directory for the command
processBuilder.directory(workingDirectory);
// execute the actual commands
try {
Process process = processBuilder.start();
// capture the output stream of the command
BufferedReader reader = new BufferedReader(new InputStreamReader(process.getInputStream()));
StringJoiner sj = new StringJoiner(System.getProperty("line.separator"));
reader.lines().iterator().forEachRemaining(sj::add);
capturedCommandOutput = sj.toString();
System.out.println("The output of this command ="+ capturedCommandOutput);
// here you connect the output of your command to any new input, e.g. if you get prompted for `yes`
new Thread(new SyncPipe(process.getErrorStream(), System.err)).start();
new Thread(new SyncPipe(process.getInputStream(), System.out)).start();
PrintWriter stdin = new PrintWriter(process.getOutputStream());
//This is not necessary but can be used to answer yes to being prompted
if (ansYes) {
System.out.println("WITH YES!");
stdin.println("yes");
}
// write any other commands you want here
stdin.close();
// this lets you know whether the command execution led to an error(!=0), or not (=0).
int returnCode = process.waitFor();
System.out.println("Return code = " + returnCode);
} catch (IOException e1) {
e1.printStackTrace();
}
return capturedCommandOutput;
}
/**
* source: https://stackoverflow.com/questions/7369664/using-export-in-java
* #param processBuilder
* #param varName
* #param varContent
* #return
*/
private static ProcessBuilder setEnvironmentVariable(ProcessBuilder processBuilder){
String varName = "variableName";
String varContent = "/mnt/c/testfolder a/";
Map<String, String> env = processBuilder.environment();
System.out.println("Setting environment variable "+varName+"="+varContent);
env.put(varName, varContent);
processBuilder.environment().put(varName, varContent);
return processBuilder;
}
}
class SyncPipe implements Runnable
{
/**
* This class pipes the output of your command to any new input you generated
* with stdin. For example, suppose you run cp /mnt/c/a.txt /mnt/b/
* but for some reason you are prompted: "do you really want to copy there yes/no?
* then you can answer yes since your input is piped to the output of your
* original command. (At least that is my practical interpretation might be wrong.)
* #param istrm
* #param ostrm
*/
public SyncPipe(InputStream istrm, OutputStream ostrm) {
istrm_ = istrm;
ostrm_ = ostrm;
}
public void run() {
try
{
final byte[] buffer = new byte[1024];
for (int length = 0; (length = istrm_.read(buffer)) != -1; )
{
ostrm_.write(buffer, 0, length);
}
}
catch (Exception e)
{
e.printStackTrace();
}
}
private final OutputStream ostrm_;
private final InputStream istrm_;
}
I am trying to untar a file on a Unix machine, using a Java batch application.
Source Code:
String fileName = "x98_dms_12";
Runtime.getRuntime().exec("gunzip "+ fileName + ".tar.gz");
System.out.println(" Gunzip:"+"gunzip "+ fileName + ".tar.gz");
Runtime.getRuntime().exec("tar -xvf "+ fileName + ".tar");
System.out.println(" Extract:tar -xvf "+ fileName + ".tar");
Problem Description:
When I run the batch program it does not (completely) work. Only the gunzip command works, converting my fileName.tar.gz to fileName.tar. But the untar command does not seem to do anything, and there is no error or exception in my log or Unix console.
When I run the same commands in a Unix prompt they work fine.
Notes:
The path of execution is correct because it converts my *.tar.gz to *.tar
I cannot use "tar -zxvf fileName.tar.gz" since the attribute "z" does not work on my system.
There is no error or exception thrown.
Please do help.
A couple of things:
The tar command will expand a file relative to your working directory, which might need to be set for your Java Process objects
You should wait for the unzip process to complete before launching into the untar process
You should process the output streams from the processes.
Here is a working example that you can extend/adapt. It uses a separate class to deal with the process output streams:
class StreamGobbler implements Runnable {
private final Process process;
public StreamGobbler(final Process process) {
super();
this.process = process;
}
#Override
public void run() {
try {
final BufferedReader reader = new BufferedReader(
new InputStreamReader(process.getInputStream()));
String line = null;
while ((line = reader.readLine()) != null) {
System.out.println(line);
}
reader.close();
} catch (final Exception e) {
e.printStackTrace();
}
}
}
public void extractTarball(final File workingDir, final String archiveName)
throws Exception {
final String gzFileName = archiveName + ".tar.gz";
final String tarFileName = archiveName + ".tar";
final ProcessBuilder builder = new ProcessBuilder();
builder.redirectErrorStream(true);
builder.directory(workingDir);
builder.command("gunzip", gzFileName);
final Process unzipProcess = builder.start();
new Thread(new StreamGobbler(unzipProcess)).start();
if (unzipProcess.waitFor() == 0) {
System.out.println("Unzip complete, now untarring");
builder.command("tar", "xvf", tarFileName);
final Process untarProcess = builder.start();
new Thread(new StreamGobbler(untarProcess)).start();
System.out.println("Finished untar process. Exit status "
+ untarProcess.waitFor());
}
}
The code below will print the output of the command executed. Check if it returns any error.
Process p = Runtime.getRuntime().exec("tar -xvf "+ fileName + ".tar");
BufferedReader br = new BufferedReader(new InputStreamReader(p.getInputStream()));
String line = null;
while ((line = br.readLine()) != null) {
System.out.println(line);
}
The problem is the commands which we give is UNIX command so it wont work in windows environment. I had written a script file to overcome this problem thanks all for you help. The Runtime.getRuntime.exec() will take some time to execute the command given so after each exec() give thread.wait(3000) to complete the process and goto next thread.