We Keep Coding

Starting and monitoring spark application from a Java application

I am working on an application to execute spark batch application from a java application.
There is one main class which starts the thread to start spark application. It uses zookeeper to find the leader among machines which would start the spark application. Main method looks like this:
public static void main(String[] args) throws IOException {
final int id = Integer.valueOf(args[0]);
final String zkURL = args[1];
final ExecutorService service = Executors.newSingleThreadExecutor();
final Future<?> status = service.submit(new ProcessNode(id, zkURL));
try {
status.get();
} catch (InterruptedException | ExecutionException e) {
LOG.fatal(e.getMessage(), e);
service.shutdown();
}
Once the leader is selected , following code would run on it to start spark application.
protected Boolean executeCommand() {
try {
final Runtime rt = Runtime.getRuntime();
final Process proc = rt.exec("sh start-sparkapp.sh");
final int exitVal = proc.waitFor();
BufferedReader buf = new BufferedReader(new InputStreamReader(proc.getInputStream()));
String line = "";
while ((line=buf.readLine())!=null) {
System.out.println(line);
}
System.out.println(" commandToExecute exited with code: " + exitVal);
proc.destroy();
} catch (final Exception e) {
System.out.println("Exception occurred while Launching process : " + e.getMessage());
return Boolean.FALSE;
}
return Boolean.TRUE;
}
But this starts a long running spark job. So I believe, next part of code would be executed only when spark job is finished. My requirement is as soon as spark application is started , the control goes to the next part of code, where I am monitoring the status of same spark application. i.e I start the spark application and monitor the status of spark application from same java application.
Assume I have a method montior which monitors the status of application
public String monitor(ApplicationId id)
Any suggestion how to achieve this?

Since you will be monitoring your Spark application using the method public String monitor(ApplicationId id) , I am assuming you do not want your current thread to wait on the process using proc.waitFor(). Additionally, you do not want to print the normal output of the process to your console. Both these operations make your thread wait on the spawned process. Moreover, your monitor method should take the process-id of the spawned process, rather than the spark applicationId, as input.
So the modified code could look like:
protected Boolean executeCommand() {
try {
final Runtime rt = Runtime.getRuntime();
final Process proc = rt.exec("sh start-sparkapp.sh");
/*
*Call to method monitor(ProcessId id)
*/
} catch (final Exception e) {
System.out.println("Exception occurred while Launching process : " + e.getMessage());
return Boolean.FALSE;
}
return Boolean.TRUE;
}

Run BASH command in JAVA in background

I made a function that executes a command from BASH, and i want to make it run in background and never stop the execution of the main program.
I could use screen -AmdS screen_thread123 php script.php but the main ideea is that i learn and understand how threads work.
I have a basic knowledge about this, but right now i want to create a quick dynamic thread like the example of bellow :
public static void exec_command_background(String command) throws IOException, InterruptedException
{
List<String> listCommands = new ArrayList<String>();
String[] arrayExplodedCommands = command.split(" ");
// it should work also with listCommands.addAll(Arrays.asList(arrayExplodedCommands));
for(String element : arrayExplodedCommands)
{
listCommands.add(element);
}
new Thread(new Runnable(){
public void run()
{
try
{
ProcessBuilder ps = new ProcessBuilder(listCommands);
ps.redirectErrorStream(true);
Process p = ps.start();
p.waitFor();
}
catch (IOException e)
{
}
finally
{
}
}
}).start();
}
and it gives me this error
NologinScanner.java:206: error: local variable listCommands is accessed from within inner class; needs to be declared final
ProcessBuilder ps = new ProcessBuilder(listCommands);
1 error
Why is that and how can i solve it? I mean how can i access the variable listCommands from this block?
new Thread(new Runnable(){
public void run()
{
try
{
// code here
}
catch (IOException e)
{
}
finally
{
}
}
}).start();
}
Thanks.

You don't need that inner class (and you don't want to waitFor)... just use
for(String element : arrayExplodedCommands)
{
listCommands.add(element);
}
ProcessBuilder ps = new ProcessBuilder(listCommands);
ps.redirectErrorStream(true);
Process p = ps.start();
// That's it.
As for your question of accessing the variable listCommands in your original block; make the reference final - like so
final List<String> listCommands = new ArrayList<String>();

Unit Testing Apache Commons Daemon in Java

Iv researched around and cant seem to find any decent resource to help me JUnit test an apache commons daemon written in Java. I would like to be able to test that when a daemon starts it starts without fail and when it shuts down its shuts down without fail.
Here is some example code of the daemon starting and stopping after a period:
Update
public class MyDaemon implements Daemon
{
private Logger myLogger = LogManager.getLogger(FileLoggerImpl.class);
private List<FileLogger> loggers;
private List<Thread> threads;
public void init(DaemonContext arg0) throws DaemonInitException, Exception
{
myLogger.info("Starting Logger");
loggers = new ArrayList<FileLogger>();
threads = new ArrayList<Thread>();
myLogger.info("Finished starting Logger");
}
public void start() throws Exception
{
if(threads.size()>0 || loggers.size()>0)
stop();
for(int i = 0; i < 1; i++)
{
FileLogger logger = new FileLoggerImpl(Integer.toString(i));
Thread thread = new Thread(logger);
loggers.add(logger);
threads.add(thread);
thread.start();
}
}
public void stop() throws Exception
{
myLogger.info("Cleaning up threads...");
for(int i = 0; i < 1; i++)
{
FileLogger logger = loggers.get(i);
Thread thread = threads.get(i);
logger.isExecuting(false);
thread.join();
}
myLogger.info("Stopping thread");
}
public void destroy()
{
myLogger.info("Destroying resources...");
loggers = null;
threads = null;
myLogger.info("Destroyed resources.");
}
public static void main(String argsv[])
throws Exception
{
MyDaemon myDaemon = new MyDaemon();
myDaemon.init(null);
myDaemon.start();
Thread.sleep(30000);
myDaemon.stop();
}
}
The Logger Class reads in a text file and writes random lines from this text file to a log file *
public void run()
{
String fileLocation = "/textFileLocation";
while(isExecuting)
{
try {
logger.info(getRandomLineOpt(fileLocation));
} catch (IOException e) {
logger.warn("File :" + fileLocation +" not found!");
e.printStackTrace();
}
pause(DELAY_SECONDS);
}
}
public String getRandomLine(String fileLoc) throws IOException
{
BufferedReader reader = new BufferedReader(new FileReader(fileLoc));
//Commons IO
ArrayList<String> lines = new ArrayList<String>();
String line =null;
while( (line = reader.readLine())!= null )
lines.add(line);
return lines.get(new Random().nextInt(lines.size()));
}
Any help much appreciated.

Since your example code is missing some important facts we can only guess what you are trying to do.
I assume that MyDaemon is a subclass of Thread. If I'm correct, then
you shouldn't use stop for the shutdown (read
http://docs.oracle.com/javase/1.5.0/docs/guide/misc/threadPrimitiveDeprecation.html
for more information about this).
Next point: to do some Unit-testing you need a "unit" to test, i.e. you need some methods which should be tested where you can specify the expected output for certain input parameters.
Last point: Since your code ends with the stop call, you cannot determine if the Daemon has been stopped by the stop call or by the shutdown of the whole Virtual machine after the end of the main method has been reached.

Reading output from java.lang.Process - There is nothing to read

I'm trying to execute terminal command in linux trough Java and i cant get any input from inputStream.
This is my code
ProcessBuilder build = new ProcessBuilder("/usr/bin/xterm", "find /home");
Process pr = null;
BufferedReader buf;
try {
build.redirectErrorStream(true);
pr = build.start();
buf = new BufferedReader(new InputStreamReader( pr.getInputStream()));
String line = buf.readLine();
pr.waitFor();
while (true) {
System.out.println(line + "sadasdas");
line = buf.readLine();
}
} catch (Exception e) {
e.printStackTrace();
}
Process is executed and immediately terminal closes, and no output is catched and printed. On the other hand if i will compose an unknown command i get all the lines with tips how to use commands. Same problem i had with windows cmd. I was trying to use getRuntime.exec(cmd) method but the end is the same.
I've also tried to created separate threads for process and reader which looks like this
public class kurdee
{
public static Thread thread;
public kurdee()
{
List cmd = new LinkedList();
cmd.add(new String("/usr/bin/xterm"));
cmd.add(new String("find"));
thisProc thispr = new thisProc(cmd);
this.thread = new Thread(thispr);
thread.start();
reader rd = new reader(thispr.proc);
Thread thread1 = new Thread(rd);
thread1.start();}
public static void main(String args[])
{
java.awt.EventQueue.invokeLater(new Runnable() {
public void run() {
kurdee kurd = new kurdee();
}
});
}
}
class reader implements Runnable
{
private BufferedReader buf;
private Process proc;
public reader(Process proc)
{
this.proc=proc;
this.buf = new BufferedReader(new InputStreamReader(proc.getInputStream()));
}
public void run()
{
String line="";
System.out.println("Thread is alive");
try{
//Thread.sleep(1000);
line = buf.readLine();
}catch(Exception ex){System.out.println(ex + " before first while started");}
while(kurdee.thread.isAlive())
{
System.out.println("Thread is alive");
while(line!=null)
{
try{
//System.out.println(proc.exitValue());
System.out.println(line + " asd");
line=buf.readLine();
}catch(Exception e){System.out.println(e + " Inner while loop");}
}
}
}
}
class thisProc implements Runnable
{
private ProcessBuilder build;
public static Process proc=null;
public thisProc(List<String> args)
{
this.build = new ProcessBuilder(args);
build.redirectErrorStream(true);
try{
this.proc = build.start();
}catch(Exception ex){System.out.println(ex + " proc class");}
}
public void run()
{
try{
proc.waitFor();
}catch(Exception ex){System.out.println(ex + " proc class");}
}
}
But with any combination of invoking threads etc i make there is still nothing to read.
I'm trying to use command "find /home -xdev -samefile file" to get all hard links to file so maybe there is an easier way.

xterm is not the way to execute processes in unix, it is not a shell. a shell is something like "/bin/sh". however, "find" is a normal unix executable, so you should just execute that directly, e.g. new ProcessBuilder("find", "/home"). and yes, you should always process the streams on separate threads, as recommended by this article.

First, don't try to execute the command with xterm, that's pointless; just do it directly. Secondly, be careful when you compose your array of command strings to put one word into each string; passing, for example "find /home" as a single string among many to ProcessBuilder is going to error out.

To check if a file is written completely

How do I know if a software is done writing a file if I am executing that software from java?For example, I am executing geniatagger.exe with an input file RawText that will produce an output file TAGGEDTEXT.txt. When geniatagger.exe is finished writing the TAGGEDTEXT.txt file, I can do some other staffs with this file. The problem is- how can I know that geniatagger is finished writing the text file?
try{
Runtime rt = Runtime.getRuntime();
Process p = rt.exec("geniatagger.exe -i "+ RawText+ " -o TAGGEDTEXT.txt");
}

You can't, or at least not reliably.
In this particular case your best bet is to watch the Process complete.
You get the process' return code as a bonus, this could tell you if an error occurred.
If you are actually talking about this GENIA tagger, below is a practical example which demonstrates various topics (see explanation about numbered comments beneath the code). The code was tested with v1.0 for Linux and demonstrates how to safely run a process which expects both input and output stream piping to work correctly.
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.util.concurrent.Callable;
import org.apache.commons.io.IOUtils;
public class GeniaTagger {
/**
* #param args
*/
public static void main(String[] args) {
tagFile(new File("inputText.txt"), new File("outputText.txt"));
}
public static void tagFile(File input, File output) {
FileInputStream ifs = null;
FileOutputStream ofs = null;
try {
ifs = new FileInputStream(input);
ofs = new FileOutputStream(output);
final FileInputStream ifsRef = ifs;
final FileOutputStream ofsRef = ofs;
// {1}
ProcessBuilder pb = new ProcessBuilder("geniatagger.exe");
final Process pr = pb.start();
// {2}
runInThread(new Callable<Void>() {
public Void call() throws Exception {
IOUtils.copy(ifsRef, pr.getOutputStream());
IOUtils.closeQuietly(pr.getOutputStream()); // {3}
return null;
}
});
runInThread(new Callable<Void>() {
public Void call() throws Exception {
IOUtils.copy(pr.getInputStream(), ofsRef); // {4}
return null;
}
});
runInThread(new Callable<Void>() {
public Void call() throws Exception {
IOUtils.copy(pr.getErrorStream(), System.err);
return null;
}
});
// {5}
pr.waitFor();
// output file is written at this point.
} catch (Exception e) {
e.printStackTrace();
} finally {
// {6}
IOUtils.closeQuietly(ifs);
IOUtils.closeQuietly(ofs);
}
}
public static void runInThread(final Callable<?> c) {
new Thread() {
public void run() {
try {
c.call();
} catch (Exception e) {
e.printStackTrace();
} finally {
}
}
}.start();
}
}
Use a ProcessBuilder to start your process, it has a better interface than plain-old Runtime.getRuntime().exec(...).
Set up stream piping in different threads, otherwhise the waitFor() call in ({5}) might never complete.
Note that I piped a FileInputStream to the process. According to the afore-mentioned GENIA page, this command expects actual input instead of a -i parameter. The OutputStream which connects to the process must be closed, otherwhise the program will keep running!
Copy the result of the process to a FileOutputStream, the result file your are waiting for.
Let the main thread wait until the process completes.
Clean up all streams.

If the program exits after generating the output file then you can call Process.waitFor() to let it run to completion then you can process the file. Note that you will likely have to drain both the standard output and error streams (at least on Windows) for the process to finish.
[Edit]
Here is an example, untested and likely fraught with problems:
// ...
Process p = rt.exec("geniatagger.exe -i "+ RawText+ " -o TAGGEDTEXT.txt");
drain(p.getInputStream());
drain(p.getErrorStream());
int exitCode = p.waitFor();
// Now you should be able to process the output file.
}
private static void drain(InputStream in) throws IOException {
while (in.read() != -1);
}