Starting and monitoring spark application from a Java application - java

I am working on an application to execute spark batch application from a java application.
There is one main class which starts the thread to start spark application. It uses zookeeper to find the leader among machines which would start the spark application. Main method looks like this:
public static void main(String[] args) throws IOException {
final int id = Integer.valueOf(args[0]);
final String zkURL = args[1];
final ExecutorService service = Executors.newSingleThreadExecutor();
final Future<?> status = service.submit(new ProcessNode(id, zkURL));
try {
status.get();
} catch (InterruptedException | ExecutionException e) {
LOG.fatal(e.getMessage(), e);
service.shutdown();
}
Once the leader is selected , following code would run on it to start spark application.
protected Boolean executeCommand() {
try {
final Runtime rt = Runtime.getRuntime();
final Process proc = rt.exec("sh start-sparkapp.sh");
final int exitVal = proc.waitFor();
BufferedReader buf = new BufferedReader(new InputStreamReader(proc.getInputStream()));
String line = "";
while ((line=buf.readLine())!=null) {
System.out.println(line);
}
System.out.println(" commandToExecute exited with code: " + exitVal);
proc.destroy();
} catch (final Exception e) {
System.out.println("Exception occurred while Launching process : " + e.getMessage());
return Boolean.FALSE;
}
return Boolean.TRUE;
}
But this starts a long running spark job. So I believe, next part of code would be executed only when spark job is finished. My requirement is as soon as spark application is started , the control goes to the next part of code, where I am monitoring the status of same spark application. i.e I start the spark application and monitor the status of spark application from same java application.
Assume I have a method montior which monitors the status of application
public String monitor(ApplicationId id)
Any suggestion how to achieve this?

Since you will be monitoring your Spark application using the method public String monitor(ApplicationId id) , I am assuming you do not want your current thread to wait on the process using proc.waitFor(). Additionally, you do not want to print the normal output of the process to your console. Both these operations make your thread wait on the spawned process. Moreover, your monitor method should take the process-id of the spawned process, rather than the spark applicationId, as input.
So the modified code could look like:
protected Boolean executeCommand() {
try {
final Runtime rt = Runtime.getRuntime();
final Process proc = rt.exec("sh start-sparkapp.sh");
/*
*Call to method monitor(ProcessId id)
*/
} catch (final Exception e) {
System.out.println("Exception occurred while Launching process : " + e.getMessage());
return Boolean.FALSE;
}
return Boolean.TRUE;
}

Related

ParameterException when spawning second Selenium Grid node as another process from Grid process programmatically

I'm setting up Selenium Grid which could spawn new instances of Nodes from HTTP Request on Grid Servlet endpoint: SpawnNodeServlet. That Servlet on GET request is creating new Selenium Grid Node with configuration to registry it to hub. By this I could add nodes when I want just by sending GET request under http://localhost:4444/grid/admin/SpawnNodeServlet. Everything works fine until I want to spawn second or next nodes. Only first one works properly, after that I got ParameterException (all codes below). Parameters should be fine, due to fact first node registering properly. Any ideas what could be wrong? I think, that trouble could be in process creation implementation.
I tried executing jars from Runtime exec as plain command but that also didn't work properly.
final Runtime runtime = Runtime.getRuntime();
final Process command = runtime.exec(executionArgs.toArray(new String[0]));
Below is main code of InstanceExecutor which allow to create new Node instance:
public class InstanceExecutor {
private final Logger logger = Logger.getLogger(getClass().getName());
private BufferedReader errorBufferedReader;
private BufferedReader outputBufferedReader;
private int exitValue;
public void execute(List<String> args) throws InstanceExecutorException {
final List<String> executionArgs = new ArrayList<String>();
executionArgs.add(0, "java");
executionArgs.addAll(args);
try {
final ProcessBuilder processBuilder = new ProcessBuilder(executionArgs.toArray(new String[0]));
Process process = processBuilder.start();
logger.info("processBuilder.start()");
this.errorBufferedReader = new BufferedReader(new InputStreamReader(process.getErrorStream()));
this.outputBufferedReader = new BufferedReader(new InputStreamReader(process.getInputStream()));
logger.info("BufferedReader's");
process.waitFor();
this.exitValue = process.exitValue();
if (this.exitValue != 0) {
throw new IOException("Failed to execute jar, " + this.getExecutionLog());
}
} catch (final IOException | InterruptedException e) {
throw new InstanceExecutorException(e);
}
}
public String getExecutionLog() {
StringBuilder error = new StringBuilder();
String line;
try {
while((line = this.errorBufferedReader.readLine()) != null) {
error
.append("\n")
.append(line);
}
} catch (final IOException ignored) { }
StringBuilder output = new StringBuilder();
try {
while((line = this.outputBufferedReader.readLine()) != null) {
output
.append("\n")
.append(line);
}
} catch (final IOException ignored) { }
try {
this.errorBufferedReader.close();
this.outputBufferedReader.close();
} catch (final IOException ignored) { }
return "exitValue: " + this.exitValue + ", error: " + error + ", output: " + output;
}
}
This class of InstanceExecutor is used in doGet of SpawnNodeServlet:
public class SpawnNodeServlet extends RegistryBasedServlet {
public SpawnNodeServlet() {
this(null);
}
public SpawnNodeServlet(GridRegistry registry) {
super(registry);
}
private final Logger logger = Logger.getLogger(getClass().getName());
private InstanceExecutor instanceExecutor = new InstanceExecutor();
private final List<String> nodeArgs = new ArrayList<String>();
private void sendResponse(HttpServletResponse response) throws IOException {
PrintWriter out = response.getWriter();
response.setContentType("application/json");
response.setCharacterEncoding("UTF-8");
response.setStatus(200);
out.flush();
}
#Override
protected void doGet(HttpServletRequest request, HttpServletResponse response) throws IOException {
logger.info("Spawning additional node");
nodeArgs.add("-Dwebdriver.chrome.driver=\"libs//chromedriver\"");
nodeArgs.add("-cp");
nodeArgs.add("hub/target/hub-1.0.0-jar-with-dependencies.jar:node/target/node-1.0.0-jar-with-dependencies.jar");
nodeArgs.add("org.openqa.grid.selenium.GridLauncherV3");
nodeArgs.add("-role");
nodeArgs.add("node");
nodeArgs.add("-nodeConfig");
nodeArgs.add("node/config.json");
try {
instanceExecutor.execute(nodeArgs);
} catch (InstanceExecutorException e) {
throw new IOException(e);
}
sendResponse(response);
}
}
This is console log after two times I executed GET request on http://localhost:4444/grid/admin/SpawnNodeServlet:
3:27:46.126 INFO [SpawnNodeServlet.doGet] - Spawning additional node
23:27:46.141 INFO [InstanceExecutor.execute] - processBuilder.start()
23:27:46.142 INFO [InstanceExecutor.execute] - BufferedReader's
23:27:47.018 INFO [DefaultGridRegistry.add] - Registered a node http://169.254.168.67:8992
23:27:47.018 INFO [DefaultProxy.startPolling] - startPolling()
23:30:33.954 INFO [SpawnNodeServlet.doGet] - Spawning additional node
23:30:33.959 INFO [InstanceExecutor.execute] - processBuilder.start()
23:30:33.960 INFO [InstanceExecutor.execute] - BufferedReader's
2019-09-08 23:30:34.203:WARN:osjs.HttpChannel:qtp1128948651-14: /grid/admin/SpawnNodeServlet java.io.IOException: exception.InstanceExecutorException: java.io.IOException: Failed to execute jar, exitValue: 1, error:
Exception in thread "main" com.beust.jcommander.ParameterException: Was passed main parameter '-Dwebdriver.chrome.driver="libs//chromedriver"' but no main parameter was defined in your arg class
at com.beust.jcommander.JCommander.initMainParameterValue(JCommander.java:936)
at com.beust.jcommander.JCommander.parseValues(JCommander.java:752)
at com.beust.jcommander.JCommander.parse(JCommander.java:340)
at com.beust.jcommander.JCommander.parse(JCommander.java:319)
at org.openqa.grid.selenium.GridLauncherV3.parse(GridLauncherV3.java:218)
at org.openqa.grid.selenium.GridLauncherV3.lambda$buildLaunchers$7(GridLauncherV3.java:271)
at org.openqa.grid.selenium.GridLauncherV3.lambda$launch$0(GridLauncherV3.java:86)
at java.base/java.util.Optional.map(Optional.java:265)
at org.openqa.grid.selenium.GridLauncherV3.launch(GridLauncherV3.java:86)
at org.openqa.grid.selenium.GridLauncherV3.main(GridLauncherV3.java:70), output:
As you can see, first execution properly spawns Node, but second one, get this error.
How this process builder logic should be changed to not wait for process end (I tried to remove it and only create process but it also didn't work) and has ability to spawn more than 1 Selenium Grid Node?
In SpawnNodeServlet you should remove class field nodeArgs, and define nodeArgs as local method variable.
You have instance field:
private final List<String> nodeArgs = new ArrayList<String>();
After first service invocation list contains:
"-Dwebdriver.chrome.driver=\"libs//chromedriver\""
"-cp"
"hub/target/hub-1.0.0-jar-with-dependencies.jar:node/target/node-1.0.0-jar-with-dependencies.jar"
"org.openqa.grid.selenium.GridLauncherV3"
"-role"
"node"
"-nodeConfig"
"node/config.json"
and after second invocation:
"-Dwebdriver.chrome.driver=\"libs//chromedriver\""
"-cp"
"hub/target/hub-1.0.0-jar-with-dependencies.jar:node/target/node-1.0.0-jar-with-dependencies.jar"
"org.openqa.grid.selenium.GridLauncherV3"
"-role"
"node"
"-nodeConfig"
"node/config.json"
"-Dwebdriver.chrome.driver=\"libs//chromedriver\""
"-cp"
"hub/target/hub-1.0.0-jar-with-dependencies.jar:node/target/node-1.0.0-jar-with-dependencies.jar"
"org.openqa.grid.selenium.GridLauncherV3"
"-role"
"node"
"-nodeConfig"
"node/config.json"
Then you are passing this list to the executor:
instanceExecutor.execute(nodeArgs);
And this is not valid java argument list.

Scheduled process stops working after several executions

I have a program running on a linux machine. That program has a class, that schedules a thread for execution every minute. The thread supposed to perform several calculations, create a text file and then synchronize the file to a different machine using rsync.
The class looks like this:
public class Generator {
private ScheduledThreadPoolExecutor SchEventPool = new ScheduledThreadPoolExecutor(5);
public void generate() {
LocalDateTime ldt = LocalDateTime.now();
int delay = 60 - ldt.getSecond();
SchEventPool.scheduleWithFixedDelay(new HelperClass(), delay, 60, SECONDS);
}
private class HelperClass implements Runnable{
#Override
public void run() {
// some calculations
// .
// .
// .
FileBuild();
}
private void FileBuild(){
try{
String filepath = System.getProperty("user.dir") + "/SomeDir/somefile.txt";
File f = new File;
BufferedWriter writer = new BufferedWriter(new FileWriter(f));
for (int i = 0; i < 100; i++){
writer.write("write something into the file" + "\n");
}
writer.close();
} catch (IOException ex) {ex.printStackTrace();}
// synchronize the file
synchronize();
}
public void synchronize(){
try {
String[] cmd = new String[]{"rsync", "--remove-source-files", "-avre", "/usr/bin/sudo /usr/bin/ssh -i /home/user/.ssh/id_rsa", "/home/user/project/SomeDir, "user#server.ip:/home/user/project/AnotherDir/"};
Process p = new ProcessBuilder().command(cmd).start();
} catch (IOException ex) {ex.printStackTrace();}
}
}
When i run the main program and invoke the generate() method, the process runs successfully for about 30 minutes, sometimes more sometimes less, and then the execution of the scheduled thread stops. Without any errors or any warnings.
I should mention that the file that is being synchronized to the second machine is about 1500 KB, very small file.
I have also monitored the processors load during running and business as usual, server is not overloaded.
What could be the issue here?
Why would the scheduled thread stops being executed?
Any help would be appreciated. Thank you.

OutOfMemoryError: unable to create new native thread using ExecutorService

I launched my instance overnight to see how it handled things and when I came by this morning, I was facing a
Exception in thread "pool-535-thread-7" java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:691)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:943)
at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:992)[info] application - Connecting to server A
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
The aim of my code is quite simple : Every 5 minutes, I connect to a list of remote server, send a request (via socket) and that's it.
Here's my code :
My "cron" task :
/** will create a new instance of ExecutorService every 5 minutes, loading all the websites in the database to check their status **/
/** Maybe that's where the problem is ? I need to empty (GC ?) this ExecutorService ? **/
Akka.system().scheduler().schedule(
Duration.create(0, TimeUnit.MILLISECONDS), // Initial delay 0 milliseconds
Duration.create(5, TimeUnit.MINUTES), // Frequency 5 minutes
new Runnable() {
public void run() {
// We get the list of websites to check
Query<Website> query = Ebean.createQuery(Website.class, "WHERE disabled = false AND removed IS NULL");
query.order("created ASC");
List<Website> websites = query.findList(); // Can be 1, 10, 100, 1000. In my test case, I had only 9 websites.
ExecutorService executor = Executors.newFixedThreadPool(NTHREDS);
for (Website website : websites) {
CheckWebsite task = new CheckWebsite(website);
executor.execute(task);
}
// This will make the executor accept no new threads
// and finish all existing threads in the queue
executor.shutdown();
}
},
Akka.system().dispatcher()
);
My CheckWebsite class :
public class CheckWebsite implements Runnable {
private Website website;
public CheckWebsite(Website website) {
this.website = website;
}
#Override
public void run() {
WebsiteLog log = website.checkState(); // This is where the request is made, I copy paste the code just after
if (log == null) {
Logger.error("OHOH, WebsiteLog should not be null for website.checkState() in CheckWebsite class :s");
return;
}
try {
log.save();
catch (Exception e) {
Logger.info ("An error occured :/");
Logger.info(e.getMessage());
e.printStackTrace();
}
}
}
My checkState() method in Website.class :
public WebsiteLog checkState() {
// Since I use Socket and the connection can hang indefinitely, I use an other ExecutorService in order to limit the time spent
// The duration is defined via Connector.timeout, Which will be the next code.
ExecutorService executor = Executors.newFixedThreadPool(1);
Connector connector = new Connector(this);
try {
final long startTime = System.nanoTime();
Future<String> future = executor.submit(connector);
String response = future.get(Connector.timeout, TimeUnit.MILLISECONDS);
long duration = System.nanoTime() - startTime;
return PlatformLog.getLastOccurence(this, response, ((int) duration/ 1000000));
}
catch (Exception e) {
return PlatformLog.getLastOccurence(this, null, null);
}
}
Here's the Connector.class. I removed useless part here (like Catches) :
public class Connector implements Callable<String> {
public final static int timeout = 2500; // WE use a timeout of 2.5s, which should be enough
private Website website;
public Connector(Website website) {
this.website = website;
}
#Override
public String call() throws Exception {
Logger.info ("Connecting to " + website.getAddress() + ":" + website.getPort());
Socket socket = new Socket();
try {
socket.connect(new InetSocketAddress(website.getIp(), website.getPort()), (timeout - 50));
BufferedReader input = new BufferedReader(new InputStreamReader(socket.getInputStream()));
String response = input.readLine();
socket.close();
return response;
}
catch (Exception e) {
e.printStackTrace();
throw e;
}
finally {
// I take the precaution to close the socket here in order to avoid a memory leak
// But if the previous ExecutorService force the close of this thread before
// I can't guarantee it will be closed :/
if (socket != null && !socket.isClosed()) {
socket.close();
}
}
}
}
I'm new to Java multithreading so I probably made big mistake. I suspect some area that could be potentially the reason, but my lack of knowledge requires me to ask for your help :)
As a summary, here's the potentials areas :
Creating a new ExecutorService every 5 minutes. Maybe I can reuse the old one ? Or do I need to close the current one when finished (if so, how ?).
The fact that I create an ExecutorService that will create an ExecutorService (in the checkstate() method)
The fact that the Connector class can be (violently) stopped by the ExecutorService running it, if it takes too long, resulting in a socket not closed (and then a memory leak) ?
Also, as you can see, the exception occured for the thread "pool-535-thread-7" which mean it didn't happen soon.
I store the last_occured check in the database, and the creation of the log entry (in WebsiteLog), the delta is around 5 hours (so, for every 5 minutes, the thread crashed after around 60 calls).
Update : Here's the revisited checkState method to include the shutdown call :
public PlatformLog checkState() {
ExecutorService executor = Executors.newFixedThreadPool(1);
Connector connector = new Connector(this);
String response = null;
Long duration = null;
try {
final long startTime = System.nanoTime();
Future<String> future = executor.submit(connector);
response = future.get(Connector.timeout, TimeUnit.MILLISECONDS);
duration = System.nanoTime() - startTime;
}
catch (Exception e) {}
executor.shutdown();
if (duration != null) {
return WebsiteLog.getLastOccurence(this, response, (duration.intValue()/ 1000000));
}
else {
return WebsiteLog.getLastOccurence(this, response, null);
}
}
I'm not sure this is the only problem, but you are creating an ExecutorService in your checkState() method but you don't shut it down.
According to the JavaDocs for Executors.newFixedThreadPool():
The threads in the pool will exist until it is explicitly shutdown.
The threads staying alive will cause the ExecutorService not to be garbage collected (which would call shutdown() on your behalf. Hence you are leaking a thread each time this is called.

Reading output from java.lang.Process - There is nothing to read

I'm trying to execute terminal command in linux trough Java and i cant get any input from inputStream.
This is my code
ProcessBuilder build = new ProcessBuilder("/usr/bin/xterm", "find /home");
Process pr = null;
BufferedReader buf;
try {
build.redirectErrorStream(true);
pr = build.start();
buf = new BufferedReader(new InputStreamReader( pr.getInputStream()));
String line = buf.readLine();
pr.waitFor();
while (true) {
System.out.println(line + "sadasdas");
line = buf.readLine();
}
} catch (Exception e) {
e.printStackTrace();
}
Process is executed and immediately terminal closes, and no output is catched and printed. On the other hand if i will compose an unknown command i get all the lines with tips how to use commands. Same problem i had with windows cmd. I was trying to use getRuntime.exec(cmd) method but the end is the same.
I've also tried to created separate threads for process and reader which looks like this
public class kurdee
{
public static Thread thread;
public kurdee()
{
List cmd = new LinkedList();
cmd.add(new String("/usr/bin/xterm"));
cmd.add(new String("find"));
thisProc thispr = new thisProc(cmd);
this.thread = new Thread(thispr);
thread.start();
reader rd = new reader(thispr.proc);
Thread thread1 = new Thread(rd);
thread1.start();}
public static void main(String args[])
{
java.awt.EventQueue.invokeLater(new Runnable() {
public void run() {
kurdee kurd = new kurdee();
}
});
}
}
class reader implements Runnable
{
private BufferedReader buf;
private Process proc;
public reader(Process proc)
{
this.proc=proc;
this.buf = new BufferedReader(new InputStreamReader(proc.getInputStream()));
}
public void run()
{
String line="";
System.out.println("Thread is alive");
try{
//Thread.sleep(1000);
line = buf.readLine();
}catch(Exception ex){System.out.println(ex + " before first while started");}
while(kurdee.thread.isAlive())
{
System.out.println("Thread is alive");
while(line!=null)
{
try{
//System.out.println(proc.exitValue());
System.out.println(line + " asd");
line=buf.readLine();
}catch(Exception e){System.out.println(e + " Inner while loop");}
}
}
}
}
class thisProc implements Runnable
{
private ProcessBuilder build;
public static Process proc=null;
public thisProc(List<String> args)
{
this.build = new ProcessBuilder(args);
build.redirectErrorStream(true);
try{
this.proc = build.start();
}catch(Exception ex){System.out.println(ex + " proc class");}
}
public void run()
{
try{
proc.waitFor();
}catch(Exception ex){System.out.println(ex + " proc class");}
}
}
But with any combination of invoking threads etc i make there is still nothing to read.
I'm trying to use command "find /home -xdev -samefile file" to get all hard links to file so maybe there is an easier way.
xterm is not the way to execute processes in unix, it is not a shell. a shell is something like "/bin/sh". however, "find" is a normal unix executable, so you should just execute that directly, e.g. new ProcessBuilder("find", "/home"). and yes, you should always process the streams on separate threads, as recommended by this article.
First, don't try to execute the command with xterm, that's pointless; just do it directly. Secondly, be careful when you compose your array of command strings to put one word into each string; passing, for example "find /home" as a single string among many to ProcessBuilder is going to error out.

unable to kill a process using Process.destroy()

I'm just experiminting with Runtime and Process classes of java. I'm trying to open a application like windows word using Runtime.exec() and then waiting some time and trying to destroy it using Process.destroy() method.
MS Word is opening but it's not closing throwing below exception in console
exception::java.lang.IllegalMonitorStateException: current thread not owner
Below is my code
import java.util.*;
public class StringTesting {
public void open()
{
try{
Runtime runtime = Runtime.getRuntime();
Process proc =runtime.exec("C:\\Program Files\\Microsoft Office\\Office12\\winword.exe");
StringTesting st = new StringTesting();
st.wait(5000);
// now destroy the process
proc.destroy();
System.out.println(" after destroy");
}
catch(Exception e)
{
System.out.println(" exception::"+e);
}
}
public static void main(String a[])
{
StringTesting st = new StringTesting();
st.open();
}
}
The problem here is that you cannot invoke Object.wait() without holding the monitor for that object:
StringTesting st = new StringTesting();
synchronized (st) {
st.wait(5000);
}

Categories