Spark submit in java(SparkLauncher) - java

I made spark+hadoop yarn enviroment and spark-submit command works well. So I made SparkLauncher java code to do this in my application jar, BUT somehow it doesn't work (actually computer fan is spinning at first but not as long as i did with spark-submit.)
It seems not work well (no application log in hadoop web ui, unlike spark-submit). I cannot see any error log when I do with 'SparkLauncher'. without log message, I can do nothing with it.
Here is how I made it so far.
public class Main {
public static void main(String[] args) {
Process spark = null;
try
{
spark = new SparkLauncher()
.setAppResource("/usr/local/spark/examples/jars/spark-examples*.jar")
.setMainClass("org.apache.spark.examples.SparkPi")
.setMaster("yarn")
.setDeployMode( "cluster")
.launch();
}
catch( IOException e)
{
e.printStackTrace();
}
}
}
executed it with ( java -jar example.jar)

I had the same problem at first. I think the main issue is that you forgot about the waitFor().
Also, it's really helpfull to extract your errorMessage and deal with it (e.g. log it or checking it while debuging ) within your java code. To allow this, you should create a streamReader thread as follows:
InputStreamReaderRunnable errorStreamReaderRunnable = new InputStreamReaderRunnable(spark.getErrorStream(), "error");
Thread errorThread = new Thread(errorStreamReaderRunnable, "LogStreamReader error");
errorThread.start();
int result= spark.waitFor();
if(result!=0) {
String errorMessage = extractExceptionMessage(errorStreamReaderRunnable.getMessage());
LOGGER.error(errorMessage);
}
This should be after your launch() command and inside your try block. Hope it helps

Related

Runtime exec command working when when run from terminal but not eclipse

I'm trying to use tesseract to do OCR on an image in java. I realize there are wrappers like Tess4J that provide a bunch more functionality and stuff, but I've been struggling to get it set up properly. Simply running a one-line command with Runtime is really all I need anyways since this is just a personal little project and doesn't need to work on other computers or anything.
I have this code:
import java.io.IOException;
public class Test {
public static void main(String[] args) {
System.out.println(scan("full-path-to-test-image"));
}
public static String scan(String imgPath) {
String contents = "";
String cmd = "[full-path-to-tesseract-binary] " + imgPath + " stdout";
try { contents = execCmd(cmd); }
catch (IOException e) { e.printStackTrace(); }
return contents;
}
public static String execCmd(String cmd) throws java.io.IOException {
java.util.Scanner s = new java.util.Scanner(Runtime.getRuntime().exec(cmd).getInputStream()).useDelimiter("\\A");
return s.hasNext() ? s.next() : "";
}
}
When it's compiled and run directly from terminal, it works perfectly. When I open the exact same file in eclipse, however, it gives an IOException:
java.io.IOException: Cannot run program "tesseract": error=2, No such file or directory
What's going on? Thank you for any help.
Check the working folder in the run configuration for the Test class in Eclipse. I bet it's different from the one when you run the same program from a terminal.

Java - Printing to cmd

So my problem here is that I'm not too sure how to print a set of commands to cmd. I have a batch file that runs a Minecraft server, and I need to be able to run commands through the command prompt that shows up when I run the batch file, which will in turn perform commands to the server.
Here is my code so far:
package com.Kaelinator;
import java.io.IOException;
public class ServerManager {
public static void main(String[] args){
try {
System.out.println("Opening");
Runtime runTime = Runtime.getRuntime();
Process process = runTime.exec("cmd /C start /min " + "C:/Users/Owner/Desktop/rekt/Run.bat");
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
e.printStackTrace();
}
//System.out.println("Closing");
//process.destroy();
} catch (IOException e){
e.printStackTrace();
}
}
}
Yes, I understand that all this does so far is open the batch file. :P I am expecting something like this that I need to add to my code:
process.InputStream(command1);
But I am certain that there is more to it, something along the lines of bufferedWriters or something like that...
Whenever I try to get answers from Google, the answers always have a whole load of extra code, or have something completely different about them.
Thanks!

Catch Java Exception with PHP shell_exec

Is there any way to catch a java exception from a JAR call by a php script? I have (!) to use some external JAVA libraries, which throws Exceptions. The Question is how can i extract valuable information for developers and show some explanation to the customer?
PHP Script
try{
$cmd = "java -jar myjar.jar";
$output = shell_exec(escapeshellcmd($cmd));
}
catch(Javaexception $e){
//do some error handling ....
}
Java Jar
//....
public class Main{
public static void main(String[] args){
throw new Exception("Testexception");
}
}
//...
shell_exec only returns a String or Null, so you can't catch the Java Exception like that, try to create your own Exception and "catch it" with a if
$cmd = "java -jar myjar.jar";
$output = shell_exec(escapeshellcmd($cmd));
if($output == 'expected output') throw new MyException();
Basic answer: not directly.
You can use exit statuses to communicate processing status from the Java application.
In Java:
public class Main{
public static void main(String[] args){
try {
// work
} catch (MyException e) {
// handle error
// write exception to file or STDOUT
System.exit(1);
}
}
}
In PHP you can get the exit status via return_var in exec call:
string exec ( string $command [, array &$output [, int &$return_var ]] )
You can also write Exception details to STDOUT or file and process its content in the PHP when status code is non 0.

Application stops responding when using Process.waitFor()

I'm launching an external application from within my Java application (8u11). However the application becomes non-responsive to UI input up under Windows XP and Windows 7 with the standard hour glass/spinner.
I've narrowed this problem down to whether or not I use Process.waitFor(). If I call it I see the problems, if I don't it works fine. Also the application un-freezes if I then quit the Java application.
My question is why is this the case - how can calling waitFor() possibly effect the internal runnings of a child process? And how can I avoid this problem?
The application in question is LinPhone.exe but I don't believe the issue is specific to the application - there must be some general way in which it handles standard IO etc which the way I'm interfering with by calling waitFor().
I need to use Process.waitFor() so I can track when the application has exited.
I've simplified the issue to this SCCEE.
import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;
import java.io.InputStreamReader;
public class LinphoneTest {
public static void main(String[] args) throws IOException,
InterruptedException {
String phoneAppPath = "C:\\Program Files\\Linphone\\bin\\linphone.exe";
ProcessBuilder processBuilder = new ProcessBuilder(phoneAppPath);
// move up from bin/linephone.exe
File workingDir = new File(phoneAppPath).getParentFile()
.getParentFile();
processBuilder.directory(workingDir);
processBuilder.redirectErrorStream();
Process process = processBuilder.start();
final BufferedReader stdout = new BufferedReader(
new InputStreamReader(process.getInputStream()));
String line = null;
try {
while (((line = stdout.readLine()) != null)) {
System.out.println(line);
}
} catch (IOException e) {
e.printStackTrace();
}
new Thread(() -> {
try {
process.waitFor();
} catch (InterruptedException e) {
e.printStackTrace();
}
}, "process wait").start();
Thread.sleep(Long.MAX_VALUE);
}
}
To summarise discussion, I was not handling standard error streams causing linphone to lockup as its output buffer for standard error became full especially as linphone produces a lot of verbose output on standard error. When RunTime.exec wont provides excellent summary of the pitfalls involved in calling processes from Java.
There was also a typo, I was calling the non-standard named getter
processBuilder.redirectErrorStream();
when I should have been calling the "setter"
processBuilder.redirectErrorStream(true);

Prevent launching multiple instances of a java application

I want to prevent the user from running my java application multiple times in parallel.
To prevent this, I have created a lock file when am opening the application, and delete the lock file when closing the application.
When the application is running, you can not open an another instance of jar. However, if you kill the application through task manager, the window closing event in the application is not triggered and the lock file is not deleted.
How can I make sure the lock file method works or what other mechanism could I use?
You could use a FileLock, this also works in environments where multiple users share ports:
String userHome = System.getProperty("user.home");
File file = new File(userHome, "my.lock");
try {
FileChannel fc = FileChannel.open(file.toPath(),
StandardOpenOption.CREATE,
StandardOpenOption.WRITE);
FileLock lock = fc.tryLock();
if (lock == null) {
System.out.println("another instance is running");
}
} catch (IOException e) {
throw new Error(e);
}
Also survives Garbage Collection.
The lock is released once your process ends, doesn't matter if regular exit or crash or whatever.
Similar discussion is at
http://www.daniweb.com/software-development/java/threads/83331
Bind a ServerSocket. If it fails to bind then abort the startup. Since a ServerSocket can be bound only once, only single instsances of the program will be able to run.
And before you ask, no. Just because you bind a ServerSocket, does not mean you are open to network traffic. That only comes into effect once the program starts "listening" to the port with accept().
I see two options you can try:
Use a Java shutdown hook
Have your lock file hold the main process number. The process should exist when you lanuch another instance. If it's not found in your system, you can assume that the lock can be dismissed and overwritten.
Creating a server socket, bounds to a specific port with a ServerSocket instance as the application starts is a straight way.
Note that ServerSocket.accept() blocks, so running it in its own thread makes sense to not block the main Thread.
Here is an example with a exception thrown as detected :
public static void main(String[] args) {
assertNoOtherInstanceRunning();
... // application code then
}
public static void assertNoOtherInstanceRunning() {
new Thread(() -> {
try {
new ServerSocket(9000).accept();
} catch (IOException e) {
throw new RuntimeException("the application is probably already started", e);
}
}).start();
}
You could write the process id of the process that created the lock file into the file.
When you encounter an existing lock file, you do not just quit, but you check if the process with that id is still alive. If not, you can go ahead.
You can create a Server socket like
new ServerSocket(65535, 1, InetAddress.getLocalHost());
at very beginning of your code. Then if AddressAlreadyInUse exception caught in main block you can display the appropriate message.
There are already available java methods in File class to achieve the same. The method is deleteOnExit() which ensure the file is automatically deleted when the JVM exits. However, it does not cater to forcible terminations. One should use FileLock in case of forcible termination.
For more details check, https://docs.oracle.com/javase/7/docs/api/java/io/File.html
Thus code snippet which could be used in the main method can be like :
public static void main(String args[]) throws Exception {
File f = new File("checkFile");
if (!f.exists()) {
f.createNewFile();
} else {
System.out.println("App already running" );
return;
}
f.deleteOnExit();
// whatever your app is supposed to do
System.out.println("Blah Blah")
}
..what other mechanism could I use?
If the app. has a GUI it can be launched using Java Web Start. The JNLP API provided to web-start offers the SingleInstanceService. Here is my demo. of the SingleInstanceService.
You can write something like this.
If file exists try to delete it. if it is not able to delete. We can say that application is already running.
Now create the same file again and redirect the sysout and syserr.
This works for me
Simple lock and advanced lock
I developed 2 solutions for this problem. I was also looking for an easy way of doing this without using any libraries and a lot of code.
My solutions are based on: https://stackoverflow.com/a/46705579/10686802 which I have improved upon. Therefore I would like to thank #akshaya pandey and #rbento
Simple file lock
package YOUR_PACKAGE_NAME;
import java.io.File;
import java.io.IOException;
/**
* Minimal reproducible example (MRE) - Example of a simple lock file.
* #author Remzi Cavdar - ict#remzi.info - #Remzi1993
*/
public class Main {
public static void main(String[] args) {
/*
* Prevents the user of starting multiple instances of the application.
* This is done by creating a temporary file in the app directory.
* The temp file should be excluded from git and is called App.lock in this example.
*/
final File FILE = new File("App.lock");
try {
if (FILE.createNewFile()) {
System.out.println("Starting application");
} else {
System.err.println("The application is already running!");
return;
}
} catch (IOException e) {
throw new RuntimeException(e);
}
/*
* Register a shutdown hook to delete the lock file when the application is closed. Even when forcefully closed
* with the task manager. (Tested on Windows 11 with JavaFX 19)
*/
FILE.deleteOnExit();
// Whatever your app is supposed to do
}
}
Advanced lock
package YOUR_PACKAGE_NAME;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.nio.channels.FileChannel;
import java.nio.channels.FileLock;
/**
* Minimal reproducible example (MRE) - Example of a more advanced lock system.
* #author Remzi Cavdar - ict#remzi.info - #Remzi1993
*/
public class Main {
public static void main(String[] args) {
/*
* Prevents the user of starting multiple instances of the application.
* This is done by creating a temporary file in the app directory.
* The temp file should be excluded from git and is called App.lock in this example.
*/
final File FILE = new File("App.lock");
if (FILE.exists()) {
System.err.println("The application is already running!");
return;
}
try (
FileOutputStream fileOutputStream = new FileOutputStream(FILE);
FileChannel channel = fileOutputStream.getChannel();
FileLock lock = channel.lock()
) {
System.out.println("Starting application");
} catch (FileNotFoundException e) {
throw new RuntimeException(e);
} catch (IOException e) {
throw new RuntimeException(e);
}
/*
* Register a shutdown hook to delete the lock file when the application is closed. Even when forcefully closed
* with the task manager. (Tested on Windows 11 with JavaFX 19)
*/
FILE.deleteOnExit();
// Whatever your app is supposed to do
}
}
Testing
Tested on: 31-10-2022
Tested OS: Windows 11 - Version 21H2 (OS Build 22000.1098)
Tested with: OpenJDK 19 - Eclipse Temurin JDK with Hotspot 19+36(x64)
I closed the application and also forcefully closed the application with task manager on Windows both times the lock file seems to be deleted upon (force) close.
I struggled with this same problem for a while... none of the ideas presented here worked for me. In all cases, the lock (file, socket or otherwise) did not persist into the 2nd process instance, so the 2nd instance still ran.
So I decided to try an old school approach to simply crate a .pid file with the process id of the first process. Then any 2nd process would quit if it finds the .pid file, and also the process number specified in the file is confirmed to be still running. This approach worked for me.
There is a fair bit of code, which I provide here in full for your use... a complete solution.
package common.environment;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.io.*;
import java.nio.charset.Charset;
public class SingleAppInstance
{
private static final #Nonnull Logger log = LogManager.getLogger(SingleAppInstance.class.getName());
/**
* Enforces that only a single instance of the given component is running. This
* is resilient to crashes, unexpected reboots and other forceful termination
* scenarios.
*
* #param componentName = Name of this component, for disambiguation with other
* components that may run simultaneously with this one.
* #return = true if the program is the only instance and is allowed to run.
*/
public static boolean isOnlyInstanceOf(#Nonnull String componentName)
{
boolean result = false;
// Make sure the directory exists
String dirPath = getHomePath();
try
{
FileUtil.createDirectories(dirPath);
}
catch (IOException e)
{
throw new RuntimeException(String.format("Unable to create directory: [%s]", dirPath));
}
File pidFile = new File(dirPath, componentName + ".pid");
// Try to read a prior, existing pid from the pid file. Returns null if the file doesn't exist.
String oldPid = FileUtil.readFile(pidFile);
// See if such a process is running.
if (oldPid != null && ProcessChecker.isStillAllive(oldPid))
{
log.error(String.format("An instance of %s is already running", componentName));
}
// If that process isn't running, create a new lock file for the current process.
else
{
// Write current pid to the file.
long thisPid = ProcessHandle.current().pid();
FileUtil.createFile(pidFile.getAbsolutePath(), String.valueOf(thisPid));
// Try to be tidy. Note: This won't happen on exit if forcibly terminated, so we don't depend on it.
pidFile.deleteOnExit();
result = true;
}
return result;
}
public static #Nonnull String getHomePath()
{
// Returns a path like C:/Users/Person/
return System.getProperty("user.home") + "/";
}
}
class ProcessChecker
{
private static final #Nonnull Logger log = LogManager.getLogger(io.cpucoin.core.platform.ProcessChecker.class.getName());
static boolean isStillAllive(#Nonnull String pidStr)
{
String OS = System.getProperty("os.name").toLowerCase();
String command;
if (OS.contains("win"))
{
log.debug("Check alive Windows mode. Pid: [{}]", pidStr);
command = "cmd /c tasklist /FI \"PID eq " + pidStr + "\"";
}
else if (OS.contains("nix") || OS.contains("nux"))
{
log.debug("Check alive Linux/Unix mode. Pid: [{}]", pidStr);
command = "ps -p " + pidStr;
}
else
{
log.warn("Unsupported OS: Check alive for Pid: [{}] return false", pidStr);
return false;
}
return isProcessIdRunning(pidStr, command); // call generic implementation
}
private static boolean isProcessIdRunning(#Nonnull String pid, #Nonnull String command)
{
log.debug("Command [{}]", command);
try
{
Runtime rt = Runtime.getRuntime();
Process pr = rt.exec(command);
InputStreamReader isReader = new InputStreamReader(pr.getInputStream());
BufferedReader bReader = new BufferedReader(isReader);
String strLine;
while ((strLine = bReader.readLine()) != null)
{
if (strLine.contains(" " + pid + " "))
{
return true;
}
}
return false;
}
catch (Exception ex)
{
log.warn("Got exception using system command [{}].", command, ex);
return true;
}
}
}
class FileUtil
{
static void createDirectories(#Nonnull String dirPath) throws IOException
{
File dir = new File(dirPath);
if (dir.mkdirs()) /* If false, directories already exist so nothing to do. */
{
if (!dir.exists())
{
throw new IOException(String.format("Failed to create directory (access permissions problem?): [%s]", dirPath));
}
}
}
static void createFile(#Nonnull String fullPathToFile, #Nonnull String contents)
{
try (PrintWriter writer = new PrintWriter(fullPathToFile, Charset.defaultCharset()))
{
writer.print(contents);
}
catch (IOException e)
{
throw new RuntimeException(String.format("Unable to create file at %s! %s", fullPathToFile, e.getMessage()), e);
}
}
static #Nullable String readFile(#Nonnull File file)
{
try
{
try (BufferedReader fileReader = new BufferedReader(new FileReader(file)))
{
StringBuilder result = new StringBuilder();
String line;
while ((line = fileReader.readLine()) != null)
{
result.append(line);
if (fileReader.ready())
result.append("\n");
}
return result.toString();
}
}
catch (IOException e)
{
return null;
}
}
}
To use it, simply invoke it like this:
if (!SingleAppInstance.isOnlyInstanceOf("my-component"))
{
// quit
}
I hope you find this helpful.
Finally I found really simple library to achieve this. You can you use JUniqe.
The JUnique library can be used to prevent a user to run at the same
time more instances of the same Java application.
This is an example how to use it from the documentation
public static void main(String[] args) {
String appId = "myapplicationid";
boolean alreadyRunning;
try {
JUnique.acquireLock(appId);
alreadyRunning = false;
} catch (AlreadyLockedException e) {
alreadyRunning = true;
}
if (!alreadyRunning) {
// Start sequence here
}
}
here is a pretty rudimental approach.
If your application is launched from a script, check the running java/javaw processes and their command line before launch
In windows
REM check if there is a javaw process running your.main.class
REM if found, go to the end of the script and skip the launch of a new instance
WMIC path win32_process WHERE "Name='javaw.exe'" get CommandLine 2>nul | findstr your.main.class >nul 2>&1
if %ERRORLEVEL% EQU 0 goto:eof
javaw your.main.class

Categories