We Keep Coding

Error: Could not find or load main class org.apache.spark.launcher.Main in Java Spark

I am trying to run the spark application which is written in java in GKP. For the same I am able to build the image and placed in the container. But while running the spark application with spark-submit command I am facing an error which is
Error: Could not find or load main class org.apache.spark.launcher.Main
The java and spark versions i am using for this was jdk-11 and spark-3.2.1
I am running this application via IntelliJ with maven. Also tried adding the spark-launcher maven dependency still the issue exists.
Can I know where it is going wrong with this versions.
NOTE : I can see the spark-launcher jar in the spark-3.2.1 jar folder as well.

I had that error message. It probably may have several root causes but this how I investigated and solved the problem (on linux):
instead of launching spark-submit, try using bash -x spark-submit to
see which line fails.
do that process several times ( since spark-submit calls nested
scripts ) until you find the underlying process called : in my case
something like :
/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -cp
'/opt/spark-2.2.0-bin-hadoop2.7/conf/:/opt/spark-2.2.0-bin-hadoop2.7/jars/*'
-Xmx1g org.apache.spark.deploy.SparkSubmit --class org.apache.spark.repl.Main --name 'Spark shell' spark-shell
So, spark-submit launches a java process and can't find the org.apache.spark.launcher.Main class using the files in /opt/spark-2.2.0-bin-hadoop2.7/jars/* (see the -cp option above). I did an ls in this jars folder and counted 4 files instead of the whole spark distrib (~200 files). It was probably a problem during the installation process. So I reinstalled spark, checked the jar folder and it worked like a charm.
So, you should:
check the java command (cp option)
check your jars folder ( does it contain ths at least all the
spark-*.jar
?)
Hope it helps.

How to determine which command Netbeans is executing to run the project?

I'm developing an Java application using Netbeans and Maven. I'm struggling a lot to run the project as standalone application. Launching the project from Netbeans is all fine but running the executable jar yields an error (failing to load a data file).
I need to know exactly which command(s) Netbeans is executing to run the application. However, the output window of Netbeans only shows me what the application writes "back". Is there a way to figure out/display the command(s) that Netbeans is using to run the project?
Thanks
EDIT: Running the java application and loading files is not the issue here. I'm able to run the application and the libraries are loading correctly. The issue here is to understand how Netbans runs the application (by looking at the commands that are executed).

Netbeans outputs the command it invokes as the very first line of the output.
I use Netbeans 8.2 and that's how it looks like when I hit big green Run button in maven project:
The first line reads (formatted for better readablility):
cd D:\test;
"JAVA_HOME=C:\\Program Files\\Java\\jdk1.8.0_112"
"M2_HOME=C:\\Program Files\\apache-maven-3.3.3"
cmd /c "\"\"C:\\Program Files\\apache-maven-3.3.3\\bin\\mvn.cmd\"
-Dexec.args=\"-classpath %classpath com.test.AppStarter\"
-Dexec.executable=\"C:\\Program Files\\Java\\jdk1.8.0_112\\bin\\java.exe\"
-Dexec.workingdir=D:\\test\\target\\dist
-Dmaven.ext.class.path=\"C:\\Program Files\\NetBeans 8.2\\java\\maven-nblib\\netbeans-eventspy.jar\"
-Dfile.encoding=UTF-8
org.codehaus.mojo:exec-maven-plugin:1.2.1:exec\""
From that line I can tell that Netbeans:
goes to the project's directory
sets environment variables (JAVA_HOME and M2_HOME)
then executes cmd that executes mvn (by its full path)
with bunch of -D arguments (which specify working directory and AppStarter as a class to execute)
and exec-maven- plugin with target exec.

When NetBeans compiles a program it creates a folder-hierarchy with compiled .class files. When creating a JAR archive ZIPs those into the archive. When executing the program, (or debugging,) NetBeans runs the .class files off the folders, not in the JAR file.
These could cause different problems, like different PATH. This can cause "file not found" errors with relative paths.

what is your operatingsystem?
windows:
then wmic may be the tool, which could show you the complete commandline. you will find examples here command line of process by name
linux:
try
ps -ef | grep java

I am getting the following error when I run mvn on OS X Mavericks: Cannot run program "/bin/sh": error=2, No such file or directory

I am running OS X Mavericks on a MacBook Pro.
I have maven 3.0.5 installed.
When I run mvn, I am getting the following error message.
Error while executing process. Cannot run program "/bin/sh": error-2, No such file or directory
And before you ask:
PATH="/usr/local/apache-maven-3.0.5/bin:/Library/Java/JavaVirtualMachines/jdk1.7.0_79.jdk/Contents/Home/bin:/usr/local/bin:/bin:/usr/bin"
M2_HOME="/usr/local/apache-maven-3.0.5"
M2="/usr/local/apache-maven-3.0.5/bin"
MAVEN_OPTS="-Xms256m -Xmx768m -XX:MaxPermSize=512m"
I am also seeing the following near the beginning of the mvn output:
[WARNING] Some problems were encountered while building the effective settings
[WARNING] Failed to use environment variables for interpolation: Cannot run program "env": error=0, spawn failed #

I had the same problem: exec() failed with ANY executable. Finally remembered that I was using a private jre. That jre was a copy of the original one, just with bin/java set as executable.
I made lib/jspawnhelper executable and it worked!

If you are using eclipse , this is due to the java version. point java to your OS java instead of eclipse java. this will be solved

It seems your system is missing the "sh" shell to start interpreting the script.
I just looked up installed shells on Mac OSX and got this result:
/bin/zsh
/bin/ksh
/bin/tcsh
/bin/bash
/bin/sh
So when "/bin/sh" is missing, try "/bin/bash" for example.
Better:
Check if "/bin/sh" exist
If exist check if the user which runs the command can execute the shell
Because it can be possible that the shell binary exist but isn't executable for you. Then you have to make it executable for the user trying mvn.
Hope that helps ;)

Running Rake Tasks In WAS without JRuby

Related to : Executing rake tasks on an exploded war on tomcat without jruby being installed
I'm trying to run rake tasks in my Tomcat server that doesn't have JRuby installed. I'm using warbler to create a war file.
Using the answer to the linked question, I ran:
java -cp lib/jruby-core*.jar:lib/jruby-stdlib*.jar org.jruby.Main -S rake -T
This gets me the error:
Exception in thread "main" java.lang.NoClassDefFoundError: org/jruby/Main
Caused by: java.lang.ClassNotFoundException: org.jruby.Main
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
ls lib gets me:
ems-gems-activerecord-jdbc-adapter-1.2.2-lib-arjdbc-jdbc-adapter_java.jar
gems-gems-jdbc-sqlite3-3.7.2-lib-sqlite-jdbc-3.7.2.jar
gems-gems-jruby-jars-1.6.8-lib-jruby-core-1.6.8.jar
gems-gems-jruby-jars-1.6.8-lib-jruby-stdlib-1.6.8.jar
gems-gems-jruby-rack-1.1.10-lib-jruby-rack-1.1.10.jar
gems-gems-json-1.7.5-java-lib-json-ext-generator.jar
gems-gems-json-1.7.5-java-lib-json-ext-parser.jar
gems-gems-therubyrhino_jar-1.7.4-jar-rhino-1.7R4.jar
gems-gems-warbler-1.3.6-lib-warbler_jar.jar
jruby-core-1.6.8.jar
jruby-rack-1.1.10.jar
jruby-stdlib-1.6.8.jar
ojdbc6.jar
Opening up the jruby-core-1.6.8.jar file, I can see that there is indeed a org/jruby/Main.class file.
As one can see from the file listing, there is no jruby-complete jar file, so I can't run the command from https://stackoverflow.com/a/9982556/684934
What am I doing wrong, and is there by now a better way to do this?

I was working on a similar problem 2 months ago, so things may have changed, but I needed to include all the jars in my class path, had to use bin stubs, and had to set GEM_HOME to get everything working. It may have been simpler, but the posts you referenced didn't work for me either.
I actually had jruby installed, (but I was only using it to build the concatenated class path), so my setup was something like:
cd /path/to/application/
export GEM_HOME=/path/to/application/gems
export CLASSPATH=$(jruby -e 'puts Dir["lib/*.jar"].join(":")')
RAILS_ENV=production java -cp $CLASSPATH org.jruby.Main bin/rake -T
Also useful, the jruby-jars gem can be included in your gemfile to set the version of jruby that warbler includes (I was using gem 'jruby-jars', '1.7.0.preview2' as 1.7.0 wasn't released yet)

deploying java through Cygwin

My Java development done in a windows machine and i run my processes on a centos machine.
I have a bash script that build all my Jars and SCP them to my centos machine. i run this bash script in Cygwin (java -version is 1.5.0_12), but when i try to run the process in my centos machine, the jVM can't open the Jars. also, running jar -tf throws:
java.util.zip.ZipException: error in opening zip file
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.<init>(ZipFile.java:114)
at java.util.zip.ZipFile.<init>(ZipFile.java:75)
at sun.tools.jar.Main.list(Main.java:979)
at sun.tools.jar.Main.run(Main.java:224)
at sun.tools.jar.Main.main(Main.java:1149)
so the only way i can deploy is running mvn commands in cmd.exe in order to build my jars and then copy everything using WinSCP (that way i have no problems in the jars).
is there any known problem running mvn in cygwin?
(running mvn -version returned
Apache Maven 2.2.1 (r801777; 2009-08-06 22:16:01+0300)
Java version: 1.6.0_26)
thank you

Solved it.
i found the solution in cygwin sets file permission to 000
Edit /etc/fstab and add this line at the end of the file:
none /cygdrive cygdrive binary,noacl,posix=0,user 0 0
Then close all Cygwin processes, open a new terminal and ls -l on your files again.

Maven being a Java application runs the same whether launched via Cygwin script or cmd.exe. The Java executable in this case is the same tool.
First, you might want to post the copy command you are using in the bash script. Secondly, have you checked the permissions on the jar files once they are pushed to the CentOS box? Are the files actually readable to the process owner when sent via your bash script and are the owners/permissions the same as when copied using WinSCP?