When I run the command I got the error:
root#1290ff36814b:/opt/airflow/dags# python test.py
/opt/spark-3.3.1/bin/spark-class: line 71: /opt/jdk1.8.0/bin/java: No such file or directory
/opt/spark-3.3.1/bin/spark-class: line 96: CMD: bad array subscript
Traceback (most recent call last):
File "test.py", line 16, in <module>
.appName('test') \
File "/opt/spark-3.3.1/python/pyspark/sql/session.py", line 269, in getOrCreate
sc = SparkContext.getOrCreate(sparkConf)
File "/opt/spark-3.3.1/python/pyspark/context.py", line 483, in getOrCreate
SparkContext(conf=conf or SparkConf())
File "/opt/spark-3.3.1/python/pyspark/context.py", line 195, in __init__
SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
File "/opt/spark-3.3.1/python/pyspark/context.py", line 417, in _ensure_initialized
SparkContext._gateway = gateway or launch_gateway(conf)
File "/opt/spark-3.3.1/python/pyspark/java_gateway.py", line 106, in launch_gateway
raise RuntimeError("Java gateway process exited before sending its port number")
RuntimeError: Java gateway process exited before sending its port number
This is a docker container and /opt/jdk1.8.0 is a volume, and the volume is visible inside container and also de java binary.
root#1290ff36814b:/opt/airflow/dags# ls -l /opt/jdk1.8.0/bin/java
-rwxr-xr-x 1 uucp 143 8464 Dec 15 2018 /opt/jdk1.8.0/bin/java
root#1290ff36814b:/opt/airflow/dags# ls -l /opt/jdk1.8.0/
total 25996
-r--r--r-- 1 uucp 143 3244 Dec 15 2018 COPYRIGHT
-r--r--r-- 1 uucp 143 40 Dec 15 2018 LICENSE
-r--r--r-- 1 uucp 143 159 Dec 15 2018 README.html
-rw-r--r-- 1 uucp 143 112724 Dec 12 2018 THIRDPARTYLICENSEREADME-JAVAFX.txt
-r--r--r-- 1 uucp 143 153824 Dec 15 2018 THIRDPARTYLICENSEREADME.txt
drwxr-xr-x 2 uucp 143 4096 Dec 15 2018 bin
drwxr-xr-x 3 uucp 143 132 Dec 15 2018 include
-rw-r--r-- 1 uucp 143 5211154 Dec 12 2018 javafx-src.zip
drwxr-xr-x 5 uucp 143 4096 Dec 15 2018 jre
drwxr-xr-x 5 uucp 143 4096 Dec 15 2018 lib
drwxr-xr-x 4 uucp 143 47 Dec 15 2018 man
-rw-r--r-- 1 uucp 143 424 Dec 15 2018 release
-rw-r--r-- 1 uucp 143 21105955 Dec 15 2018 src.zip
root#1290ff36814b:/opt/airflow/dags#
Why this happen?
I already tried to set JAVA_HOME and PATH, but still the same problem.
I just realize it has one space inside my python code where I export the JAVA_HOME. After remove it finally the code worked.
import os
import sys
os.environ["SPARK_HOME"]='/opt/spark-3.3.1'
os.environ["JAVA_HOME"]=' /opt/jdk1.8.0'
sys.path+=[os.environ["SPARK_HOME"]+"/python"]
sys.path+=[os.environ["SPARK_HOME"]+"/python/build"]
sys.path+=[os.environ["SPARK_HOME"]+"/python/lib/py4j-0.10.9.5-src.zip"]
Related
I've got a relatively simple jar-with-dependencies being built with Maven that is way larger than it seems like it should be. It's around 20MB, and in order to figure out what's taking up so much space, I've done the following:
First, I ran mvn dependency:tree. Then I checked in my .m2 cache for the size of each of the jars in the dependency tree. If I add up all of those sizes, it comes to about 8MB. How can I figure out where the other 12MB are coming from?
One thing I noticed in looking through my .m2 was that for many of the dependencies, they'll have something like this:
total 5224
-rw-r--r-- 1 user 289B Jul 25 2016 _remote.repositories
-rw-r--r-- 1 user 1.7M Jul 25 2016 commons-compress-1.12-javadoc.jar
-rw-r--r-- 1 user 407B Jul 25 2016 commons-compress-1.12-javadoc.jar.lastUpdated
-rw-r--r-- 1 user 40B Jul 25 2016 commons-compress-1.12-javadoc.jar.sha1
-rw-r--r-- 1 user 427K Jul 25 2016 commons-compress-1.12-sources.jar
-rw-r--r-- 1 user 407B Jul 25 2016 commons-compress-1.12-sources.jar.lastUpdated
-rw-r--r-- 1 user 40B Jul 25 2016 commons-compress-1.12-sources.jar.sha1
-rw-r--r-- 1 user 432K Jul 22 2016 commons-compress-1.12.jar
-rw-r--r-- 1 user 407B Jul 22 2016 commons-compress-1.12.jar.lastUpdated
-rw-r--r-- 1 user 40B Jul 22 2016 commons-compress-1.12.jar.sha1
-rw-r--r-- 1 user 13K Jul 22 2016 commons-compress-1.12.pom
-rw-r--r-- 1 user 407B Jul 22 2016 commons-compress-1.12.pom.lastUpdated
-rw-r--r-- 1 user 40B Jul 22 2016 commons-compress-1.12.pom.sha1
What are the -sources and -javadoc jars? Are those included in my uber jar? Because if every one of my dependencies uses the -javadoc jar instead of the standard one, that gets me a lot closer to 20MB.
Run
jar tvvf <your_simple-jar-with-dependencies.jar>
Or, open it with any zip compatible archiver and examine the contents to determine what is being included. Alternatively, run maven with -X for more extensive runtime information.
When I do
bin/zkServer.sh start #It shows it has started
ZooKeeper JMX enabled by default
Using config: /data/sparkHA/zookeeper-3.4.9/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
Afetr few second when i check status, I got
ZooKeeper JMX enabled by default
Using config: /data/sparkHA/zookeeper-3.4.9/bin/../conf/zoo.cfg
Error contacting service. It is probably not running.
My zookeeper.out says
nohup: failed to run command â/usr/bin/java/bin/javaâ: Not a directory
But my JAVA_HOME in bashrc is usr/bin/java, How come one extra /bin/java is added , that results in an invalid directory.
Also echo $JAVA_HOME outputs
/usr/bin/java
How to approach this error. Please Help. Thanks.
Also tried setting JAVA_HOME in zkServer.sh also by following Zookeeper not starting, nohup error but getting same error.
Your JAVA_HOME point to /usr/bin/java file but it should point to root directory of your JDK. For example for me this is:
➜ ~ echo $JAVA_HOME
/Library/Java/JavaVirtualMachines/jdk1.8.0_112.jdk/Contents/Home
➜ ~ ll $JAVA_HOME
total 52064
-rw-rw-r-- 1 root wheel 3.2K Oct 1 09:00 COPYRIGHT
-rw-rw-r-- 1 root wheel 40B Oct 1 09:01 LICENSE
-rw-rw-r-- 1 root wheel 159B Oct 1 09:01 README.html
-rwxrwxr-x 1 root wheel 108K Sep 22 22:49 THIRDPARTYLICENSEREADME-JAVAFX.txt
-rw-rw-r-- 1 root wheel 173K Oct 1 09:01 THIRDPARTYLICENSEREADME.txt
drwxrwxr-x 46 root wheel 1.5K Oct 1 09:04 bin
drwxrwxr-x 9 root wheel 306B Oct 1 09:00 db
drwxrwxr-x 9 root wheel 306B Oct 1 09:00 include
-rwxrwxr-x 1 root wheel 4.9M Sep 22 22:49 javafx-src.zip
drwxrwxr-x 10 root wheel 340B Oct 1 09:02 jre
drwxrwxr-x 14 root wheel 476B Oct 1 09:02 lib
drwxrwxr-x 5 root wheel 170B Oct 1 09:01 man
-rw-rw-r-- 1 root wheel 529B Oct 1 09:01 release
-rw-rw-r-- 1 root wheel 20M Oct 1 09:01 src.zip
So, try to set valid path to JDK root directory. I think it should fix your problem.
Anyone can tell me where can I find the full version of JFreeChart? I downloaded the JCommon and JFreeChart as well (from the sourceforge page), but the JFreeChart contains only the jfreechart-demo.jar, so even if I correctly added the libraries in Eclipse, it doesn't work either...
Many Thanks!
When unpacked, the required JAR files are in the lib directory.
$ ls -ln lib
total 5800
-rw-r--r--# 1 501 20 45024 Jul 31 2014 hamcrest-core-1.3.jar
-rw-r--r--# 1 501 20 330246 Jul 31 2014 jcommon-1.0.23.jar
-rw-r--r--# 1 501 20 13796 Jul 31 2014 jfreechart-1.0.19-experimental.jar
-rw-r--r--# 1 501 20 79793 Jul 31 2014 jfreechart-1.0.19-swt.jar
-rw-r--r--# 1 501 20 1570157 Jul 31 2014 jfreechart-1.0.19.jar
-rw-r--r--# 1 501 20 50769 Jul 31 2014 jfreesvg-2.0.jar
-rw-r--r--# 1 501 20 245039 Jul 31 2014 junit-4.11.jar
-rw-r--r--# 1 501 20 445025 Jul 31 2014 orsoncharts-1.4-eval-nofx.jar
-rw-r--r--# 1 501 20 72444 Jul 31 2014 orsonpdf-1.6-eval.jar
-rw-r--r--# 1 501 20 80054 Jul 31 2014 servlet.jar
-rw-r--r--# 1 501 20 17492 Jul 31 2014 swtgraphics2d.jar
For example, the following command line runs TimeSeriesChartDemo1 on Unix; separate class path entries with a semicolon (;) on Windows.
$ java -cp .:lib/* org.jfree.chart.demo.TimeSeriesChartDemo1
I am trying to run jconsole from my machine but I can't find the location of the .sh . I have tried finding it in different directories and setting JAVA_HOME to the jdk.
$which java
/usr/bin/java
$java -version
java version "1.7.0_51"
OpenJDK Runtime Environment (rhel-2.4.4.1.el6_5-x86_64 u51-b02)
OpenJDK 64-Bit Server VM (build 24.45-b08, mixed mode)
$whereis java
java: /usr/bin/java /etc/java /usr/lib/java /usr/share/java /usr/share/man/man1/java.1.gz
I looked in /usr/lib/jvm and I found
drwxr-xr-x. 3 root root 4096 Apr 23 2013 java-1.5.0-gcj-1.5.0.0
drwxr-xr-x. 3 root root 4096 Jan 27 2014 java-1.6.0-openjdk-1.6.0.0.x86_64
drwxr-xr-x. 3 root root 4096 Feb 5 2014 java-1.7.0-openjdk-1.7.0.51.x86_64
lrwxrwxrwx. 1 root root 21 Feb 5 2014 jre -> /etc/alternatives/jre
lrwxrwxrwx. 1 root root 27 Apr 23 2013 jre-1.5.0 -> /etc/alternatives/jre_1.5.0
lrwxrwxrwx. 1 root root 26 Apr 23 2013 jre-1.5.0-gcj -> java-1.5.0-gcj-1.5.0.0/jre
lrwxrwxrwx. 1 root root 27 Feb 5 2014 jre-1.6.0 -> /etc/alternatives/jre_1.6.0
lrwxrwxrwx. 1 root root 37 Feb 5 2014 jre-1.6.0-openjdk.x86_64 -> java-1.6.0-openjdk- 1.6.0.0.x86_64/jre
lrwxrwxrwx. 1 root root 27 Feb 5 2014 jre-1.7.0 -> /etc/alternatives/jre_1.7.0
lrwxrwxrwx. 1 root root 38 Feb 5 2014 jre-1.7.0-openjdk.x86_64 -> java-1.7.0-openjdk-1.7.0.51.x86_64/jre
lrwxrwxrwx. 1 root root 25 Apr 23 2013 jre-gcj -> /etc/alternatives/jre_gcj
lrwxrwxrwx. 1 root root 29 Feb 5 2014 jre-openjdk -> /etc/alternatives/jre_openjdk
and in java-1.7.0-openjdk-1.7.0.51.x86_64/jre/bin/, I only found
[root#portal01 bin]# ll
total 200
-rwxr-xr-x. 1 root root 9536 Jan 14 2014 java
-rwxr-xr-x. 1 root root 9656 Jan 14 2014 keytool
-rwxr-xr-x. 1 root root 9856 Jan 14 2014 orbd
-rwxr-xr-x. 1 root root 9688 Jan 14 2014 pack200
-rwxr-xr-x. 1 root root 10984 Jan 14 2014 policytool
-rwxr-xr-x. 1 root root 9648 Jan 14 2014 rmid
-rwxr-xr-x. 1 root root 9656 Jan 14 2014 rmiregistry
-rwxr-xr-x. 1 root root 9688 Jan 14 2014 servertool
-rwxr-xr-x. 1 root root 9896 Jan 14 2014 tnameserv
-rwxr-xr-x. 1 root root 92264 Jan 14 2014 unpack200
..even though there are 200 files. How can I find and run jconsole?
jconsole is part of the Java SDK, it appears you only have the JRE installed.
You can run jconsole directly from your terminal by running the command jconsole.
reena#IM-1500:~$ jconsole
It will launch java monitoring console dialog.
From https://serverfault.com/questions/179908/fedora-jconsole-location:
Try running: yum provides */jconsole
that will tell you what package you need as well as where it's located in the package.
Once you set the path of JDK in Linux then this command execute easily might be you had not included the JAVA_HOME with PATH variable.
plz follow this discussion
If you have Java JDK installed in your Linux machine, usually jconsole is located at /usr/bin/ as a symbolic link.
Example from my Linux VM running Centos6
$ which jconsole
/usr/bin/jconsole
$ ls -l /usr/bin/jconsole
lrwxrwxrwx. 1 root root 26 Sep 4 10:05 /usr/bin/jconsole -> /etc/alternatives/jconsole
$ ls -l /etc/alternatives/jconsole
lrwxrwxrwx. 1 root root 51 Sep 4 10:05 /etc/alternatives/jconsole -> /usr/lib/jvm/java-1.7.0-openjdk.x86_64/bin/jconsole
But you mentioned you got the a command not found error when you typed jconsole at the prompt. Can you find if jconsole is available in one of the jdk's bin directory?
$ ls -l /usr/lib/jvm/java-1.7.0-openjdk.x86_64/bin | grep jconsole
If yes, then add this in your PATH environment variable like so
$ export PATH=$PATH:/usr/lib/jvm/java-1.7.0-openjdk.x86_64/bin
or put this in your ~/.bashrc if you are using Bash.
If jconsole is not there, then maybe you need to try another jdk which you have installed or re-install the jdk.
Please install JDK Development Package to get Jconsole
I try to build a android app with android studio. Gradle throws this error when i try to build:
Gradle invocation completed successfully with 1 error(s) in 10 sec
and the message gradle build window shows:
Error:Execution failed for task
':SlidingUpPanelLayout:processReleaseResources'.
java.io.IOException: Cannot run program "/home/[...]/android-studio/sdk/build-tools/19.0.2/aapt": error=2, No
such file or directory
but ls -l shows me this files:
~/development/android-studio/sdk/build-tools/19.0.2$ ls -l
total 49348
-rwxr-xr-x 1 david david 1109734 Apr 12 12:29 aapt
-rwxr-xr-x 1 david david 271551 Apr 12 12:29 aidl
-rwxr-xr-x 1 david david 3570836 Apr 12 12:29 arm-linux-androideabi-ld
-rwxr-xr-x 1 david david 36534 Apr 12 12:29 bcc_compat
-rwxr-xr-x 1 david david 445402 Apr 12 12:29 dexdump
-rwxr-xr-x 1 david david 2577 Apr 12 12:29 dx
-rwxr-xr-x 1 david david 3570836 Apr 12 12:29 i686-linux-android-ld
drwxr-xr-x 2 david david 4096 Apr 12 12:29 lib
-rwxr-xr-x 1 david david 332357 Apr 12 12:29 libbcc.so
-rwxr-xr-x 1 david david 932160 Apr 12 12:29 libbcinfo.so
-rwxr-xr-x 1 david david 15372369 Apr 12 12:29 libclang.so
-rwxr-xr-x 1 david david 21101517 Apr 12 12:29 libLLVM.so
-rwxr-xr-x 1 david david 1667769 Apr 12 12:29 llvm-rs-cc
-rwxr-xr-x 1 david david 1919720 Apr 12 12:29 mipsel-linux-android-ld
-rw-r--r-- 1 david david 10800 Apr 12 12:29 NOTICE.txt
drwxr-xr-x 5 david david 4096 Apr 12 12:29 renderscript
-rw-r--r-- 1 david david 16649 Apr 12 12:29 source.properties
why say gradle the the aapt is not a file? whats wrong with it?
this helped me link
sudo apt-get install lib32stdc++6
sudo apt-get install lib32z1