I am running the following shell script,
#!/bin/bash
HADOOP_HOME='/home/hadoop/hadoop-0.20.2'
HIVE_HOME='/home/hadoop/hive-0.8.0'
echo -e '1\x01foo' > /tmp/a.txt
echo -e '2\x01bar' >> /tmp/a.txt
HADOOP_CORE={{ls $HADOOP_HOME/hadoop-0.20.2-core.jar}}
CLASSPATH=.:$HADOOP_CORE:$HIVE_HOME/conf
for i in ${HIVE_HOME}/lib/*.jar ; do
CLASSPATH=$CLASSPATH:$i
done
java -cp $CLASSPATH HiveJdbcClient
In this code every variable is set properly. I have written code as per instructions given here. My hive version is 0.8.0.
When I run my code it gives me the following error,
./test.sh
./test.sh: line 8: /home/hadoop/hadoop-0.20.2/hadoop-0.20.2-core.jar}}: No such file or directory
Exception in thread "main" java.sql.SQLException: Could not establish connection to localhost:10000/default: java.net.ConnectException: Connection refused
at org.apache.hadoop.hive.jdbc.HiveConnection.<init>(HiveConnection.java:98)
at org.apache.hadoop.hive.jdbc.HiveDriver.connect(HiveDriver.java:104)
at java.sql.DriverManager.getConnection(DriverManager.java:582)
at java.sql.DriverManager.getConnection(DriverManager.java:185)
at HiveJdbcClient.main(HiveJdbcClient.java:22)
Can anyone please explain me what this error means?
The error you are seeing indicates that Java could not connect to the port with the JDBC server.
You should start it by following the instructions on the HiveServer wiki page. Something like:
$HIVE_HOME/bin/hive --service hiveserver
should work.
code Banger, here tricky thing is that Hive was not running in the background at all.
Check it is running.
First export the hive port, then start the hive.
$ export HIVE_PORT=10000
Good luck!
Related
I am new to Zeppelin and want to install it on my Windows10Pro/WSL machine.
These installation scripts are used https://github.com/x4ax/lxss-install-zeppelin .
Since it is three years old, I had to modify it a bit, so I have:
Ubuntu 20.04, bash
zeppelin-0.9.0-bin-all
hadoop-3.3.0
spark-3.0.1-bin-hadoop2.7
I link python3 to python
Everything is installed, hadoop and spark are successfully tested with provided scripts. At the end I managed to see "Welcome to Zeppelin!" landing page.
First, I go to the provided tutorial python notes, "1.IPython Basic" and run first cell with %md only. I get the error message":
"
org.apache.zeppelin.interpreter.InterpreterException: java.io.IOException: Fail to launch interpreter process:
null
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:129)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:271)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:444)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:72)
at org.apache.zeppelin.scheduler.Job.run(Job.java:172)
at org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:132)
at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:182)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Fail to launch interpreter process:
null
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:126)
at org.apache.zeppelin.interpreter.ManagedInterpreterGroup.getOrCreateInterpreterProcess(ManagedInterpreterGroup.java:68)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:104)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:154)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
... 13 more
Info from md-shared.log
INFO [2021-01-14 18:54:46,610] ({RemoteInterpreterServer-Thread} RemoteInterpreterServer.java[run]:193) - Launching ThriftServer at 169.254.120.3:52579
INFO [2021-01-14 18:54:47,785] ({RegisterThread} RemoteInterpreterServer.java[run]:609) - Registering interpreter process
ERROR [2021-01-14 18:54:47,790] ({RegisterThread} RemoteInterpreterServer.java[run]:613) - Error while registering interpreter: RegisterInfo(host:169.254.120.3, port:52579, interpreterGroupId:md-shared_process), cause: {}
java.lang.RuntimeException: java.io.IOException: org.apache.zeppelin.shaded.org.apache.thrift.transport.TTransportException: java.net.SocketException: Network is unreachable (connect failed)
Which means that there is some network problem
Steps to find solution:
From logs I see the command like this, which is run by RemoteInterpreter java-object:
/usr/local/zeppelin/bin/interpreter.sh -d /usr/local/zeppelin/interpreter/md -c 169.254.120.3 -p 52579 -r : -i md-shared_process -l /usr/local/zeppelin/local-repo/md -g md
It runs silently.
zeppelin-daemon.sh start/stop run ok. 'status' also shows correct status. So, restart does not help.
Reinstall of zeppelin and of wsl did not help.
I also tested problem while firewall was shutdown.
I am puzzled.
By looking here Hello world in zeppelin failed I managed to run md interpreter by fixing in conf/zeppelin-env.sh
ZEPPELIN_LOCAL_IP=127.0.0.1
I saw some statements (I cannot confirm them!) that:
Microsoft WSL blocks random addressing or
WSL listens to localhost only if it is really local.
PS. Now, I have difficulty to run python interpreter, but that is another problem.
(Besides, it can be linked to my aliasing of python with python3 or similar shell settings).
I'm starting the h2 server using this command:
java -Xms256M -Xmx512M -Xss1024K -cp "C:\workspaces\ProjectA\ModuleA\target;C:\Program Files (x86)\H2\bin\h2-1.4.199.jar" org.h2.tools.Server -tcp -tcpPort 80 -tcpAllowOthers -baseDir C:/database/h2/data
where in C:\workspaces\ProjectA\ModuleA\target I got jar where is my trigger class(I checked with the total commander, and I'm 100% sure it is there)
Then I open my h2 database and try to run sql on table for which I have set up trigger, but it throws:
Caused by: org.h2.message.DbException: Class "com.test.projecta.trigger.Trigger" not found [90086-199]
at org.h2.message.DbException.get(DbException.java:194).
Why is it thrown? As I understand trigger should be added to classpath and it is, so why it is not seen?
Solved: It was the problem with the way I was connecting to the database.I was specifying JDBC URL like jdbc:h2:C:\database\h2\data\database instead of doing it this way jdbc:h2:tcp://localhost:80/database
I'm using CentOS and have downloaded Elasticsearch 6.2.1. I created a new user "elastic" and when I run ./bin/elasticsearch I get the error:
Could not find or load main class org.elasticsearch.tools.launchers.JavaVersionChecker
I tried placing this user in an admin group ("wheel"), and the same problem occurs. If I try it with "sudo ./bin/elasticsearch" I get:
[2018-02-15T17:42:39,776][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:125) ~[elasticsearch-6.2.1.jar:6.2.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-6.2.1.jar:6.2.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.2.1.jar:6.2.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.2.1.jar:6.2.1]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.2.1.jar:6.2.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-6.2.1.jar:6.2.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) ~[elasticsearch-6.2.1.jar:6.2.1]
Caused by: java.lang.RuntimeException: can not run elasticsearch as root
at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:105) ~[elasticsearch-6.2.1.jar:6.2.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:172) ~[elasticsearch-6.2.1.jar:6.2.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) ~[elasticsearch-6.2.1.jar:6.2.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-6.2.1.jar:6.2.1]
I searched a bit and saw that it this error can be due to the java version, but it seems to be up to date:
[elastic#sandbox-hdp elasticsearch-6.1.1]$ sudo update-alternatives --config java
There are 3 programs which provide 'java'.
Selection Command
1 /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java
*+ 2 /usr/lib/jvm/jre-1.8.0-openjdk.x86_64/bin/java
3 /usr/lib/jvm/jre-1.5.0-gcj/bin/java
This also happens if I try Elasticsearch 6.1
Any suggestions would be appreciated.
In case you still have this problem... or just for curiosity... take a look at the file ./bin/elasticsearch-env.sh, around the line 70, look for the command:
"$JAVA" -cp "$ES_CLASSPATH" org.elasticsearch.tools.launchers.JavaVersionChecker
Check if it is correctly "spelled" for your environment, write this command just before it and see what you've got:
echo "$JAVA" -cp "$ES_CLASSPATH" org.elasticsearch.tools.launchers.JavaVersionChecker
in my case... I was running in a Git Bash inside a Windows machine, and the command was pointing to:
/c/Users/Ualter/Developer/ELKStack/elasticsearch-6.2.2/lib/*
Changing to:
/Users/Ualter/Developer/ELKStack/elasticsearch-6.2.2/lib/*
As I was running on a Windows S.O. it get up and running OK the Elasticsearch.
I was trying hard to use elasticsearch 6.7.1 on Windows.
I had this exact error when running bin/elasticsearch.bat .
Long story short, there was probably an issue with my java version and elastic search, it couldn't run the jars (using Java 8 211).
I installed the last ElasticSearch 6.x from here :
https://www.elastic.co/downloads/past-releases#elasticsearch
And unset the existing ES_HOME and ES_CLASSPATH environment variables that I created, and all went well.
I also encounter such strange question, the first time it raises exception for running elastic as root, after I add a new user elastic and run it as elastic , I see this exception "Could not find or load main class org.elasticsearch.tools.launchers.JavaVersionChecker
".
I solved this problem by moving the program dirs to the non /root/ dirs, for example, I put it to /opt/elk/, and it works.
So I guess elastic may not allow to run as root for security , thus we can not put the program in the root user's dir. Hope it can make some tips for you.
I'm installing elasticsearch plugin es-sql on windows.
In my case, run the cmd as administrator role.
Seems like you don't have right permissions. Try to set them to 774 (rwxrwxr--) and check again:
sudo chmod 774 -R elasticsearch-6.3.2/
I have a java program ( eclipse , Maven , windows7) to fill Hbase which exist on a VirtualBox from a service,
the program works fine when I run it from eclipse.
But when i created an executable jar using the assembly plugin in Maven and run it from the cmd or from cygwin I get this error
> [2016-05-03 14:46:44,663][DEBUG] Reading reply sessionid:0x154769ed563000a, packet:: clientPath:null serverPath:null finished:false header:: 300,4 replyHeader:: 300,3632,-101 request:: '/hbase-unsecure/meta-region-server,F response:: [2016-05-03 14:46:44,663][DEBUG] hconnection-0x6b63f5ff-0x154769ed563000a, quorum=sandbox:2181, baseZNode=/hbase-unsecure Unable to get data of znode /hbase-unsecure/meta-region-server because node does not exist (not an error) [2016-05-03 14:46:44,663][DEBUG] Looked up meta region location, connection=org.apache.hadoop.hbase.client.ZooKeeperRegistry#695e5335; servers = null [2016-05-03 14:46:44,689][DEBUG] Closing scanner id=-1
>org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the locations
at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:312)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:61)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:155)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:811)
at org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:602)
at org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:366)
at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:303)
at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:313)
Could any of you help me with this?
Some more information like the following may be helpful. Run these commands in the virtualbox where you have the "hbase" command
hbase zkCli -server sandbox:2181
Once you connect, you can run, in the zkcli prompt:
ls /
Please paste the output of the above command. The baseZnode of "/hbase-unsecure " does not seem to exist. In general the base znode is "/hbase".
Kindly cross check the classpath of your running program by the java code, as suggested by #RamPrasadG
System.out.println(System.getProperty("java.classpath"))
I have noticed an issue that crops up repeatedly: using an env var for the classpath on the java command line within a shell script does not work.
First off, let us see what does work: both using hard-coded classpath in a script as follows: (note: the "classpath is" statement is printed within the java program itself)
steve#mithril:/shared$ java -classpath .:/shared/mysql-connector-java-5.1.25-bin.jar DbPing com.mysql.jdbc.Driver jdbc:mysql://localhost:3306/mysql user password
classpath is .:/shared/mysql-connector-java-5.1.25-bin.jar
Attempting connection to com.mysql.jdbc.Driver jdbc:mysql://localhost:3306/mysql password
Connecting to user using URL=jdbc:mysql://localhost:3306/mysql
Successfully connected.
and also using env var directly within the shell:
steve#mithril:/shared$ export CP=.:/shared/mysql-connector-java-5.1.25-bin.jar
steve#mithril:/shared$ java -classpath $CP DbPing com.mysql.jdbc.Driver jdbc:mysql://localhost:3306/mysql user password
classpath is .:/shared/mysql-connector-java-5.1.25-bin.jar
Attempting connection to com.mysql.jdbc.Driver jdbc:mysql://localhost:3306/mysql password
Connecting to user using URL=jdbc:mysql://localhost:3306/mysql
Successfully connected.
What does *not * work: running the same commands as shown above within a shell script:
steve#mithril:/shared$ cat dbping.mysql
CP=.:/shared/mysql-connector-java-5.1.25-bin.jar
echo $CP
java -classpath $CP DbPing com.mysql.jdbc.Driver jdbc:mysql://localhost:3306/mysql user password
#java -classpath .:/shared/mysql-connector-java-5.1.25-bin.jar DbPing com.mysql.jdbc.Driver jdbc:mysql://localhost:3306/mysql user password
steve#mithril:/shared$ ./dbping.mysql
.:/shared/mysql-connector-java-5.1.25-bin.jar
classpath is .:/shared/mysql-connector-java-5.1.25-bin.jar
Attempting connection to com.mysql.jdbc.Driver jdbc:mysql://localhost:3306/mysql password
Could not load db driver com.mysql.jdbc.Driver
java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:190)
at DbPing.getConnection(DbPing.java:34)
at DbPing.main(DbPing.java:22)
Exception in thread "main" java.sql.SQLException: com.mysql.jdbc.Driver
at DbPing.getConnection(DbPing.java:41)
at DbPing.main(DbPing.java:22)
Follow-up: The script had windows style newlines in it. Apparently the \r clobbered the internal environment variable. found this using
od -cx
. I am going to give credit to stephen c anyways, since his prodding got me on the right track to finding the solution
The symptoms you describe are rather puzzling. I can't see the problem (the script looks correct), but I've got an idea on how to start tracking it down.
Get rid of the commented out lines from the script.
Add a #!/bin/sh line to the start of the script to make sure that you are actually using the right shell to execute it. (It is always a good idea to do this ... even if you think that you will get the right shell by default. That might change, depending on the platform.)
To figure out what the shell is doing, add set -vx as after the #!/bin/sh line.
The "-v" says echo each script line read, and the "-x" says echo the actual command lines that are executed. This will tell you exactly what commands are being run ... so that you can figure out what the command arguments really are.