We Keep Coding

"Unavaliable io exception" when connecting to remote Bazel master on bazel-buildfarm

I want to setup a small POC remote area with 1x master (192.168.60.99) and 1x worker (192.168.60.98) using bazel-buildfarm. Both are CentOS 7 machines provisioned with Vagrant. When connection from a Ubuntu workstation (third machine) in the network, the following error occurs:
$ bazel build --verbose_failures //projects/myproj:app
Starting local Bazel server and connecting to it...
INFO: Options provided by the client:
Inherited 'common' options: --isatty=1 --terminal_columns=229
INFO: Reading rc options for 'build' from /home/user/tests/ecommerce/.bazelrc:
'build' options: --strategy=TypeScriptCompile=worker --strategy=AngularTemplateCompile=worker --symlink_prefix=dist/ --define=compile=legacy --incompatible_strict_action_env --experimental_allow_incremental_repository_updates --distdir=third_party/_distdir
INFO: Reading rc options for 'build' from /home/user/.bazelrc:
'build' options: --spawn_strategy=remote --genrule_strategy=remote --strategy=Javac=remote --strategy=Closure=remote --remote_executor=192.168.60.99:8980
INFO: Writing tracer profile to '/home/user/.cache/bazel/_bazel_user/24700f1ad3e201a00a1c26bd59dc6502/command.profile.gz'
INFO: Invocation ID: 569b59ca-edcb-4922-92a0-b6f0b5ca2819
ERROR: Failed to query remote execution capabilities: UNAVAILABLE: io exception
The network connection is working and I even can connect to Bazel using telnet:
telnet 192.168.60.99 8980
Trying 192.168.60.99...
Connected to 192.168.60.99.
Escape character is '^]'.
.bazelrc file of the third Ubuntu machine:
$ cat ~/.bazelrc
build --spawn_strategy=remote --genrule_strategy=remote --strategy=Javac=remote --strategy=Closure=remote --remote_executor=192.168.60.99:8980
Buildfarm setup
Both got a clon of the buildfarm git repo. The example config files were used. Just on the server I replaced localhost by 192.168.60.99 (master server ip).
I know that bazel run is not recommended. But in lack of better alternatives that works, my idea is to get the documented way working first (Bazel itself doesn't mention any alternative). Since not even bazel run works, I think that something is wrong with my installation.
All machines use version 1.1.0, which is the latest stable one at the time of writing. It's definitely an issue with bazel-buildfarm, since the local build works fine on the Ubuntu machine.
Master server
bazel run //src/main/java/build/buildfarm:buildfarm-server $(pwd)/examples/server.config.example
Worker
bazel run //src/main/java/build/buildfarm:buildfarm-operationqueue-worker $(pwd)/examples/worker.config.example --distdir ~/distdir/
The distdir is a workaround for our company proxy, that manipulates files with MITM attacks. Since Bazel doesn't allow this, I downloaded the affected file for it's jdk manually:
[vagrant#localhost bazel-buildfarm]$ l ~/distdir/
total 188M
-rw-rw-r--. 1 vagrant vagrant 188M Jan 17 2019 zulu11.2.3-jdk11.0.1-linux_x64.tar.gz

If Bazel >= 1.0 is used , we need to specify the protocol grpc in .bazelrc like this:
--remote_executor=grpc://192.168.60.99:8980
Without the protocol, the UNAVAILABLE: io exception occurs. There is currently no documentation about this issue.

Failing to set up Zookeeper cluster for Pulsar

I am trying to set up a Zookeeper cluster for Pulsar. I am following the instructions here, but I keep failing.
In my setup, I have two nodes, that should be part of the cluster. Since I need to deploy bookie to the same nodes, I executed
$ PULSAR_EXTRA_OPTS="-Dstats_server_port=8001" bin/pulsar-daemon start zookeeper
to start zookeeper. Afterwards, I am trying to init the cluster using this command:
bin/pulsar initialize-cluster-metadata \
--cluster pulsar-cluster-1 \
--zookeeper 10.100.100.77:2181 \
--configuration-store 10.100.100.77:2181 \
--web-service-url http://10.100.100.77:8080 \
--broker-service-url pulsar://10.100.100.77:6650 \
But I keep getting this error:
17:12:24.146 [main-SendThread(10.100.100.77:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: 10.100.100.77/10.100.100.77:2181: Verbindungsaufbau abgelehnt
17:12:25.251 [main-SendThread(10.100.100.77:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 10.100.100.77/10.100.100.77:2181. Will not attempt to authenticate using SASL (unknown error)
I read here that I need to have an odd number of nodes, so I added a virtual machine on one of the nodes. When I start Zookeeper on it, it doesn't print an error message, but but shows:
$ PULSAR_EXTRA_OPTS="-Dstats_server_port=8001" bin/pulsar-daemon start zookeeper
doing start zookeeper ...
starting zookeeper, logging to /home/host1/apache-pulsar-2.4.0/logs/pulsar-zookeeper-host1-VirtualBox.log
OpenJDK 64-Bit Server VM warning: Option AggressiveOpts was deprecated in version 11.0 and will likely be removed in a future release.
[AppClassLoader#27c170f0] info AspectJ Weaver Version 1.9.2 built on Wednesday Oct 24, 2018 at 15:43:33 GMT
[AppClassLoader#27c170f0] info register classloader jdk.internal.loader.ClassLoaders$AppClassLoader#27c170f0
[AppClassLoader#27c170f0] info using configuration file:/home/host1/apache-pulsar-2.4.0/lib/org.apache.pulsar-pulsar-zookeeper-utils-2.4.0.jar!/META-INF/aop.xml
[AppClassLoader#27c170f0] info using configuration file:/home/host1/apache-pulsar-2.4.0/lib/org.apache.pulsar-pulsar-zookeeper-2.4.0.jar!/META-INF/aop.xml
[AppClassLoader#27c170f0] info register aspect org.apache.pulsar.zookeeper.SerializeUtilsAspect
[AppClassLoader#27c170f0] info register aspect org.apache.pulsar.broker.zookeeper.aspectj.ClientCnxnAspect
However the Zookeeper service is not started, even if the setup is very similar to its host and I can't make up why.
Any Ideas how I could proceed from here? Thanks in advance!

The first error you posted seems to indicate that the connection to 10.100.100.77:2181 is refused "Verbindungsaufbau abgelehnt", and therefore the ZK server isn't listening on that server and port. You should first confirm that ZK is up and running and check the ZK log for any errors.
HTH

I found the soulution. The original error was indeed caused by having an odd number of nodes. The third (virtual) one wouldn't start, because of a mislocation of Zookepers data-directory. Now that the third server started, also the configuration passed successfully.

Where to see console log in Openshift?

Recently, I've deployed my JSP project into Openshift server. Now my wish is to see the Console log.
Suppose, if I print System.out.println("Message"); into my JSP project, how do I see that message printed into Console log in Openshift server?
EDITED:
DL is deprecated, please use Fiddle
rajendra # http://code-programmersplace.rhcloud.com/
(uuid: 54d19a5be0b8cd9bf9000082)
-------------------------------------------------
Domain: programmersplace
Created: Feb 04 9:34 AM
Gears: 1 (defaults to small)
Git URL: ssh://54d19a5be0b8cd9bf9000082#code-programmersplace.rhcloud.com/~/gi
t/rajendra.git/
SSH: 54d19a5be0b8cd9bf9000082#code-programmersplace.rhcloud.com
Deployment: auto (on git push)
jbossews-2.0 (Tomcat 7 (JBoss EWS 2.0))
---------------------------------------
Gears: 1 small
You have access to 1 application.
C:\Users\rajendra>

The first thing you need is to connect via SSH to your application on OpenShift. If the name of your app is awesome, run the following command:
rhc ssh -a awesome
If you've forgotten the name of your application, execute rhc apps in order to see your current apps. See the lines with something similar to hereisthename # http://... or .../~/git/hereisthename.git/.
Once you're connected via SSH, you can see the log using the tail command:
Tomcat 7 (JBoss EWS 2.0)
tail -f -n 100 app-root/logs/jbossews.log
JBoss Application Server 7
tail -f -n 100 app-root/logs/jbossas.log
The OpenShift Client Tools are required. See Installing the OpenShift Client Tools. See also Getting Started with OpenShift Online.
RELATED: rhc ssh [No system SSH available] error

"Uncaught Could not connect" error when using Openshift to deploy .war for PHP/Java bridge

Problem: Recieve 'Could not connect' error when using Openshift to deploy .war for PHP/Java bridge
Below I describe every step I've taken to get to this error, thanks in advance.
Full error: (IP concealed)
"Fatal error: Uncaught Could not connect to the JEE server W.X.Y.Z:8080. Please start it. Or define('JAVA_HOSTS',9267); define('JAVA_SERVLET',false); before including 'Java.inc' and try again. Error message: Connection refused (111) thrown in /home/stevenw1/public_html/softwareProject/real/inc/Java.inc on line 989"
Players: "Java_Bridge.war" = my java .war file that php will use to call methods from
"W.X.Y.Z" = Unchanging IP address of deployed .war file
"GUIProfile" = a Java class with #WebServlet("/GUIProfile") as public class GUIProfile extends HttpServlet
"tomcattest.php" = test file that calls GUIProfile's static method 'validate'
Deployed Java_Bridge.war and tomcattest.php work fine on my local tomcat, just not yet on Openshift.
Senario:
0. followed command line steps from Openshift 2012 java tomcat application
1. Using Openshift's rhc I have created my first app using "rhc app create -a tomcat6 -t jbossews-1.0"
2. cd ~/tomcat6/; git rm -rf ./src/ pom.xml
3. cp ~/Java_Bridge.war ./webapps/; git add ./webapps/Java_Bridge.war; git commit -m "first try"; git push
//notable output includes:
remote: Stopping jbossews cartridge
remote: Sending SIGTERM to jboss:386662 ...
remote: Skipping Maven build due to absence of pom.xml
remote: Preparing build for deployment
remote: Deployment id is bd83d6eb
remote: Activating deployment
remote: + '[' Java_Bridge.war ']'
remote: Starting jbossews cartridge
remote: Found W.X.Y.Z:8080 listening port //(not actual IP)
remote: Git Post-Receive Result: success
remote: Activation status: success
remote: Deployment completed with status: success
to ssh://thisisnottheactualuser#tomcat6-notthenamehereeither.rhcloud.com
4. Thinking the .war was deployed I ran a tomcattest.php that calls a java method 'validate', from java class GUIProfile that extends httpservlet, using the php/java bridge Java.inc
//of course using the same IP from above...
cat tomcattest.php;
<?php
define("JAVA_HOSTS", "W.X.Y.Z:8080");
define("JAVA_SERVLET", "/Java_Bridge/GUIProfile");
require_once("Java.inc");
$valiationOutput = java_context()->getServlet()->validate("hello");
echo $valiationOutput;
?>
EDIT: later tried this also in all instances: same results exactly
cat alternatetest.php
<?php
define("JAVA_HOSTS", "W.X.Y.Z:8080");
require_once("Java.inc");
echo java("java.lang.System")->getProperties();
?>
The output of calling tomcattest.php is the 'Fatal error' you see at the top.
Double Check:
5. ssh thisisnottheactualuser#tomcat6-notthenamehereeither.rhcloud.com
6. env | grep "IP"
OPENSHIFT_JBOSSEWS_IP=W.X.Y.Z
7. env | grep "PORT"
OPENSHIFT_JBOSSEWS_HTTP_PORT=8080
OPENSHIFT_JBOSSEWS_JPDA_PORT=8787
8. changed port in tomcattest.php to 8787, didn't change a thing of course.
9. restarted from the top, this time keeping ./src/ & pom.xml, same 'Deployment completed with status: success', same error using tomcattest.php with new listening IP...
10. tried solution from 'openshift youtube vid 2012' seemingly outdated. Followed step by step:
onced ssh'ed, no "tomcat" dir, I used cd $OPENSHIFT_DATA_DIR instead to access data dir
wget tomcat, changed ports to >15000, sh startup.sh && tail -f ../logs/*
notable errors in the result include:
SEVERE: Failed to initialize connector [Connector[AJP/1.3-15009]]
org.apache.catalina.LifecycleException: Failed to initialize component
[Connector[AJP/1.3-15009]] SEVERE: Failed to initialize connector
[Connector[AJP/1.3-15009]] org.apache.catalina.LifecycleException:
Failed to initialize component [Connector[AJP/1.3-15009]] SEVERE:
Failed to initialize end point associated with ProtocolHandler
["ajp-bio-15009"] java.net.BindException: Permission denied
:15009 SEVERE: Failed to initialize end point associated with
ProtocolHandler ["ajp-bio-15009"] java.net.BindException: Permission
denied :15009
11. restarted step 6, this time left IP's as they were, changing only localhost to tomcat-stevenwernercs.rhcloud.com, same result but with original IP's
12. ran out of options online, then I posed this question...
13. then I edited it a lot...
14. patiently wait :)
That is everything I have done, I am not sure why php isnt finding Java.
I looked at previous questions didn't find anything helpful.
Open to anything, thank you

Gears in a non-scable application are not allowed to communicate on any non http/ws port. So if your php java bridge is trying to communicate with the tomcat instance on 15009 (from what i see above) that is not going to work. You could only make requests on 80/443/8000/8443. Can you verify what port the java bridge works over?

Alternate solution: https://bitnami.com/cloud
I got it working using Bitnami with Amazon AWS, it provides free service for 1 year, documentation that is easy to follow, and plentiful and up to date, It took less than an hour.
Disclaimer: Using ssh tunnel for port 8080 i configured tomcat as if on localhost, and moved my hosted .php to the
same microserver that also hosted the tomcat. (I gave up on trying to host php and tomcat separately.)
I have hosted tomcat separately before on my university's tomcat server. But that was a node I had full access to.
I would still like to try Openshifts free solution, although the current resources and methods I've seen seem to no longer be valid. And unfortunately the workarounds are not intuitive enough and are currently over my head. Thank you.

Tomcat is not shutting down

I am unable to shutdown tomcat. When I do ./shutdown.sh, I don't see any error, but the process still seems to be running. When I kill the process (kill -9), the process gets killed. After this, if I do a startup, and shutdown again, I see the same problem. How do I gracefully shut it down? Why is this happening in the first place? Please help me.
devServer:bin dev$ ps -ef|grep tomcat
501 34933 1 0 11:24PM test 0:02.02 /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java -Djava.util.logging.config.file=/Users/dev/admin/runtime/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/Users/dev/admin/runtime/endorsed -classpath /Users/dev/admin/runtime/bin/bootstrap.jar:/Users/dev/admin/runtime/bin/tomcat-juli.jar -Dcatalina.base=/Users/dev/admin/runtime -Dcatalina.home=/Users/dev/admin/runtime -Djava.io.tmpdir=/Users/dev/admin/runtime/temp org.apache.catalina.startup.Bootstrap start
501 34982 33786 0 11:45PM test 0:00.00 grep tomcat
devServer:bin dev$ ./shutdown.sh
Using CATALINA_BASE: /Users/dev/admin/runtime
Using CATALINA_HOME: /Users/dev/admin/runtime
Using CATALINA_TMPDIR: /Users/dev/admin/runtime/temp
Using JRE_HOME: /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
Using CLASSPATH: /Users/dev/admin/runtime/bin/bootstrap.jar:/Users/dev/admin/runtime/bin/tomcat-juli.jar
devServer:bin dev$ ps -ef|grep tomcat
501 34933 1 0 11:24PM test 0:02.03 /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java -Djava.util.logging.config.file=/Users/dev/admin/runtime/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/Users/dev/admin/runtime/endorsed -classpath /Users/dev/admin/runtime/bin/bootstrap.jar:/Users/dev/admin/runtime/bin/tomcat-juli.jar -Dcatalina.base=/Users/dev/admin/runtime -Dcatalina.home=/Users/dev/admin/runtime -Djava.io.tmpdir=/Users/dev/admin/runtime/temp org.apache.catalina.startup.Bootstrap start
501 34992 33786 0 11:45PM test 0:00.00 grep tomcat
devServer:bin dev$
Also, I see my catalina.out is rather too short. There is no port number info getting written to it at all. This is all I see everytime I startup.
Feb 15, 2014 11:24:23 PM org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: .:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java
Here is my log file -
devServer:logs dev$ cat catalina.2014-02-15.log
Feb 15, 2014 11:24:23 PM org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: .:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java
Something seems to be terribly messed up. I can't understand what. Please advise.
I am running tomcat 7.0.50 on Mac OS.

Judging by the logs, it is evident that tomcat is not starting up at all as it should be. I can advise a couple of other tests as well like to
Verify whether or not the ports are getting bound by the tomcat service or not. You can do this by issueing the command netstat -tuplen on the terminal and check to see if port 8080 is getting bound by the tomcat instance or not.
Check to see if you can ping the instance. Try to ping the tomcat instance from a local machine. Then try to ping the public ip of the server to verify whether that server is responding correctly or not
Verify that you have, infact, the complete tomcat template as is used by your other servers. Verify the tomcat installtion folder and compare it with the other tomcat templates as used in rest of your servers. Make sure all folders are there, all configuration (default tomcat config) files are present.
(Just a hunch) Verify read/write/execute privileges as well
Remove your application, shutdown tomcat, empty the logs, and then restart tomcat and re-deploy your application and then view the logs to see if you get some error
Point 3 seems more likely that you have an incomplete tomcat instance, so I advise you to make a copy of the tomcat instance used in any other server and use it in this server to make sure that the server environment is infact how it should be