We Keep Coding

Apache Flink: Standalone Cluster tries to connect with username "flink"

For my master thesis I'm trying to set up a flink standalone cluster on 4 nodes. I've worked along the documentation which pretty neatly explains how to set it up. But when I start the cluster there is a warning and when I'm trying to run a job, there is an error with the same message:
akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka.tcp://flink#MYHOSTNAME:6123/user/jobmanager#-818199108]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.messages.JobManagerMessages$LeaderSessionMessage"
Increasing the timeout didn't work. When I open the taskmanagers in web UI, all of them have the following pattern:
akka.tcp://flink#MYHOSTNAME:33779/user/taskmanager
Does anyone have an idea how to solve this to get the cluster working? Thanks in advance!
One last thing: There isn't a user "flink" on the cluster and won't be created. So any advices without telling me I should create that user would be very appreciated! Thanks!

Not sure if it is still relevant, but the way i did it (using Flink 1.5.3):
I setup a HA standalone cluster with 3 master (JobManager) and 20 slaves (TaskManager) in the following way.
Define your conf/masters file (hostname:8081 per line)
Define your conf/slaves file (each taskmanager hostname per line)
Define in the flink-conf.yaml on each master machine its own jobmanager.rpc.address hostname
Define in the flink-conf.yaml on each slave machine the jobmanager.rpc.address as localhost
Once every is set, execute the bin/start-cluster.sh on any of the master host.
If you need HA then you need to setup a zookeeper quorum and modify the corresponding properties regarding HA (high-availability, high-availability.storageDir, high-availability.zookeeper.quorum)

How to configure Java client connecting to AWS EMR spark cluster

I'm trying to write a simple spark application, and when i run it locally it works with setting the master as
.master("local[2]")
But after configuring spark cluster on AWS (EMR) i can't connet to the master url:
.master("spark://<master url>:7077")
Is this the way to do it? am i missing something here?
The cluster is up and running, and when i tried adding my application as a step jar, so it will run directly in the cluster it worked. But i want to be able to run it from a remote machine.
would appreciate some help here,
Thanks

To run from a remote machine, you will need to open the appropriate ports in the Security Group assigned to your EMR master node. You will need to add at least 7077.
If by "remote" you mean one that isn't in your AWS environment, you will also need to setup a way to route traffic to it from the outside.

How to know status of Kafka broker in java?

i am working on apache storm which has a topolgy main class. This topology contains the kafkaSpout which listen a kafka topic over a kafka broker. Now before i submit this topology i want to make sure the status of the kafka broker which has the topic. But i didnt found any way to do it? How a kafka brokers status can be known from storm tolopogy class ? Please help...

If you simply want a quick way to know if it is running or not you can just run the start command again with the same config:
bin/kafka-server-start.sh config/server.properties
If it's running then you should get an exception about the port already being in use.
Not foolproof, so better would be to use Zookeeper as mentioned above:
Personally I use intellij which has a Zookeeper plugin which helps you browse the brokers/topics running within it. Probably something similar for Eclipse or other IDEs.
(IntelliJ)
Go to File > Settings > type zookeeper in the search, then install and click ok (may need to restart)
Go to File > Settings > type zookeeper in the search. Click enable then put in the address where your zookeeper server is running and apply changes. (Note you may need to check the port is correct too)
You should now see your zookeeper server as a tab on the left side of the IDE.
This should show you your broker and topics, consumers etc
Hope that helps!

If you have configured storm-ui, then that should give you a brief information about the running cluster, including informations such as currently running topologies, available free slots, supervisor info etc.
Programitically you can write a thrift client to retrieve those information from the storm cluster. You can possibly choose almost any language to develope your own client.
Check out this article for further reference

Depending on what kind of status you want to have, for most cases you would actually retrieve this from Zookeeper. In Zookeeper you can see registered brokers, topics and other useful things which might be what you're looking for.
Another solution would be to deploy a small regular consumer which would be able to perform those checks for you.

WebSockets not working on OpenShift Tomcat 7.0 and WildFly Application Server 8.1.0.CR1

I 've spent the whole night trying to run a simple echo test on Tomcat 7.
The test I try to run runs locally as expected. But when deployed to the openshift the connection opens and then immediately closed. I've been looking for the right setup on the internet and found questions and answers for them for exactly the same problem but the solutions don't work for me.
The example of what I have as a test can be checked at http://pureikka-belgampaul.rhcloud.com/pi-8/index.html
At the bottom of the viewarea there's a debug console reporting the statsus of the websocket.
So I feel like I have not made a simple mistake in the setup and it runs locally with only the change of the host name and the port number for the websocket.
Could it be a routing trouble after the upgrade of the connection?
I deploy my war file directly to the webapps directory.
I also think I got websockets working at one moment but i'm not so sure anymore. I could be looking at the localhost after all.
Eventually I decided to try wildfly8 gear because it ships with support for websockets. I checked out the https://github.com/javaee-samples/javaee7-samples repo and deployed the chat demo to the local wildfly8 and the example worked from the first run as it is supposed. Great. But the same war file fails to work correctly on the openshift (http://wildfly8-belgampaul.rhcloud.com/chat3/)
I tried to use ports 80 or 8000 in both setups.
Need someone's help? advice or question to send me back to the right direction
Maybe someone has solved a similar issue?

Web Sockets on OpenShift work over ports 8000 for ws and 8443 for wss, it is not a "work around" it's the way websockets work on this system.

Here are the most useful links providing just enough light about the problem
https://www.openshift.com/blogs/paas-websockets (general info about websockets on openshift)
https://www.openshift.com/blogs/deploy-websocket-web-applications-with-jboss-wildfly (diy example)
to summarize:
websocket support out-of-the-box is the most reliable with node.js.

Could not connect to ActiveMQ service running on AWS - EC2

I am trying to connect to aws-ec2 instance having activemq service running. I am trying to connect it using tcp transport connector using an application built under java + maven + git setting.
I am doing this exercise to add network latency to compare activemq with aws-sqs, which will help us in deciding which one to use for our requirements.
what I have tried so far:
I am able to run this application locally as I scp'ed into ec2 added my code there and it ran locally, which led me to believe that amq service is running.
I also ran the same setting from my home network it is working fine, so home computer -> ec2 connection is also working.
It is just not working from my office network, which led me to believe that it is failing because of some kind of firewall setting?
A simple netstat -a | grep 61616 gave me that it is listening on port 61616.
Does anyone know where I am going wrong? Or suggest some more debugging techniques?
EDIT: Security rule at aws-ec2