we have kafka 0.8.2.1 running on a single docker container. This is one of several containers set up to work together with docker compose. The docker compose also exposes the kafka 9092 port to the host machine.
The advertised.host.name of the kafka server is set to kafka and all the other containers can talk to it fine using this name.
The problem is that java test programs cannot send messages to the kafka server from the host machine.
Using kafka.javaapi.producer.Producer errors
Using org.apache.kafka.clients.producer.KafkaProducer hangs for ages then errors
In both cases if I add an entry to the /etc/hosts file on the host machine it works fine. But this is not ideal, I would like anyone to be able to run these tests without messing with their hosts file.
So is there a way in a java Kafka producer to override the hostname/ip specified in the metadata. We only have one instance of kafka so there is no issue in getting the "right one".
On the FAQ page it implies this can be done:
In another rare case where the binding host/port is different from the
host/port for client connection, you can set advertised.host.name and
advertised.port for client connection
But there are no details how...
Or failing that a more general solution.
Is there a way to set a hosts entry in the java runtime environment? Without messing with system /etc/hosts files?
Thanks
It is possible to set up a dockerised broker so that it can talk to consumers and producers both in containers and on the host machine.
Set advertised.host.name in the dockerised broker to its docker-compose network IP address, and bootstrap.servers for any non-dockerised producers or consumers to localhost.
To get the compose network IP you can either specify a static IP in the compose file or set your container to run a shell command before starting kafka to get it. The command I use is ip -4 addr show scope global dev $(route -n | grep 'UG[ \t]' | awk '{print $8}' | head -1) | grep inet | awk '{print $2}' | cut -d / -f 1, but there are probably nicer ways.
Related
I am using Apache ActiveMQ Artemis that is embedded within JBoss EAP 7.2.0 server. I have created a queue and and external application (external to the JBoss server) is listening to that queue. In Runtime tab in JBoss Admin Console, I can see that my queue has 1 listener, but after a few days I see that the number of listeners increases to 2 or 3.
Is there any way by which I can get details of the processes that are listening to the queue?
In your specific use case, with only remote consumers on the same host of your EAP server, you can get all connected processes using the following command (note that this doesn't cover the case where you also have internal consumers with the same process id of your server and/or remote consumers on external hosts).
$ lsof -Pni | grep ">127.0.0.1:8080"
java 25322 fvaleri 230u IPv4 0x97a3c5aa109b7ebd 0t0 TCP 127.0.0.1:55058->127.0.0.1:8080 (ESTABLISHED)
Then, each of this connections may have a number of sessions opened, that you can find out with the following command (replace TestQueue with your queue's name).
$ $EAP_HOME/bin/jboss-cli.sh -c --command="/subsystem=messaging-activemq/server=default/jms-queue=TestQueue:list-consumers-as-json"
I am writing end-to-end tests for an application. I start an instance of an application, a Kafka instance, and a Zookeeper (all Dockerized) and then I interact with the application API to test its functionality. I need to test an event consumer's functionality in this application. I publish events from my tests and the application is expected to handle them.
Problem: If I run the application locally (not in Docker) and run tests that would produce events, the consumer in the application code handles events correctly. In this case, the consumer and the test have bootstrapServers set to localhost:9092. But if the application is run as a Dockerized instance it doesn't see the events. In this case bootstrapServers are set to kafka:9092 in the application and localhost:9092 in the test where kafka is a Docker container name. The kafka container exposes its 9092 port to the host so that the same instance of Kafka can be accessed from inside a Docker container and from the host (running my tests).
The only difference in the code is localhost vs kafka set as bootstrap servers. In both scenarios consumers and producers start successfully; events are published without errors. It is just that in one case the consumer doesn't receive events.
Question: How to make Dockerized consumers see events posted from the host machine?
Note: I have a properly configured Docker network which includes the application instance, Zookeeper, and Kafka. They all "see" each other. The corresponding ports of kafka and zookeeper are exposed to the host.
Kafka ports: 0.0.0.0:9092->9092/tcp. Zookeeper ports: 22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp.
I am using wurstmeister/kafka and wurstmeister/zookeeper Docker images (I cannot replace them).
Any ideas/thoughts are appreciated. How would you debug it?
UPDATE: The issue was with KAFKA_ADVERTISED_LISTENERS and KAFKA_LISTENERS env variables that were set to different ports for INSIDE and OUTSIDE communications. The solution was to use a correct port in the application code when it is run inside a Docker container.
Thes kind of issues are usually related to the way Kafka handles the broker's address.
When you start a Kafka broker it binds itself on 0.0.0.0:9092 and register itself on Zookeeper with the address <hostname>:9092. When you connect with a client, Zookeeper will be contacted to fetch the address of the specific broker.
This means that when you start a Kafka container you have a situation like the following:
container name: kafka
network name: kafkanet
hostname: kafka
registration on zookeeper: kafka:9092
Now if you connect a client to your Kafka from a container inside the kafkanet network, the address you get back from Zookeeper is kafka:9092 which is resolvable through the kafkanet network.
However if you connect to Kafka from outside docker (i.e. using the localhost:9092 endpoint mapped by docker), you still get back the kafka:9092 address which is not resolvable.
In order to address this issue you can specify the advertised.host.name and advertised.port in the broker configuration in such a way that the address is resolvable by all the client (see documentation).
What is usually done is to set advertised.host.name as <container-name>.<network> (in your case something like kafka.kafkanet) so that any container connected to the network is able to correctly resolve the IP of the Kafka broker.
In your case however you have a mixed network configuration, as some components live inside docker (hence able to resolve the kafkanet network) while others live outside it. If it were a production system my suggestion would be to set the advertised.host.name to the DNS/IP of the host machine and always rely on docker port mapping to reach the Kafka broker.
From my understanding however you only need this setup to test things out, so the easiest thing would be to "trick" the system living outside docker. Using the naming specified above, this means simply to add to your /etc/hosts (or windows equivalent) the line 127.0.0.1 kafka.kafkanet.
This way when your client living outside docker connects to Kafka the following should happen:
client -> Kafka via localhost:9092
kafka queries Zookeeper and return the host kafka.kafkanet
client resolves kafka.kafkanet to 127.0.0.1
client -> Kafka via 127.0.0.1:9092
EDIT
As pointed out in a comment, newer Kafka version now use the concept of listeners and advertised.listeners which are used in place of host.name and advertised.host.name (which are deprecated and only used in case the the above ones are not specified). The general idea is the same however:
host.name: specifies the host to which the Kafka broker should bind itself to (works in conjunction with port
listeners: specifies all the endpoints to which the Kafka broker should bind (for instance PLAINTEXT://0.0.0.0:9092,SSL://0.0.0.0:9091)
advertised.host.name: specifies how the broker is advertised to client (i.e. which address client should use to connect to it)
avertised.listeners: specifies all the advertised endpoints (for instance PLAINTEXT://kafka.example.com:9092,SSL://kafka.example.com:9091)
In both cases for client to be able to successfully communicate with Kafka they need to be able to resolve and connect to the advertised hostname and port.
In both cases if not specified they are automatically derived by the broker using the hostname of the machine the broker is running on.
You kept referencing 8092. Was that intentional? Kafka runs on 9092. Easiest test is to download the same version of Kafka and manually run its kafka-console-consumer and kafka-console-producer scripts to see if you can pub-sub from your host machine.
did you try "host.docker.internal" in dockerized application?
You could create a docker network for your containers and then containers will be able to resolve each other hostnames and communicate.
Note: this is usable with docker-compose as well with standalone containers
I am trying to get some CPU sampling working on a remote Java-in-Docker process.
I've already looked at the related questions here, and tried everything, to no avail, so I'm posting my setup here.
I have a Java process (openjdk-8) running in a Docker container on a Google Compute Engine (GCE) instance. The GCE instance and container are both running Debian-9. I want to attach VisualVM or JConsole to my Java process.
I am able to run my docker container locally and connect with both visualvm and jconsole using localhost:9010.
I start the container in the VM startup script with:
docker run -d -p 9010:9010 <my container>
The Dockerfile also has:
EXPOSE 9010
The Java process, started by the Dockerfile CMD, has the following relevant args:
"-Dcom.sun.management.jmxremote", \
"-Dcom.sun.management.jmxremote.port=9010", \
"-Dcom.sun.management.jmxremote.rmi.port=9010", \
"-Dcom.sun.management.jmxremote.local.only=false", \
"-Dcom.sun.management.jmxremote.authenticate=false", \
"-Dcom.sun.management.jmxremote.ssl=false", \
I have opened up port 9010 in my gcloud firewall using:
gcloud compute firewall-rules create jmx-port --allow=tcp:9010,udp:9010
I have verified with netcat that the port is open and I can make a TCP connection to it.
I have other ports open from the same Docker container, with clients connecting successfully to those ports. They were exposed and mapped to the host ports the same way (-p port:port) and opened in the firewall the same way.
I am passing the external IP address of the GCE instance. For instance, if I do:
gcloud compute instances list
and it tells me:
NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS
my-server-b23j us-central1-d n1-standard-1 10.240.0.2 108.357.213.99 RUNNING
Then I will use the argument:
108.357.213.99:9010
as the remote jmx connection host:port pair.
VisualVM and JConsole both tell me they can't connect to the remote JMX service. In both cases, I decline the secure connection, and then they say:
Cannot connect to 108.357.213.99:9010 using
service:jmx:rmi:////jndi/rmi://108.357.213.99:9010/jmxrmi
In desperation, I added a firewall rule that enables TCP/UDP connections on all ports 0-65535, but it did not make a difference -- they still could not connect.
I've read that JMX-RMI opens up anonymous ports, and that you can (at least partly?) disable this behavior by specifying both:
"-Dcom.sun.management.jmxremote.port=9010", \
"-Dcom.sun.management.jmxremote.rmi.port=9010", \
However, it doesn't do the trick in my case.
I've read here that you need to specify the rmi server hostname:
-Djava.rmi.server.hostname='192.168.99.100'
but my server IP is ephemeral -- it is assigned by Google Compute Engine when I create the instance, and so I can't hardwire it into the Dockerfile with the rest of the Java args.
Am I going to have to get a static IP address to make this work?
One possible solution would be to ssh into your GCE box and port-forward port 9010. This can be done from the local console with:
gcloud compute ssh name-of-your-gce-engine -- -L 9010:localhost:9010
Then in jconsole or jvisualvm you connect to localhost:9010. Using localhost here means that jconsole/jvisualvm will connect to your local machine, this connect is tunneled by ssh into your GCE engine and there to the host and port defined in the -L argument, which is localhost:9010, but from the GCE-engine's view. Meaning that you will end up at your application.
You still have to set the rmi server name before starting your program, but you must use
-Djava.rmi.server.hostname='localhost'
so that RMI will tell jconsole/jvisualvm to use localhost and this will then resolve to your local tunneled endpoint. And of course you still need these:
"-Dcom.sun.management.jmxremote", \
"-Dcom.sun.management.jmxremote.port=9010", \
"-Dcom.sun.management.jmxremote.rmi.port=9010", \
"-Dcom.sun.management.jmxremote.local.only=false", \
"-Dcom.sun.management.jmxremote.authenticate=false", \
"-Dcom.sun.management.jmxremote.ssl=false", \
I've a rest service running in docker container on port 5000 which is used to produce message over kafka topic running out of docker container.
I've configure my producer client with below properties :-
bootstrap.servers=localhost:9093
And I've started my contained with below command:-
docker run -d -p 127.0.0.1:5000:5000 <contained id>
I've also made below configuration to advertise kafka host and port
advertised.host.name=localhost
advertised.port=9093
Despite of having all configuration when I try to produce to a kafka topic then I get below error:-
org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.
Can someone please point where the actual problem is?
In real life, advertised.host.name should never be localhost.
In your case, you run your Docker container in bridge networking mode, so it won't be able to reach the broker via localhost, as it will point to the container network, not the host machine.
To make it work you should set the advertised.host.name and bootstrap.servers to the IP address returned by ifconfig docker0 (might be not docker0 in your case but you get the point).
Alternatively, you may probably run your container with --net=host, but I think you'd better properly configure the advertised host name.
I developing under Spring3.1 standalone env.
I am trying to connect my application remotely via jconsole.
It's working locally but when I deploy my application into the linux machine it gets time out.
I am using Daemon in order to run my environment.
this is what I add in the run.sh script:
-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.port=6969 \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.authenticate=false \
com.mypackage.daemon.FixDaemon
and inside applicationContext.xml:
<context:mbean-server />
<context:mbean-export />
now on the linux machine after doing netstat thats what we see:
[root# logs]# netstat -an | grep 6969
tcp 0 0 :::6969 :::* LISTEN
so it's seems like it does listening.
but when I add my ip:6969 inside the jconsole interface I get connection failed popup.
any idea what am I doing wrong?
thanks,
ray.
First try to add also this option to your application:
-Djava.rmi.server.hostname=<ip>
Also keep in mind jconsole is using RMI for the communication. This means jconsole first connects to ip:6969. Then server generates a random port X which is passed back to the jconsole. Jconsole then opens another connection to ip:X. Since X is random, there is no way you can open this specific port in the firewall. You have either to open all ports or use a socks proxy which is another subject.
Try connecting to that port using telnet from your machine. If this does not succeed it usually is because of a firewall dropping packets. You will have to talk to your network administrator to open up that port.
Note: You will have to open up two ports. One for binding the RMIRegistry and another one to export the RMI objects. RMI usually exports objects on random high ports. But this will not work in a firewall-ed environment hence you would have to configure the port on which it is exported. This is done by using a RMI URL.
If you are running this on Linux then do a hostname -i, if it returns 127.0.0.1 then fix /etc/hosts. The FAQ entry for JConsole has more information on this.
Another option I would strongly suggest is to look at Jolokia which does not involve changing the firewall configuration but still provides the JMX goodies over HTTP.