spring-boot Kafka integration in docker using docker-compose - java

I am trying to connect Kafka in spring boot inside Docker but it shows an error "could not be established. Broker may not be available."
Kafka and zookeeper server running successfully but I am unable to connect broker inside spring-boot.
here is my docker-compose.yml
version: "3"
services:
zookeeper:
image: wurstmeister/zookeeper
container_name: zookeeper
ports:
- 2181:2181
kafka:
image: wurstmeister/kafka
container_name: kafka
ports:
- 9092:9092
environment:
KAFKA_ADVERTISED_HOST_NAME: kafka
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
volumes:
- /var/run/docker.sock:/var/run/docker.sock
depends_on:
- zookeeper
spring_boot:
build: .
depends_on:
- kafka
application.yml from Kafka producer
kafka:
boot:
server: kafka:9092
topic:
name: myTopic-kafkasender
server:
port: 8081
Please suggest what I am doing wrong here

Related

Docker-compose: There is not are not services registered in Eureka Server

I am trying to run all microservices from a docker-compose file and I am facing issues when running the containers due to the issue that have between the Eureka discovery server and api-gateway with the other services. There is a way that can make the discover-server (Eureka) communicate with the other services? Many thanks in advance.
version: "3"
services:
discovery-server:
image: renosbardis/discovery-service:latest
container_name: discovery-server
ports:
- "8761:8761"
environment:
- SPRING_PROFILES_ACTIVE=docker
networks:
- test-network
api-gateway:
image: renosbardis/api-gateway:latest
container_name: api-gateway
ports:
- "8888:8888"
environment:
- SPRING_PROFILES_ACTIVE=docker
- LOGGING_LEVEL_ORG_SPRINGFRAMEWORK_SECURITY= TRACE
depends_on:
- discovery-server
networks:
- test-network
accounts-service:
image: renosbardis/accounts-service:latest
container_name: accounts-service
ports:
- "8081:8081"
environment:
- SPRING_PROFILES_ACTIVE=docker
depends_on:
- discovery-server
- api-gateway
networks:
- test-network
rabbitmq:
image: rabbitmq:3-management-alpine
container_name: rabbitmq
ports:
- "5672:5672"
- "15672:15672"
environment:
AMQP_URL: 'amqp://rabbitmq?connection_attempts=5&retry_delay=5'
RABBITMQ_DEFAULT_USER: "guest"
RABBITMQ_DEFAULT_PASS: "guest"
depends_on:
- discovery-server
- api-gateway
networks:
- test-network
customers-service:
image: renosbardis/customer-service:latest
container_name: customers-service
ports:
- "8083:8083"
environment:
- SPRING_PROFILES_ACTIVE=docker
depends_on:
- discovery-server
- api-gateway
networks:
- test-network
transactions-service:
image: renosbardis/transaction-service:latest
container_name: transactions-service
ports:
- "8084:8084"
environment:
- SPRING_PROFILES_ACTIVE=docker
depends_on:
- discovery-server
- api-gateway
networks:
- test-network
notification-service:
image: renosbardis/notification-service:latest
container_name: notification-service
ports:
- "8085:8085"
environment:
- SPRING_PROFILES_ACTIVE=docker
depends_on:
- discovery-server
- api-gateway
networks:
- test-network
networks:
test-network:
driver: bridge
There is nothing wrong with the docker-compose configuration file, the first thing you need to check is that the api-gateway can access the discovery-server properly
You can go into the containers and ping or telnet to test whether the network between the containers is connected.
use docker exec -it name /bin/bash
The startup log shows that your api-gateway configuration is incorrect, as it should be
discover-server:8761//eureka no localhost
api-gateway | 2022-12-28 02:25:27.549 INFO 1 --- [nfoReplicator-0] c.n.d.s.t.d.RedirectingEurekaHttpClient : Request execution error. endpoint=DefaultEndpoint{ serviceUrl='http://localhost:8761/eureka/}, exception=I/O error on POST request for "http://localhost:8761/eureka/apps/API-GATEWAY": Connect to localhost:8761 [localhost/127.0.0.1] failed: Connection refused (Connection refused); nested exception is org.apache.http.conn.HttpHostConnectException: Connect to localhost:8761 [localhost/127.0.0.1] failed: Connection refused (Connection refused) stacktrace=org.springframework.web.client.ResourceAccessException: I/O error on POST request for "http://localhost:8761/eureka/apps/API-GATEWAY": Connect to localhost:8761 [localhost/127.0.0.1] failed: Connection refused (Connection refused); nested exception is org.apache.http.conn.HttpHostConnectException: Connect to localhost:8761 [localhost/127.0.0.1] failed: Connection refused (Connection refused)
edit 1
I pulled your api-gateway image and started
And executed
docker cp api-gateway:/app/resources/application.properties ./
Here is the configuration that went wrong
eureka.client.serviceUrl.defaultZone=http://localhost:8761/eureka
You just have to change it to
eureka.client.serviceUrl.defaultZone=http://discovery-server:8761/eureka
edit 2
Add the following configuration to api-gatway configuration file to see if it works
eureka.client.fetch-registry=true
eureka.client.registry-with-eureka=true
edit 3
I find that most of your problems are configuration file errors in the mirror
This section uses customer-service mirroring as an example
Its port is set to 0
hostname is localhost
defaultZone is localhost:8761/eureka
This is the configuration file of customer-service that I modified
# Server port
server.port = 8083
eureka.instance.hostname = customer-service
spring.application.name = customer-service
# Memory Database for development Environment
spring.h2.console.enabled=true
spring.h2.console.path=/h2-console
spring.datasource.url=jdbc:h2:mem:customerdb;DB_CLOSE_ON_EXIT=FALSE
spring.datasource.driverClassName=org.h2.Driver
spring.datasource.username=sa
spring.datasource.password=
spring.jpa.hibernate.hibernate.dialect=org.hibernate.dialect.H2Dialect
eureka.client.service-url.defaultZone=http://discovery-server:8761/eureka
eureka.client.fetch-registry= true
eureka.client.register-with-eureka= true
This is my visit 0.0.0.0:8888/api/v1/customer/2 the returned json all working properly
{
"customerID": 2,
"name": "John",
"surname": "Doe",
"balance": 50
}

Connecting spring boot docker container to MQTT docker container

I'm trying to connect my containerized spring boot application with another containerized MQTT broker. Both of them are on their own projects as follows:
mqtt docker-compose.yml:
version: '3.9'
services:
mqttbroker:
container_name: mqttbroker
restart: always
volumes:
- ./config:/mosquitto/config
- ./data:/mosquitto/data
- ./log:/mosquitto/log
ports:
- 8883:8883
networks:
- mynetwork
volumes:
config:
data:
log:
mqtt Dockerfile
FROM eclipse-mosquitto
WORKDIR /mosquitto
COPY . .
EXPOSE 8883
And then the spring boot project is like:
spring boot docker-compose.yml
version: '3.8'
services:
myapp:
build: .
container_name: myapp
ports:
- '8082:8082'
stdin_open: true
tty: true
networks:
- mynetwork
In my application.properties I try to connect to the MQTT broker like:
mosquitto.url=tcp://mqttbroker:8883 and I get connection refused. However, if I run the spring boot application locally, I can connect to the docker container with mosquitto.url=tcp://localhost:8883.
I would rather have all the configurations in my docker-compose files to decrease manual codes.
I really appreciate your help in advance!
Yes, You do have to share the network as external.
you cab check the networks you have with this command : docker network ls
You would notice that they each have their own networks.
adding this to spring boot docker-compose file should fix the issue :
networks:
default:
name: mynetwork
external: true
For more information about docker compose networks

Eureka server cannot discover clients when running services on docker containers

I'm trying to develop application which have 3 microservices, postgres database and eureka server for discovering clients. Locally all works perfect. The problem appears when i'm deploying services on docker containers. One container per service, for postgres and for eureka. I'm create a network in docker and join all containers. The question is what should I do to discover my microservices in eureka server? If anyone will want to help I will provide more details to resolve the problem.
That's my docker-compose
version: "3.7"
networks:
default:
name: microservices_nw
driver: bridge
services:
postgresService:
image: postgres:latest
container_name: postgres
expose:
- 5432
volumes:
- ./init.sql:/docker-entrypoint-initdb.d/init.sql
environment:
- POSTGRES_PASSWORD=xxx
- POSTGRES_USER=xxx
- POSTGRES_DB=work
restart: unless-stopped
# EUREKA SERVER#####
eureka-server:
image: eureka-server:latest
container_name: eureka-server
expose:
- 8761
restart: unless-stopped
# APP*****************************************
credit-microservice:
image: credit-microservice:latest
container_name: credit-microservice
expose:
- 8600
environment:
- SPRING_DATASOURCE_URL=jdbc:postgresql://postgresService/work
- SPRING_DATASOURCE_USERNAME=xxx
- SPRING_DATASOURCE_PASSWORD=xxx
restart: unless-stopped
product-micorservice:
image: product-microservice:latest
container_name: product-micorservice
expose:
- 8080
environment:
- SPRING_DATASOURCE_URL=jdbc:postgresql://postgresService/work
- SPRING_DATASOURCE_USERNAME=xxx
- SPRING_DATASOURCE_PASSWORD=xxx
restart: unless-stopped
customer-microservice:
image: customer-microservice:latest
container_name: customer-microservice
expose:
- 8081
environment:
- SPRING_DATASOURCE_URL=jdbc:postgresql://postgresService/work
- SPRING_DATASOURCE_USERNAME=xxx
- SPRING_DATASOURCE_PASSWORD=xxx
- EUREKA_INSTANCE_PREFER_IP_ADDRESS=true
- EUREKA_CLIENT_SERVICEURL_DEFAULTZONE=http://eureka-server/eureka
depends_on:
- eureka-server
restart: unless-stopped
Dockerfile for customer-microservice
FROM openjdk:8
ADD target/customer-service.jar customer-service.jar
EXPOSE 8081
CMD java -jar customer-service.jar
Dockerfile for product-microservice
FROM openjdk:8
ADD target/product-service.jar product-service.jar
EXPOSE 8080
CMD java -jar product-service.jar
Dockerfile for eureka-server
FROM openjdk:8
ADD target/eureka-server.jar eureka-server.jar
EXPOSE 8761
ENTRYPOINT ["java", "-jar", "eureka-server.jar"]
Dockerfile for credit-microservice
FROM openjdk:8-jre
ADD target/credit-service.jar credit-service.jar
EXPOSE 8600
ENTRYPOINT ["java","-jar", "credit-service.jar"]
And application.yml for customer-service
server:
port: 8081
eureka:
client:
service-url:
default-zone: ${EUREKA_SERVER:http://localhost:8761/eureka}
## PostgreSQL
spring:
application:
name: customer-service
datasource:
driver-class-name: org.postgresql.Driver
url: jdbc:postgresql://localhost:5432/work
username: xxx
password: xxx
jpa:
hibernate.ddl-auto: validate
properties:
hibernate.jdbc.lob.non_contextual_creation: true

Zookeeper connection failing cp-rest-proxy with spotify kafka image

I have been using the kafka image provided by spotify to run kafka locally. I'm currently trying to use it with cp-kafka-rest and schema-registry images.
I need help resolving this issue:
ERROR (Log Group: kafka_rest_1_609fd108dcf4)
[main-SendThread(zookeeper:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server zookeeper:2181, unexpected error, closing socket connection and attempting reconnect
java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:101)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622)
at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:277)
at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:287)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1021)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1064)
Docker Compose
version: '3.5'
services:
kafka:
image: 'spotify/kafka'
hostname: kafka
environment:
- ADVERTISED_HOST=kafka
- ADVERTISED_PORT=9092
ports:
- "9092:9092"
- "2181:2181"
volumes:
- /var/run/docker.sock:/var/run/docker.sock
networks:
- one
kafka_rest:
image: 'confluentinc/cp-kafka-rest:5.1.0'
hostname: kafka_rest
environment:
- KAFKA_REST_ZOOKEEPER_CONNECT=zookeeper:2181
- KAFKA_REST_LISTENERS=http://0.0.0.0:8082
- KAFKA_REST_SCHEMA_REGISTRY_URL=http:schema-registry:8081
- KAFKA_REST_HOST_NAME=kafka-rest
networks:
- one
schema_registry:
hostname: schema-registry
image: 'confluentinc/cp-schema-registry:5.1.0'
environment:
- SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL=zookeeper:2181
- SCHEMA_REGISTRY_HOST_NAME=schema-registry
- SCHEMA_REGISTRY_LISTENERS=http://0.0.0.0:8081
networks:
- one
networks:
one:
name: rest_network
You have no zookepeer container - it is actually your "kafka" service image that includes both Zookeeper and Kafka servers, so zookeeper:2181 should rather be kafka:2181
However, I would recommend not using the spotify images, as they are significantly outdated
You can find a fully functional Docker Compose example of the entire Confluent 5.1.0 Platform on Github
Here is the revelvant configuration you are looking for
---
version: '2'
services:
zookeeper:
image: confluentinc/cp-zookeeper:5.1.0
hostname: zookeeper
container_name: zookeeper
ports:
- "2181:2181"
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
broker:
image: confluentinc/cp-enterprise-kafka:5.1.0
hostname: broker
container_name: broker
depends_on:
- zookeeper
ports:
- "9092:9092"
- "29092:29092"
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:9092,PLAINTEXT_HOST://localhost:29092
KAFKA_METRIC_REPORTERS: io.confluent.metrics.reporter.ConfluentMetricsReporter
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: broker:9092
CONFLUENT_METRICS_REPORTER_ZOOKEEPER_CONNECT: zookeeper:2181
CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS: 1
CONFLUENT_METRICS_ENABLE: 'true'
CONFLUENT_SUPPORT_CUSTOMER_ID: 'anonymous'
schema-registry:
image: confluentinc/cp-schema-registry:5.1.0
hostname: schema-registry
container_name: schema-registry
depends_on:
- zookeeper
- broker
ports:
- "8081:8081"
environment:
SCHEMA_REGISTRY_HOST_NAME: schema-registry
SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: 'zookeeper:2181'
rest-proxy:
image: confluentinc/cp-kafka-rest:5.1.0
depends_on:
- zookeeper
- broker
- schema-registry
ports:
- 8082:8082
hostname: rest-proxy
container_name: rest-proxy
environment:
KAFKA_REST_HOST_NAME: rest-proxy
KAFKA_REST_BOOTSTRAP_SERVERS: 'broker:9092'
KAFKA_REST_LISTENERS: "http://0.0.0.0:8082"
KAFKA_REST_SCHEMA_REGISTRY_URL: 'http://schema-registry:8081'
You need to add zookeeper service to your docker compose file
zookeeper:
image: confluent/zookeeper
ports:
- "2181:2181"
environment:
zk_id: "1"
network_mode: "host"

Docker-compose up : Connection refused

I'm using Docker-compose in my project when i'm trying to inside a project to connect to a host in my VBox "localhost:22000" it causes an Exception connection refused
version: '2'
services:
mongodb:
image: mongo
ports:
- "27017:27017"
command: mongod --smallfiles
rabbitmq:
image: rabbitmq:3.5.3-management
ports:
- "5672:5672"
- "15672:15672"
broker:
image: java:openjdk-8u91-jdk
depends_on:
- mongodb
- rabbitmq
working_dir: /app
volumes:
- ./blockchain-rabbitmq/target/:/app
command: java -jar /app/blockchain-rabbitmq-0.0.1.jar
ports:
- "8484:8484"
links:
- rabbitmq
- mongodb
environment:
SPRING_DATA_MONGODB_URI: mongodb://mongodb/ethereum
RABBIT_HOST: rabbitmq
SPRING_RABBITMQ_HOST: rabbitmq
nfb:
image: java:openjdk-8u91-jdk
depends_on:
- mongodb
- broker
working_dir: /app
volumes:
- ./coordinator/target/:/app
command: java -jar /app/coordinator-0.0.1.jar
ports:
- "8383:8383"
links:
- mongodb
environment:
SPRING_DATA_MONGODB_URI: mongodb://mongodb/ethereum
is there a way to expose some hosts or port ( i got this kind of exception before that when i'm dealing with mongo and rabbit mq and i resolved it buy using links and env vars )

Categories