I am getting the UnknownHostException while trying to deploy a Spring Boot application to a K8s Cluster in AWS EKS.
Scenario:
I have a AWS EKS Cluster running on a VPC (A) and the RDS is running on a VPC (B).
I have created the VPC peering connection between the two VPCs, enabling DNS resolution and configuring the routing tables accordingly.
I also created a K8s Service mapping the external RDS endpoint.
If I try to connect to the RDS instance from EKS Cluster using a busybox like this:
$ kubectl run -i --tty --rm debug --image=busybox --restart=Never -- sh
with any of the following options, it works:
/ # nc mysql-service 3306
/ # nc mysql-service.default.svc.cluster.local 3306
/ # nc xxxxxxx.xxxxxxx.us-east-1.rds.amazonaws.com 3306
But when I deploy my Spring Boot application it doesn't work.
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:488)
at com.mysql.cj.core.exceptions.ExceptionFactory.createException(ExceptionFactory.java:54)
at com.mysql.cj.core.exceptions.ExceptionFactory.createException(ExceptionFactory.java:93)
at com.mysql.cj.core.exceptions.ExceptionFactory.createException(ExceptionFactory.java:133)
at com.mysql.cj.core.exceptions.ExceptionFactory.createCommunicationsException(ExceptionFactory.java:149)
at com.mysql.cj.mysqla.io.MysqlaSocketConnection.connect(MysqlaSocketConnection.java:83)
at com.mysql.cj.mysqla.MysqlaSession.connect(MysqlaSession.java:144)
at com.mysql.cj.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:1676)
... 58 common frames omitted
Caused by: java.net.UnknownHostException: mysql-service
Any suggestions?
Thanks!
I figured it out! The whole VPC Peering configuration is working. The K8s external service is also working. The problem was in my Spring Boot application.
I had configured the data source URL with an environment variable which was causing the error. For some reason, Spring doesn't resolve the host configured with an environment variable.
I just changed the spring configuration from:
spring:
datasource:
url: jdbc:mysql://${DB_HOST}:3306/db_name
to:
spring:
datasource:
url: jdbc:mysql://mysql-service:3306/db_name
and now it works!
Related
I'm attempting to create a Docker container using quay.io/keycloak/keycloak:20.0.1 image the that will connect to a Postgres Database listening on IPv6. Working with a downloaded zip of Keycloak, the application is able to make a connection if I export JAVA_OPTS as follows:
$ export JAVA_OPTS=-Djava.net.preferIPv6Addresses=true
If I attempt to do the same within Docker, however, I receive java.net.UnknownHostException. I've tried both the hostname which should resolve, as well as the IP that is behind the hostname.
If I try to use the IP within the Docker container, I receive:
Caused by: java.net.SocketException: Protocol family unavailable
I'm not sure how to proceed from here, thanks!
UPDATE: This seems to extend beyond just Keycloak, I’ve attempted connecting to the same DB via container using both Spring Boot and Quarkus and receive the same error.
I am newbie to Kubernetes and had a long time configuring my application to be hosted on Kubernetes cluster hosted on AWS EKS.
Status-quo: I am pretty sure that the service of type LoadBalancer is up and running. It has its pod and all the stuff running. The application is simple Java application with input. You can try accessing it by pulling an image from Docker Hub via:
docker run -i ardulat/mckinsey
Question: how can I run the Java application (not Spring, not REST) that is being hosted on Kubernetes cluster?
Already tried:
curl -v <EXTERNAL-IP>:<PORT> that outputs:
* Trying 3.134.148.191...
* TCP_NODELAY set
* Connected to a8154210d09da11ea9c3806983848f2f-1085657314.us-east-2.elb.amazonaws.com (3.134.148.191) port 8080 (#0)
> GET / HTTP/1.1
> Host: a8154210d09da11ea9c3806983848f2f-1085657314.us-east-2.elb.amazonaws.com:8080
> User-Agent: curl/7.63.0
> Accept: */*
>
* Empty reply from server
* Connection #0 to host a8154210d09da11ea9c3806983848f2f-1085657314.us-east-2.elb.amazonaws.com left intact
curl: (52) Empty reply from server
nc -v <EXTERNAL-IP> <PORT> that outputs:
found 0 associations
found 1 connections:
1: flags=82<CONNECTED,PREFERRED>
outif en0
src 172.20.22.42 port 63865
dst 3.13.128.24 port 8080
rank info not available
TCP aux info available
Connection to a8154210d09da11ea9c3806983848f2f-1085657314.us-east-2.elb.amazonaws.com port 8080 [tcp/http-alt] succeeded!
Therefore, I assume that connection works and the service is up and running except I am trying to connect to the Java (.jar) application in the wrong way. Do you have any suggestions?
You should change your dockerfile and change CMD to ENTRYPOINT which is nicely explained here.
I would also recommend reading Define a Command and Arguments for a Container.
CMD sets default command and/or parameters, which can be overwritten from command line when docker container runs.
ENTRYPOINT configures a container that will run as an executable.
Your dockerfile might look like this:
FROM java:8
WORKDIR /
ADD Anuar.jar Anuar.jar
EXPOSE 8080
ENTRYPOINT ["java","-jar","Anuar.jar"]
Your service might look like this:
apiVersion: v1
kind: Service
metadata:
name: javaservice
labels:
app: javaservice
spec:
type: LoadBalancer
selector:
app: javaservice
ports:
- protocol: TCP
port: 8080
name: http
Also it's important which LoadBalancer you want to use as on AWS there is Classic Load Balancer which is default and Network Load Balancer. You can read more about it on Internal load balancer and check the AWS documentation for Load Balancing.
Amazon EKS supports the Network Load Balancer and the Classic Load Balancer through the Kubernetes service of type LoadBalancer. The configuration of your load balancer is controlled by annotations that are added to the manifest for your service.
By default, Classic Load Balancers are used for LoadBalancer type services. To use the Network Load Balancer instead, apply the following annotation to your service:
service.beta.kubernetes.io/aws-load-balancer-type: nlb
For more information about using Network Load Balancer with Kubernetes, see Network Load Balancer support on AWS in the Kubernetes documentation.
By default, services of type LoadBalancer create public-facing load balancers. To use an internal load balancer, apply the following annotation to your service:
service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
For internal load balancers, your Amazon EKS cluster must be configured to use at least one private subnet in your VPC. Kubernetes examines the route table for your subnets to identify whether they are public or private. Public subnets have a route directly to the internet using an internet gateway, but private subnets do not.
I deployed a Spring Boot app on AWS Elastic Beanstalk. I am facing a 502 Bad Gateway error. I cannot find anything useful from the logs
/var/log/nginx/error.log
2019/02/10 02:12:54 [error] 3257#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: ...., server: , request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:5000/", host: "...."
I do not have any html files for front-end. I just want to upload deploy the api to share the documentation from swagger ui.
It's because server is listening to 5000, Adding "server.port=5000" to application.properties fixed the issue.
This happens because the application load balancer by default points to the Port 80 of the nginx server in EC2 instance. The nginx is configured to forward requests to Port 5000 by default, whereas out application server runs on Port 8080.
Default Nginx Configuration
Expected Nginx Configuration
This can be fixed using an environment property named PORT and pointing it to 8080
Go to configuration > Environment Properties and add the property
Refer AWS Document: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/java-se-nginx.html
Another option to fix this is to point application load balancer to the application server port(8080) directly instead of the nginx(80.
You can configure 8080 as the process port.
Another way to fix this would be to set port to 5000 in the spring boot application by using server.port property.
My issue was my Java version didn't match the platform I'm running with Elastic Beanstalk, even tho my server.port was on 5000. My Java version was 11, and my platform was only Java 8 for Amazon Linux. So changing it to 8 in my base pom.xml fixed it.
I am getting this issue when I deploy redis server and spring server in two different servers and try to connect to spring server through my application.
App Details:
We have mobile application which publish gps coordinates to spring server using stomp. In the spring server we create jedis pubsub connection and publish those gps data to our web application and web users subscribe to those jedis pubsub connections.
Library versions:
stomp:1.7.1
jedis: 2.8.1
spring: 4.3.0
Working scenarios:
*Deploy spring server in my local machine and redis server in remote production server.
*Deploy spring server in remote server and redis server in same remote server where spring server is deployed.
Partially Working scenarios:
*Deploy spring server in remote server and redis server in different remote server where spring server is deployed. In this scenario I monitor redis server using redis cli and I can see the "HGETALL", "PUBLISH" key words with its data. But the same time I got following error in spring server:
Caused by:
java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at redis.clients.jedis.Connection.connect(Connection.java:158)
... 4 more
redis.clients.jedis.exceptions.JedisConnectionException: java.net.ConnectException: Connection refused (Connection refused)
at redis.clients.jedis.Connection.connect(Connection.java:164)
at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:80)
at redis.clients.jedis.Connection.setTimeoutInfinite(Connection.java:68)
at redis.clients.jedis.Jedis.subscribe(Jedis.java:2626)
at shipxpress.pubsubservice.controllers.SubscriberThread.run(MainController.java:227)
The spring server can successfully depoyed to the server and exception occurs when jedis try to publish or subscripe to the connection.
I can succesfully ping to redis server from the server where we deploy the spring server through the redis-cli and the redis server prtected-mode is no and and bind ip is set to 0.0.0.0
Links:
I went through following links but no luck
Redis bind to more than one IP
https://github.com/xetorthio/jedis/issues/1405
Cannot connect to redis using jedis
This issue means that the Spring remote server is not able to communicate with the other remote server(Redis server) on the default Redis port.
Maybe you could confirm this by trying to access the Remote Redis server from the other remote Server using netstat from the console.
I had the same issue, and the simplest solution was
Find the following line in your redis.conf file and comment it out:
bind 127.0.0.1
By adding a # in front of it:
# bind 127.0.0.1
and change the line protected-mode yes to be protected-mode no
save your redis.conf and restart redis using the config file
redis-server /configFileLocation
How to make Spring Cloud Consul register with external IP?
I run spring boot applications with spring cloud consul in docker.
The application is running inside a docker network on a docker swarm.
This means that if spring consul registers with the hostname or the IP address it will get an hostname or ip address that is internal to the docker swarm. My consul server is outside the docker swarm.
You can set spring.cloud.consul.discovery.ipAddress externally and set the property spring.cloud.consul.discovery.preferIpAddress=true.
Alternatively, you can control which network interfaces to derive an ip address from.
In Brixton you can ignore interfaces:
spring:
cloud:
inetutils:
ignoredInterfaces:
- docker0
- veth.*
In Camden (currently RC1) you can also whitelist networks:
spring:
cloud:
inetutils:
preferredNetworks:
- 192.168
- 10.0
Run the container with -p 12345:12345 to publish the port so that you can access the container from your host machine through the specified port