I deployed a Spring Boot app on AWS Elastic Beanstalk. I am facing a 502 Bad Gateway error. I cannot find anything useful from the logs
/var/log/nginx/error.log
2019/02/10 02:12:54 [error] 3257#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: ...., server: , request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:5000/", host: "...."
I do not have any html files for front-end. I just want to upload deploy the api to share the documentation from swagger ui.
It's because server is listening to 5000, Adding "server.port=5000" to application.properties fixed the issue.
This happens because the application load balancer by default points to the Port 80 of the nginx server in EC2 instance. The nginx is configured to forward requests to Port 5000 by default, whereas out application server runs on Port 8080.
Default Nginx Configuration
Expected Nginx Configuration
This can be fixed using an environment property named PORT and pointing it to 8080
Go to configuration > Environment Properties and add the property
Refer AWS Document: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/java-se-nginx.html
Another option to fix this is to point application load balancer to the application server port(8080) directly instead of the nginx(80.
You can configure 8080 as the process port.
Another way to fix this would be to set port to 5000 in the spring boot application by using server.port property.
My issue was my Java version didn't match the platform I'm running with Elastic Beanstalk, even tho my server.port was on 5000. My Java version was 11, and my platform was only Java 8 for Amazon Linux. So changing it to 8 in my base pom.xml fixed it.
Related
I am newbie to Kubernetes and had a long time configuring my application to be hosted on Kubernetes cluster hosted on AWS EKS.
Status-quo: I am pretty sure that the service of type LoadBalancer is up and running. It has its pod and all the stuff running. The application is simple Java application with input. You can try accessing it by pulling an image from Docker Hub via:
docker run -i ardulat/mckinsey
Question: how can I run the Java application (not Spring, not REST) that is being hosted on Kubernetes cluster?
Already tried:
curl -v <EXTERNAL-IP>:<PORT> that outputs:
* Trying 3.134.148.191...
* TCP_NODELAY set
* Connected to a8154210d09da11ea9c3806983848f2f-1085657314.us-east-2.elb.amazonaws.com (3.134.148.191) port 8080 (#0)
> GET / HTTP/1.1
> Host: a8154210d09da11ea9c3806983848f2f-1085657314.us-east-2.elb.amazonaws.com:8080
> User-Agent: curl/7.63.0
> Accept: */*
>
* Empty reply from server
* Connection #0 to host a8154210d09da11ea9c3806983848f2f-1085657314.us-east-2.elb.amazonaws.com left intact
curl: (52) Empty reply from server
nc -v <EXTERNAL-IP> <PORT> that outputs:
found 0 associations
found 1 connections:
1: flags=82<CONNECTED,PREFERRED>
outif en0
src 172.20.22.42 port 63865
dst 3.13.128.24 port 8080
rank info not available
TCP aux info available
Connection to a8154210d09da11ea9c3806983848f2f-1085657314.us-east-2.elb.amazonaws.com port 8080 [tcp/http-alt] succeeded!
Therefore, I assume that connection works and the service is up and running except I am trying to connect to the Java (.jar) application in the wrong way. Do you have any suggestions?
You should change your dockerfile and change CMD to ENTRYPOINT which is nicely explained here.
I would also recommend reading Define a Command and Arguments for a Container.
CMD sets default command and/or parameters, which can be overwritten from command line when docker container runs.
ENTRYPOINT configures a container that will run as an executable.
Your dockerfile might look like this:
FROM java:8
WORKDIR /
ADD Anuar.jar Anuar.jar
EXPOSE 8080
ENTRYPOINT ["java","-jar","Anuar.jar"]
Your service might look like this:
apiVersion: v1
kind: Service
metadata:
name: javaservice
labels:
app: javaservice
spec:
type: LoadBalancer
selector:
app: javaservice
ports:
- protocol: TCP
port: 8080
name: http
Also it's important which LoadBalancer you want to use as on AWS there is Classic Load Balancer which is default and Network Load Balancer. You can read more about it on Internal load balancer and check the AWS documentation for Load Balancing.
Amazon EKS supports the Network Load Balancer and the Classic Load Balancer through the Kubernetes service of type LoadBalancer. The configuration of your load balancer is controlled by annotations that are added to the manifest for your service.
By default, Classic Load Balancers are used for LoadBalancer type services. To use the Network Load Balancer instead, apply the following annotation to your service:
service.beta.kubernetes.io/aws-load-balancer-type: nlb
For more information about using Network Load Balancer with Kubernetes, see Network Load Balancer support on AWS in the Kubernetes documentation.
By default, services of type LoadBalancer create public-facing load balancers. To use an internal load balancer, apply the following annotation to your service:
service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
For internal load balancers, your Amazon EKS cluster must be configured to use at least one private subnet in your VPC. Kubernetes examines the route table for your subnets to identify whether they are public or private. Public subnets have a route directly to the internet using an internet gateway, but private subnets do not.
I am trying to deploy simple Spring Boot app on Jelastic. So I have created environment with Java 8, Postgres 9.5, Apache 2.4, deployed war-file and nothing works.
I receive the following error:
Proxy Error
The proxy server received an invalid response from an
upstream server. The proxy server could not handle the request GET
/localize/. Reason: Error reading from remote server
Apache/2.4.6 (CentOS) Server at node30... Port 80
It might be for the following reason: my server configurated on port 5555. However, I don't see any options that would tell Jelastic about the neccesary port.
I am connected to dormitory subweb. I have 2 machines with external ip adresses. ( I am able to ping them or connect with them via SSH) Raspberry Pi running ubuntu mate where my Spring Boot server is hosted and other machine running Windows 10. Problem is that Spring server do not respond to requests from any source beside dormitory web (DW). I found that Issue is 8080 port closed. when I scan for open ports from beside DW I can only see 22 port opened, Computers inside DW see 8080 and 22 ports as opened and are able to respond HTTP server.
From what I have read here I have to set server.address property to be accessible from "outside"
My spring application.properties
#mysql test config
spring.datasource.url=jdbc:mysql://localhost:3306/springboot
spring.datasource.username=root
spring.datasource.password=root
spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver`
spring.jpa.show-sql=true
spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.MySQLDialect
spring.jpa.hibernate.ddl-auto = update
spring.jpa.hibernate.naming-strategy=org.hibernate.cfg.DefaultNamingStrategy
#initialization
spring.datasource.schema=classpath:/data/schema.sql
#server
server.address= my.adres.here
server.port = 8080
I am sure that sever reads this file properly.
I also tried turn off firewall temporarily
sudo ufw disable
to check if it blocks connection on this port but thats not the issue.
I am not sure where the real problem may be localized. Could it be something wrong with Spring Boot config? Or maybe it is some networking issue.
as #Robert Masen suggested problem was dormitory network blocking connection on 8080 port.
How to make Spring Cloud Consul register with external IP?
I run spring boot applications with spring cloud consul in docker.
The application is running inside a docker network on a docker swarm.
This means that if spring consul registers with the hostname or the IP address it will get an hostname or ip address that is internal to the docker swarm. My consul server is outside the docker swarm.
You can set spring.cloud.consul.discovery.ipAddress externally and set the property spring.cloud.consul.discovery.preferIpAddress=true.
Alternatively, you can control which network interfaces to derive an ip address from.
In Brixton you can ignore interfaces:
spring:
cloud:
inetutils:
ignoredInterfaces:
- docker0
- veth.*
In Camden (currently RC1) you can also whitelist networks:
spring:
cloud:
inetutils:
preferredNetworks:
- 192.168
- 10.0
Run the container with -p 12345:12345 to publish the port so that you can access the container from your host machine through the specified port
I have a Tomcat 7.0 webapp running inside a docker container on AWS Elastic Beanstalk (EB) (I followed the tutorial here).
When I browse to my EB url myapplication.elasticbeanstalk.com, I get a 502 Bad Gateway by served by nginx. So its immediately clear that my port 80 is not forwarding to my container. When I browse to myapplication.elasticbeanstalk.com:8888 (another port I exposed in my Dockerfile) the connection is refused (ERR_CONNECTION_REFUSED). So I SSH'ed into the AWS instance and checked the docker logs, which show that my Tomcat server has started successfully, yet obviously hasn't processed any requests.
Does anyone have any idea my port 8888 appears not to be forwarding to my container?
Executing the command (on the AWS instance):
sudo docker ps -a
gives:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c353e236da7a aws_beanstalk/current-app:latest "catalina.sh run" 28 minutes ago Up 13 minutes 80/tcp, 8080/tcp, 8888/tcp sharp_leakey
which shows port 80, 8080, and 8888 as being open on the docker container.
My Dockerfile is fairly simple:
FROM tomcat:7.0
EXPOSE 8080
EXPOSE 8888
EXPOSE 80
and my Dockerrun.aws.json file is:
{
"AWSEBDockerrunVersion": "1",
"Image": {
"Name": "myusername/mycontainer-repo"
},
"Authentication": {
"Bucket": "mybucket",
"Key": "docker/.dockercfg"
},
"Ports": [
{
"ContainerPort": "8888"
}
]
}
Does anyone see where I could be going wrong?
I'm not even sure where to look at this point.
Also, my AWS security group for the instance is open on port 80, 8080, and 8888.
Any advice would be greatly appreciated! I'm at a loss here.
Update 1:
Minor update, although I am still having trouble.
After SSH'ing into my AWS EB instance, I inspected the Docker container to grab the IP of the container:
sudo docker inspect c353e236da7a
which gave me the IP as 172.17.0.6.
Then, again from the AWS instance, I ran a curl command:
curl 172.17.0.6:8080/homepage
which works, and returns the HTML of homepage! However, curl 172.17.0.6:8888/homepage does not work (so I'm not sure what the "ContainerPort" : "8888" means in the Dockerrun.aws.json file then).
However, I still have the question, why aren't my :8080 requests being forwarded to the container Tomcat webserver? As above, myapplication.elasticbeanstalk.com:8080/homepage still receives a connection refused (ERR_CONNECTION_REFUSED).
myapplication.elasticbeanstalk.com
Is a load balancer, not your instance. Elastic beanstalk launches a load balancer to autoscale your instances. Therefore when you are connecting to myapplication.elasticbeanstalk.com:8888 You are actually connecting to an instance that has only port 80 open. The load balancer then fowards traffic to an instance listening on port 8080.
You should be able to access your web application by just using the url without a port: myapplication.elasticbeanstalk.com
The reason this doesn't work is because you told your docker container to use port 8080, but told Beanstalk to forward to port 8888. Sure, all your ports are open, but tomcat is only running on port 8080.
The ports section in the dockerrun.aws.json doesnt tell your app which port to run on, it tells the load balancer which port to forward to.
Ports – (required when you specify the Image key) Lists the ports to expose on the Docker container. AWS Elastic Beanstalk uses ContainerPort value to connect the Docker container to the reverse proxy running on the host.
You can specify multiple container ports, but AWS Elastic Beanstalk uses only the first one to connect your container to the host's reverse proxy and route requests from the public Internet.
as seen here.
Or, in other words, the 8888 that you told beanstalk to forward to is working correctly, but your app is actually running on port 8080. You should change the dockerrun.aws.json to use the port 8080 instead.
I'm done this using fixing of nginx's listen ports.
So, you have to add .ebextensions directory into root of your app and put your config file here (in my example it's 00-bypass-nginx-proxy.config):
files:
"/tmp/change_nginx_port.sh":
mode: "000755"
owner: root
group: root
content: |
#!/bin/sh
# change listen port from 80 to 8761
sed -i '7s/.*/ listen 8761;/' /etc/nginx/sites-available/elasticbeanstalk-nginx-docker-proxy.conf
# restart nginx
service nginx restart
container_commands:
00setup-nginx:
command: "/tmp/change_nginx_port.sh"
Your service now will be available on port 8761. Pay attention to sed part of script, there is hardcoded line number which could be differ on your environment.