We Keep Coding

Spring Boot in Docker container behind Nginx reverse proxy not working

I have a problem with a Spring Boot application running inside docker container, which is running behind nginx reverse proxy.
To check that docker and nginx are working correctly I've tried folloging, which worked as expected:
docker run -d -p 9001:80 nginx:alpine
My nginx config looks like
server {
server_name xyz.com;
listen 80;
location / {
proxy_pass http://localhost:9001;
}
}
Now I'm able to reach the nginx welcome page from the internet via xyz.com and let's assume the server's IP address is 123.123.1.1, via http://123.123.1.1:9001 as expected.
Also on the server itself, I get the welcome page via
curl http://localhost:9001
I think that shows, that docker and nginx are working as expected.
Let's go further down towards my problem.
Now I have build a simple Spring Boot application, which is bind to address 0.0.0.0 and port 9001. (Of course I have stopped the nginx container before going on...)
After the maven build, I have started the application with
java -jar app.jar
After it started up, I can reach it on the server with
curl http://localhost:9001
and from the internet via xyz.com and http://123.123.1.1:9001 - everything fine as expected.
Now, I've stopped the application und build a Dockerfile:
FROM amazoncorretto:11-alpine-jdk
COPY target/app.jar app.jar
ENTRYPOINT ["java", "-jar","/app.jar"]
First building the image with docker build -t app:latest . and afterwards run the container with docker run -p 9001:9001 -d app:latest. So far so good, the container starts up - fine.
And here is the problem:
I'm not able to reach the application neither via xyz.com nor on the server witch curl http://localhost:9001! But via http://123.123.1.1:9001 it worked!
I've tried to check it inside the container docker exec -it [container] sh and afterwards # wget localhost:9001 worked fine. # netstat -tulpn on the container itself brings following
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:9001 0.0.0.0:* LISTEN 1/java
Also on the server $ netstat -tulpn shows the following:
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:9001 0.0.0.0:* LISTEN 607173/docker-proxy
tcp6 0 0 :::9001 :::* LISTEN 607180/docker-proxy
Maybe something with iptables, but not sure what.
Hopefully somebody out there has an idea to help me.
Thanks in advance!
Update
When I used port 80 inside the container (The Spring Boot application listen now on port 80 instead of 9001) - magically it works! Tried port 8080 inside the container also - this didn't work!
Looks like a port problem, maybe someone can explain where the problem is?!

JMX - remote management into docker-compose container - Failed to retrieve RMIServer stub

I want to do REAL remote JMX management into a docker container running a Spring Boot application:
architecture sketch
I've read a lot of documentation and my understanding is that this should be the server-side configuration:
java \
-Djava.rmi.server.hostname=10.0.2.15 \
-Dcom.sun.management.jmxremote.port=8600 \
-Dcom.sun.management.jmxremote.rmi.port=8601 \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.authenticate=false \
-Dcom.sun.management.jmxremote.local.only=false \
-jar my-spring-boot-app.jar
The url to use in JVisualVM should be service:jmx:rmi://10.0.2.15:8601/jndi/rmi://10.0.2.15:8600/jmxrmi.
BUT THIS FAILS (Failed to retrieve RMIServer stub) within JVisualVM (started on machine 1) - this is the log output:
Caused: java.io.IOException: Failed to retrieve RMIServer stub at
javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369)
at
com.sun.tools.visualvm.jmx.impl.JmxModelImpl$ProxyClient.tryConnect(JmxModelImpl.java:549)
[catch] at
com.sun.tools.visualvm.jmx.impl.JmxModelImpl$ProxyClient.connect(JmxModelImpl.java:486)
at
com.sun.tools.visualvm.jmx.impl.JmxModelImpl.connect(JmxModelImpl.java:214)
IT WORKS if I change the server application configuration to -Djava.rmi.server.hostname=172.19.0.6 (I use a BRDIGE docker network ... therefore routing to 172.19.0.6 is possible). With this configuration I am able to do JMX monitoring if the JVisualVM is started on the Docker Host (machine 2). But this is NO REAL REMOTE management because routing to 172.19.0.6 is usually impossible.
Some additional informations:
Port 8600, 8601 are exposed and are shown as LISTEN:
pfh#workbench ~/temp/ % netstat -taupen | grep 860
tcp6 0 0 :::8600 :::* LISTEN 0 254349 -
tcp6 0 0 :::8601 :::* LISTEN 0 254334 -
and telnet 10.0.2.15 8600 from machine 1 is possible.
I get the same wrong behavior with Java 1.8.0_111 and 1.7.0_80 on docker containers and docker host (running JVisualVM).
BTW: this configuration works if the Spring Boot application is running on machine 2 directly (without Docker).
I know that JMX usually negotiates random ports ... I try to make them explicit in my configuration. There is also one additional property -Dcom.sun.aas.jconsole.server.cbport=8602 that can be set but this did not solve the problem.
Where is my fault?

In my problem description I concealed the the docker container was started via docker-composewith this configuration:
my-spring-boot-service:
...
ports:
- "8610:8610"
- "8611:8611"
... and this results in open ports which seem to be bound to all interfaces as you can see via docker inspect my-spring-boot-app:
"NetworkSettings": {
"Bridge": "",
"SandboxID": "ac1a27e2696fd4ac2fcddf6e0935716304e348203ddbe1a0f8e31114cc6e289b",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {
"8610/tcp": [
{
"HostIp": "0.0.0.0",
"HostPort": "8610"
}
],
"8611/tcp": [
{
"HostIp": "0.0.0.0",
"HostPort": "8611"
}
],
I cannot see a problem here ... but this seems to be the problem because if I start the container via docker itself (as suggested by #zapl)
docker run -p 8610:8610 -p 8611:8611 my-spring-boot-app-image
IT WORKS - BUT NOT THE WAY I WANT - I want to use docker-compose.
There is a difference between the two deployment ... docker inspect network <foo>.
... on the working docker network looks this way:
"Options": {
"com.docker.network.bridge.default_bridge": "true",
"com.docker.network.bridge.enable_icc": "true",
"com.docker.network.bridge.enable_ip_masquerade": "true",
"com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
"com.docker.network.bridge.name": "docker0",
"com.docker.network.driver.mtu": "1500"
},
on the non-working docker-compose network looks this way:
"Options": {},
Both container configurations use no explicit defined network but the default one.
QUESTIONS: Is there a configuration missing? Should I define a network explicitly in docker-compose?
docker-elk is a docker-compose based deployment. I configured the described configuration of the JMX interface and was able to remote JMX this machine.
My JMX configuration is exactly the same - MINE IS NOT WORKING :-(
OS/Arch: linux/amd64
docker version: 1.12.2
docker-compose version: 1.8.0, build f3628c7 and 1.9.0, build 2585387

maybe I should switch to JMXMP instead of JMXRMI - https://github.com/oracle/docker-images/tree/master/OracleCoherence/docs/5.monitoring

HBase 1.2.1 standalone in Docker unable to connect

I want to connect to HBase running in standalone in a docker, using Java and the HBase API
I use this code to connect :
Configuration config = HBaseConfiguration.create();
config.set("hbase.zookeeper.quorum", "163.172.142.199");
config.set("hbase.zookeeper.property.clientPort","2181");
HBaseAdmin.checkHBaseAvailable(config);
Here is my /etc/hosts file
127.0.0.1 localhost
XXX.XXX.XXX.XXX hbase-srv
Here is the /etc/hosts file from my docker (named hbase-srv)
XXX.XXX.XXX.XXX hbase-srv
With this configuration, I get a connection refused error :
INFO | Initiating client connection, connectString=163.172.142.199:2181 sessionTimeout=90000 watcher=hconnection-0x6aba2b860x0, quorum=163.172.142.199:2181, baseZNode=/hbase
INFO | Opening socket connection to server 163.172.142.199/163.172.142.199:2181. Will not attempt to authenticate using SASL (unknown error)
INFO | Socket connection established to 163.172.142.199/163.172.142.199:2181, initiating session
INFO | Session establishment complete on server 163.172.142.199/163.172.142.199:2181, sessionid = 0x15602f8d8dc0002, negotiated timeout = 40000
INFO | Closing zookeeper sessionid=0x15602f8d8dc0002
INFO | Session: 0x15602f8d8dc0002 closed
INFO | EventThread shut down
org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: java.net.ConnectException: Connection refused
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1560)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1580)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1737)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.isMasterRunning(ConnectionManager.java:948)
at org.apache.hadoop.hbase.client.HBaseAdmin.checkHBaseAvailable(HBaseAdmin.java:3159)
at hbase.Benchmark.main(Benchmark.java:26)
However, if I remove the lines XXX.XXX.XXX.XXX hbase-srv from both /etc/hosts files I get the error unknown host : hbase-srv
I have also checked, I can successfully telnet to my hbase docker on the client port.
On the docker, all the ports used by HBase are opened and binded to the same number (60000 on 60000, 2181 on 2181, etc).
I also wanted to add that all was fine when I used this configuration on localhost.
If you can't give me an answer to my problem, could you at least give me a procedure to deploy a standalone hbase on a docker.
UPDATE : Here is my Docker file
FROM java:openjdk-8
ADD hbase-1.2.1 /hbase-1.2.1
WORKDIR /hbase-1.2.1
# ZooKeeper
EXPOSE 2181
# HMaster
EXPOSE 60000
# HMaster Web
EXPOSE 60010
# RegionServer
EXPOSE 60020
# RegionServer Web
EXPOSE 60030
EXPOSE 16010
RUN chmod 755 /hbase-1.2.1/bin/start-hbase.sh
CMD ["/hbase-1.2.1/bin/start-hbase.sh"]
My HBase shell is working, I also tried to open the port using iptables for tcp and udp but still the same problem

There are two problems with your Dockerfile:
use hbase master start instead of start-hbase.sh
regionserver is actually not running on 60020
The 2nd problem is not so easy to solve. If run hbase standalone with version >= 1.2.0 (not sure, I'm running 1.2.0), hbase will use ephemeral port instead of the default port or the port you provide in hbase-site.xml which makes it very hard to provide hbase service in docker using the original version.
I add a property named hbase.localcluster.port.ephemeral and managed to build a standalone hbase in docker, which you can reference here.

FitNesse with Jenkins: BindException - Possible port clash?

It happens quite often (at least once under five runs) that running a FitNesse test results in this error message:
Unable to start test system 'slim': fitnesse.slim.SlimError: Got invalid slim header from client. Read the following:
the following seems to be empty thus the error message ends after the colon.
The log shows a BindException outside our own code:
Can not bind to port 8086. Aborting.
java.net.BindException: Address already in use
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376)
at java.net.ServerSocket.bind(ServerSocket.java:376)
at java.net.ServerSocket.<init>(ServerSocket.java:237)
at java.net.ServerSocket.<init>(ServerSocket.java:128)
at fitnesse.socketservice.SocketFactory.createServerSocket(SocketFactory.java:33)
at fitnesse.slim.SlimService.<init>(SlimService.java:102)
at fitnesse.slim.SlimService.startWithFactory(SlimService.java:75)
at fitnesse.slim.SlimService.main(SlimService.java:56)
The reason for this seems to be a port clash. FitNesse and Jenkins run on the same machine.
FitNesse runs on port 8090 under the user "jenkins" with Java 7:
% netstat -plten | grep 8090
tcp6 0 0 :::8090 :::* LISTEN 113 213626367 17072/java
% ps aux | grep 17072
jenkins 17072 0.5 0.5 9299292 130676 ? Sl 13:25 0:04 /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java -jar /opt/FitServer/fitnesse.jar -p 8090
Port 8086 is used by Jenkins with the latest Java 6 version. This instance is needed for all our projects which build on a remote host without Java 7 installed. The error occurs when running the test from the browser (address: jenkins:8090/TestEnvironment/TestSuite) as well as automatic jobs from Jenkins.
% netstat -plten | grep 8086
tcp6 0 0 :::8086 :::* LISTEN 118 134031785 32343/java
% ps aux | grep 32343
jenkins+ 32343 0.7 1.5 15745712 384180 ? Sl Jun01 197:53 /usr/bin/java -Djava.awt.headless=true -jar /usr/share/jenkins-java6/jenkins-java6.war --webroot=/var/cache/jenkins-java6/war --httpPort=8086 --ajp13Port=-1
Now for the questions:
How do I get rid of this error?
Why does FitNesse connects to port 8086? Is there a configuration for that?
What has FitNesse do to with our Jenkins Java 6 instance? I don't really see the link here.
Is the port 8090 for FitNesse not well chosen? I tried to change it to 8050 but the error occurs as well.

The default port used by Slim is indeed close to the 8086 you report: see the section 'Configure SLIM' on http://www.fitnesse.org/FitNesse.FullReferenceGuide.UserGuide.WritingAcceptanceTests.SliM
You should be able to get it to use a different port (range) by overriding the slim.port property. Either via supplying an extra property to your FitNesse startup script, or placing it in `plugins.properties':
Those properties can be either provided by a wiki page, on the command line (e.g. -Dslim.port=9000) or in the plugins.properties file.

Cannot connect to localhost using service:jmx/rmi//..... When Using Visual VM with Tomcat within Vagrant

I've got a vagrant VM which is running tomcat and I wanted to connect VisualVM to monitor the performance of the VM during some tests.
I followed this guide to setup this configuration which recommended creating a setenv.sh script and placing it within /usr/share/tomcat7/bin. I found that setenv wasn't being run, so I placed the following in /etc/tomcat/tomcat.conf instead:
JAVA_OPTS="-Dcom.sun.management.jmxremote=true - Dcom.sun.management.jmxremote.port=9901 -Dcom.sun.management.jmxremote.ssl=false
m.sun.management.jmxremote.authenticate=false -Djava.rmi.server.hostname=localhost -Dcom.sun.management.jmxremote.rmi.port=9909"
I believe that the config is working as when running the following, I can see the ports active:
[vagrant#localhost ~]$ netstat -lntu | grep 99
tcp6 0 0 :::9901 :::* LISTEN
tcp6 0 0 :::9909 :::* LISTEN
Here is my config from the vagrant file:
# Ports enabled for VisualVM to work.
config.vm.network "forwarded_port", guest: 9901, host: 9905
config.vm.network "forwarded_port", guest: 9909, host: 9907
However after entering the following config, or close variations of (have tried 127.0.0.1 & enabling "do not require SSL" all I get is " ".
I have also tried ports 9901, 9905, 9909.
Any thoughts on how to solve this?