Docker-swap container connects to host port
I have a cluster swarm in which I created a global service to run on all docker hosts in the cluster.
The goal is for every container instance for this service to connect to the port that the docker host is listening on.
For more information, I am following this Docker Daemon Metrics to familiarize myself with the new Docker API for all hosts, then proxying that host port to the overlay network so Prometheus can flush metrics from all swarm hosts.
I have read several gockub issues docker # 8395 # 32101 # 32277 # 1143 - from this my understanding is the same as described in Docker Daemon Metrics . To connect to the host from the swarm container, I have to use the docker-gwbridge network, which is 172.18.0.1 by default.
Each container in my swarm has a network interface for the docker-gwbridge network:
326: eth0@if327: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue
link/ether 02:42:0a:ff:00:06 brd ff:ff:ff:ff:ff:ff
inet 10.255.0.6/16 scope global eth0
valid_lft forever preferred_lft forever
inet 10.255.0.5/32 scope global eth0
valid_lft forever preferred_lft forever
333: eth1@if334: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue
link/ether 02:42:ac:12:00:04 brd ff:ff:ff:ff:ff:ff
inet 172.18.0.4/16 scope global eth1
valid_lft forever preferred_lft forever
Additionally, each container in the swarm has a default route through 172.0.0.1:
/prometheus # ip route show 0.0.0.0/0 | grep -Eo 'via \S+' | awk '{ print $2 }'
172.18.0.1
/prometheus # netstat -nr | grep '^0\.0\.0\.0' | awk '{print $2}'
172.18.0.1
/prometheus # ip route
default via 172.18.0.1 dev eth1
10.0.1.0/24 dev eth2 src 10.0.1.9
10.255.0.0/16 dev eth0 src 10.255.0.6
172.18.0.0/16 dev eth1 src 172.18.0.4
Regardless, I am unable to reach 172.18.0.1 from the container:
/ # wget -O- 172.18.0.1:4999
Connecting to 172.18.0.1:4999 (172.18.0.1:4999)
wget: can't connect to remote host (172.18.0.1): No route to host
On the host, I can access the docker metrics API at 172.18.0.1. I can ping and I can make a successful HTTP request.
- Can anyone shed some light on why this doesn't work from within a container, as described in the Docker Daemon Metrics guide?
- If the container has a network interface on the 172.18.0.1 network and has routes configured for 172.18.0.1, why pings fail to execute 172.18.0.1 from the container?
- If this is not a valid approach for accessing the host port from a swarm container, then how do you do it?
EDIT: Just realized that I didn't give all the information in the original post. I am running docker swarm on a CentOS 7.2 host with docker version 17.04.0-ce, build 4845c56. My kernel is build 4.9.11 with vxlan and ipvs modules enabled.
After some further digging, I noticed it was a firewall issue. I found that not only could I not be able to ping 172.18.0.1 from containers, but I could not ping my host machine at all! I tried my domain name, FQDN for the server and even its public IP, but the container was unable to ping the host (there is network access, since I can ping google, etc.).
I disabled firewalld on my host and then restarted the docker daemon. After that, I was able to ping my host from containers (both the domain name and 172.18.0.1). Unfortunately this is not a solution for me. I need to determine what firewall rules I need to set in order to allow data transfer in containers without requiring firewalld to be disabled.
source to share
First, I owe you a huge THANK YOU. Before I read your EDIT part, I spent literally day and night trying to solve a similar problem and never really understood that the devil is a firewall.
Without disabling the firewall, I solved the problem on Ubunt 16.04 using
sudo ufw allow in on docker_gwbridge
sudo ufw allow out on docker_gwbridge
sudo ufw enable
I'm not very familiar with CentOS, but I believe the following should help you, or at least serve as a hint
sudo firewall-cmd --permanent --zone=trusted --change-interface=docker_gwbridge
sudo systemctl restart firewalld
You may need to restart docker as well.
source to share