Docker Swarm Container is unable to connect to host network
On one of my project, I was deploying a Docker Swarm service which pushes data to an Elastic Search VM. I have worked on this scenario several times. But I did face this issue for the first time.
The docker service gets deployed on an Overlay network, But the container is unable to connect to the Elastic search host on the same network. I have been troubleshooting for hours check the Firewall, Security group, Even installed different docker version. When i deploy service with host network, I’m able to access the other host. But the issue is when you bind a port in a service with host network, you cannot create multiple replicas.
After few hours of troubleshooting, I found the issue with the Overlay network’s default subnet is conflicting with the host network subnet. In this solution we need to remove the services and network and redeploy. So make sure to copy your deployment configuration before this operation
Solution:
My host IP is 10.0.0.4
Here my overlay network name is “frontend”. To verify the subnet, run the following command.
#docker network inspect frontend
[
{
“Name”: “frontend”,
“Id”: “jkf9a20q1auoo3tj0y4p1t018”,
“Created”: “2021–02–27T07:03:57.266691687Z”,
“Scope”: “swarm”,
“Driver”: “overlay”,
“EnableIPv6”: false,
“IPAM”: {
“Driver”: “default”,
“Options”: null,
“Config”: [
{
“Subnet”: “10.0.0.0/24”,
“Gateway”: “10.0.0.1”
}
]
},
“Internal”: false,
“Attachable”: false,
“Ingress”: false,
“ConfigFrom”: {
“Network”: “”
},
“ConfigOnly”: false,
“Containers”: null,
“Options”: {
“com.docker.network.driver.overlay.vxlanid_list”: “4096”
},
“Labels”: null
}
]
Now, Remove your services and the ingress network. We cannot remove the network If the services are running on the ingress network.
# docker service rm <service name>
#docker network rm frontend
#docker network rm ingress
After removing, recreate the ingress Overlay with a different subnet
#docker network create — driver overlay — ingress — subnet=172.16.0.0/16 — gateway=172.16.0.2 — opt com.docker.network.driver.mtu=1200 ingress
#docker network inspect ingress
[
{
“Name”: “ingress”,
“Id”: “5g9tgkkydk9qj710kuq2iztvx”,
“Created”: “2021–02–27T07:10:38.263862822Z”,
“Scope”: “swarm”,
“Driver”: “overlay”,
“EnableIPv6”: false,
“IPAM”: {
“Driver”: “default”,
“Options”: null,
“Config”: [
{
“Subnet”: “172.16.0.0/16”,
“Gateway”: “172.16.0.2”
}
]
},
“Internal”: false,
“Attachable”: false,
“Ingress”: true,
“ConfigFrom”: {
“Network”: “”
},
“ConfigOnly”: false,
“Containers”: null,
“Options”: {
“com.docker.network.driver.mtu”: “1200”,
“com.docker.network.driver.overlay.vxlanid_list”: “4097”
},
“Labels”: null
}
]
So its confirmed the Overlay network is created in a different subnet. You need to create your Overlay network for the services
# docker network create frontend -d overlay
Now Im deploying my test Nginx service and check the host connectivity.
You may refer the below for the swarm compose file
#docker stack deploy -c<(curl -fsSL https://raw.githubusercontent.com/sivaprakash123/swarm-manifests/main/nginx-overlay.yml) mystack
List your running service containers
#docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4ed8b85fd17b nginx:latest “/docker-entrypoint.…” About a minute ago Up 58 seconds 80/tcp mystack_gateway.1.a
Copy the container ID and login to container shell to verify the host connection
#docker exec -it 4ed8b85fd17b bash
#curl 10.0.0.5:9200
{
“name” : “ElasticsearchVM”,
“cluster_name” : “elasticsearch”,
“cluster_uuid” : “p3YcPDnqR82RaGdREg19Qw”,
“version” : {
“number” : “7.11.1”,
“build_flavor” : “default”,
“build_type” : “deb”,
“build_hash” : “ff17057114c2199c9c1bbecc727003a907c0db7a”,
“build_date” : “2021–02–15T13:44:09.394032Z”,
“build_snapshot” : false,
“lucene_version” : “8.7.0”,
“minimum_wire_compatibility_version” : “6.8.0”,
“minimum_index_compatibility_version” : “6.0.0-beta1”
},
“tagline” : “You Know, for Search”
}
This should solve the issue. If you have any queries, please do comment.