Introduction

In this section, we will explain the network used when operating the Docker swarm cluster. The Docker overlay network uses VXLAN technology to connect containers in the L2 network located on different hosts. In the introduction, we’re talking about architecture, external traffic going to Docker services, how DNS services works, different types of load balancing, and finally security and encryption.

Overlay Driver Network Architecture

Docker overlay network driver is an integral part of the Docker installation that handles high-quality multi-host communication. Docker Swarm initialization creates an overlay ingress network. Docker Engine automatically creates a VXLAN support without the need for additional setup.

Engine also creates an additional network to access the outside world. To summarize, the overlay network is only used between containers, while the additional network serves to connect the container to the outside world, and communication between the containers is not allowed on this network. The Docker engine does not create an overlay network on hosts that do run services.

Docker Network Control Plane

The normal functioning and communication of the overlay network data layer require an additional layer to monitor, propagate the network state, detect errors, and quickly converge within the Docker swarm cluster. And here comes the Docker Network Control Plane, which is based on Gossip and SWIM protocols.

Docker Network Data Plane

The layer on which the actual traffic takes place is based on an industry-recognized VXLAN protocol that connects multi-host containers across L2 and L3 physical networks. VXLAN encapsulates Ethernet frame with the additional information that is required for a packet to successfully reach its destination in a multi-host environment.

VXLAN has been an integral part of the Linux kernel for some time now, so there is no additional load on resources like CPU. VXLAN separates the overlay layer from the physical layer making this network highly independent and making applications highly mobile.

1. Container 2 sends DNS requests for container1. Because container reside on same network embedded DNS servers responds to container2 with info
2. Container2 generates an Ethernet frame with MAC and IP address of container1
3. To resolve MAC address, Container 2 sends ARP request which is answered by the operating system
4. Ethernet frame is encapsulated with VXLAN header containing source and destination IP and MAC address of hosts.
5. Encapsulated packet is send through VXLAN tunnel and routed by physical devices
6. Packet arrives at host1 physical interface, is encapsulated and the original packet is forwarded to the container by the destination IP address

Example 1 (Create overlay network)

In this example, we will create an overlay network. It is assumed that a 3-node swarm cluster has already been created. More on this in the Docker swarm article.

When swarm cluster is initialized, a default ingress overlay network is created:

[root@swmanager ~]# docker network ls NETWORK ID NAME DRIVER SCOPE 956c386c86c1 bridge local 733f0523e928 host local aegdyx3wfcv3 ingress overlay swarm e57587b7fb95 none null local

If you want to create own overlay network it done this way:

[root@swmanager ~]# docker network create -d overlay customoverlay zmaql0urnd458cgbhyxjghx5i

Example 3 (Creating an Overlay Network for Communication between Services and standalone Containers)

If you want to create an overlay network to communicate services and standalone containers with standalone containers on other docker daemons, use the attach parameter:

[root@swmanager ~]# docker network create -d overlay –attachable overlay-to-attach m2zhfr0ssqgxtbs817ng77hks

Create container centos and attach it to overlay-to-attach network:

[root@swmanager ~]# docker run -d –name centos –network overlay-to-attach centos ping docker.com 0ce108196f14c225339acb7f4528179ad21b4229390586a39651a5a2a46c6955

On second node (worker) check that overlay-to-attach network does not exist:

[root@swworker1 ~]# docker network ls NETWORK ID NAME DRIVER SCOPE 28669ea03274 bridge bridge local 0b59bda12c63 custombridge bridge local yqw6qmtzielg customingress overlay swarm edd78417b9df docker_gwbridge bridge local 733f0523e928 host local 7eeeed2e81e1 mymacvlan macvlan local e57587b7fb95 none null local

Create a second centos2 container on another node and join it to overlay-to-attach network:

[root@swworker1 ~]# docker run -dit –name centos2 –network overlay-to-attach centos ping docker.com d6e6c89f62e661f0daf89df40e8b7b383cd8ff8419c2980259f229acc31d5e5e

From first container ping second one:

[root@swmanager ~]# docker exec centos ping centos2 PING centos2 (10.0.1.7) 56(84) bytes of data. 64 bytes from centos2.overlay-to-attach (10.0.1.7): icmp_seq=1 ttl=64 time=0.658 ms 64 bytes from centos2.overlay-to-attach (10.0.1.7): icmp_seq=2 ttl=64 time=0.607 ms 64 bytes from centos2.overlay-to-attach (10.0.1.7): icmp_seq=3 ttl=64 time=1.09 ms 64 bytes from centos2.overlay-to-attach (10.0.1.7): icmp_seq=4 ttl=64 time=1.19 ms 64 bytes from centos2.overlay-to-attach (10.0.1.7): icmp_seq=5 ttl=64 time=1.39 ms ^C

Example 3 (Edit default overlay network)

When initializing the Docker swarm cluster, depending on the IPAM driver, a subnet network is defined. If default overaly network subnet is in conflict with your existing infrastructure, the predefined subnet can be changed.

Check which subnet is currently assigned to the default network:

[root@swmanager ~]# docker network inspect ingress | grep -i subnet “Subnet”: “10.255.0.0/16”,

Delete current ingress network:

[root@swmanager ~]# docker network rm ingress WARNING! Before removing the routing-mesh network, make sure all the nodes in your swarm run the same docker engine version. Otherwise, removal may not be effective and functionality of newly create ingress networks will be impaired. Are you sure you want to continue? [y/N] y Ingress

Create overlay network with custom subnet:

[root@swmanager ~]# docker network create -d overlay –ingress –subnet=172.16.0.0/16 –gateway 172.16.0.1 customingress yqw6qmtzielgey21wymklisot

Check list of available networks:

[root@swmanager ~]# docker network ls NETWORK ID NAME DRIVER SCOPE 956c386c86c1 bridge local 0b59bda12c63 custombridge bridge local yqw6qmtzielg customingress overlay swarm zmaql0urnd45 customoverlay overlay swarm 7d74a19eb071 docker_gwbridge bridge local 1topkmis3kib encrypted-overlay overlay swarm 733f0523e928 host local 7eeeed2e81e1 mymacvlan macvlan local e57587b7fb95 none null local m2zhfr0ssqgx overlay-to-attach overlay swarm

Docker Service exposed to outside world

Once port is exposed for Docker service, it is reachable from outside world. If the service is defined by two or more replicas, then Docker load balancing mode is activated, as part of the routing mesh configuration. Each host participates in load balancing regardless of whether the service is started on it.

Alternative way to access service from outside world is to bypass routing mesh configuration. Service is exposed on dedicated host. In this case, make sure that the service is active on that host because it will not be available otherwise.

Example1 (Exposing ports from outside world)

Create service nginx and expose port 80 which is mapped to port 8080:

[root@swmanager ~]# docker service create –name nginx –replicas 2 -p 80:8080 nginx hbv0ty0g29xxu9daiilad1kvr overall progress: 2 out of 2 tasks 1/2: running [==================================================>] 2/2: running [==================================================>] verify: Service converged

DNS services in Docker Swarm cluster

Docker Engine has an internal DNS server with internal KV store used to store records of service names and IP addresses. It is important to note that the IP address that binds to the service name is Virtual IP address of the service cluster, that is, the IP address that represents the cluster and not the individual container.

1) If the services are on the same subnet and use specific subnets, an internal DNS server is used

2) If the services are not in the same subnet, an external DNS server is used

Docker Native Load Balancing

Docker has an internal load balancing mechanism that includes internal balancing within the Docker service, balancing with routing mesh and use of external load balancing.

Internal Load Balancing

Internal load balancing is activated immediately during service creation and Docker balancing between tasks (containers) in the service. Balancing is done on the name DNS service name that is resolved from the Virtual IP address of the service itself. No special adjustments need to be made because Docker does the balancing automatically. Alternative way is done via DNS round robin technique where balancing is done by the individual IP address of each container.

Routing mesh Load Balancing (L4)

Routing mesh mode does load balancing in the way that every host participates in balancing regardless of service running on it. To have this working, you should expose container ports to outside world. The Docker Engine for L4 Load balancer uses integrated Linux kernel functionality – IPVS (IP Virtual Server – embedded L4 load balancer) and iptables. Usually an external load balancer is added to this combination as one of the services (for example nginx) or a physical component.

External load balancing (L7)

Load balancing that runs on the seventh ISO / OSI layer.

IP Address Management

IP addresses and subnets are assigned via an IPAM driver that assigns local non-routable IP addresses for bridge and overlay networks, while IP addresses on the MACVLAN are part of the physical network and depend on the configuration of the physical network itself.

Docker Network Security and Encryption

Security and encryption are implemented on data and control plane.

Control Plane Security

Docker data protection is based on a distributed firewall that is implemented with iptables functionality which is integral part of the Linux kernel. The firewall blocks traffic between different networks and controls open ports to the outside world. The following rules apply:

Inside same container subnet

Traffic is allowed in all directions for all default Docker networks.

Between two different container subnet

Traffic is blocked in all directions for all Docker networks.

From service to the outside world (Egress traffic)

Egress traffic is allowed to the outside world with host statefull firewall which controls responses. Applies to all Docker network defaults.

Traffic from outside world to services

Not allowed by default for all default Docker network except MACVLAN network where incoming traffic is allowed.

Encryption

Data-level encryption is implemented using IPSEC framework and the AES encryption standard. Traffic is encrypted after it leaves the container and before traffic reaches the physical network interface on the host. It is decrypted before entering the destination container.

Data Plane Network Encryption

Data Plane is a secured with Integrated Public Key Infrastructure (PKI). All manager-manager or manager-worker communication is encrypted with the TLS protocol. The leading node manager, the node from which the Docker swarm was initially created, is certificate authority (CA) that issues and sends certificates that is used for authentication of potential members of Docker swarm.

Example (Creating encrypted overlay network)

Use parameter –opt encrypted to create encrypted network:

[root@swmanager ~]# docker network create –opt encrypted -d overlay –attachable encrypted-overlay 1topkmis3kibaccof53voae7c