Shulou(Shulou.com)05/31 Report--

Today, the editor will share with you the relevant knowledge about the principle of Docker Swarm service discovery and load balancing. The content is detailed and the logic is clear. I believe most people still know too much about this, so share this article for your reference. I hope you can get something after reading this article. Let's take a look at it.

The technology used

Docker uses the functions of linux kernel iptables and ipvs to achieve service discovery and load balancing.

Iptables is a packet filtering technology available in the linux kernel, which can be used to classify, modify, and forward decisions based on the content of packets.

Ipvs is a transport-level load balancer available in the linux kernel.

Preparatory work

Swarm cluster: [manager] node1, [worker] node2

Client image: registry.cn-hangzhou.aliyuncs.com/anoy/ubuntu

Server image: registry.cn-hangzhou.aliyuncs.com/anoy/vote

As shown in the figure, we will deploy the "client" service and the "vote" service in the swarm cluster, where the "vote" service deploys multiple copies. When the client requests the "vote" service, the output includes the server's container id, which makes it easier to demonstrate the network request.

Cluster status

[root@node1 ~] # docker node lsid hostname status availability manager status engine versionrnr2i1y2of3n5vy2vzh2vkzq0 * node1 ready active leader 18.03.1-ceqvik057dvphx5s06evmswahaf node2 ready active 18.03.1-ce

Create an overlay network using the following command:

Docker network create-driver overlay overlay1

Load balancing based on dns

The following figure describes how dns-based load balancing works:

Dns server is embedded in the docker engine. Docker dns parses the service name "vote" and returns a list of container id addresses (sorted at random). The client usually picks the first ip access, so load balancing may occur between different instances of the server.

Use the following command to create two services "client" and "vote" based on dns load balancer:

Docker service create-- endpoint-mode dnsrr-- replicas 1-- name client-- network overlay1 registry.cn-hangzhou.aliyuncs.com/anoy/ubuntu ping anoyi.comdocker service create-- endpoint-mode dnsrr-- name vote-- network overlay1-- replicas 2 registry.cn-hangzhou.aliyuncs.com/anoy/vote

View service information:

[root@node1 ~] # docker service lsid name mode replicas image ports2mrj3pqyioc3 client replicated 1 registry.cn-hangzhou.aliyuncs.com/anoy/ubuntu:latest826s79tsixuh vote replicated 2 registry.cn-hangzhou.aliyuncs.com/anoy/ubuntu:latest826s79tsixuh vote replicated 2 registry.cn-hangzhou.aliyuncs.com/anoy/vote:latest [root@node1 ~] # docker service ps clientid name image Node desired state current state error portsf74i688vbh12 client.1 registry.cn-hangzhou.aliyuncs.com/anoy/ubuntu:latest node2 running running 2 minutes ago [root@node1 ~] # docker service ps voteid name image node desired state current state error ports7iiuzl2a63hy vote.1 registry.cn-hangzhou .aliyuncs.com / anoy/vote:latest node1 running running 47 seconds agouyhxxqfdima7 vote.2 registry.cn-hangzhou.aliyuncs.com/anoy/vote:latest node2 running running about a minute ago

You can see that "client" runs on node2, enters the client container on node2, and uses dig to resolve the service name "vote", as shown below, "vote" resolves to 10.0.0.6 and 10.0.0.5

[root@node2 ~] # docker pscontainer id image command created status ports names1eed67d37cbb registry.cn-hangzhou.aliyuncs.com/anoy/vote:latest "gunicorn app:app-b …" About a minute ago up about a minute 80/tcp vote.2.uyhxxqfdima7smos5pki84wul436702b21a1c registry.cn-hangzhou.aliyuncs.com/anoy/ubuntu:latest "ping anoyi.com" 3 minutes ago up 3 minutes client.1.f74i688vbh12on8oniufht633 [root@node2] # docker exec-it 436702b21a1c / bin/bashroot@436702b21a1c:/# dig vote;; answer section:vote. 600 in a 10.0.0.5vote. 600 in a 10.0.0.6

Use ping to parse the "vote" service, as shown below, alternately to 10.0.0.6 and 10.0.0.5

Root@436702b21a1c:/# ping-C1 voteping vote (10.0.0.6) 56 (84) bytes of data.64 bytes from vote.2.uyhxxqfdima7smos5pki84wul.overlay1 (10.0.0.6): icmp_seq=1 ttl=64 time=0.087 msroot@436702b21a1c:/# ping-C1 voteping vote (10.0.0.5) 56 (84) bytes of data.64 bytes from vote.1.7iiuzl2a63hyj084qgufc175v.overlay1 (10.0.0.5): icmp_seq=1 ttl=64 time=0.767 ms

If you use curl, as shown below, the request can also be resolved to a different container

Root@436702b21a1c:/# curl vote | grep-I "container id"% total% received% xferd average speed time current dload upload total spent left speed100 3162 3162 00 7542 0 -:-7546 processed by container id 9b42319d4f13root@436702b21a1c:/# curl vote | grep-I "container id"% total% received% xferd average speed time current Dload upload total spent left speed100 3162 3162 00 452k 0 -:-514k processed by container id 1eed67d37cbb

There are the following problems with load balancing based on dns:

Some applications cache dns hostnames to ip address mappings, which can cause applications to time out when mapping changes

Having a non-zero dns ttl value causes a delay when the dns entry reflects the latest details

Load balancing based on vip

Load balancing based on vip overcomes some problems of load balancing based on dns. In this approach, each service has an ip address, and the ip address is mapped to the ip addresses of multiple containers associated with the service. In this case, the service ip associated with the service does not change, even if the container associated with the service dies and restarts.

The following figure describes how vip-based load balancing works:

Dns server resolves the service name "vote" to vip, and uses iptables and ipvs,vip to achieve load balancing between the two server "vote" containers.

Create two vip mode services "client" and "vote" using the following command:

Docker service create-replicas 1-name client-network overlay1 registry.cn-hangzhou.aliyuncs.com/anoy/ubuntu ping anoyi.comdocker service create-name vote-network overlay1-replicas 2 registry.cn-hangzhou.aliyuncs.com/anoy/vote

View these two services and their service ip:

[root@node1 ~] # docker service inspect-- format {{endpoint.virtualips}} vote [{tetug0isdx1gri62g7cfm889i 10.0.0.9 pact 24}] [root@node1 ~] # docker service inspect-- format {{endpoint.virtualips}} client [{tetug0isdx1gri62g7cfm889i 10.0.0.7 and 24}]

Using the following command in the container of "client", you can see that the service name "vote" is mapped to vip "10.0.0.9"

[root@node2] # docker exec-it f3d1c4ef53f8 / bin/bashroot@f3d1c4ef53f8:/# dig vote;; answer section:vote. 600 in a 10.0.0.9

Service ip "10.0.0.9" uses iptables and ipvs load balancing of the linux kernel to two containers. Iptables implements firewall rules and ipvs implements load balancing. To prove this, we need to use nsenter to enter the container's cyberspace (namespace). To do this, we need to find the namespace of the network.

The following is the network namespace on node2:

[root@node2 ~] # cd / run/docker/netns/ [root@node2 netns] # ls1-tetug0isdx 1-vyy22w04t6 be7330b99a27 d67fa9efb59e ingress_sbox

The first two namespaces are for the overlay network, and the latter are for containers. The following command is used to find the network namespace of the "client" container:

[root@node2 netns] # docker pscontainer id image command created status ports names43a789312e70 registry.cn-hangzhou.aliyuncs.com/anoy/vote:latest "gunicorn app:app-b …" 3 minutes ago up 3 minutes 80/tcp vote.1.u46ms31e8zjdxtwrxvaec8zubf3d1c4ef53f8 registry.cn-hangzhou.aliyuncs.com/anoy/ubuntu:latest "ping anoyi.com" 4 minutes ago up 4 minutes client.1.ycox088aek5ajejezubwsjqf2 [root@node2 netns] # docker inspect f3d1c4ef53f8 | grep-I sandbox "sandboxid": "be7330b99a274a03a7f58e9e991346dc6f048836a1682c7244a6068acbfb664c", "sandboxkey": "/ var/run/docker/netns/be7330b99a27"

Sandboxid is the network namespace of the client container.

Using the following command, we can enter the network command space of the "client" container:

Nsenter-net=f3d1c4ef53f8 sh

Below, we can see the forwarding rules and ipvs output of iptables:

Sh-4.2# iptables-nvl-t manglechain output (policy accept 606 packets 50867 bytes) pkts bytes target prot opt in out source destination 0 0 mark all-- * * 0.0.0.7 mark set 0x102 0 0 mark all-- * * 0.0.0.9 mark set 0x103sh-4.2# ipvsadmip virtual server version 1.2.1 (size=4096) prot localaddress:port scheduler flags- > remoteaddress:port forward weight activeconn inactconnfwm 258 rr-> node2:0 masq 10 0fwm 259 rr-> 10.0.0.10 0fwm 0 masq 100-> 10.0.0.11 rr 0 masq 100

Service ip "10.0.0.9" uses the iptables output chain to get the tag 0x103 (hexadecimal-> decimal: 259), and then ipvs uses this tag and balances its load to "10.0.0.10" and "10.0.0.11".

Check the ip of the two containers of the vote service as follows, that is, vip "10.0.0.9" load balance to different container instances:

[root@node2 netns] # docker inspect vote.1.u46ms31e8zjdxtwrxvaec8zub | grep ipv4 "ipv4address": "10.0.0.10" [root@node1 ~] # docker inspect vote.2.tutj19i4iwu1xn7arsaq815cu | grep ipv4 "ipv4address": "10.0.0.11"

Enter the container of the client service and request the vote service using curl. The output is as follows, that is, the request is distributed to different containers:

Root@f3d1c4ef53f8:/# curl vote | grep-I "container id"% total% received% xferd average speed time current dload upload total spent left speed100 3162 3162 00 14409 0 -:-14438 processed by container id c2af209c4e90root@f3d1c4ef53f8:/# curl vote | grep-I "container id"% total% received% xferd average speed time current Dload upload total spent left speed100 3162 3162 00 165k 0 -:-171k processed by container id 43a789312e70

Routing Grid (routing mesh)

Using the routing grid, the ports exposed by the service are exposed to all working nodes in the swarm cluster. Docker does this by creating a "ingress" overlay network, and all nodes use the inherent sandbox network namespace to become part of the "ingress" overlay network by default.

The following figure describes how routing mesh implements load balancing:

First, the hostname or ip is mapped to iptables and ipvs in sandbox ip,sandbox, which are responsible for balancing the request load to two vote containers. The ingress sandbox network namespace resides on all working nodes in the swarm cluster and assists in routing grid functionality by balancing the load of host-mapped ports to the back-end container.

Use the following command to create a vote service and use a routing grid to expose ports to all nodes:

The copy code is as follows:

Docker service create-- name vote-- network overlay1-- replicas 2-p 8080 registry.cn-hangzhou.aliyuncs.com/anoy/vote

The following figure shows the mapping between sandbox, container, and the network of each node:

As shown in the figure, the sandbox and vote containers are part of the "ingress" network, which helps with routing grids. The client container and the vote container are part of the "overlay1" network and help with internal load balancing. All containers are part of the default "docker_gwbridge" network.

Following the nat rules in iptables, host traffic on port 8080 is sent to sandbox in node1:

[root@node1] # iptables-nvl-t natchain docker-ingress (2 references) pkts bytes target prot opt in out source destination 0 0 dnat tcp-- * * 0.0.0.0qq0 0.0.0.0qq0 tcp dpt:8080 to:172.18.0.2:8080 315 18876 return all-- * * 0.0.0.0Univer 0 0.0.0.0Universe 0

Enter the sandbox Network Namespace (ingress_sbox) on node1 and view the forwarding rules and ipvs output of iptables:

[root@node1 netns] # nsenter-- net=ingress_sbox shsh-4.2# iptables-nvl-t manglechain prerouting (policy accept 0 packets 0 bytes) pkts bytes target prot opt in out source destination 0 0 mark tcp-- * * 0.0.0.0 size=4096 0 tcp dpt:8080 mark set 0x105sh-4.2# ipvsadmip virtual server version 1.2.1 (size=4096) prot localaddress:port scheduler flags-> remoteaddress:port forward weight activeconn inactconnfwm 261 rr-> 10.255.0.5 mark tcp 0 masq 100 0 -> 10.255.0.6 masq 0 100

Port 8080 is marked 0x105 (hexadecimal-> decimal: 261), and ipvs uses this flag to load balance it to "10.255.0.5" and "10.255.0.6".

Check the ip of the two containers of the vote service as follows, that is, the traffic of CVM port 8080 will be balanced to different container instances:

[root@node1 netns] # docker inspect 6173afd5fab8 | grep ipv4 "ipv4address": "10.255.0.6"ipv4address": "10.0.0.14" [root@node2 ~] # docker inspect b07e95c5c681 | grep ipv4 "ipv4address": "10.255.0.5"ipv4address": "10.0.0.13"

Verify the load balancer and request the vote service on node1 through the ip and port 8080 of node2:

[root@node1 netns] # curl node2:8080 | grep-I "container id"% total% received% xferd average speed time current dload upload total spent left speed100 3162 3162 00 199k 0-192 k processed by container id 6173afd5fab8 [root@node1 netns] # curl node2:8080 | grep-I "container id"% total% received% xferd average speed Time current dload upload total spent left speed100 3162 3162 00 7551 0 -:-7546 processed by container id b07e95c5c681

Request the vote service on node2 through node1's ip and port 8080:

[root@node2 ~] # curl node1:8080 | grep-I "container id"% total% received% xferd average speed time current dload upload total spent left speed100 3162 3162 00 7531 0-7546 processed by container id 6173afd5fab8 [root@node2 ~] # curl node1:8080 | grep-I "container id"% total% received% xferd average speed Time current dload upload total spent left speed100 3162 3162 00 169k 0-171k processed by container id b07e95c5c681 is all the content of the article "what is the principle of Docker Swarm service discovery and load balancing" Thank you for reading! I believe you will gain a lot after reading this article. The editor will update different knowledge for you every day. If you want to learn more knowledge, please pay attention to the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.