Jusene's Blog

Docker flannel网络

docker
字数统计: 1.6k阅读时长: 8 min
2018/04/13 Share

flannel

flannel是CoreOS开发的容器网络解决方案。flannel为每个host分配一个subnet，容器从此subnet中分配IP,这些IP可以在host间路由，容器间无需NAT和port mapping就可以跨主机通信。

每个subnet都是从一个更大的IP池中划分的，flannel会在每个主机上运行一个叫flanneld的agent，起职责是从从池子中分配subnet，为了在各个主机间共享信息，flannel用etcd存放网络配置、已分配的subnet、host的IP等信息。

数据的转发由backend实现的。flannel提供多种backend，最常用的有vxlan和host-gw。

flannel vxlan演示

1.创建etcd数据库

1
~]# wget -c https://github.com/coreos/etcd/releases/download/v3.2.9/etcd-v3.2.9-linux-amd64.tar.gz
2
~]# tar xf etcd-v3.2.9-linux-amd64.tar.gz
3
~]# cp etcd-v3.2.9-linux-amd64/etcd* /usr/bin/
4
~]# vim  /usr/lib/systemd/system/etcd.service
5
[Unit]
6
Description=Etcd Server
7
After=network.target
8
9
[Service]
10
Type=notify
11
ExecStart=/usr/bin/etcd --name=flannel --data-dir=/var/lib/etcd/ --listen-client-urls=http://10.211.55.16:2379,http://127.0.0.1:2379 --listen-peer-urls=http://10.211.55.16:2380  --advertise-client-urls=http://10.211.55.16:2379 --initial-cluster-token=etcd-cluster  --initial-cluster-state=new
12
Restart=on-failure
13
LimitNOFILE=65536
14
15
[Install]
16
WantedBy=multi-user.target
17
18
~]# systemctl start etcd
19
#测试
20
~]# etcdctl set foo bar
21
bar
22
~]# etcdctl get foo
23
bar
24
## 配置flannel配置
25
~]# cat flannel-config.json
26
{
27
    "Network":"10.2.0.0/16",
28
    "SubnetLen":24,
29
    "Backend":{
30
        "Type":"vxlan"
31
    }
32
}
33
~]# etcdctl set /docker/network/config < flannel-config.json
34
~]# etcdctl get /docker/network/confi
35
{
36
"Network":"10.2.0.0/16",
37
"SubnetLen":24,
38
"Backend": {
39
 "Type": "vxlan"
40
}
41
}

2.配置flannel agent

host1:

1
~]# wget https://github.com/coreos/flannel/releases/download/v0.8.0/flannel-v0.8.0-linux-amd64.tar.gz
2
~]# tar xf flannel-v0.8.0-linux-amd64.tar.gz -C /usr/local/bin/
3
~]# vim /usr/lib/systemd/system/flanneld.service
4
[Unit]
5
Description=Flanneld overlay address etcd agent
6
After=network.target
7
After=network-online.target
8
Wants=network-online.target
9
After=etcd.service
10
Before=docker.service
11
12
[Service]
13
Type=notify
14
ExecStart=/usr/local/bin/flanneld \
15
-etcd-endpoints=http://10.211.55.16:2379 \
16
-etcd-prefix=/docker/network \
17
-iface=eth0
18
ExecStartPost=/usr/local/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker
19
Restart=on-failure
20
21
[Install]
22
WantedBy=multi-user.target
23
RequiredBy=docker.service
24
~]# systemctl start flanneld

host2也进行相同的配置即可

3.查看网络变化

host1:

1
~]# ip a show flannel.1
2
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN 
3
    link/ether 32:f7:c6:54:52:ee brd ff:ff:ff:ff:ff:ff
4
    inet 10.2.71.0/32 scope global flannel.1
5
       valid_lft forever preferred_lft forever
6
    inet6 fe80::30f7:c6ff:fe54:52ee/64 scope link 
7
       valid_lft forever preferred_lft forever
8
# 一个新的interface flannel.1被新建，而且被分配了ip
9
~]# ip r
10
....
11
10.2.0.0/16 dev flannel.1 
12
....
13
# 自动添加了一条route，来之10.2.0.0/16的网络数据包都往这个接口走

4.配置docker连接flannel
host1:

1
~]# cat /usr/lib/systemd/system/docker.service
2
...
3
ExecStart=/usr/bin/dockerd --bip=10.2.71.1/24 --mtu=1450
4
... 
5
#这里的配置必须与/run/flannel/subnet.env中的FLANNEL_SUBNET和FLANNEL_MTU一致
6
~]# cat /run/flannel/subnet.env
7
FLANNEL_NETWORK=10.2.0.0/16
8
FLANNEL_SUBNET=10.2.71.1/24
9
FLANNEL_MTU=1450
10
FLANNEL_IPMASQ=false
11
~]# systemctl daemon-reload
12
~]# systemctl start docker.service

5.查看网络变化

1
~]# ip a show docker0
2
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
3
    link/ether 02:42:ab:10:a0:7d brd ff:ff:ff:ff:ff:ff
4
    inet 10.2.71.1/24 brd 10.2.71.255 scope global docker0
5
       valid_lft forever preferred_lft forever
6
    inet6 fe80::42:abff:fe10:a07d/64 scope link 
7
       valid_lft forever preferred_lft forever
8
~]# ip r
9
...
10
10.2.0.0/16 dev flannel.1 
11
10.2.71.0/24 dev docker0 proto kernel scope link src 10.2.71.1 
12
...

6.将容器连接到flannel网络

host1:

1
~]# docker run -itd --name bbox1 buxybox
2
~]# docker exec bbox1 ip r
3
...
4
default via 10.2.71.1 dev eth0 
5
10.2.71.0/24 dev eth0 scope link  src 10.2.71.2
6
...

host2:

1
~]# docker run -itd --name bbox2 buxybox
2
~]# docker exec bbox2 ip r
3
default via 10.2.81.1 dev eth0 
4
10.2.81.0/24 dev eth0 scope link  src 10.2.81.2 
5
~]# docker exec bbox2 ping 10.2.71.2
6
64 bytes from 10.2.71.2: seq=0 ttl=62 time=0.874 ms
7
64 bytes from 10.2.71.2: seq=1 ttl=62 time=0.686 ms
8
64 bytes from 10.2.71.2: seq=2 ttl=62 time=0.573 ms
9
....
10
~]# docker exec bbox2 traceroute 10.2.71.2
11
traceroute to 10.2.71.2 (10.2.71.2), 30 hops max, 46 byte packets
12
 1  10.2.81.1 (10.2.81.1)  0.007 ms  0.010 ms  0.003 ms
13
 2  10.2.71.0 (10.2.71.0)  0.241 ms  0.318 ms  0.185 ms
14
 3  10.2.71.2 (10.2.71.2)  0.177 ms  0.304 ms  0.196 ms
  1. bbox1与bbox2不是一个subnet，数据包发送给默认网关10.2.81.1
  2. 根据host2的路由表，数据包会发给flannel.1
  3. flannel.1将数据包封装成vxvlan，通过eth0发送给host1
  4. host1收到包解封，发现数据包的目的是10.2.71.2，根据路由表将数据包发给flannel.1,并通过docker0到底bbox1

flannel为每个主机分配独立的subnet，但flannel.1将这些subnet连接起来，相互路由。本质上，flannel将各主机上相互独立的docker0容器网络组成了一个互通的大网络，flannel没有提供隔离网络。

fannel host-gw演示

与vxlan不同，host-gw不会封装数据包，而是在主机的路由表中创建到其他主机subnet的路由条目，从而实现容器的跨主机通信。

1.修改flannel-config.json

1
~]# cat flannel-config.json
2
{
3
    "Network":"10.2.0.0/16",
4
    "SubnetLen":24,
5
    "Backend":{
6
        "Type":"host-gw"
7
    }
8
}
9
~]# etcdctl set /docker/network/config < flannel-config.json

2.重启flanneld
host1:

1
~]# systemctl restart flanned
2
~}# ip r
3
....
4
10.2.0.0/16 dev flannel.1
5
10.2.81.0/24 via 10.211.55.18 dev eth0
6
....

3.host-gw的mtu改成了1500，重启docker
host1:

1
~]# cat /run/flannel/subnet.env 
2
FLANNEL_NETWORK=10.2.0.0/16
3
FLANNEL_SUBNET=10.2.71.1/24
4
FLANNEL_MTU=1500
5
FLANNEL_IPMASQ=false

3.测试网络

1
~]# docker exec bbox2 traceroute 10.2.71.2
2
traceroute to 10.2.71.2 (10.2.71.2), 30 hops max, 46 byte packets
3
 1  10.2.81.1 (10.2.81.1)  0.006 ms  0.010 ms  0.005 ms
4
 2  10.211.55.17 (10.211.55.17)  0.228 ms  0.288 ms  0.165 ms
5
 3  10.2.71.2 (10.2.71.2)  0.284 ms  0.262 ms  0.167 ms
  1. bbox1与bbox2不是一个subnet，数据包发送给默认网关10.2.81.1
  2. 查看路由10.2.71.0/24的网段地址发往10.211.55.17(host1)通过eth0
  3. host1收到包解封，发现数据包的目的是10.2.71.2，根据路由表将数据包发给flannel.1,并通过docker0到底bbox1

host-gw和vxlan比较

  1. host-gw 把每个主机都配置成网关，主机知道其他主机的 subnet 和转发地址。vxlan 则在主机间建立隧道，不同主机的容器都在一个大的网段内（比如 10.2.0.0/16）。

  2. 虽然 vxlan 与 host-gw 使用不同的机制建立主机之间连接，但对于容器则无需任何改变，bbox1 仍然可以与 bbox2 通信。

  3. 由于 vxlan 需要对数据进行额外打包和拆包，性能会稍逊于 host-gw。

原文作者：Zhang Jusene

原文链接：http://jusene.github.io/2018/04/13/docker-flannel/

发表日期：April 13th 2018, 10:19:37 pm

更新日期：November 30th 2019, 1:54:24 am

版权声明：本文采用知识共享署名-非商业性使用 4.0 国际许可协议进行许可

CATALOG
  1. 1. flannel
  2. 2. flannel vxlan演示
  3. 3. fannel host-gw演示
Total : 190
2020
2019
2018
2017
python boot troubshooting docker drbd ssh web ELK kubernetes Kubernetes git ansible tools dns corosync consul Django etcd security ftp Go bash bigdata grub heartbeat Java kickstart kvm lvs mongodb mysql mesh nc nfs mogilefs openresty upgrade puppet rabbitmq saltstack rsync screen socat svn tcpdump fastdfs haproxy iscsi keepalived cache openssl vpn optimization prometheus redis RPM cncf supervisord kernel JAVA
python Linux troubshooting Docker docker storage nginx Nosql kubernetes Flask Kubernetes ansible Shell cluster Django Go bigdata boot Java hypervisor mysql mesh tools openresty puppet rabbitmq saltstack optimization cncf varnish tomcat JAVA