Jusene's Blog

Docker flannel网络

字数统计: 1.6k阅读时长: 8 min
2018/04/13 Share

flannel

flannel是CoreOS开发的容器网络解决方案。flannel为每个host分配一个subnet,容器从此subnet中分配IP,这些IP可以在host间路由,容器间无需NAT和port mapping就可以跨主机通信。

每个subnet都是从一个更大的IP池中划分的,flannel会在每个主机上运行一个叫flanneld的agent,起职责是从从池子中分配subnet,为了在各个主机间共享信息,flannel用etcd存放网络配置、已分配的subnet、host的IP等信息。

数据的转发由backend实现的。flannel提供多种backend,最常用的有vxlan和host-gw。

flannel vxlan演示

1.创建etcd数据库

1
~]# wget -c https://github.com/coreos/etcd/releases/download/v3.2.9/etcd-v3.2.9-linux-amd64.tar.gz
2
~]# tar xf etcd-v3.2.9-linux-amd64.tar.gz
3
~]# cp etcd-v3.2.9-linux-amd64/etcd* /usr/bin/
4
~]# vim  /usr/lib/systemd/system/etcd.service
5
[Unit]
6
Description=Etcd Server
7
After=network.target
8
9
[Service]
10
Type=notify
11
ExecStart=/usr/bin/etcd --name=flannel --data-dir=/var/lib/etcd/ --listen-client-urls=http://10.211.55.16:2379,http://127.0.0.1:2379 --listen-peer-urls=http://10.211.55.16:2380  --advertise-client-urls=http://10.211.55.16:2379 --initial-cluster-token=etcd-cluster  --initial-cluster-state=new
12
Restart=on-failure
13
LimitNOFILE=65536
14
15
[Install]
16
WantedBy=multi-user.target
17
18
~]# systemctl start etcd
19
#测试
20
~]# etcdctl set foo bar
21
bar
22
~]# etcdctl get foo
23
bar
24
## 配置flannel配置
25
~]# cat flannel-config.json
26
{
27
    "Network":"10.2.0.0/16",
28
    "SubnetLen":24,
29
    "Backend":{
30
        "Type":"vxlan"
31
    }
32
}
33
~]# etcdctl set /docker/network/config < flannel-config.json
34
~]# etcdctl get /docker/network/confi
35
{
36
"Network":"10.2.0.0/16",
37
"SubnetLen":24,
38
"Backend": {
39
 "Type": "vxlan"
40
}
41
}

2.配置flannel agent

host1:

1
~]# wget https://github.com/coreos/flannel/releases/download/v0.8.0/flannel-v0.8.0-linux-amd64.tar.gz
2
~]# tar xf flannel-v0.8.0-linux-amd64.tar.gz -C /usr/local/bin/
3
~]# vim /usr/lib/systemd/system/flanneld.service
4
[Unit]
5
Description=Flanneld overlay address etcd agent
6
After=network.target
7
After=network-online.target
8
Wants=network-online.target
9
After=etcd.service
10
Before=docker.service
11
12
[Service]
13
Type=notify
14
ExecStart=/usr/local/bin/flanneld \
15
-etcd-endpoints=http://10.211.55.16:2379 \
16
-etcd-prefix=/docker/network \
17
-iface=eth0
18
ExecStartPost=/usr/local/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker
19
Restart=on-failure
20
21
[Install]
22
WantedBy=multi-user.target
23
RequiredBy=docker.service
24
~]# systemctl start flanneld

host2也进行相同的配置即可

3.查看网络变化

host1:

1
~]# ip a show flannel.1
2
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN 
3
    link/ether 32:f7:c6:54:52:ee brd ff:ff:ff:ff:ff:ff
4
    inet 10.2.71.0/32 scope global flannel.1
5
       valid_lft forever preferred_lft forever
6
    inet6 fe80::30f7:c6ff:fe54:52ee/64 scope link 
7
       valid_lft forever preferred_lft forever
8
# 一个新的interface flannel.1被新建,而且被分配了ip
9
~]# ip r
10
....
11
10.2.0.0/16 dev flannel.1 
12
....
13
# 自动添加了一条route,来之10.2.0.0/16的网络数据包都往这个接口走

4.配置docker连接flannel
host1:

1
~]# cat /usr/lib/systemd/system/docker.service
2
...
3
ExecStart=/usr/bin/dockerd --bip=10.2.71.1/24 --mtu=1450
4
... 
5
#这里的配置必须与/run/flannel/subnet.env中的FLANNEL_SUBNET和FLANNEL_MTU一致
6
~]# cat /run/flannel/subnet.env
7
FLANNEL_NETWORK=10.2.0.0/16
8
FLANNEL_SUBNET=10.2.71.1/24
9
FLANNEL_MTU=1450
10
FLANNEL_IPMASQ=false
11
~]# systemctl daemon-reload
12
~]# systemctl start docker.service

5.查看网络变化

1
~]# ip a show docker0
2
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
3
    link/ether 02:42:ab:10:a0:7d brd ff:ff:ff:ff:ff:ff
4
    inet 10.2.71.1/24 brd 10.2.71.255 scope global docker0
5
       valid_lft forever preferred_lft forever
6
    inet6 fe80::42:abff:fe10:a07d/64 scope link 
7
       valid_lft forever preferred_lft forever
8
~]# ip r
9
...
10
10.2.0.0/16 dev flannel.1 
11
10.2.71.0/24 dev docker0 proto kernel scope link src 10.2.71.1 
12
...

6.将容器连接到flannel网络

host1:

1
~]# docker run -itd --name bbox1 buxybox
2
~]# docker exec bbox1 ip r
3
...
4
default via 10.2.71.1 dev eth0 
5
10.2.71.0/24 dev eth0 scope link  src 10.2.71.2
6
...

host2:

1
~]# docker run -itd --name bbox2 buxybox
2
~]# docker exec bbox2 ip r
3
default via 10.2.81.1 dev eth0 
4
10.2.81.0/24 dev eth0 scope link  src 10.2.81.2 
5
~]# docker exec bbox2 ping 10.2.71.2
6
64 bytes from 10.2.71.2: seq=0 ttl=62 time=0.874 ms
7
64 bytes from 10.2.71.2: seq=1 ttl=62 time=0.686 ms
8
64 bytes from 10.2.71.2: seq=2 ttl=62 time=0.573 ms
9
....
10
~]# docker exec bbox2 traceroute 10.2.71.2
11
traceroute to 10.2.71.2 (10.2.71.2), 30 hops max, 46 byte packets
12
 1  10.2.81.1 (10.2.81.1)  0.007 ms  0.010 ms  0.003 ms
13
 2  10.2.71.0 (10.2.71.0)  0.241 ms  0.318 ms  0.185 ms
14
 3  10.2.71.2 (10.2.71.2)  0.177 ms  0.304 ms  0.196 ms
  1. bbox1与bbox2不是一个subnet,数据包发送给默认网关10.2.81.1
  2. 根据host2的路由表,数据包会发给flannel.1
  3. flannel.1将数据包封装成vxvlan,通过eth0发送给host1
  4. host1收到包解封,发现数据包的目的是10.2.71.2,根据路由表将数据包发给flannel.1,并通过docker0到底bbox1

flannel为每个主机分配独立的subnet,但flannel.1将这些subnet连接起来,相互路由。本质上,flannel将各主机上相互独立的docker0容器网络组成了一个互通的大网络,flannel没有提供隔离网络。

fannel host-gw演示

与vxlan不同,host-gw不会封装数据包,而是在主机的路由表中创建到其他主机subnet的路由条目,从而实现容器的跨主机通信。

1.修改flannel-config.json

1
~]# cat flannel-config.json
2
{
3
    "Network":"10.2.0.0/16",
4
    "SubnetLen":24,
5
    "Backend":{
6
        "Type":"host-gw"
7
    }
8
}
9
~]# etcdctl set /docker/network/config < flannel-config.json

2.重启flanneld
host1:

1
~]# systemctl restart flanned
2
~}# ip r
3
....
4
10.2.0.0/16 dev flannel.1
5
10.2.81.0/24 via 10.211.55.18 dev eth0
6
....

3.host-gw的mtu改成了1500,重启docker
host1:

1
~]# cat /run/flannel/subnet.env 
2
FLANNEL_NETWORK=10.2.0.0/16
3
FLANNEL_SUBNET=10.2.71.1/24
4
FLANNEL_MTU=1500
5
FLANNEL_IPMASQ=false

3.测试网络

1
~]# docker exec bbox2 traceroute 10.2.71.2
2
traceroute to 10.2.71.2 (10.2.71.2), 30 hops max, 46 byte packets
3
 1  10.2.81.1 (10.2.81.1)  0.006 ms  0.010 ms  0.005 ms
4
 2  10.211.55.17 (10.211.55.17)  0.228 ms  0.288 ms  0.165 ms
5
 3  10.2.71.2 (10.2.71.2)  0.284 ms  0.262 ms  0.167 ms
  1. bbox1与bbox2不是一个subnet,数据包发送给默认网关10.2.81.1
  2. 查看路由10.2.71.0/24的网段地址发往10.211.55.17(host1)通过eth0
  3. host1收到包解封,发现数据包的目的是10.2.71.2,根据路由表将数据包发给flannel.1,并通过docker0到底bbox1

host-gw和vxlan比较

  1. host-gw 把每个主机都配置成网关,主机知道其他主机的 subnet 和转发地址。vxlan 则在主机间建立隧道,不同主机的容器都在一个大的网段内(比如 10.2.0.0/16)。

  2. 虽然 vxlan 与 host-gw 使用不同的机制建立主机之间连接,但对于容器则无需任何改变,bbox1 仍然可以与 bbox2 通信。

  3. 由于 vxlan 需要对数据进行额外打包和拆包,性能会稍逊于 host-gw。

CATALOG
  1. 1. flannel
  2. 2. flannel vxlan演示
  3. 3. fannel host-gw演示