nginx负载均衡配置可以通过upstream实现,但是我们知道每次改变后端服务就需要手工修改配置文件,首先是管理麻烦且容易出错,而且对于upstream服务上线不能自动注册到nginx upstream列表。因此,我们需要一种服务发现,可以自动注册到nginx上,从而实现upstream服务的自动发现。
常见的方案有:
- Consul+Consul-template
- Etcd+Confd
Consul-Template是Consul官方提供,而且Consul还提供了多数据中心,故障检测,web管理界面,所以我就选这个方案来完成。
consul是一款开源的分布式服务注册和发现系统,通过http api可以使得服务注册、发现实现起来非常简单:
- 服务注册: 服务实现者可以通过HTTP API或DNS方式,将服务注册到consul
- 服务发现: 服务消费者可以通过HTTP API或DNS方式,从Consul获取服务的IP和PORT
- 故障检测: 支持如TCP、HTTP等方式的健康检查机制,从而当服务有故障时自动摘除
- K/V存储: 使用K/V存储实现动态配置中心,其使用HTTP长轮询实现变更触发和配置更改
- 多数据中心: 支持多数据中心,可以按照数据中心注册和服务发现,即支持只消费本地机房服务,使用多数据中心集群还可以避免数据中心的单点故障
- Raft算法: 使用Raft算法实现集群数据一致性
架构图:
Consul Server
1 | ~]# ./consul agent -server -bootstrap-expect 1 -bind=10.211.55.16 -data-dir=. -advertise=10.211.55.16 |
2 | BootstrapExpect is set to 1; this is the same as Bootstrap mode. |
3 | bootstrap = true: do not enable unless necessary |
4 | ==> Starting Consul agent... |
5 | ==> Consul agent running! |
6 | Version: 'v1.2.1' |
7 | Node ID: 'e60c134d-8cfb-fc4f-b47e-c1b27ba9a8be' |
8 | Node name: 'node1' |
9 | Datacenter: 'dc1' (Segment: '<all>') |
10 | Server: true (Bootstrap: true) |
11 | Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, DNS: 8600) |
12 | Cluster Addr: 10.211.55.16 (LAN: 8301, WAN: 8302) |
13 | Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false |
14 | |
15 | ==> Log data will now stream in as it occurs: |
16 | |
17 | 2018/09/18 20:11:58 [INFO] raft: Initial configuration (index=1): [{Suffrage:Voter ID:e60c134d-8cfb-fc4f-b47e-c1b27ba9a8be Address:10.211.55.16:8300}] |
18 | 2018/09/18 20:11:58 [INFO] raft: Node at 10.211.55.16:8300 [Follower] entering Follower state (Leader: "") |
19 | 2018/09/18 20:11:58 [INFO] serf: EventMemberJoin: node1.dc1 10.211.55.16 |
20 | 2018/09/18 20:11:58 [INFO] serf: EventMemberJoin: node1 10.211.55.16 |
21 | 2018/09/18 20:11:58 [INFO] consul: Adding LAN server node1 (Addr: tcp/10.211.55.16:8300) (DC: dc1) |
22 | 2018/09/18 20:11:58 [INFO] agent: Started DNS server 127.0.0.1:8600 (udp) |
23 | 2018/09/18 20:11:58 [INFO] consul: Handled member-join event for server "node1.dc1" in area "wan" |
24 | 2018/09/18 20:11:58 [WARN] agent/proxy: running as root, will not start managed proxies |
25 | 2018/09/18 20:11:58 [INFO] agent: Started DNS server 127.0.0.1:8600 (tcp) |
26 | 2018/09/18 20:11:58 [INFO] agent: Started HTTP server on 127.0.0.1:8500 (tcp) |
27 | 2018/09/18 20:11:58 [INFO] agent: started state syncer |
28 | 2018/09/18 20:12:05 [WARN] raft: Heartbeat timeout from "" reached, starting election |
29 | 2018/09/18 20:12:05 [INFO] raft: Node at 10.211.55.16:8300 [Candidate] entering Candidate state in term 2 |
30 | 2018/09/18 20:12:05 [INFO] raft: Election won. Tally: 1 |
31 | 2018/09/18 20:12:05 [INFO] raft: Node at 10.211.55.16:8300 [Leader] entering Leader state |
32 | 2018/09/18 20:12:05 [INFO] consul: cluster leadership acquired |
33 | 2018/09/18 20:12:05 [INFO] consul: New leader elected: node1 |
34 | 2018/09/18 20:12:05 [INFO] consul: member 'node1' joined, marking health alive |
35 | 2018/09/18 20:12:05 [INFO] agent: Synced node info |
注册服务
1 | ~]# curl -XPUT http://127.0.0.1:8500/v1/catalog/register -d ' |
2 | {"Datacenter": "dc1", |
3 | "Node": "nginx", |
4 | "Address": "127.0.0.1", |
5 | "Service": { |
6 | "Id": "127.0.0.1:81", |
7 | "Service": "test_nginx", |
8 | "tags": ["dev","upstream"], |
9 | "Port": 81 |
10 | } |
11 | }' |
12 | ~]# curl -XPUT http://127.0.0.1:8500/v1/catalog/register -d ' |
13 | {"Datacenter": "dc1", |
14 | "Node": "nginx", |
15 | "Address": "127.0.0.1", |
16 | "Service": { |
17 | "Id": "127.0.0.1:82", |
18 | "Service": "test_nginx", |
19 | "tags": ["dev","upstream"], |
20 | "Port": 82 |
21 | } |
22 | }' |
Datacenter指定数据中心,Address指定服务IP,Service.Id指定服务唯一标识,Service.Service指定服务分组,Service.tags指定服务标签,Service.Port指定服务端口
通过如下摘除服务:
1 | ~]# curl -XPUT http://127.0.0.1:8500/v1/catalog/deregister -d ' |
2 | {"Datacenter": "dc1", |
3 | "Node": "nginx", |
4 | "ServiceID": "127.0.0.1:82" |
5 | }' |
通过HTTP API发现服务:
1 | ~]# curl http://127.0.0.1:8500/v1/catalog/service/test_nginx |
Consule-template
1 | ~]# cat item.test.upstream.tmp |
2 | upstream test_upstream { |
3 | |
4 | {{range service "dev.test_nginx@dc1"}} |
5 | server {{.Address}}:{{.Port}} weight=1; |
6 | {{end}} |
7 | } |
service 指定格式为: 标签.服务@数据中心,然后通过循环输出Address和Port,从而生成Consul-template配置。
1 | ~]# ./consul-template -consul-addr 127.0.0.1:8500 -template ./item.test.upstream.tmp:/etc/nginx/conf.d/test.upstream.conf:"nginx -s reload" |
2 | ~]# cat /etc/nginx/conf.d/test.upstream.conf |
3 | upstream test_upstream { |
4 | |
5 | |
6 | server 127.0.0.1:81 weight=1; |
7 | |
8 | server 127.0.0.1:82 weight=1; |
9 | |
10 | } |
11 | ~]# curl -XPUT http://127.0.0.1:8500/v1/catalog/deregister -d ' |
12 | {"Datacenter": "dc1", |
13 | "Node": "nginx", |
14 | "ServiceID": "127.0.0.1:82" |
15 | }' |
16 | ~]# cat /etc/nginx/conf.d/test.upstream.conf |
17 | upstream test_upstream { |
18 | |
19 | server 127.0.0.1:81 weight=1; |
20 | |
21 | } |
upsync-module
nginx-stream-upsync-module提供四层动态负载均衡,nginx-upstream-module提供七层动态负载均衡,动态更新上游服务器不需要reload nginx。
首先编译需要加入module:
nginx-stream-upsync-module: https://github.com/xiaokai-wang/nginx-stream-upsync-module
nginx-upstream-module: https://github.com/weibocom/nginx-upsync-module
./configure --prefix=/usr/local/nginx --with-stream --add-module=./nginx-stream-upsync-module
- upstream配置
1 | upstream db_backend { |
2 | upsync 127.0.0.1:8500/v1/kv/upstreams/db_backend upsync_timeout=6m upsync_interval=500ms upsync_type=consul strong_dependency=off; |
3 | upsync_dump_path /tmp/db_backend.conf; |
4 | } |
upsync指令指定consul哪个路径下拉取上游服务器配置:upsync_timeout配置拉取上游服务器的配置的超市时间;upsync_interval配置从consul拉取上游服务器配置的间隔时间;upsync_type指定使用consul配置服务器;strong_dependency配置nginx在启动时是否强制依赖配置服务器;upsync_dump_path指定从consul拉取的上游服务器后持久化到的位置,这样即使consul服务器出现问题,本地还是有备份的。
- 从consul添加上游服务器
1 | ~]# curl -XPUT -d '{"weight": 1, "max_fails": 2, "fail_timeout": 10}' http://127.0.0.1:8500/v1/kv/upstreams/db_backend/127.0.0.1:3306 |
2 | ~]# curl -XPUT -d '{"weight": 1, "max_fails": 2, "fail_timeout": 10}' http://127.0.0.1:8500/v1/kv/upstreams/db_backend/127.0.0.1:3307 |
从consul删除上游服务器
1
~]# curl -X DELETE http://127.0.0.1:8500/v1/kv/upstreams/db_backend/127.0.0.1:3306
upstream_show
1
server {
2
listen 1234;
3
upstream_show;
4
}
配置upstream_show指令后,可以通过curl 127.0.0.1:1234/upstream_show来查看当前动态负载均衡上游服务器列表。