Jusene's Blog

HTTP动态负载均衡

字数统计: 1.6k阅读时长: 8 min
2018/11/11 Share

nginx负载均衡配置可以通过upstream实现,但是我们知道每次改变后端服务就需要手工修改配置文件,首先是管理麻烦且容易出错,而且对于upstream服务上线不能自动注册到nginx upstream列表。因此,我们需要一种服务发现,可以自动注册到nginx上,从而实现upstream服务的自动发现。

常见的方案有:

  • Consul+Consul-template
  • Etcd+Confd

Consul-Template是Consul官方提供,而且Consul还提供了多数据中心,故障检测,web管理界面,所以我就选这个方案来完成。

consul是一款开源的分布式服务注册和发现系统,通过http api可以使得服务注册、发现实现起来非常简单:

  • 服务注册: 服务实现者可以通过HTTP API或DNS方式,将服务注册到consul
  • 服务发现: 服务消费者可以通过HTTP API或DNS方式,从Consul获取服务的IP和PORT
  • 故障检测: 支持如TCP、HTTP等方式的健康检查机制,从而当服务有故障时自动摘除
  • K/V存储: 使用K/V存储实现动态配置中心,其使用HTTP长轮询实现变更触发和配置更改
  • 多数据中心: 支持多数据中心,可以按照数据中心注册和服务发现,即支持只消费本地机房服务,使用多数据中心集群还可以避免数据中心的单点故障
  • Raft算法: 使用Raft算法实现集群数据一致性

架构图:

Consul Server

1
 ~]# ./consul agent -server -bootstrap-expect 1 -bind=10.211.55.16 -data-dir=. -advertise=10.211.55.16 
2
BootstrapExpect is set to 1; this is the same as Bootstrap mode.
3
bootstrap = true: do not enable unless necessary
4
==> Starting Consul agent...
5
==> Consul agent running!
6
           Version: 'v1.2.1'
7
           Node ID: 'e60c134d-8cfb-fc4f-b47e-c1b27ba9a8be'
8
         Node name: 'node1'
9
        Datacenter: 'dc1' (Segment: '<all>')
10
            Server: true (Bootstrap: true)
11
       Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, DNS: 8600)
12
      Cluster Addr: 10.211.55.16 (LAN: 8301, WAN: 8302)
13
           Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
14
15
==> Log data will now stream in as it occurs:
16
17
    2018/09/18 20:11:58 [INFO] raft: Initial configuration (index=1): [{Suffrage:Voter ID:e60c134d-8cfb-fc4f-b47e-c1b27ba9a8be Address:10.211.55.16:8300}]
18
    2018/09/18 20:11:58 [INFO] raft: Node at 10.211.55.16:8300 [Follower] entering Follower state (Leader: "")
19
    2018/09/18 20:11:58 [INFO] serf: EventMemberJoin: node1.dc1 10.211.55.16
20
    2018/09/18 20:11:58 [INFO] serf: EventMemberJoin: node1 10.211.55.16
21
    2018/09/18 20:11:58 [INFO] consul: Adding LAN server node1 (Addr: tcp/10.211.55.16:8300) (DC: dc1)
22
    2018/09/18 20:11:58 [INFO] agent: Started DNS server 127.0.0.1:8600 (udp)
23
    2018/09/18 20:11:58 [INFO] consul: Handled member-join event for server "node1.dc1" in area "wan"
24
    2018/09/18 20:11:58 [WARN] agent/proxy: running as root, will not start managed proxies
25
    2018/09/18 20:11:58 [INFO] agent: Started DNS server 127.0.0.1:8600 (tcp)
26
    2018/09/18 20:11:58 [INFO] agent: Started HTTP server on 127.0.0.1:8500 (tcp)
27
    2018/09/18 20:11:58 [INFO] agent: started state syncer
28
    2018/09/18 20:12:05 [WARN] raft: Heartbeat timeout from "" reached, starting election
29
    2018/09/18 20:12:05 [INFO] raft: Node at 10.211.55.16:8300 [Candidate] entering Candidate state in term 2
30
    2018/09/18 20:12:05 [INFO] raft: Election won. Tally: 1
31
    2018/09/18 20:12:05 [INFO] raft: Node at 10.211.55.16:8300 [Leader] entering Leader state
32
    2018/09/18 20:12:05 [INFO] consul: cluster leadership acquired
33
    2018/09/18 20:12:05 [INFO] consul: New leader elected: node1
34
    2018/09/18 20:12:05 [INFO] consul: member 'node1' joined, marking health alive
35
    2018/09/18 20:12:05 [INFO] agent: Synced node info

注册服务

1
~]# curl -XPUT http://127.0.0.1:8500/v1/catalog/register -d '
2
{"Datacenter": "dc1",
3
"Node": "nginx",
4
"Address": "127.0.0.1",
5
"Service": {
6
        "Id": "127.0.0.1:81",
7
        "Service": "test_nginx",
8
        "tags": ["dev","upstream"],
9
        "Port": 81
10
    }
11
}'
12
~]# curl -XPUT http://127.0.0.1:8500/v1/catalog/register -d '
13
{"Datacenter": "dc1",
14
"Node": "nginx",
15
"Address": "127.0.0.1",
16
"Service": {
17
        "Id": "127.0.0.1:82",
18
        "Service": "test_nginx",
19
        "tags": ["dev","upstream"],
20
        "Port": 82
21
    }
22
}'

Datacenter指定数据中心,Address指定服务IP,Service.Id指定服务唯一标识,Service.Service指定服务分组,Service.tags指定服务标签,Service.Port指定服务端口

通过如下摘除服务:

1
~]# curl -XPUT http://127.0.0.1:8500/v1/catalog/deregister -d '
2
{"Datacenter": "dc1",
3
"Node": "nginx",
4
"ServiceID": "127.0.0.1:82"
5
}'

通过HTTP API发现服务:

1
~]# curl http://127.0.0.1:8500/v1/catalog/service/test_nginx

Consule-template

1
~]# cat item.test.upstream.tmp
2
upstream test_upstream {
3
   
4
    {{range service "dev.test_nginx@dc1"}}
5
        server {{.Address}}:{{.Port}} weight=1;
6
    {{end}}
7
}

service 指定格式为: 标签.服务@数据中心,然后通过循环输出Address和Port,从而生成Consul-template配置。

1
~]# ./consul-template -consul-addr 127.0.0.1:8500 -template ./item.test.upstream.tmp:/etc/nginx/conf.d/test.upstream.conf:"nginx -s reload"
2
~]# cat /etc/nginx/conf.d/test.upstream.conf
3
upstream test_upstream {
4
    
5
    
6
        server 127.0.0.1:81 weight=1;
7
    
8
        server 127.0.0.1:82 weight=1;
9
    
10
}
11
~]# curl -XPUT http://127.0.0.1:8500/v1/catalog/deregister -d '
12
{"Datacenter": "dc1",
13
"Node": "nginx",
14
"ServiceID": "127.0.0.1:82"
15
}'
16
~]# cat /etc/nginx/conf.d/test.upstream.conf                       
17
upstream test_upstream {
18
    
19
        server 127.0.0.1:81 weight=1;
20
    
21
}

upsync-module

nginx-stream-upsync-module提供四层动态负载均衡,nginx-upstream-module提供七层动态负载均衡,动态更新上游服务器不需要reload nginx。

首先编译需要加入module:
nginx-stream-upsync-module: https://github.com/xiaokai-wang/nginx-stream-upsync-module
nginx-upstream-module: https://github.com/weibocom/nginx-upsync-module

./configure --prefix=/usr/local/nginx --with-stream --add-module=./nginx-stream-upsync-module

  1. upstream配置
1
upstream db_backend {
2
    upsync 127.0.0.1:8500/v1/kv/upstreams/db_backend upsync_timeout=6m upsync_interval=500ms upsync_type=consul strong_dependency=off;
3
    upsync_dump_path /tmp/db_backend.conf;
4
}

upsync指令指定consul哪个路径下拉取上游服务器配置:upsync_timeout配置拉取上游服务器的配置的超市时间;upsync_interval配置从consul拉取上游服务器配置的间隔时间;upsync_type指定使用consul配置服务器;strong_dependency配置nginx在启动时是否强制依赖配置服务器;upsync_dump_path指定从consul拉取的上游服务器后持久化到的位置,这样即使consul服务器出现问题,本地还是有备份的。

  1. 从consul添加上游服务器
1
~]# curl -XPUT -d '{"weight": 1, "max_fails": 2, "fail_timeout": 10}' http://127.0.0.1:8500/v1/kv/upstreams/db_backend/127.0.0.1:3306
2
~]# curl -XPUT -d '{"weight": 1, "max_fails": 2, "fail_timeout": 10}' http://127.0.0.1:8500/v1/kv/upstreams/db_backend/127.0.0.1:3307
  1. 从consul删除上游服务器

    1
    ~]# curl -X DELETE http://127.0.0.1:8500/v1/kv/upstreams/db_backend/127.0.0.1:3306
  2. upstream_show

    1
    server {
    2
        listen 1234;
    3
        upstream_show;
    4
    }

配置upstream_show指令后,可以通过curl 127.0.0.1:1234/upstream_show来查看当前动态负载均衡上游服务器列表。

CATALOG
  1. 1. Consul Server
  2. 2. Consule-template
  3. 3. upsync-module