Jusene's Blog

FastDFS分布式存储实现

fastdfs
字数统计: 3.6k阅读时长: 17 min
2017/07/22 Share

FastDFS

FastDFS是一个开源的轻量级分布式文件系统，整体功能与MogileFS十分相似，由C编写，无论在性能，冗余备份，负载均衡等方面都比MogileFS提升了不少，整个文件系统也只可以通过API进行访问。

  • Storage server

storage server以组为单位组织，一个group内包含多台storage机器，数据互为备份，存储空间以group内容量最小的storage机器为主。以group为单位组织存储能方便的进行应用隔离、负载均衡、副本数定制(group内storage server数量即为该group的副本数)，比如将不同应用数据存到不同的group就能隔离应用数据，同时还可以根据应用的访问特性来将应用的访问特性来将应用分配到不同的group来做负载均衡；缺点是group的容量受单机存储容量的限制，同时group内的机器坏掉时，数据恢复只能依赖group内的其他机器，使得恢复时间会很长。

group内的每个storage server的存储依赖于本地文件系统，storage可配置多个数据存储目录。storage server接受写文件的请求时，会根据配置好的规则，选择其中一个存储目录来存储文件，为了避免每个目录下的文件数过多，在storage第一次启动时，会在每个数据存储目录下创建2级子目录，每级256个，总共65535个文件，新写的文件以hash的方式被路由到其中某个子目录下，然后将文件数据直接作为一个本地存储到该目录中。

  • Tracker server

Tracker是FasrDFS的协调者，负责管理所有的storage server和group，每个storage server启动后会连接Tracker,告知自己所属的group等信息，并保持周期性的心跳，tracker根据storage server的心跳信息，建立group和storage server的映射表。

Tracker需要管理的元信息很少，会全部存储在内存中；另外tracker上的元信息都是由storage server汇报的信息生成的，本身不需要持久化任何数据，这样使得tracker非常容易扩展，直接增加tracker机器即可扩展为tracker cluster来服务，cluster里每个tracker之间是完全对等的，所有的tracker都接受stroage的心跳信息，生成元数据信息来提供读写服务。

  • Client

客户端，业务请求发起方，通过专用接口基于tcp协议与tracker以及storage server进行交互,存储返回的file_id。

Upload file

  1. 选择tracker server
    当集群中不止一个tracker server时，由于tracker之间完全对等的关系，客户端在upload文件时可以任意选择一个tracker

  2. 选择存储group
    当tracker受到uplode file的请求时，会为该文件分配一个可以存储文件的group：1.Round robin 2.Specified group 3.Load balance

  3. 选择存储的storage server
    当选定group后，tracker会在group内选择storage server给客户端：1.Round robin 2.First server ordered by ip 3.First server ordered by priority

  4. 选择storage path
    当分配好storage server后，客户端将storage发送写文件请求，storage将会为文件分配一个数据存储目录：1.Round robin 2.剩余空间最多优先

  5. 生产Fileid
    选定存储目录之后，storage会为文件生成Fileid,由storage server ip、文件创建时间、文件大小、文件的crc32和一个随机数拼接而成，然后将这个二进制进行base64编码，转换成可打印的字符。

  6. 选择两级目录
    当选定存储目录之后，storage会为文件分配一个field，每个存储目录下有两级256x256的子目录，storage会按照fileld进行两次hash，路由到其中一个子目录，然后将文件以field为文件名存储到该子目录下。

  7. 生成文件名
    当文件存储到某一个子目录下后，即为该文件存储成功，接下来会为该文件生成一个文件名，文件名由group、存储目录、两级子目录、field、文件后缀名拼接而成。

  8. 文件同步
    写文件时，客户端将文件写至group内的一个storage server即为写文件成功，storage server写文件后，会由后台线程将文件同步至同group内的其他storage server，每个storage写文件后，同时会写一份binlog，binlog里不包含文件数据，只包含文件名等元信息，这份binlog用于后台同步，storage会记录向group内其他storage同步的进度，以便重启后能接上次的进度继续同步；进度以时间戳的方式进行记录，所以最好能保证集群内所有server的时钟保持同步。 storage的同步进度会作为元数据的一部分汇报到tracker上，tracke在选择读storage的时候会以同步进度作为参考。

Download file

跟upload file一样，在download file时客户端可以选择任意tracker server。

tracker发送download请求给某个tracker，必须带上文件名信息，tracke从文件名中解析出文件的group、大小、创建时间等信息，然后为该请求选择一个storage用来服务读请求。由于group内的文件同步时在后台异步进行的，所以有可能出现在读到时候，文件还没有同步到某些storage server上，为了尽量避免访问到这样的storage，tracker按照如下规则选择group内可读的storage。

  1. 该文件上传到的源头storage - 源头storage只要存活着，肯定包含这个文件，源头的地址被编码在文件名中。 2. 文件创建时间戳==storage被同步到的时间戳 且(当前时间-文件创建时间戳) > 文件同步最大时间（如5分钟) - 文件创建后，认为经过最大同步时间后，肯定已经同步到其他storage了。 3. 文件创建时间戳 < storage被同步到的时间戳。 - 同步时间戳之前的文件确定已经同步了 4. (当前时间-文件创建时间戳) > 同步延迟阀值（如一天）。 - 经过同步延迟阈值时间，认为文件肯定已经同步了。

FastDFS实例

  • 10.211.55.24 nginx tracker
  • 10.211.55.25 storage1
  • 10.211.55.26 storage2

FastDFS下载链接：https://github.com/happyfish100/fastdfs
FastDFS依赖链接：https://github.com/happyfish100/libfastcommon.git

  • 安装tracker
1
~]# git clone https://github.com/happyfish100/libfastcommon.git
2
~]# cd libfastcommon
3
~]# ./make.sh
4
~]# ./make.sh install
5
~]# git clone https://github.com/happyfish100/fastdfs
6
~]# cd fastdfs
7
~]# ./make.sh
8
~]# ./make.sh install
9
~]# cp /etc/fdfs/tracker.comf.sample /etc/fdfs/tracker.conf
10
~]# cat /etc/fdfs/tracker.conf
11
disabled=false
12
bind_addr=0.0.0.0
13
port=22122
14
connect_timeout=30
15
network_timeout=60
16
base_path=/data/fdfs/tracker      #设置tracker的数据目录和日志文件
17
max_connections=256
18
accept_threads=1                  #接受请求的线程
19
work_threads=4                    #工作的线程
20
min_buff_size = 8KB
21
max_buff_size = 128KB
22
store_lookup=2                    #上传文件选择组的规则  0.轮询 1.指定组 2.选择最多空闲空间的组
23
store_group=group2				  #选择1时，指定的组
24
store_server=0                    #上传文件选择storage server 0.轮询 1.根据ip排序的第一台 2.根据优先级，越小越优先
25
store_path=2					  #上传文件选择的storage的挂载点  0.轮询  2.负载均衡，选择最大的容量的上传
26
download_server=0                 #下载文件选择storage server  0.轮询  1.上传到那台storage server，从那台下载
27
reserved_storage_space = 10%      #存储挂在点预留多少空间来为系统或者其他应用
28
log_level=info                    
29
run_by_group=tracker              #进程以哪个用户组运行
30
run_by_user=tracker               #进程以哪个用户运行
31
allow_hosts=*                     #访问控制
32
sync_log_buff_interval = 10       #同步日志的频率
33
check_active_interval = 120       #检测storage server的存活频率
34
thread_stack_size = 64KB          
35
storage_ip_changed_auto_adjust = true
36
storage_sync_file_max_delay = 86400
37
storage_sync_file_max_time = 300
38
use_trunk_file = false            #是否裁剪文件
39
slot_min_size = 256
40
slot_max_size = 16MB
41
trunk_file_size = 64MB
42
trunk_create_file_advance = false
43
trunk_create_file_time_base = 02:00
44
trunk_create_file_interval = 86400
45
trunk_create_file_space_threshold = 20G
46
trunk_init_check_occupying = false
47
trunk_init_reload_from_binlog = false
48
trunk_compress_binlog_min_interval = 0
49
use_storage_id = false
50
storage_ids_filename = storage_ids.conf
51
id_type_in_filename = ip
52
store_slave_file_use_link = false
53
rotate_error_log = false
54
error_log_rotate_time=00:00
55
rotate_error_log_size = 0
56
log_file_keep_days = 0
57
use_connection_pool = false
58
connection_pool_max_idle_time = 3600
59
http.server_port=8080
60
http.check_alive_interval=30
61
http.check_alive_type=tcp
62
http.check_alive_uri=/status.html
63
~]# service fdfs_trackerd start
  • 安装storage
1
~]# cp /etc/fdfs/storage.conf.sample /etc/fdfs/storage.conf
2
~]# cat /etc/fdfs/storage.conf
3
disabled=false
4
group_name=group1                  #定义组的名字
5
bind_addr=0.0.0.0
6
client_bind=true
7
port=23000
8
connect_timeout=30
9
network_timeout=60
10
heart_beat_interval=30             #心跳信息频率
11
stat_report_interval=60            #磁盘用量信息频率
12
base_path=/data/fdfs/storage       #日志目录
13
max_connections=256
14
buff_size = 256KB
15
accept_threads=1
16
work_threads=4
17
disk_rw_separated = true           #读写分离
18
disk_reader_threads = 1
19
disk_writer_threads = 1
20
sync_wait_msec=50                  #当无法同步文件时，隔多久重试
21
sync_interval=0
22
sync_start_time=00:00
23
sync_end_time=23:59
24
write_mark_file_freq=500
25
store_path_count=1
26
store_path0=/data/fdfs/storage/0   #存储路径
27
subdir_count_per_path=256
28
tracker_server=10.211.55.24:22122
29
log_level=info
30
run_by_group=storage
31
run_by_user=storage
32
allow_hosts=*
33
file_distribute_path_mode=0        #文件分布的模式 0.轮询  1.按照hash随机分布
34
file_distribute_rotate_count=100   #当模式为0，分布的文件到达这个值，会会滚到下一个路径
35
fsync_after_written_bytes=0
36
sync_log_buff_interval=10
37
sync_binlog_buff_interval=10
38
sync_stat_file_interval=300
39
thread_stack_size=512KB
40
upload_priority=10                 #上传的优先级
41
if_alias_prefix=
42
check_file_duplicate=0
43
file_signature_method=hash
44
key_namespace=FastDFS
45
keep_alive=0
46
use_access_log = false
47
rotate_access_log = false
48
access_log_rotate_time=00:00
49
rotate_error_log = false
50
error_log_rotate_time=00:00
51
rotate_access_log_size = 0
52
rotate_error_log_size = 0
53
log_file_keep_days = 0
54
file_sync_skip_invalid_record=false
55
use_connection_pool = false
56
connection_pool_max_idle_time = 3600
57
http.domain_name=
58
http.server_port=8888
59
~]# service fdfs_storaged start
  • 查看状态
1
~]# fdfs_monitor /etc/fdfs/storage.conf
2
[2017-06-10 19:50:47] DEBUG - base_path=/data/fdfs/storage, connect_timeout=30, network_timeout=60, tracker_server_count=1, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0
3
4
server_count=1, server_index=0
5
6
tracker server is 10.211.55.24:22122
7
8
group count: 1
9
10
Group 1:
11
group name = group1
12
disk total space = 50268 MB
13
disk free space = 42976 MB
14
trunk free space = 0 MB
15
storage server count = 2
16
active server count = 2
17
storage server port = 23000
18
storage HTTP port = 8888
19
store path count = 1
20
subdir count per path = 256
21
current write server index = 0
22
current trunk file id = 0
23
24
        Storage 1:
25
                id = 10.211.55.25
26
                ip_addr = 10.211.55.25 (node1)  ACTIVE
27
                http domain = 
28
                version = 5.11
29
                join time = 2017-06-10 18:22:00
30
                up time = 2017-06-10 18:22:00
31
                total storage = 50268 MB
32
                free storage = 42976 MB
33
                upload priority = 10
34
                store_path_count = 1
35
                subdir_count_per_path = 256
36
                storage_port = 23000
37
                storage_http_port = 8888
38
                current_write_path = 0
39
                source storage id = 10.211.55.25
40
                if_trunk_server = 0
41
                connection.alloc_count = 256
42
                connection.current_count = 1
43
                connection.max_count = 1
44
                total_upload_count = 0
45
                success_upload_count = 0
46
                total_append_count = 0
47
                success_append_count = 0
48
                total_modify_count = 0
49
                success_modify_count = 0
50
                total_truncate_count = 0
51
                success_truncate_count = 0
52
                total_set_meta_count = 0
53
                success_set_meta_count = 0
54
                total_delete_count = 0
55
                success_delete_count = 0
56
                total_download_count = 0
57
                success_download_count = 0
58
                total_get_meta_count = 0
59
                success_get_meta_count = 0
60
                total_create_link_count = 0
61
                success_create_link_count = 0
62
                total_delete_link_count = 0
63
                success_delete_link_count = 0
64
                total_upload_bytes = 0
65
                success_upload_bytes = 0
66
                total_append_bytes = 0
67
                success_append_bytes = 0
68
                total_modify_bytes = 0
69
                success_modify_bytes = 0
70
                stotal_download_bytes = 0
71
                success_download_bytes = 0
72
                total_sync_in_bytes = 0
73
                success_sync_in_bytes = 0
74
                total_sync_out_bytes = 0
75
                success_sync_out_bytes = 0
76
                total_file_open_count = 0
77
                success_file_open_count = 0
78
                total_file_read_count = 0
79
                success_file_read_count = 0
80
                total_file_write_count = 0
81
                success_file_write_count = 0
82
                last_heart_beat_time = 2017-06-10 19:50:43
83
                last_source_update = 1969-12-31 19:00:00
84
                last_sync_update = 1969-12-31 19:00:00
85
                last_synced_timestamp = 1969-12-31 19:00:00 
86
        Storage 2:
87
                id = 10.211.55.26
88
                ip_addr = 10.211.55.26 (node2)  ACTIVE
89
                http domain = 
90
                version = 5.11
91
                join time = 2017-06-11 11:45:12
92
                up time = 2017-06-11 11:45:12
93
                total storage = 50268 MB
94
                free storage = 46044 MB
95
                upload priority = 10
96
                store_path_count = 1
97
                subdir_count_per_path = 256
98
                storage_port = 23000
99
                storage_http_port = 8888
100
                current_write_path = 0
101
                source storage id = 10.211.55.26
102
                if_trunk_server = 0
103
                connection.alloc_count = 256
104
                connection.current_count = 1
105
                connection.max_count = 1
106
                total_upload_count = 0
107
                success_upload_count = 0
108
                total_append_count = 0
109
                success_append_count = 0
110
                total_modify_count = 0
111
                success_modify_count = 0
112
                total_truncate_count = 0
113
                success_truncate_count = 0
114
                total_set_meta_count = 0
115
                success_set_meta_count = 0
116
                total_delete_count = 0
117
                success_delete_count = 0
118
                total_download_count = 0
119
                success_download_count = 0
120
                total_get_meta_count = 0
121
                success_get_meta_count = 0
122
                total_create_link_count = 0
123
                success_create_link_count = 0
124
                total_delete_link_count = 0
125
                success_delete_link_count = 0
126
                total_upload_bytes = 0
127
                success_upload_bytes = 0
128
                total_append_bytes = 0
129
                success_append_bytes = 0
130
                total_modify_bytes = 0
131
                success_modify_bytes = 0
132
                stotal_download_bytes = 0
133
                success_download_bytes = 0
134
                total_sync_in_bytes = 0
135
                success_sync_in_bytes = 0
136
                total_sync_out_bytes = 0
137
                success_sync_out_bytes = 0
138
                total_file_open_count = 0
139
                success_file_open_count = 0
140
                total_file_read_count = 0
141
                success_file_read_count = 0
142
                total_file_write_count = 0
143
                success_file_write_count = 0
144
                last_heart_beat_time = 2017-06-10 19:50:37
145
                last_source_update = 1969-12-31 19:00:00
146
                last_sync_update = 1969-12-31 19:00:00
147
                last_synced_timestamp = 1969-12-31 19:00:00
  • 上传文件
1
~]# cat /etc/fdfs/client.conf
2
connect_timeout=30
3
network_timeout=60
4
base_path=/data/fdfs/client
5
tracker_server=10.211.55.24:22122
6
log_level=info
7
use_connection_pool = false
8
connection_pool_max_idle_time = 3600
9
load_fdfs_parameters_from_tracker=false
10
use_storage_id = false
11
storage_ids_filename = storage_ids.conf
12
http.tracker_server_port=80
13
~]# fdfs_upload_file /etc/fdfs/client.conf /etc/fstab 
14
group1/M00/00/00/CtM3MFk8h1mACqW3AAADXPbJpH84824215
15
~]# fdfs_file_info /etc/fdfs/client.conf group1/M00/00/00/CtM3MFk8h1mACqW3AAADXPbJpH84824215
16
source storage id: 0
17
source ip address: 10.211.55.25
18
file create timestamp: 2017-06-10 19:57:13
19
file size: 860
20
file crc32: 4140409983 (0xF6C9A47F)
21
~]# ll /data/fdfs/storage/0/data/00/00/CtM3MFk8h1mACqW3AAADXPbJpH84824215 
22
-rw-r--r--. 1 root root 860 Jun 10 19:57 /data/fdfs/storage/0/data/00/00/CtM3MFk8h1mACqW3AAADXPbJpH84824215
  • 在storage安装nginx
1
~]# git clone https://github.com/happyfish100/fastdfs-nginx-module
2
~]# tar xf  nginx-1.12.0.tar.gz 
3
~]# cd nginx-1.12.0
4
~]# ./configure --prefix=/usr/local/nginx --with-http_stub_status_module --with-http_ssl_module --add-module=../fastdfs-nginx-module/src --user=www
5
~]# make -j 4 && make install
6
~]# cd 
7
~]# cp fastdfs/conf/{http.conf,mime.types} /etc/fdfs
8
~]# cp fastdfs-nginx-module/src/mod_fastdfs.conf /etc/fdfs
9
~]# cd /etc/fdfs
10
~]# cat mod_fastdfs.conf 
11
connect_timeout=2
12
network_timeout=30
13
base_path=/data/fdfs/storage
14
load_fdfs_parameters_from_tracker=true
15
storage_sync_file_max_delay = 86400
16
use_storage_id = false
17
storage_ids_filename = storage_ids.conf
18
tracker_server=10.211.55.24:22122
19
storage_server_port=23000                  #本地storage的端口
20
group_name=group1                          #本地的group
21
url_have_group_name = true                 #开启
22
store_path_count=1
23
store_path0=/data/fdfs/storage/0
24
log_level=info
25
log_filename=
26
response_mode=proxy
27
if_alias_prefix=
28
flv_support = true
29
flv_extension = flv
30
group_count = 0
31
~]# vim /usr/local/nginx/conf/nginx.conf
32
location /group1/M00 {
33
	root /data/fdfs/storage/data;
34
	ngx_fastdfs_module;
35
}
36
~]# ln -s /data/fdfs/storage/data/ /data/fdfs/storage/data/M00
37
~]# cd /usr/local/nginx/sbin
38
~]# ./nginx -t && ./nginx
  • 测试
1
~]# echo "hello fastdfs" > index.html
2
~]# fdfs_text /etc/fdfs/client.conf index.html
3
This is FastDFS client test program v5.11
4
5
Copyright (C) 2008, Happy Fish / YuQing
6
7
FastDFS may be copied only under the terms of the GNU General
8
Public License V3, which may be found in the FastDFS source kit.
9
Please visit the FastDFS Home Page http://www.csource.org/ 
10
for more detail.
11
12
[2017-06-10 21:14:42] DEBUG - base_path=/data/fdfs/client, connect_timeout=30, network_timeout=60, tracker_server_count=1, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0
13
14
tracker_query_storage_store_list_without_group: 
15
        server 1. group_name=, ip_addr=10.211.55.25, port=23000
16
        server 2. group_name=, ip_addr=10.211.55.26, port=23000
17
18
group_name=group1, ip_addr=10.211.55.25, port=23000
19
storage_upload_by_filename
20
group_name=group1, remote_filename=M00/00/00/CtM3MFk8mYKAdz3jAAAADplm3rU73.html
21
source ip address: 10.211.55.25
22
file timestamp=2017-06-10 21:14:42
23
file size=14
24
file crc32=2573655733
25
example file url: http://10.211.55.25/group1/M00/00/00/CtM3MFk8mYKAdz3jAAAADplm3rU73.html
26
storage_upload_slave_by_filename
27
group_name=group1, remote_filename=M00/00/00/CtM3MFk8mYKAdz3jAAAADplm3rU73_big.html
28
source ip address: 10.211.55.25
29
file timestamp=2017-06-10 21:14:42
30
file size=14
31
file crc32=2573655733
32
example file url: http://10.211.55.25/group1/M00/00/00/CtM3MFk8mYKAdz3jAAAADplm3rU73_big.html
33
~]# curl http://10.211.55.25/group1/M00/00/00/CtM3MFk8mYKAdz3jAAAADplm3rU73_big.html
34
hello fastdfs

原文作者：Zhang Jusene

原文链接：http://jusene.github.io/2017/07/22/fastdfs/

发表日期：July 22nd 2017, 4:43:58 pm

更新日期：November 30th 2019, 1:54:24 am

版权声明：本文采用知识共享署名-非商业性使用 4.0 国际许可协议进行许可

CATALOG
  1. 1. FastDFS
    1. 1.1. Upload file
    2. 1.2. Download file
  2. 2. FastDFS实例
Total : 190
2020
2019
2018
2017
python boot troubshooting docker drbd ssh web ELK kubernetes Kubernetes git ansible tools dns corosync consul Django etcd security ftp Go bash bigdata grub heartbeat Java kickstart kvm lvs mongodb mysql mesh nc nfs mogilefs openresty upgrade puppet rabbitmq saltstack rsync screen socat svn tcpdump fastdfs haproxy iscsi keepalived cache openssl vpn optimization prometheus redis RPM cncf supervisord kernel JAVA
python Linux troubshooting Docker docker storage nginx Nosql kubernetes Flask Kubernetes ansible Shell cluster Django Go bigdata boot Java hypervisor mysql mesh tools openresty puppet rabbitmq saltstack optimization cncf varnish tomcat JAVA