Jusene's Blog

MogileFS 分布式存储

mogilefs
字数统计: 2.5k阅读时长: 13 min
2017/07/19 Share

分布式

不知不觉进入了云计算大数据时代，在不知不觉中数据成为了最重要的财富，而时代的变迁，传统的单一的架构设计已经完全无法满足现在对数据的需求，分布式架构设计走上了历史的舞台，由多节点共同完成单一任务，使任务的效率提升了千倍万倍。

分布式这个词听起来很是高大上，其实运维每天都在接触分布式架构，如lnmp，lamp等，将多个节点共同组合成为一个网站的交互任务，简单来说将各个服务器分配的任务越是单一，使服务器的运行效率提升到极致，就可以越大限度的提升上整个分布式架构的运行能力。

分布式存储和分布式文件系统

分布式存储和分布式文件系统最大的区别，文件系统是可以挂载的，而存储则是无文件系统入口的，所以只可以通过API来访问，常见的分布式存储有：

  • GFS: Google File System
    分布式的鼻祖级别系统，由Google内部开发，发表有技术实现论文，为后来的分布式存储系统提供了技术实现基础

  • HDFS: Hadoop Distribution File System
    Hadoop的分布式存储系统，通过GFS的论文的开源实现

  • GlusterFS
    不温不火的分布式存储系统，被redhat收购，去中心化，没有元数据节点

  • Ceph
    Linux内核级的实现的分布式文件系统，收录Linux内核，正在被广泛的试验，未来的新贵

  • MogileFS
    适合存储海量小文件，使用perl语言编写，国内有人使用c语言重写并开源为FastDFS

  • TFS
    Taobao FileSystem,基于HDFS开发适合于存储海量小文件

MogileFS实现的原理

MogileFS:

  • tracker：借助数据库保存各个节点的元数据信息，便于检索定位数据位置并监控各个节点，告知客户端存储区块的位置并指挥storage节点复制数据副本，进程为mogilefsd
  • database：为tracker节点储存节点的元数据信息
  • storage：将指定域中的键转换为特有的文件名存储于特定的设备文件中，转换后文件名为值，storage自动维护键值之间对应关系，storage节点使用http进行数据传输，依赖perbal，进程为mogstored，perbal
  • Domain: 一个域中的键是唯一的，一个mogilefs可以有多个域来存储不同类型的文件
  • Class: 复制的最小单位，管理文件属性，定义文件存储在不同设备上的份数
  • device: 一个存储节点，可以有多个device，就是用来存放文件的目录，每个设备都有一个设备ID,需要在mogilefs配置文件中docroot配置，设备不能删除，只能将设备的状态置为dead，置为dead之后的数据无法恢复，并且设备ID也是无法使用

MogileFS基础框架:

MogileFS

MogileFS CentOS6 RPM

测试环境：
tracker: 10.211.55.24
database: 10.211.55.25
storage node1: 10.211.55.26
storage node2: 10.211.55.27

每台都必须配置epel源，具体配置不累赘了。

tracker和storage nodes

1
~]# cd mogilefs
2
~]# yum install -y  *.rpm perl-IO-AIO

database

1
~]# yum install -y mysql-server mysql
2
~]# mysql
3
Welcome to the MySQL monitor.  Commands end with ; or \g.
4
Your MySQL connection id is 6
5
Server version: 5.1.73 Source distribution
6
7
Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.
8
9
Oracle is a registered trademark of Oracle Corporation and/or its
10
affiliates. Other names may be trademarks of their respective
11
owners.
12
13
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
14
15
mysql> grant all on *.* to mogilefs@'10.211.55.%' identified by 'mogpass' with grant option;
16
17
mysql> flush privilesgs;

tracker初始化数据库

1
~]# ~]# mogdbsetup --dbrootuser=mogilefs --dbrootpass=mogpass --dbhost=10.211.55.25 --dbname=mog
2
3
This will attempt to setup or upgrade your MogileFS database.
4
It won't destroy existing data.
5
Run with --help for more information.  Run with --yes to shut up these prompts.
6
7
Continue? [N/y]: y
8
9
Create/Upgrade database name 'mog'? [Y/n]: y
10
11
Grant all privileges to user 'mogile', connecting from anywhere, to the mogilefs database 'mog'? [Y/n]: y

database验证下

1
~]# ]# mysql
2
Welcome to the MySQL monitor.  Commands end with ; or \g.
3
Your MySQL connection id is 26
4
Server version: 5.1.73 Source distribution
5
6
Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.
7
8
Oracle is a registered trademark of Oracle Corporation and/or its
9
affiliates. Other names may be trademarks of their respective
10
owners.
11
12
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
13
14
mysql> use mog
15
Reading table information for completion of table and column names
16
You can turn off this feature to get a quicker startup with -A
17
18
Database changed
19
mysql> show tables;
20
+----------------------+
21
| Tables_in_mog        |
22
+----------------------+
23
| checksum             |
24
| class                |
25
| device               |
26
| domain               |
27
| file                 |
28
| file_on              |
29
| file_on_corrupt      |
30
| file_to_delete       |
31
| file_to_delete2      |
32
| file_to_delete_later |
33
| file_to_queue        |
34
| file_to_replicate    |
35
| fsck_log             |
36
| host                 |
37
| server_settings      |
38
| tempfile             |
39
| unreachable_fids     |
40
+----------------------+
41
17 rows in set (0.00 sec)

tracker

1
~]# cat /etc/mogilefs/mogilefsd.conf 
2
...
3
db_dsn = DBI:mysql:mog:host=10.211.55.25
4
db_user = mogilefs
5
db_pass = mogpass
6
listen = 0.0.0.0:7001
7
...
8
~]# service mogilefsd start

storage nodes

1
~]# cat /etc/mogilefs/mogstored.conf
2
maxconns = 10000
3
httplisten = 0.0.0.0:7500
4
mgmtlisten = 0.0.0.0:7501
5
docroot = /data/mogdata
6
~]# mkidr -pv /data/mogdata
7
~]# cd /data/mogdata
8
~]# mkdir dev1 dev2
9
~]# chown -R mogilefs.mogilefs /data/mogdata/
10
~]# service mogstored start

tracker

1
~]# mogstats --db_dsn="DBI:mysql:mog:host=10.211.55.25" --db_user="mogilefs" --db_pass="mogpass" --states='all'
2
Fetching statistics... (all)
3
4
Statistics for devices...
5
  device     host                   files     status
6
  ---------- ---------------- ------------ ----------
7
  ---------- ---------------- ------------ ----------
8
9
Statistics for file ids...
10
  Max file id: none
11
12
Statistics for files...
13
  domain               class           files    size (m)  fullsize (m)
14
  -------------------- ----------- ---------- ----------- -------------
15
  -------------------- ----------- ---------- ----------- -------------
16
17
Statistics for replication...
18
  domain               class        devcount      files
19
  -------------------- ----------- ---------- ----------
20
  -------------------- ----------- ---------- ----------
21
22
Statistics for replication queue...
23
  status                      count
24
  -------------------- ------------
25
  -------------------- ------------
26
27
Statistics for delete queue...
28
  status                      count
29
  -------------------- ------------
30
  -------------------- ------------
31
32
Statistics for general queues...
33
  queue           status                      count
34
  --------------- -------------------- ------------
35
  --------------- -------------------- ------------
36
37
done
38
~]# mogadm --help
39
Usage:  (enter any command prefix, leaving off options, for further help)
40
41
  mogadm check                     Check the state of the MogileFS world.
42
  mogadm stats                     Show MogileFS system statistics.  (DEPRECATED: use mogstats instead)
43
  mogadm host ...
44
         host add ...              Add a host to MogileFS.
45
         host delete ...           Delete a host.
46
         host list                 List all hosts.
47
         host mark ...             Change the status of a host.  (equivalent to 'modify --status')
48
         host modify ...           Modify a host's properties.
49
  mogadm device ...
50
         device add ...            Add a device to a host.
51
         device list ...           List all devices, for each host.
52
         device mark ...           Mark a device as {alive,dead,down,drain,readonly}
53
         device modify ...         Modify a device's properties.
54
         device summary ...        List the summary of devices, for each host.
55
  mogadm domain ...
56
         domain add ...            Add a domain (namespace)
57
         domain delete ...         Delete a domain.
58
         domain list               List all hosts.
59
  mogadm class ...
60
         class add ...             Add a file class to a domain.
61
         class delete ...          Delete a file class from a domain.
62
         class list                List all classes, for each domain.
63
         class modify ...          Modify properties of a file class.
64
  mogadm slave ...
65
         slave add ...             Add a slave node for store usage
66
         slave delete ...          Delete a slave node for store usage
67
         slave list                List current store slave nodes.
68
         slave modify ...          Modify a slave node for store usage
69
  mogadm fsck ...
70
         fsck clearlog             Clear the fsck log
71
         fsck printlog             Display the fsck log
72
         fsck reset ...            Reset fsck position back to the beginning
73
         fsck start                Start (or resume) background fsck
74
         fsck status               Show fsck status
75
         fsck stop                 Stop (pause) background fsck
76
         fsck taillog              Tail the fsck log
77
  mogadm rebalance ...
78
         rebalance policy ...      Add or adjust the current policy
79
         rebalance reset           Reset an existing policy
80
         rebalance settings        Display rebalance settings
81
         rebalance start           Start a rebalance job
82
         rebalance status          Show status of current rebalance job
83
         rebalance stop            Stop a rebalance job
84
         rebalance test            Show what devices the current policy would match
85
  mogadm settings ...
86
         settings list             List all server settings
87
         settings set ...          Set server setting 'key' to 'value'.

  • 添加host

    1
    ~]# mogadm host add --help
    2
    3
    ERROR: Unexpected option.  Expected argument 'hostname'
    4
    5
    Help for 'host-add' command:
    6
    7
      mogadm host add <hostname> [opts]                  Add a host to MogileFS.
    8
    9
          <hostname>           Hostname of machine
    10
          --altip=s            Alternate IP that is machine is reachable from
    11
          --altmask=s          Netmask which, when matches client, uses alt IP
    12
          --getport=i          Alternate HTTP port serving readonly traffic
    13
          --ip=s               IP address of machine.
    14
          --port=i             HTTP port of mogstored
    15
          --status=s           One of {alive,down}.  Default 'down'
    16
    17
    ~]# mogadm host add node1 --ip=10.211.55.26 --port=7500 --status=alive
    18
    ~]# mogadm host add node2 --ip=10.211.55.27 --port=7500 --status=alive
    19
    ~]# ~]# mogadm host list
    20
    node1 [1]: alive
    21
      IP:       10.211.55.26:7500
    22
    23
    node2 [2]: alive
    24
      IP:       10.211.55.27:7500

  • 添加设备

    1
    ~]# mogadm device list
    2
    node1 [1]: alive
    3
                        used(G)    free(G)   total(G)  weight(%)
    4
    5
    node2 [2]: alive
    6
                        used(G)    free(G)   total(G)  weight(%)
    7
    ~]# mogadm device add --help
    8
    9
    ERROR: Unexpected option.  Expected argument 'hostname'
    10
    11
    Help for 'device-add' command:
    12
    13
      mogadm device add <hostname> <devid> [opts]        Add a device to a host.
    14
    15
          <devid>              Numeric devid.  Never reuse these.
    16
          <hostname>           Hostname to add a device
    17
          --status=s           One of 'alive' or 'down'.  Defaults to 'alive'.
    18
    19
    ~]# mogadm device add node1 1 --status=alive
    20
    ~]# mogadm device add node2 2 --status=alive
    21
    ~]# mogadm device list
    22
    ~]# mogadm device list
    23
    node1 [1]: alive
    24
                        used(G)    free(G)   total(G)  weight(%)
    25
       dev2:   alive      1.139     45.451     46.590        100
    26
    27
    node2 [2]: alive
    28
                        used(G)    free(G)   total(G)  weight(%)
    29
       dev1:   alive      3.992     42.598     46.590        100

  • 创建域

    1
    ~]# mogadm domain list
    2
        domain               class                mindevcount   replpolicy   hashtype
    3
        -------------------- -------------------- ------------- ------------ -------
    4
    5
    ~]# mogadm domain add files
    6
    ~]# mogadm domain add imgs
    7
    ~]# mogadm domain list
    8
     domain               class                mindevcount   replpolicy   hashtype
    9
    -------------------- -------------------- ------------- ------------ -------
    10
     files                default                   2        MultipleHosts() NONE   
    11
    12
     imgs                 default                   2        MultipleHosts() NONE

  • 创建类

    1
    ~]#  mogadm class add --help
    2
    3
    ERROR: Unexpected option.  Expected argument 'domain'
    4
    5
    Help for 'class-add' command:
    6
    7
      mogadm class add <domain> <class> [opts]           Add a file class to a domain.
    8
    9
          <class>              Name of class to add.
    10
          <domain>             Domain to add class to.
    11
          --hashtype=s         Hash algorithm string ('MD5', 'NONE').
    12
          --mindevcount=i      Minimum number of replicas.
    13
          --replpolicy=s       Replication policy string.
    14
    ~]# mogadm class add files text --mindevcount=2

  • 文件操作

    1
    ~]# mogupload -help
    2
    Usage: /usr/bin/mogupload --trackers=host --domain=foo --key='/hello.jpg' --file='./hello.jpg'
    3
    ~]# mogupload --trackers=10.211.55.24 --domain=files --key='/fstab.txt' --file='/etc/fstab'
    4
    ~]# moglistkeys --trackers=10.211.55.24 --domain=files
    5
    /fstab.txt
    6
    ~]# mogfetch --trackers=10.211.55.24 --domain=files --key='/fstab.txt' --file='./fstab'
    7
    ~]# mogfileinfo --trackers=10.211.55.24 --domain=files --key='/fstab.txt'
    8
    - file: /fstab.txt
    9
         class:              default
    10
      devcount:                    2
    11
        domain:                files
    12
           fid:                    2
    13
           key:           /fstab.txt
    14
        length:                  860
    15
     - http://10.211.55.26:7500/dev1/0/000/000/0000000002.fid
    16
     - http://10.211.55.27:7500/dev2/0/000/000/0000000002.fid
    17
    ~]# curl http://10.211.55.26:7500/dev1/0/000/000/0000000002.fid
    18
    19
    #
    20
    # /etc/fstab
    21
    # Created by anaconda on Tue May 16 10:41:16 2017
    22
    #
    23
    # Accessible filesystems, by reference, are maintained under '/dev/disk'
    24
    # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
    25
    #
    26
    /dev/mapper/vg_node1-lv_root /                       ext4    defaults        1 1
    27
    UUID=923aa778-65ee-4938-a72b-5213c92037af /boot                   ext4    defaults        1 2
    28
    /dev/mapper/vg_node1-lv_home /home                   ext4    defaults        1 2
    29
    /dev/mapper/vg_node1-lv_swap swap                    swap    defaults        0 0
    30
    tmpfs                   /dev/shm                tmpfs   defaults        0 0
    31
    devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
    32
    sysfs                   /sys                    sysfs   defaults        0 0
    33
    proc                    /proc                   proc    defaults        0 0
    34
    ~] mogdelete --trackers=10.211.55.24 --domain=files --key='/fstab.txt'

原文作者：Zhang Jusene

原文链接：http://jusene.github.io/2017/07/19/mogilefs/

发表日期：July 19th 2017, 7:11:08 pm

更新日期：November 30th 2019, 1:54:24 am

版权声明：本文采用知识共享署名-非商业性使用 4.0 国际许可协议进行许可

CATALOG
  1. 1. 分布式
  2. 2. 分布式存储和分布式文件系统
  3. 3. MogileFS实现的原理
  4. 4. MogileFS
Total : 190
2020
2019
2018
2017
python boot troubshooting docker drbd ssh web ELK kubernetes Kubernetes git ansible tools dns corosync consul Django etcd security ftp Go bash bigdata grub heartbeat Java kickstart kvm lvs mongodb mysql mesh nc nfs mogilefs openresty upgrade puppet rabbitmq saltstack rsync screen socat svn tcpdump fastdfs haproxy iscsi keepalived cache openssl vpn optimization prometheus redis RPM cncf supervisord kernel JAVA
python Linux troubshooting Docker docker storage nginx Nosql kubernetes Flask Kubernetes ansible Shell cluster Django Go bigdata boot Java hypervisor mysql mesh tools openresty puppet rabbitmq saltstack optimization cncf varnish tomcat JAVA