Shulou(Shulou.com)06/02 Report--

MFS introduction

MFS (MooseFS) is a fault-tolerant, highly available and scalable sea-level distributed file system. MFS spreads the data across multiple servers, but customers see only one source. MFS, like other UNIX-like file systems, contains hierarchies and file attributes that can create special files (block devices, character devices, pipes, sockets), symbolic links, and hard links.

MFS principle

(1) composition of MFS file system

1)。 Metadata server (Master): responsible for managing the file system and maintaining metadata throughout the system.

2)。 Metadata log server (MetaLogger): back up the change log file of the Master server, the file type is changelog_ml.*.mfs. When the Master server data is lost or corrupted, you can get the file from the log server and recover it.

3)。 Data storage server (Chunk Server): the server that actually stores data. When a file is stored, the file is saved in blocks and copied between database servers. The more data servers, the more capacity you can use, the higher the reliability, and the better the performance.

4)。 Client (Client): the MFS file system can be mounted just like NFS, and the operation is the same.

(2) the process of reading data by MFS.

1)。 The client issues a read request to the metadata server.

2)。 The metadata server informs the client where the required data is stored (the IP address and Chunk number of the Chunk Server).

3)。 The client sends data to a known Chunk Server request.

4)。 Chunk Server sends data to the client.

(3)。 The process by which MFS writes data.

1)。 The client issues a write request to the metadata server.

2)。 The metadata server interacts with the Chunk Server (only if the required chunked Chunks exists), and the unit data server only creates a new chunked Chunks on some servers, and the Chunk Server informs the metadata server that the operation is successful.

3)。 The metadata server tells the client which Chunk Server and which Chunks the data can be written to.

4)。 The customer writes data to the specified Chunk Server.

5)。 The Chunk Server synchronizes data with other Chunk Server. After the synchronization is successful, the Chunk Server informs the client that the data is written successfully.

6)。 The client informs the metadata server that the write is complete.

Experimental environment

In this experiment, 5 servers are used to simulate and build the MFS file system. The specific topology diagram is as follows:

The experimental environment is as follows

Host operating system IP address main software Master ServerCentos 7192.168.91.147mfs-1.6.27-5.tar.gzMetalogger ServerCentos 7192.168.91.149mfs-1.6.27-5.tar.gzChunk Server 1Centos 7192.168.91.148mfs-1.6.27-5.tar.gzChunk Server 2Centos 7192.168.91.150mfs-1.6.27-5.tar.gzClientCentos 7192.168.91.146mfs-1.6.27-5.tar.gz

Fuse-2.9.2.tar.gz

Link: https://pan.baidu.com/s/17N08A1JTAitFDJhJTJmwlQ (MFS package)

Extraction code: b7z9

(1) turn off the firewall on the five servers and install the compiler environment 1. Set up Master server

[root@localhost ~] # yum install gcc gcc-c++ zlib-devel-y

[root@localhost] # useradd-s / sbin/nologin mfs

[root@localhost ~] # systemctl stop firewalld

[root@localhost ~] # setenforce 0

(2) create users

[root@master Y2C] # useradd-s / sbin/nologin mfs / / create mfs users

(3) install the source package

[root@master Y2C] # tar zxvf mfs-1.6.27-5.tar.gz-C / opt/

[root@master Y2C] # cd / opt/mfs-1.6.27/

[root@master mfs-1.6.27] #. / configure\

>-- prefix=/usr/local/mfs\ / / specify the installation path

>-- with-default-user=mfs\ / / specifies that the default user is mfs

>-- with-default-group=mfs\ / / specify the default group

>-- disable-mfschunkserver\ / / disable Chunk Server

>-- disable-mfsmount / / disable the client

[root@master mfs-1.6.27] # make & & make install

(4) copy the file (the configuration file does not need to be modified in the master server)

[root@master mfs-1.6.27] # cd / usr/local/mfs/etc/mfs/

[root@master mfs] # ls

Mfsexports.cfg.dist mfsmaster.cfg.dist mfsmetalogger.cfg.dist mfstopology.cfg.dist / / generate 4 configuration files

[root@master mfs] # cp mfsexports.cfg.dist mfsexports.cfg / / permission profile

[root@master mfs] # cp mfsmaster.cfg.dist mfsmaster.cfg / / master configuration file

[root@master mfs] # cp mfstopology.cfg.dist mfstopology.cfg / / metadata log file

[root@master mfs] # cd / usr/local/mfs/var/mfs/

[root@master mfs] # ls

Metadata.mfs.empty

[root@master mfs] # metadata files automatically generated during cp metadata.mfs.empty metadata.mfs / / master operation

(5) start Master Server

[root@master mfs] # / usr/local/mfs/sbin/mfsmaster start

[root@master mfs] # netstat-ntap | grep mfsmaster / / check whether the port is started (there are 3 ports)

Tcp 0 0 0.0.0 0 9419 0.0.0 0 V * LISTEN 9061/mfsmaster

Tcp 0 0 0.0.0.0 9420 0.0.0. 0 LISTEN 9061/mfsmaster

Tcp 0 0 0.0.0.0 9421 0.0.0. 0 LISTEN 9061/mfsmaster

The command to stop Master Server is / usr/local/mfs/sbin/mfsmaster-s

two。 Set up MetaLogger Server

(1) install the compiler environment and turn off the firewall.

[root@metalogger] # yum install gcc gcc-c++ zlib-devel-y

[root@metalogger] # systemctl stop firewalld

[root@metalogger] # setenforce 0

(2) create mfs users

[root@metalogger] # useradd-s / sbin/nologin mfs

(3) install the source package (same as Master Server configuration)

[root@metalogger] tar zxvf mfs-1.6.27-5.tar.gz-C / opt/

[root@metalogger] cd / opt/mfs-1.6.27/

[root@metalogger]. / configure\

-- prefix=/usr/local/mfs\

-- with-default-user=mfs\

-- with-default-group=mfs\

-- disable-mfschunkserver\

-- disable-mfsmount

[root@metalogger] make & & make install

(4) copy files

[root@metalogger mfs-1.6.27] # cd / usr/local/mfs/etc/mfs/

[root@metalogger mfs] # ls

Mfsexports.cfg.dist mfsmaster.cfg.dist mfsmetalogger.cfg.dist mfstopology.cfg.dist

[root@metalogger mfs] # cp mfsmetalogger.cfg.dist mfsmetalogger.cfg / / copy metadata log files

(5) modify the configuration file of the metadata log file

[root@metalogger mfs] # vim mfsmetalogger.cfg

# WORKING_USER = mfs / / users running masterserver

# WORKING_GROUP = mfs / / Group running masterserver

# SYSLOG_IDENT = mfsmetalogger / / indicates the log generated by mfsmetalogger

# LOCK_MEMORY = 0 / / whether to execute mlockall () to avoid masterserver process overflow (default is 0)

# NICE_LEVEL =-19 / / priority to run (default is-19 if possible; Note: processes must be started with root)

# DATA_PATH = / usr/local/mfs/var/mfs / / data storage path

# BACK_LOGS = 50 / / the number of log files changed by metadata (default is 50)

# BACK_META_KEEP_PREVIOUS = 3

# META_DOWNLOAD_FREQ = 24

# MASTER_RECONNECTION_DELAY = 5

MASTER_HOST = 192.168.91.147 / / modified to the IP address of Master server

# MASTER_PORT = 9419 / / connection port address

# MASTER_TIMEOUT = 60

(6) start mfsmetalogger

[root@metalogger mfs] # / usr/local/mfs/sbin/mfsmetalogger start

Working directory: / usr/local/mfs/var/mfs

Lockfile created and locked

Initializing mfsmetalogger modules...

Mfsmetalogger daemon initialized properly

[root@metalogger mfs] # netstat-ntap | grep mfsmetalogger / / View port

Tcp 01 192.168.91.149:40552 220.250.64.225:9419 SYN_SENT 59501/mfsmetalogger

3. Build Chunk Server (the steps for building the two Chunk Server in this experiment are the same, the specific steps are as follows)

(1) install the compilation environment

[root@chunkserver ~] # systemctl stop firewalld.service

[root@chunkserver ~] # setenforce 0

[root@chunkserver ~] # yum install gcc gcc-c++ zlib-devel-y

(2) create mfs users

[root@chunkserrver] # useradd-s / sbin/nologin mfs

(3) install the source package

[root@chunkserver] # tar zxvf mfs-1.6.27-5.tar.gz-C / opt/

[root@chunkserver Y2C] # cd / opt/mfs-1.6.27/

[root@chunkserver mfs-1.6.27] # useradd-s / sbin/nologin mfs

[root@chunkserver mfs-1.6.27] #. / configure\

>-- prefix=/usr/local/mfs\

>-- with-default-user=mfs\

>-- with-default-group=mfs\

>-- disable-mfsmaster\ / / disable mfsmaster

>-- disable-mfsmount / / disable the client

[root@chunkserver mfs] # make & & make install

(4) copy the file and modify the configuration file of Chunkserver

[root@chunkserver mfs-1.6.27] # cd / usr/local/mfs/etc/mfs/

[root@chunkserver mfs] # ls

Mfschunkserver.cfg.dist mfshdd.cfg.dist

[root@chunkserver mfs] # cp mfschunkserver.cfg.dist mfschunkserver.cfg

[root@chunkserver mfs] # cp mfshdd.cfg.dist mfshdd.cfg

[root@chunkserver mfs] # vim mfschunkserver.cfg

# WORKING_USER = mfs

# WORKING_GROUP = mfs

# SYSLOG_IDENT = mfschunkserver

# LOCK_MEMORY = 0

# NICE_LEVEL =-19

# DATA_PATH = / usr/local/mfs/var/mfs

# MASTER_RECONNECTION_DELAY = 5

# BIND_HOST = *

MASTER_HOST = 192.168.91.147 / / modified to the IP address of Master Server

# MASTER_PORT = 9420

# MASTER_TIMEOUT = 60

# CSSERV_LISTEN_HOST = *

# CSSERV_LISTEN_PORT = 9422

# HDD_CONF_FILENAME = / usr/local/mfs/etc/mfs/mfshdd.cfg

# HDD_TEST_FREQ = 10

(5) add storage directory

[root@chunkserver mfs] # vim mfshdd.cfg

# mount points of HDD drives

#

# / mnt/hd1

# / mnt/hd2

# etc.

/ data / / add a line / data, where / data is a partition for MFS, and it is best for production environments to use separate partitions or disks to close to this directory

[root@chunkserver mfs] # mkdir / data / / create / data directory

[root@chunkserver mfs] # chown-R mfs.mfs / data / / give mfs users and groups the right to execute / data

(6) start ChukServer

[root@chunkserver mfs] # / usr/local/mfs/sbin/mfschunkserver start

Working directory: / usr/local/mfs/var/mfs

Lockfile created and locked

Initializing mfschunkserver modules...

Hdd space manager: path to scan: / data/

Hdd space manager: start background hdd scanning (searching for available chunks)

Main server module: listen on *: 9422

No charts data file-initializing empty charts

Mfschunkserver daemon initialized properly

[root@chunkserver mfs] # netstat-ntap | grep mfs

Tcp 0 0 0.0.0 0 9422 0.0.0 0 V * LISTEN 43697/mfschunkserve

Tcp 0 0 192.168.91.148:52454 192.168.91.147:9420 ESTABLISHED 43697/mfschunkserve

The other ChunkServer has the same configuration.

Chunk server 2

[root@chunkserver2 mfs] # / usr/local/mfs/sbin/mfschunkserver start

Working directory: / usr/local/mfs/var/mfs

Lockfile created and locked

Initializing mfschunkserver modules...

Hdd space manager: path to scan: / data/

Hdd space manager: start background hdd scanning (searching for available chunks)

Main server module: listen on *: 9422

No charts data file-initializing empty charts

Mfschunkserver daemon initialized properly

[root@chunkserver2 mfs] # netstat-ntap | grep mfs

Tcp 0 0 0.0.0 0 9422 0.0.0 0 V * LISTEN 47743/mfschunkserve

Tcp 0 0 192.168.91.150:36058 192.168.91.147:9420 ESTABLISHED 47743/mfschunkserve

4. Build the client

(1) install the compilation environment

[root@client ~] # systemctl stop firewalld.service

[root@client ~] # setenforce 0

[root@client ~] # yum install gcc gcc-c++ zlib-devel-y

(2) installation of FUSE,MFS client depends on FUSE

[root@client Y2C] # tar zxvf fuse-2.9.2.tar.gz-C / opt/

[root@client Y2C] # cd / opt/fuse-2.9.2/

[root@client fuse-2.9.2] #. / configure

[root@client fuse-2.9.2] # make & & make install

And then set it to the environment variable

[root@client fuse-2.9.2] # vim / etc/profile

Export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig:$PKG_CONFIG_PATH / / add at the last line

[root@client fuse-2.9.2] # source / etc/profile / / refresh environment variables

(3) install MFS client

[root@client fuse-2.9.2] # useradd-s / sbin/nologin mfs / / create a mfs user

[root@client Y2C] # tar zxvf mfs-1.6.27-5.tar.gz-C / opt/

[root@client Y2C] # cd / opt/mfs-1.6.27/

[root@client mfs-1.6.27] #. / configure\

>-- prefix=/usr/local/mfs\

>-- with-default-user=mfs\

>-- with-default-group=mfs\

>-- disable-mfsmaster\ / / disable master

>-- disable-mfschunkserver\ / / disable chunkserver

>-- enable-mfsmount / / enable the client

[root@client mfs-1.6.27] # make & & make install

(4) Mount the MFS file system

[root@client mfs-1.6.27] # mkdir / opt/mfs / / create mount point

[root@client mfs-1.6.27] # modprobe fuse / / load fuse module into kernel

[root@client mfs-1.6.27] # / usr/local/mfs/bin/mfsmount / opt/mfs-H 192.168.91.147 / / Mount MFS (assigned to Master Server server)

Mfsmaster accepted connection with parameters: read-write,restricted_ip; root mapped to root:root

[root@client mfs-1.6.27] # df-h / / View the mount status

File system capacity used available used% mount point

/ dev/sda2 20G 3.6G 17G 18% /

Devtmpfs 978M 0978M 0% / dev

Tmpfs 993M 0 993M 0% / dev/shm

Tmpfs 993M 18m 975m 2% / run

Tmpfs 993M 0 993M 0% / sys/fs/cgroup

/ dev/sda5 5.0G 50m 5.0G 1% / opt

/ dev/sda3 10G 37m 10G 1% / home

/ dev/sda1 997M 158M 839M 16% / boot

Tmpfs 199m 4.0K 199m 1% / run/user/42

Tmpfs 199m 40K 199m 1% / run/user/0

/ / 192.168.91.1/rhel6 120g 50g 71G 41% / aaa

192.168.91.147purl 9421 33G 0 33G 0 / opt/mfs

To uninstall MFS, use the command umount / opt/mfs

(5) Common operation of MFS

After MFS is installed on the client, it generates the / usr/local/mfs/bin/ directory, where there are many commands that the user needs. To facilitate the use of these commands, you can add / usr/local/mfs/bin/ to the environment variable.

[root@client mfs-1.6.27] # vim / etc/profile

Export PATH=/usr/local/mfs/bin/:$PATH

[root@client mfs-1.6.27] # source / etc/profile

The mfsgetgoal command is used to query the score of the file being copied. The-r command can be used to recurse the entire directory. Goal refers to the score of the file being copied.

[root@client mfs-1.6.27] # mfsgetgoal-r / opt/mfs

/ opt/mfs:

Directories with goal 1: 1

The mfssetgoal command is used to set the score for files to be copied. The number of Chunk Server nodes in a production environment should be at least 2, and the number of file copies should be less than or equal to the number of Chunk Server servers.

[root@client mfs-1.6.27] # mfssetgoal-r 2 / opt/mfs

/ opt/mfs:

Inodes with goal changed: 1

Inodes with goal not changed: 0

Inodes with permission denied: 0

[root@client mfs-1.6.27] # mfsgetgoal-r / opt/mfs/

/ opt/mfs/:

Directories with goal 2: 1

Create a file to test as follows

[root@client mfs-1.6.27] # cd / opt/mfs

[root@client mfs] # touch test

[root@client mfs] # ls

Test

[root@client mfs] # mfsgetgoal test

Test: 2

(6) MFS monitoring

Mfscgiserv is a web server written in python, and its listening port is 9425. It can be started on Master Server by command / usr/local/mfs/sbin/mfscgiserv, and the user interest browser can fully monitor all client mounts, Chunk Server, Master Server, and various client operations.

[root@master mfs] # / usr/local/mfs/sbin/mfscgiserv

Lockfile created and locked

Starting simple cgi server (host: any, port: 9425, rootpath: / usr/local/mfs/share/mfscgi)

Access http://192.168.91.147:9425/ on the client side through a browser

Summary:

(1) MFS is a fault-tolerant network distributed file system, which distributes data on multiple physical servers and presents a unified resource to customers.

(2) the MFS file system is composed of metadata server (Master), metadata log server (MetaLogger), data storage server (Chunk Server) and client (Client).

(3) the metadata server (Master) needs to use two configuration files, namely mfsmaster.dfg and mfsexports.cfg.

(4) when the metadata server (Master) fails, the Master can be restored from the Metaloggwr.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.