Shulou(Shulou.com)06/03 Report--

Operation mode of MFS distributed storage

How MFS works

MooseFS is a distributed file system with redundant fault tolerance. It distributes data on multiple servers, ensures multiple backup copies of one data, and provides a unified structure.

Functional characteristics

For standard file operations, MooseFS behaves in the same way as other Unix-like file systems. Supported through file system features:

Hierarchy (directory tree)

Compatible with POSIX file properties

Support for special files

Symbolic links and hard links

Access control based on IP address and password

Unique characteristics

High reliability (multiple copies of data are stored on different servers)

Capacity dynamic expansion (add new hard disk or server)

Files deleted within the specified time can be recycled, similar to the Recycle Bin function.

You can create a file snapshot of an entire file or even a file being written

Four roles of the overall architecture of MFS

Master (metadata server) is responsible for the management of each data storage server, file read and write scheduling, file space recovery and recovery, multi-node copy. Stores metadata for each file (file size, attributes, location information, including all information for all unconventional files, such as directories, sockets, pipes, and device files)

Metalogger (metadata Log Server) is responsible for backing up the changelog of the Master server. The file type is changelog.*.mfs to take over the work of Master if something goes wrong

Chunk (data storage server) is responsible for connecting to Master, following Master scheduling, providing storage space, and providing data transmission for clients.

Client (client mount) mounts the data storage server managed on the remote management server (master) through the FUSE kernel interface, using the same as the local file system

# # MFS read process:

1. The client sends a read request to the metadata server.

2. The metadata server informs the client where the required data is stored (the IP address and chunk number of Chunk Server).

3. The client sends data to a known Chunk Server request.

4. Chunk Server sends data to the client.

# # MFS write process:

The client sends a write request to the metadata server.

2. The metadata server interacts with the Chunk Server (only if the required chunked Chunks exists):

Metadata servers just create new chunked chunks on some servers.

B, Chunk Servers informs the metadata server that step a has been operated successfully.

3. The metadata server tells the client which Chunk Server and which Chunks you can write data in.

4. The client writes data to the specified Chunk Server

5. Chunk Server synchronizes data with other Chunk Server.

6. Synchronization between Chunk Server is successful.

7. Chunk Server informs the client that the data is written successfully.

8. The client informs the metadata server that the write is complete.

Experimental topology

First, set up master server

1. Install dependency packages

[root@centos1 ~] # yum-y install zlib-devel

two。 Create a user

[root@centos1] # useradd mfs-s / sbin/nologin

3. Install the source package

[root@centos1] # tar xf / mnt/mfs-1.6.27-5.tar.gz-C / usr/src

[root@centos1 mfs-1.6.27] # cd / usr/src/mfs-1.6.27/

[root@centos1 mfs-1.6.27] # / configure-- prefix=/usr/local/mfs-- with-default-user=mfs-- with-default-group=mfs-- disable-mfschunkserver-- disable-mfsmount

[root@centos1 mfs-1.6.27] # make&&make install

4. Copy a file

[root@centos1 mfs-1.6.27] # cd / usr/local/mfs/etc/mfs

[root@centos1 mfs] # cp mfsmaster.cfg.dist mfsmaster.cfg

[root@centos1 mfs] # cp mfsexports.cfg.dist mfsexports.cfg

[root@centos1 mfs] # cp mfstopology.cfg mfstopology.cfg

[root@centos1 mfs] # cd / usr/local/mfs/var/mfs/

[root@centos1 mfs] # cp metadata.mfs.empty metadata.mfs

5. Configuration file (only need to know, no need to change)

① / / mfsmaster.cfg (main configuration file) / usr/local/mfs/etc/mfs directory

# WORKING_USER = mfs the user running master server

# WORKING_GROUP = mfs the group running master server

# SYSLOG_IDENT = the identity of mfsmaster master server in syslog, indicating that it is generated by master serve

# LOCK_MEMORY = 0 whether to execute mlockall () to avoid mfsmaster process overflow (default is 0)

# NICE_LEVEL =-19 priority to run (if default is-19; Note: the process must be started with root)

# EXPORTS_FILENAME = / usr/local/mfs/etc/mfsexports.cfg the location of the attached directory and its permission control files

# TOPOLOGY_FILENAME = / usr/local/mfs/etc/mfs/mfstopology.cfg

# DATA_PATH = / usr/local/mfs/var/mfs data storage path. There are roughly three types of files in this directory, changelog,sessions and stats

# BACK_LOGS = the number of changed log files for 50 metadata (default is 50)

# BACK_META_KEEP_PREVIOUS = 1

# REPLICATIONS_DELAY_INIT = 300s for delayed replication (default is 300s)

# REPLICATIONS_DELAY_DISCONNECT = 3600 chunkserver fractured replication latency (default is 3600)

# MATOML_LISTEN_HOST = * IP address of metalogger listener (default is *, which represents any IP)

# MATOML_LISTEN_PORT = 9419 metalogger listening port address (default is 9419)

# MATOML_LOG_PRESERVE_SECONDS = 600

# MATOCS_LISTEN_HOST = * IP address for chunkserver connections (default is *, representing any IP)

# MATOCS_LISTEN_PORT = 9420 Port address for chunkserver connections (default is 9420)

# MATOCU_LISTEN_HOST = * IP address for client hanging connections (default is *, representing any IP)

# MATOCU_LISTEN_PORT = 9421 Port address for client hanging connections (default is 9421)

# CHUNKS_LOOP_MAX_CPS = 100000

# CHUNKS_LOOP_MIN_TIME = loopback frequency of 300chunks (default: 300s)

② mfsexportes.cfg is the control file of the shared mfs file system. When NFS wants to share a directory, we will use the vim / etc/exports command to write the content to whom, the directory to be shared, and the attributes shared. The function of mfsexports.cfg is similar to its writing format as follows: configuration of the client.

Allow everything but "meta"

* / rw,alldirs,maproot=0

Client (client) supports formats: ip, ip/netmask, ip/ bit mask, ip-ip, *

Each entry in the file is divided into three parts:

Part one: the ip address of the client

Part II: the directory to be attached

Part III: permissions held by the client

/ / several forms of address can be specified:

* all ip addresses

N.n.n.n single ip address

N.n.n.n/b IP Network address / bit Mask

N.n.n.n/m.m.m.m IP network address / subnet mask

F.f.f.f-t.t.t.t IP segment

/ / the directory section needs to pay attention to two points:

/ identify the MooseFS root

. Represents the MFSMETA file system

/ / permission section:

Ro read-only mode sharing

Rw read and write sharing

Alldirs allows you to mount any specified subdirectory

6. Start the service / usr/local/mfs/sbin/mfsmaster start

[root@centos1 mfs] # netstat-anpt

Active Internet connections (servers and established)

Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name

Tcp 0 0 0.0.0 0 9419 0.0.0 0 V * LISTEN 7599/mfsmaster

Tcp 0 0 0.0.0.0 9420 0.0.0. 0 LISTEN 7599/mfsmaster

Tcp 0 0 0.0.0.0 9421 0.0.0. 0 LISTEN 7599/mfsmaster

Tcp 0 0 192.168.1.1:9419 192.168.1.2:37816 ESTABLISHED 7599/mfsmaster

Tcp 0 0 192.168.1.1:9420 192.168.1.3:48107 ESTABLISHED 7599/mfsmaster

Tcp 0 0:: 1:25: * LISTEN 1732/master

7. View the process

[root@centos1 mfs] # ps-ef | grep mfs

8. Stop mfsmaster

[root@centos1 mfs] # / usr/local/mfs/sbin/mfsmaster-s or stop

Second, set up metalogger server

1. Install dependency packages

[root@centos1 ~] # yum-y install zlib-devel

two。 Create a user

[root@centos1] # useradd mfs-s / sbin/nologin

3. Install the source package

[root@centos1] # tar xf / mnt/mfs-1.6.27-5.tar.gz-C / usr/src

[root@centos1 mfs-1.6.27] # cd / usr/src/mfs-1.6.27/

[root@centos1 mfs-1.6.27] # / configure-- prefix=/usr/local/mfs-- with-default-user=mfs-- with-default-group=mfs-- disable-mfschunkserver-- disable-mfsmount

[root@centos1 mfs-1.6.27] # make & & make install

4. Copy a file

[root@centos1 mfs-1.6.27] # cd / usr/local/mfs/etc

[root@centos2 mfs] # cp mfsmetalogger.cfg.dist mfsmetalogger.cfg

[root@centos2 mfs] # vim mfsmetalogger.cfg

MASTER_HOST=192.168.1.1 / / manually add the ip address of MASTER SERVER

5. Start the service

[root@centos2 mfs] # / usr/local/mfs/sbin/mfsmetalogger start

Third, set up chunk server

1. Install dependency packages

[root@centos1 ~] # yum-y install zlib-devel

two。 Create a user

[root@centos1] # useradd mfs-s / sbin/nologin

3. Installation

[root@centos1] # tar xf / mnt/mfs-1.6.27-5.tar.gz-C / usr/src

[root@centos1 mfs-1.6.27] # cd / usr/src/mfs-1.6.27/

[root@centos3 mfs-1.6.27] # / configure-- prefix=/usr/local/mfs-- with-default-user=mfs-- with-default-group=mfs-- disable-mfsmaster-- disable-mfsmount

[root@centos3 mfs-1.6.27] # make & & make install

4. Copy Profil

[root@centos3 mfs] # cd / usr/local/mfs/etc/mfs/

# cp mfschunkserver.cfg.dist mfschunkserver.cfg

[root@centos3 mfs] # cp mfshdd.cfg mfshdd.cfg

5. Modify the configuration file

[root@centos3 mfs] # vim mfschunkserver.cfg

MASTER_HOST = 192.168.1.1 / / ip address of the master server

[root@centos3 mfs] # vim mfshdd.cfg

/ data / / sets the directory where chunkserver stores data

4. Set directory permissions and start mfschunkserver

[root@centos3 mfs] # mkdir / data / / create / data

[root@centos3 mfs] # chown-R mfs:mfs / data / / set permissions

[root@centos3 mfs] # / usr/local/mfs/sbin/mfschunkserver start / / start the service

Fourth, build a second chunkserver

1. [root@centos3] # useradd mfs-s / sbin/nologin

two。 Installation

[root@centos4 mfs-1.6.27] # tar xf / mnt/mfs-1.6.27-5.tar.gz-C / usr/src

[root@centos4 mfs-1.6.27] # cd / usr/src/mfs-1.6.27/

[root@centos4 mfs-1.6.27] # / configure-- prefix=/usr/local/mfs-- with-default-user=mfs-- with-default-group=mfs-- disable-mfsmaster-- disable-mfsmount

[root@centos4 mfs-1.6.27] # make & & make install

3. Copy Profil

[root@centos3 mfs-1.6.27] # cd / usr/local/mfs/etc/mfs

[root@centos4 mfs] # cp mfschunkserver.cfg.dist mfschunkserver.cfg

[root@centos4 mfs] # cp mfshdd.cfg.dist mfshdd.cfg

4. Modify the configuration file

[root@centos3 mfs] # vim mfschunkserver.cfg

MASTER_HOST = 192.168.1.1 / / ip address of the master server

[root@centos3 mfs] # vim mfshdd.cfg

/ data / / the directory where the data is stored

5. Set directory permissions and start mfschunkserver

# mkdir / data

[root@centos3 mfs] # chown-R mfs:mfs / data

[root@centos3 mfs] # / usr/local/mfs/sbin/mfschunkserver start

V. client configuration

1. Install fuse

The function of fuse is briefly summarized. Fuse provides an interface for file operations, and then fuse listens for file operations on this interface and transmits these file operations to our own related code for processing. In other words, fuse provides a mount point, and then he listens at this mount point. Once operations (new, read, modify) occur, he will hand over control to our code, which is implemented in the mfsmount module.

[root@centos1 fuse-2.9.2] # tar xf / mnt/fuse-2.9.2.tar.gz-C / usr/src

[root@centos1 fuse-2.9.2] # cd / usr/src/fuse-2.9.2/

[root@centos1 fuse-2.9.2] #. / configure

[root@centos1 fuse-2.9.2] # make & & make install

two。 Set environment variabl

[root@centos1 fuse-2.9.2] # vim / etc/profile

The export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig:$PKG_CONFIG_PATH / / environment variable PKG_CONFIG_PATH is used to set the search path for .pc files.

[root@centos1 fuse-2.9.2] # source / etc/profile / / refresh environment variables

3. Install the mfs client

[root@centos3] # useradd-s / sbin/nologin mfs

[root@centos1 mfs-1.6.27] # tar xf / mnt/mfs-1.6.27-5.tar.gz-C / usr/src

[root@centos1 mfs-1.6.27] # cd / usr/src/mfs-1.6.27/

[root@centos1 mfs-1.6.27] # / configure-- prefix=/usr/local/mfs-- with-default-user=mfs-- with-default-group=mfs-- disable-mfsmaster-- disable-mfschunkserver-- enable-mfsmount

[root@centos1 mfs-1.6.27] # make&&make install

4. Mount the mfs file system

# mkdir / test / / create a mount point

[root@centos1 test] # modprobe fuse// loads fuse module into kernel

[root@centos1 test] # / usr/local/mfs/bin/mfsmount / test-H 192.168.1.1 pick / mount mfs

Check the mount status:

Common operations of 5.mfs

1. Set environment variabl

[root@centos1 test] # vim / etc/profile

Export PATH=/usr/local/mfs/bin:$PATH

[root@centos1 test] # source / etc/profile

two。 View and set the number of copies of files

[root@centos1 test] # mfsgetgoal-r / test

3. test

[root@centos1 test] # dd if=/dev/zero of=/test/111.data bs=200M count=1

View the write results:

[root@centos1 test] # mfssetgoal-r 2 / test for redundancy testing

[root@centos1 test] # dd if=/dev/zero of=/test/222.data bs=200M count=1

Check to see if the write results are redundant:

VI. Mfs monitoring

1. [root@centos1 mfs] # / usr/local/mfs/sbin/mfscgiserv / / master server write

two。 Monitor the test through the website:

7. MFS cluster maintenance

1. Start the MFS cluster

A safe way to start a MooseFS cluster (to avoid any read or write incorrect data or similar problems) is to follow these command steps:

1. Start the mfsmaster process

two。 Start all mfschunkserver processes

3. Start the mfsmetalogger process (if mfsmetalogger is configured)

4. When all the chunkservers are connected to the MooseFS master, any number of clients can use mfsmount to mount the export file system. You can check the master log or the CGI monitor to see if all chunkserver is connected.

two。 Stop the MFS cluster

Stop the MooseFS cluster securely:

1. Unmount the MooseFS file system on all clients (using the umount command or other equivalent command)

two。 Stop the chunkserver process with the mfschunkserver stop command

3. Stop the metalogger process with the mfsmetalogger stop command

4. Stop the master process with the mfsmaster stop command

3.MFS metadata backup

Usually metadata has two parts of data:

The main metadata file metadata.mfs, which is named metadata.mfs.back when mfsmaster is running

The metadata change log changelog.*.mfs stores file changes over the past N hours (the value of N is set by the BACK_LOGS parameter, and the parameter is set in the mfschunkserver.cfg configuration file).

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.