Shulou(Shulou.com)06/01 Report--

This article will explain in detail what MooseFS is and how to deploy the application. The content of the article is of high quality, so the editor will share it with you for reference. I hope you will have some understanding of the relevant knowledge after reading this article.

Introduction to mooseFS

The official launch of MooseFS is from 2008-05-30 to 2009-10-12. The latest version is 1.5.12.

MooseFS is a network distributed file system. It distributes data across multiple servers, but for users, all they see is a source. MFS, like other unix-like file systems, contains a hierarchical structure (directory tree), stores file attributes (permissions, last access and modification time), and can create special files (block devices, character devices, pipes, sockets), symbolic links, hard links.

The composition of mooseFS system

1.master (metadata server)

Master is responsible for managing data throughout the system. Is the maintainer of the entire system. But it has a weakness that there is only one master! In other words, if the master breaks down, the whole system will stop working! However, this is not an Achilles' heel, because after we have made a backup of the data, it is very easy to restore master.

2.chunkserver (data Storage Server)

Chunkserver is the data store in the mfs system. According to the algorithm, the real user data is divided into chunk and distributed to each chunkserver. This ensures the security of the data.

3.client

Any machine that uses the mfs file system can be called client. Client is a user of the mfs system. When client mounts the mfs filesystem locally, it can use mfs as if it were a normal disk partition.

Test environment

This article only introduces the deployment and application of moosefs, for the purpose of the experiment, and does not consider the performance factors, so all servers are implemented using virtual machines. In the future, you will have the opportunity to do a separate physical server, and then test its performance.

Topological graph

Note:

(1) chunkserver1-4 all add a virtual disk sda (disk space must be greater than 1G, I set 4G here), and mount it to the system / data. All servers upgrade the kernel to version 2.6.18-164. Why upgrade the kernel is described below.

(2) all servers have installed the full version of mfs components.

Installation and configuration

Prerequisites for installing mfs

Because the mfs client program, that is, the command to load the mfs disk system, is written in fuse, as long as you want to mount the mfs server, the necessary prerequisite is to install fuse first, so that mfs can be compiled smoothly. Another point to note: the fuse module is already built into the linux kernel in the linux 2.6.18-164.e15 version. However, this module was not included in the linux kernel prior to this version. In addition, fuse removed the source code of the fuse system module from the source package of the 2.8.0-pre1 version for the reason mentioned in the previous point. In this way, we are compiling and installing the latest version of fuse (version 2.8or above) and the linux kernel version we are using is lower than 2.6.18-164.e15 version, then the system does not contain fuse modules.

The solution:

1. Upgrade the system kernel to version 2.6.18-164

For convenience, you can directly use yum to upgrade the system kernel. Yum install kernel

Restart the system after successful installation

2. Compile and install using fuse 2.7x version

This version of fuse contains the fuse module needed by the linux kernel. Specify the-enable-kernel-module option when configuring the compilation option, and the corresponding fuse module will be compiled when make. Make intall will copy the fuse.ko to the / lib/modules/ `uname-r` / kernel/fs/fuse/ directory and use it after installation.

# modprobe-l | grep fuse/ lib/modules/2.6.18-128.el5xen/kernel/fs/fuse/fuse.ko

Check to see if the installation is normal

3. Use yum to install the current kernel version of the fuse module

Yum install-y dkms-fuse dkms

Install fuse

It is not necessary to install fuse if only the metadata server or data storage server is compiled. Only mfsmount requires fuse support (fuse's development package is required at compile time, and fuse.ko system modules are required for mounting using mfsmount). You can install fuse using source code or yum

1. Source code installation

# wget http://ncu.dl.sourceforge.net/project/fuse/fuse-2.X/2.8.1/fuse-2.8.1.tar.gz # tar-xvzf fuse-2.8.1.tar.gz # cd fuse-2.8.1 #. / configure-prefix=/usr/-libdir=/usr/lib64 # make & & make install

Since my system is 64-bit, I set the lib directory to / usr/lib64 when compiling fuse. In this way, when you compile and install MFS below, you won't get an error because you can't find the lib file for fuse.

2. Yum installation

# yum install-y fuse fuse-devel

Install mfs

# useradd mfs-s / sbin/nologin #. / configure-prefix=/usr/local/mfs-with-default-user=mfs-with-default-group=mfs-enable-mfsmount # make & & make install

View the directory structure after installation

# ll / usr/local/mfs/ total 20 drwxr-xr-x 2 root root 4096 Oct 14 15:14 bin drwxr-xr-x 2 root root 4096 Oct 14 12:13 etc drwxr-xr-x 2 root root 4096 Oct 14 15:14 sbin drwxr-xr-x 3 root root 4096 Oct 14 12:13 share drwxr-xr-x 3 root root 4096 Oct 14 12:13 var

Bin-client tool

Etc-metadata server, the configuration files of the data storage server are placed in this directory

Sbin-metadata server program mfsmaster, data storage server side service program mfschunkserver

Share-document

Var-metadata directory (can be customized to other directories in the configuration file)

Configure master (metadata server)

IP:192.168.108.108

[root@master~] # vi/usr/local/mfs/etc/mfsmaster.cfg # WORKING_USER=mfs # WORKING_GROUP=mfs # LOCK_FILE=/var/run/mfs/mfsmaster.pid # DATA_PATH=/usr/local/mfs/var/mfs # SYSLOG_IDENT=mfsmaster # BACK_LOGS=50 # REPLICATIONS_DELAY_INIT=300 # REPLICATIONS_DELAY_DISCONNECT=3600 MATOCS_LISTEN_HOST=192.168.108.108 # MATOCS_LISTEN_PORT=9420 # MATOCU_LISTEN_HOST=* # MATOCU_LISTEN_PORT=9421 # CHUNKS_LOOP_TIME=300 # CHUNKS_DEL_LIMIT=100 # CHUNKS_REP_LIMIT=15

Note: all annotated settings in this configuration file are the default configuration. Here I only change the value of MATOCS_LISTEN_HOST, that is, I change it to the native ip address: 192.168.108.108. You can also modify the settings of DATA_PATH to store the metadata directory to another partition or disk if necessary. Other parameters are simple to adjust as needed.

Master will open port 9420 and wait for mfschunkserver connection.

Start mfsmaster

[root@master~] # / usr/local/mfs/sbin/mfsmaster [root@master~] # ps-ef | grepmfsmaster | grep-vgrep mfs101321017:37?00:00:00/usr/local/mfs/sbin/mfsmaster [root@master~] # netstat-tulnp | grepmfsmaster tcp00192.168.108.108:94200.0.0.0:*LISTEN10132/mfsmaster tcp000.0.0.0:94210.0.0.0:*LISTEN10132/mfsmaster

View Syslog

[root@master~] # tail-f/var/log/messages Oct1417:37:35mastermfsmaster:config:usingdefaultvalueforoption'SYSLOG_IDENT'-'mfsmaster' Oct1417:37:35mastermfsmaster [10130]: config:usingdefaultvalueforoption'WORKING_USER'-'mfs' Oct1417:37:35mastermfsmaster [10130]: config:usingdefaultvalueforoption'WORKING_GROUP'-'mfs'... Oct1417:37:35mastermfsmaster [10132]: config:usingdefaultvalueforoption'CHUNKS_LOOP_TIME'-'300'

# A lot of the above log contents are omitted, mainly the process of the program reading the configuration.

The following section is to check the metadata and check the data storage server (every 1 minute).

# since I haven't started chunkservers yet, the display result in chunkservers status: is empty. The result of total: usedspace: is also empty.

Oct1417:38:00mastermfsmaster [10132]: inodes:45 Oct1417:38:00mastermfsmaster [10132]: dirnodes:3 Oct1417:38:00mastermfsmaster [10132]: filenodes:42 Oct1417:38:00mastermfsmaster [10132]: chunks:14 Oct1417:38:00mastermfsmaster [10132]: chunkstodelete:0 Oct1417:38:00mastermfsmaster [10132]: chunkserversstatus: Oct1417:38:00mastermfsmaster [10132]: total:usedspace:0 (0GB), totalspace:0 (0GB), usage:0.00%

Setup service starts with the system

[root@master~] # echo "/ usr/local/mfs/sbin/mfsmaster" > > / etc/rc.local

Configure chunkserver (data Storage Server)

IP:192.168.108.161~164

[root@chunkserver-1~] # vi/usr/local/mfs/etc/mfschunkserver.cfg # WORKING_USER=mfs # WORKING_GROUP=mfs # DATA_PATH=/usr/local/mfs/var/mfs # LOCK_FILE=/var/run/mfs/mfschunkserver.pid # SYSLOG_IDENT=mfschunkserver # BACK_LOGS=50 # MASTER_RECONNECTION_DELAY=30 MASTER_HOST=192.168.108.108 # MASTER_PORT=9420 # MASTER_TIMEOUT=60 # CSSERV_LISTEN_HOST=* # CSSERV_LISTEN_PORT=9422 # CSSERV_TIMEOUT=60 # CSTOCS_TIMEOUT=60 # HDD _ CONF_FILENAME=/usr/local/mfs/etc/mfshdd.cfg

Configure Storage Partition

[root@chunkserver-1~] # vi/usr/local/mfs/etc/mfshdd.cfg Delete / mnt/hd1 / mnt/hd2 / mnt/hd3 / mnt/hd4

Add a separate partition

/ data

Change the partition owner to mfs

[root@chunkserver-1~] # chownmfs.mfs/data

Note: the main configuration of the mfschunkserver server is very simple, there are no special requirements, only need to change the address of the MASTER_HOST. The configuration of the storage partition chooses a separate disk partition (the partition must be larger than 1G).

Start mfschunkserver

[root@chunkserver-1~] # / usr/local/mfs/sbin/mfschunkserver [root@chunkserver-1~] # netstat-an | grep9420 tcp00192.168.108.161:15099192.168.108.108:9420ESTABLISHED

View the system log at the same time

Oct1417:53:45vm_web_1mfschunkserver [1992]: connecting... Oct1417:53:45vm_web_1mfschunkserver [1992]: connectedtoMaster

Indicates that you have successfully connected to the master server

Then check the log of master.

Oct1417:59:00experimentmfsmaster [10132]: server1 (192.168.108.161): usedspace:560484352 (0GB), totalspace:4226125824 (3GB), usage:13.26% Oct1417:59:00experimentmfsmaster [10132]: total:usedspace:560484352 (0GB), totalspace:4226125824 (3GB), usage:13.26%

It also shows that there is a chunkserver connected to it, and gives the usage space and the remaining disk space.

After the remaining three mfschunkserver are configured using the same method, start the mfschunkserver program, and then check the master system log:

Oct1615:27:00experimentmfsmaster [10132]: inodes:9 Oct1615:27:00experimentmfsmaster [10132]: dirnodes:1 Oct1615:27:00experimentmfsmaster [10132]: filenodes:8 Oct1615:27:00experimentmfsmaster [10132]: chunks:18 Oct1615:27:00experimentmfsmaster [10132]: chunkstodelete:0 Oct1615:27:00experimentmfsmaster [10132]: chunkserversstatus: Oct1615:27:00experimentmfsmaster [10132]: server1 (192.168.108.162): usedspace:924282880 (0GB), totalspace:4226125824 (3GB) Usage:21.87% Oct1615:27:00experimentmfsmaster [10132]: server2 (192.168.108.164): usedspace:924282880 (0GB), totalspace:4226125824 (3GB), usage:21.87% Oct1615:27:00experimentmfsmaster [10132]: server3 (192.168.108.163): usedspace:924282880 (0GB), totalspace:4226125824 (3GB), usage:21.87% Oct1615:27:00experimentmfsmaster [10132]: server4 (192.168.108.161): usedspace:924028928 (0GB), totalspace:4226125824 (3GB) Usage:21.86% Oct1615:27:00experimentmfsmaster [10132]: total:usedspace:3696877568 (3GB), totalspace:16904503296 (15GB), usage:21.87%

Here you can see that four chunkserver have been connected to the master.

Client mount and tool usage

192.168.108.109

Mount MFS

Mfsmount

[root@client~] # mkdir/mnt/mfs [root@client~] # mfsmount-h292.168.108.108 * mfsmount loads partitions into the / mnt/mfs directory by default. If you want to load to another directory, use the-w parameter. Mfsmount-help usage:/usr/local/mfs/bin/mfsmount [- r] [- m] [- c] [- v0.. 2] [- hmasterhost] [- pmasterport] [- lpath] [- wmountpoint] r:readonlymode m:mountmetadata c:allowusingcache v:verboselevel defaults: h:mfsmaster Plux 9421 lvl / w:/mnt/mfs

Modify MFS file deletion delay time

Mfsrsettrashtime [root@client~] # mfsrsettrashtime100/mnt/mfs/ / mnt/mfs/: inodeswithtrashtimechanged:1 (1) inodeswithtrashtimenotchanged:0 (0) inodeswithpermissiondenied:0 (0)

View MFS file deletion delay time

Mfsrgettrashtime # mfsrgettrashtime/mnt/mfs/ / mnt/mfs/: directorieswithtrashtime100:1 (1) mfssettrashtime [root@client~] # mfssettrashtime60/mnt/mfs/ / mnt/mfs/:60 mfsgettrashtime [root@client~] # mfsgettrashtime/mnt/mfs/ / mnt/mfs/:60 sets the number of files saved mfssetgoal [root@client~] # mfssetgoal4/mnt/mfs/ / mnt/mfs/:4 View the number of file saves mfsgetgoal [root@client~] # mfsgetgoal/mnt/mfs/ / mnt/mfs /: 4mfsrsetgoal [root@client~] # mfsrsetgoal3/mnt/mfs/ / mnt/mfs/: inodeswithgoalchanged:1 (1) inodeswithgoalnotchanged:0 (0) inodeswithpermissiondenied:0 (0) mfsrgetgoal [root@client~] # mfsrgetgoal/mnt/mfs/ / mnt/mfs/: directorieswithgoal3:1 (1)

View (check) command for files (folders)

Mr. into a 10m file [root@client~] # ddif=/dev/zeroof=filebs=1Mcount=10 [root@client~] # cpfile/mnt/mfs file information view mfsfileinfo [root@client~] # mfsfileinfo/mnt/mfs/file / mnt/mfs/file: chunk0:0000000000000001_00000001/ (id:1ver:1) copy1:192.168.108.161:9422 copy2:192.168.108.162:9422 copy3:192.168.108.163:9422 file check mfscheckfile [root@client~] # mfscheckfile/ Mnt/mfs/file / mnt/mfs/file: 3copies:1chunks

Directory information view

Mfsdirinfo

[root@client~] # rm/mnt/mfs/file [root@client~] # mkdir/mnt/mfs/newdir [root@client~] # cpfile/mnt/mfs/newdir [root@client~] # mfsdirinfo/mnt/mfs/newdir / mnt/mfs/newdir/: inodes:2 (2) / / inode number, 1 directory 1 file directories:1 (1) / / 1 directory files:1 (1) / 1 file goodfiles:1 (1) / / normal number of files undergoalfiles:0 (0) missingfiles:0 (0) chunks:1 (1) goodchunks:1 (1) undergoalchunks:0 (0) missingchunks:0 (0) length:10M (10485760) size:10M (10490880) / here is the file size hddusage:30M (31472640) / / because I set the number of files to 3 So this is 3000100m [root@client~] # cp/data/filenewdir/file_2 / mnt/mfs/newdir/: inodes:3 (3) directories:1 (1) files:2 (2) goodfiles:2 (2) undergoalfiles:0 (0) missingfiles:0 (0) chunks:2 (2) goodchunks:2 (2) undergoalchunks:0 (0) missingchunks:0 (0) length:20M (20971520) size:20M (20981760) hddusage:60M (62945280)

Create a file snapshot

Mfssnapshot

[root@client/mnt/mfs/newdir] # mfssnapshotmysnapshotfile [root@client/mnt/mfs/newdir] # ll total30720-rw-r-r-1rootroot10485760Oct1616:43file-rw-r-r-1rootroot10485760Oct1616:44file_2-rw-r-r-1rootroot10485760Oct1616:58mysnapshot [root@monitor/mnt/mfs/newdir] # mfsfileinfomysnapshot mysnapshot: chunk0:0000000000000003_00000001/ (id:3ver:1) copy1:192.168.108.161:9422 copy2:192.168.108.162:9422 copy3:192.168.108.163: 9422 [root@monitor/mnt/mfs/newdir] # mfsfileinfofile file: chunk0:0000000000000003_00000001/ (id:3ver:1) copy1:192.168.108.161:9422 copy2:192.168.108.162:9422 copy3:192.168.108.163:9422

The mfsfileinfo command allows you to view the created file snapshot, which takes up only one inode and does not take up disk space, just like the ln command creates a hard link. But there seems to be something wrong with the display of mfsdirinfo:

[root@monitor/mnt/mfs/newdir] # mfsdirinfo/mnt/mfs/newdir/ / mnt/mfs/newdir/: inodes:4 (4) directories:1 (1) files:3 (3) goodfiles:3 (3) undergoalfiles:0 (0) missingfiles:0 (0) chunks:3 (3) goodchunks:3 (3) undergoalchunks:0 (0) missingchunks:0 (0) length:30M (31457280) size:30M (31472640) hddusage:90M (94417920)

Using mfsdirinfo to view the directory information, it is found that the snapshot file you just created also takes up the same disk space as the original file. But is this really the case?

View master's Syslog

Oct1617:14:00experimentmfsmaster [24445]: inodes:5 Oct1617:14:00experimentmfsmaster [24445]: dirnodes:2 Oct1617:14:00experimentmfsmaster [24445]: filenodes:3 Oct1617:14:00experimentmfsmaster [24445]: chunks:2 Oct1617:14:00experimentmfsmaster [24445]: chunkstodelete:0

Through the chunks: 2 of the log, we can know that there are 2 chunks who have just looked at the file and learned that the file file occupies 1 chunks, then 2 chunks means that there are 2 files. Looking up at the line filenodes: 3 indicates that the number of inode files is 3 means that there are 3 files, which is exactly 2 files + 1 snapshot file.

View the mfs files of each mfschunkserver

By looking at the master log and the mfs file on each data storage server, verify the results we just checked with mfsfileinfo! A file snapshot, which takes up only one inode and does not take up disk space, just like the ln command creates a hard link.

So much for sharing about what MooseFS is and how to deploy applications. I hope the above content can be helpful to you and learn more. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.