Shulou(Shulou.com)06/02 Report--

I. brief introduction of distributed storage system

With the continuous development of information technology, it brings convenience to us at the same time, the increasing magnitude of data, the increasingly complex connection between information, the increasing concurrency of data access, the higher and higher requirements for Ihand O, the increasing complexity of data types and other problems have become problems that need to be solved urgently for the rapid development of information technology. The emergence of distributed storage system has solved most of the above problems to a great extent.

Distributed storage system is to store data on multiple independent devices. Traditional network storage systems use centralized storage servers to store all data. Storage servers become the bottleneck of system performance and the focus of reliability and security, which can not meet the needs of large-scale storage applications. The distributed storage system adopts a scalable system structure, uses multiple storage servers to share the storage load, and uses the location server to locate and store information. it not only improves the reliability, availability and access efficiency of the system, but also is easy to expand.

Distributed storage system can be divided into general distributed storage and dedicated distributed storage in terms of interface types. General distributed storage means that there is no file system interface and needs to be accessed through API interface; dedicated distributed storage is also known as distributed file system, they generally have file system interface and can be mounted directly. General distributed storage includes mogilefs, fastdfs, etc., and special distributed storage systems include moosefs, etc.

II. Mogilefs

MogileFS is an open source distributed file storage system, MogileFS is suitable for storing a large number of small files in the work scene, developed by Danga Interactive company under LiveJournal, the team has developed a number of well-known open source projects, including Memcached, MogileFS, Perlbal and so on.

1.Mogilefs architecture diagram:

two。 The components that make up the Mogliefs:

1.Trackers (mogilefsd): the core component of Mogliefs, the main functions are (Replication) node file copy, (Deletion) file deletion, (Query) metadata query, (Monitor) health monitoring, (Reaper) storage failure reset and so on. It is often called a metadata server, but instead of storing metadata, it stores it in a database such as MySQL. To ensure the reliability of the architecture, there are generally multiple Trackers. Trackers can be thought of as a side-hanging agent that is only responsible for processing metadata information.

two。 Database: the database is used to store Mogliefs metadata, while Trackers manages the data. Therefore, it is usually recommended to do HA.

3.mogstored (storage node): the place where the actual files are stored. At least two copies of the actual document are usually saved.

3. Example demonstrates the topology diagram

Install Trackers and mogstored on three nodes at the same time, and select one of the nodes to install MySQL. In a production environment, it is best to deploy MySQL separately and replicate as a master and slave. Trackers and mogstored can also be deployed separately on different nodes, which need to be determined according to the actual production environment. Here is mainly to demonstrate mogilefs, not to do MySQL master-slave copy demonstration. If you want Mogilefs to be mounted, you can use FUSE to do so.

It should be noted that the file URL stored in mogilefs is very special (the production process of mogilefs file name will be explained later). For example, when storing a picture, the file URL may be in a format like 6060/0000/0000/0000/00000021.jpg, which is not very friendly to users, and users may need intuitive URL like image.hello.com/21.jpg. So Nginx is usually used instead of Mogilefs.

4. System environment and installation

Mogilefs is a relatively old but mature distributed storage. Considering that there may be compatibility problems on Centos7, Centos6 will demonstrate it here.

Operating system: CentOS release 6.6

Mogilefs:2.46

Nginx:1.10

Mysql:5.1

IP allocation:

N1:192.168.29.111,n2:192.168.29.112,n3:192.168.29.113,n4:192.168.29.114

The structure is shown in the figure above.

1. Install MySQL, mogilefsd, mogstored on N1 node and configure N1 as Trackers, Storage Node

MySQL is installed directly with yum.

~] # yum install-y mysql mysql-server

To install the Trackers and Storage Node components of Mogilefs, be sure to install Perl-related dependency packages. The dependency packages are:

Perl-Danga-Socket-1.61-1.el6.rf.noarch.rpmperl-IO-stringy-2.110-1.2.el6.rfx.noarch.rpmperl-Net-Netmask-1.9015-8.el6.noarch.rpmPerlbal-1.78-1.el6.noarch.rpmperl-Perlbal-1.78-1.el6.noarch.rpmPerlbal-doc-1.78-1.el6.noarch.rpmperl-IO-AIO-3.71-2.el6.x86_64.rpm

The above dependency packages must be installed before you can install Mogilefs. Install the components:

Yum install-y MogileFS-Server-mogstored-2.46-2.el6.noarch.rpm MogileFS-Server-mogilefsd-2.46-2.el6.noarch.rpm MogileFS-Server-2.46-2.el6.noarch.rpm

Configure MogileFS-Server-mogilefsd:

~] # vim / etc/mogilefs/mogilefsd.conf # Mogilfs Trackers's main configuration file # Enable daemon mode to work in background and use syslogdaemonize = 1 # whether to run as a daemon. # Where to store the pid of the daemon (must be the same in the init script) pidfile = / var/run/mogilefsd/mogilefsd.pid # pid file path # Database connection informationdb_dsn = DBI:mysql:mogilefs:host=192.168.29.111 # address of the database db_user = moguser # user name and password of the configuration database db_pass = 12345 user IP:PORT to listen on for mogilefs client requestslisten = 0.0.0.0 address and port of 7001 # listener # Optional If you don't define the port above.conf_port = 7001 # default port # Number of query workers to start by default.query_jobs = 10 # number of query processes # Number of delete workers to start by default.delete_jobs = "Number of replicate workers to start by default.replicate_jobs =" Number of reaper workers to start by default.# (you don't usually need to increase this) reaper_jobs = "Number of fsck workers to start by default.# (these can cause a lot of load) When fsck'ing) # fsck_jobs = "Minimum amount of space to reserve in megabytes# default: 10" Consider setting this to be larger than the largest file you# would normally be uploading.#min_free_space = 20 "Number of seconds to wait for a storage node to respond.# default:" Keep this low So busy storage nodes are quickly ignored.#node_timeout = "Number of seconds to wait to connect to a storage node.# default:" Keep this low so overloaded nodes get skipped.#conn_timeout = "Allow replication to use the secondary node get port,# if you have apache or similar configured for GET's#repl_use_get_port = 1"

After modifying the configuration, go to the database to create a root user who can connect remotely, or initialize the database with mogdbsetup:

Mysql > GRANT ALL ON mogilefs.* TO 'moguser'@'192.168.29.%' IDENTIFIED BY' 123456users; # create user moguser, have all permissions to manage the mogilefs library, and allow users of 192.168.29.* to connect remotely. Mysql > FLUSH PRIVILEGES;mysql > quit~] # mogdbsetup-dbhost=127.0.0.1-dbuser=moguser-dbpass=123456

After initialization, you can see the created mogilefs library and the tables in the MySQL:

Start mogilefs and confirm that port 7001 is listening:

~] # service mogilefsd startStarting mogilefsd [OK] ~] # ss-lnt

Note: Trackers service can be installed on N2 and N3 nodes, so that the risk of single point of failure can be eliminated and the pressure can be averaged.

3. Configure Storage Node on N1

The path to the Storage Node profile is / etc/mogilefs/mogstored.conf:

~] # vim / etc/mogilefs/mogstored.confmaxconns = 10000 # maximum number of concurrent connections httplisten = 0.0.0.0 Mogilefs 7500 # data transmission is realized through http protocol, here is the listening address and port mgmtlisten = 0.0.0.0httplisten 7501 # health monitoring listening address and port docroot = / mogliefs/mogdata # data storage path. The group and owner of the directory must be mogilefs

Create a data storage directory and modify the group and owner to mogilefs:

~] # mkdir-pv / mogliefs/mogdata~] # chown-R mogilefs.mogilefs / mogliefs/

Start mogstored to see if the process starts normally and whether the port is listening:

~] # service mogstored start~] # ss-lnt # listening ports are 7500 and 7501

4. Follow the steps of N1 to install Mogilefs on nodes N2 and n3, and copy the configuration files on N1 to N2 and n3.

~] # scp / etc/mogilefs/*.conf root@192.168.29.112:/etc/mogilefs/~] # scp / etc/mogilefs/*.conf root@192.168.29.113:/etc/mogilefs/

Start the mogstored service and confirm the listening:

~] # service mogstored start~] # ss-lnt # listening ports are 7500 and 7501

5. Used on N1 to cluster all nodes with the mogadm command.

Add a storage node and check:

1] # mogadm host add 192.168.29.111-- ip=192.168.29.111-- port=7500-- status=alive2 ~] # mogadm host add 192.168.29.112-- ip=192.168.29.112-- port=7500-- status=alive3 ~] # mogadm host add 192.168.29.113-- ip=192.168.29.113-- port=7500-- status=alive~] # mogadm check

If you want the storage in the mogilefs cluster to be recognized as different devices, you need to create a directory called dev* under the / mogliefs/mogdata directory created so that each node is used as a storage device. Mogilefs stores redundancy in different devices, and each node should be identified as a different device.

Create a dev1,dev2,dev3 directory under the / mogliefs/mogdata/ directory on N1, N2, and n3, respectively, and add devices on Trackers:

1 ~] # mogadm device add 192.168.29.111 12 ~] # mogadm device add 192.168.29.112 23 ~] # mogadm device add 192.168.29.113 3

6. Create Domain and Class

In Mogilefs, in order to facilitate file copy management on multiple nodes, it is usually not managed by files in the device, but by class (class), replication and deletion are carried out with class as the minimum unit. There are many files in each class, and the volume of the class is not fixed.

In the storage space of Mogilefs, all data files are in the same plane, so the same name cannot happen, which will affect the flexibility of Mogilefs, so the concept of Domain (namespace) is introduced. Domain contains Class, which can have the same file name in different Domain.

~] # mogadm domain add imgs # create a Domain~ named imgs] # mogadm domain add text # create a Domain~ named text] # mogadm domain list # View Domain list

You can customize the properties of Class in the format: mogadm class add [opts]

~] # mogadm class add imgs png-- mindevcount=3-- hashtype=MD5 # defines class named png in Domain imgs, copies 3 copies in different devices, and verifies with MD5 ~] # mogadm class add imgs jpg-- mindevcount=3-- hashtype=MD5 # defines class named jpg in Domain imgs, copies 3 copies in different devices, and verifies with MD5 ~] # mogadm domain list

7. Use Mogilefs to do upload and download test

Mogilefs can interact with the self-built API interface, which has many commands for managing stored data. For example, the command for uploading data is mogupload, and the command for viewing data is mogfileinfo.

Example: test uploading the file / test/123.png to the Mogilefs cluster (the file is prepared locally in advance):

~] # mogupload-- trackers=192.168.29.111-- domain=imgs-- class=png-- key='/111.png'-- file='/test/123.png' # upload the 123.png file through Trackers with IP 192.168.29.111, save it to the space where Domain is imgs,Class and png, and rename it 111.png ~] # mogfileinfo-- trackers=192.168.29.111-- domain=imgs-- class=png-- key='/111.png' # View Domain as imgs Class is the storage of files with key of 111.png in png.

At this point, the Mogilefs distributed storage cluster has been built, but if we want to enable the client to communicate with it, we need to program on the interface, which is very troublesome. Fortunately, we can use Nginx to communicate. Let's demonstrate the steps of Nginx to reverse Mogilefs.

5.Nginx reverse Mogilefs

1. Open the mogilefsd service for N2 and n3 and set all three nodes to Trackers (make sure the configuration file is the same as N1):

~] # service mogilefsd start

two。 Compile and install Nginx on the N4 node

Install the dependency package:

~] # yum install gcc gcc-c++ perl pcre-devel openssl openssl-devel

Download the Nginx compilation and installation package nginx-1.10.3.tar.gz and the Nginx_Mogilefs module nginx_mogilefs_module-1.0.4.tar.gz and expand:

~] # lsnginx-1.10.3 nginx_mogilefs_module-1.0.4nginx-1.10.3.tar.gz nginx_mogilefs_module-1.0.4.tar.gz~] # cd nginx-1.10.3./configure\ >-- prefix=/usr\ >-- sbin-path=/usr/sbin/nginx\ >-- conf-path=/etc/nginx/nginx.conf\ >-- error-log-path=/var/log/nginx/error.log\ >-- http- Log-path=/var/log/nginx/access.log\ >-- pid-path=/var/run/nginx/nginx.pid\ >-- lock-path=/var/lock/nginx.lock\ >-- user=nginx\ >-- group=nginx\ >-- with-http_ssl_module\ >-- with-http_flv_module\ >-- with-http_stub_status_module\ >-- with-http_gzip_static_module\ >-- http-client-body-temp-path=/var/tmp / nginx/client/\ >-http-proxy-temp-path=/var/tmp/nginx/proxy/\ >-- http-fastcgi-temp-path=/var/tmp/nginx/fcgi/\ >-- http-uwsgi-temp-path=/var/tmp/nginx/uwsgi\ >-- http-scgi-temp-path=/var/tmp/nginx/scgi\ >-- with-pcre\ >-- with-debug\ >-- add-module=../nginx_mogilefs_module-1.0. 4 / # remember the path where the Mogilefs module is added It's indispensable. ~] # make & make install

Add the nginx user and start nginx:

~] # useradd-s / sbin/nologin-M nginx~] # / usr/sbin/nginx

3. Configure Nginx

Single Trackers example:

Location / imgs/ {mogilefs_tracker 192.168.29.111mogilefs_tracker 7001; # single Trackers example mogilefs_domain imgs; # specify Domain mogilefs_class png jpg; # specify Class mogilefs_pass {# Transport related configuration proxy_pass $mogilefs_path; proxy_hide_header Content-Type; proxy_buffering off;}}

Multi-Trackers example:

Add a scheduling module in the http configuration section of the nginx configuration:

1 upstream mogsvr {2 server 192.168.29.111 server 192.168.29.112 server 7001 * 4 server 192.168.29.113 * * 7001 * *

Add the server configuration segment in the nginx configuration:

Location / imgs/ {mogilefs_tracker mogsvr; mogilefs_domain imgs; mogilefs_class png jpg; mogilefs_pass {proxy_pass $mogilefs_path; proxy_hide_header Content-Type; proxy_buffering off;}}

Restart nginx and access the previously uploaded image through nginx:

Summary:

An error message MogileFS::Backend: couldn't connect to mogilefsdbackend at / usr/local/share/perl/5.8.4/Client.pm line 282 is encountered when uploading files, which is due to the fact that the mogilefsd service cannot connect to MySQL. Check the connection between them to find the error.

All right, that's all of this article. I hope the content of this article has a certain reference and learning value for everyone's study or work. If you have any questions, you can leave a message and exchange. Thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.