Shulou(Shulou.com)06/01 Report--

This article mainly introduces how to install and deploy distributed file system FastDFS, which has a certain reference value. Interested friends can refer to it. I hope you can learn a lot after reading this article.

A brief introduction to FastDFS

FastDFS is an open source lightweight distributed file system, it manages files, and its functions include file storage, file synchronization, file access (file upload, file download), etc., which solves the problems of mass storage and load balancing. It is especially suitable for online services with files as carriers, such as photo album websites, video websites and so on.

FastDFS is tailored for the Internet, taking full account of redundant backup, load balancing, linear expansion and other mechanisms, and pays attention to high availability, high performance and other indicators. Using FastDFS, it is easy to build a high-performance file server cluster to provide file upload, download and other services.

The FastDFS server has two roles: tracker (tracker) and storage node (storage). The tracker mainly does the scheduling work and plays the role of load balancing on the access.

The storage node stores files and completes all the functions of file management: it stores, synchronizes and provides access interfaces, and FastDFS manages the metadata of files at the same time. The so-called meta data of a file is the relevant attributes of the file, expressed in the form of key-value pairs (key value), such as: width=1024, in which the key is width,value is 1024. A file metadata is a list of file attributes that can contain multiple key-value pairs.

Both the tracker and the storage node can consist of one or more servers. Servers in trackers and storage nodes can be added or offline at any time without affecting online services. All the servers in the tracker are peer-to-peer and can be increased or decreased at any time according to the pressure on the server.

In order to support large capacity, storage nodes (servers) are organized into volumes (or groups). The storage system consists of one or more volumes, the files between the volumes are independent of each other, and the file capacity of all volumes is the file capacity of the whole storage system. A volume can be composed of one or more storage servers, the files in the storage servers under one volume are all the same, and multiple storage servers in the volume play the role of redundant backup and load balancing.

When the server is added to the volume, the synchronization of the existing files is automatically completed by the system, and after the synchronization is completed, the system automatically switches the new server to provide services online.

You can add volumes dynamically when storage space is insufficient or running out. You only need to add one or more servers and configure them as a new volume, thus expanding the capacity of the storage system.

The file identity in FastDFS is divided into two parts: the volume name and the file name, both of which are indispensable.

Introduction of the principle

1. File upload

FastDFS provides basic file access interfaces such as upload, download, append, delete, etc., in the form of client libraries. Storage services regularly send their own storage information to Tracker services. When there are multiple Tracker services in the Tracker service cluster, the relationship between each Tracker service is peer-to-peer, so the client will randomly select a Trackre service when uploading. When the Tracker service receives a request from the client to upload a file, it allocates a group that can store the file. When group is selected, it is necessary to decide which storage service in the group is assigned to the client. When the storage service is allocated, the client sends a write request to storage, and storage will assign a data storage directory to the file. Then assign a file ID tag to the file, and then generate the file name storage file based on the above information.

two。 File synchronization

After uploading a file, the client writes the file to a storage service in group to upload the file successfully. After the storage service has finished writing the file, the background thread will synchronize the file to other storage service nodes in the same group. After each storage service writes a file, it also writes a binlog,binlog that contains no file data, only file name and other meta-information. This binlog is used for background synchronization, and storage records the progress of synchronization with other storage in the group, so that the progress can be synchronized with the last progress after restart. The progress is recorded by timestamp, so it is best to ensure that all server in the cluster are always synchronized. Finally, the synchronization progress of the Storage service is reported to the tracker service as part of the metadata, and the tracker service uses the synchronization progress as a reference when choosing to read the storage.

3. Download a file

When downloading a file, the client first asks the tracker service the storage of the downloaded file, and the parameter is the file ID (volume name and file name); then tracker returns an available storage; to the client. Finally, the client communicates with storage directly to complete the file download.

III. Environmental preparation

1. Software environment

Libevent download address: http://libevent.org/

Libfasttcommon download address: https://github.com/happyfish200/libfastcommon/releases

Fastdfs download address: https://github.com/happyfish200/fastdfs/releases

Fastdfs-nginx-module download address: https://github.com/happyfish200/fastdfs-nginx-module/releases

two。 Machine and network environment

Tracker Server1: 192.168.100.101

Storage Group1 Node1: 192.168.100.102

Storage Group1 Node2: 192.168.100.103

Components that need to be installed on Tracker nodes: libevent, libfasttcommon, fastdfs

Components that need to be installed on Storage nodes: libfasttcommon, fastdfs, nginx, fastdfs-nginx-module

Deploy the Tracker service (192.168.100.101)

1. Install libevent dependencies

Note: if the machine has a public network environment directly yum-y install libevent, this article uses the source code package to compile and install

Decompress libevent source code package: tar-zxvf libevent-2.1.11-stable.tar.gz

Compile pre-installation configuration:. / configure

Compile and install: make & & make install

Default installation location: / usr/local/lib

two。 Install libfasttcommon dependencies

Decompress libfasttcommon source code package: tar-zxvf libfastcommon-1.0.41.tar.gz

Compile and install:. / make.sh & &. / make.sh install

Default installation location: / usr/lib64

3. Install the FastDFS Tracker service

Decompress fastdfs source code package: tar-zxvf fastdfs-6.01.tar.gz

Compile and install:. / make.sh & &. / make.sh install

After installation, the service and scripts are copied to the / usr/bin directory:

After the installation, the configuration files are copied to the / etc/fdfs directory:

Delete / etc/init.d/fdfs_storaged because only Tracker services are installed on this machine

Note: compiling and installing fastdfs requires perl library dependency

Set the Tracker service to boot automatically:

Chkconfig-add fdfs_trackerd

Chkconfig fdfs_trackerd on

Deploy the Storage service (192.168.100.102Universe 103)

1. Install libfasttcommon dependencies

Decompress libfasttcommon source code package: tar-zxvf libfastcommon-1.0.41.tar.gz

Compile and install:. / make.sh & &. / make.sh install

Default installation location: / usr/lib64

two。 Install the FastDFS Storage service

Decompress fastdfs source code package: tar-zxvf fastdfs-6.01.tar.gz

Compile and install:. / make.sh & &. / make.sh install

After installation, the service and scripts are copied to the / usr/bin directory:

Copy the Storage configuration file in the / etc/fdfs directory example: cp / etc/fdfs/storage.conf.sample / etc/fdfs/storage.conf

Modify the configuration item in / etc/fdfs/storage.conf:

Base_path=/home/yuqing/fastdfs is changed to: base_path=/work/fastdfs/storage (this directory is defined for itself, will be used when starting up, and no error will be reported)

Store_path0=/home/yuqing/fastdfs is changed to: store_path0=/work/fastdfs/storage (this directory is defined for itself, will be used when starting up, and no error will be reported)

Change tracker_server=192.168.209.121:22122 to: tracker_server=192.168.100.101:22122

After the installation, the startup scripts are copied to the / etc/init.d directory:

Start the Storage service: systemctl start fdfs_storaged

Restart the Storage service: systemctl restart fdfs_storaged

Stop the Storage service: systemctl stop fdfs_storaged

Open the Storage service port:

Iptables-I INPUT-p tcp-- dport 23000-j ACCEPT

Iptables-I INPUT-p tcp-- dport 8888-j ACCEPT

Verify the current Storage service and Tracker service communication:

/ usr/bin/fdfs_monitor / etc/fdfs/storage.conf

Modify the configuration of the nginx service on each Storage machine as follows:

Use Java client to test FastDFS

1.java project Maven dependency

Project address: https://github.com/tobato/FastDFS_Client client mainly depends on SpringBoot at present Therefore, we must introduce: org.springframework.boot spring-boot-starter-parent 2.0.0.RELEASE FastDFS dependency package: com.github.tobato fastdfs-client 1.26.7 introduce FastDFS into the project: @ Import (FdfsClientConfig.class) configure Fdfs-related parameters in application.yml: # = # distributed file system FDFS configuration # = fdfs: so-timeout: 1500 connect-timeout: 600thumb-image: width: 150height: 150 tracker-list:-192.168.100.101 22122 uses interface services to operate on the Fdfs server The main interfaces include: TrackerClient-TrackerServer interface GenerateStorageClient-general file storage interface (StorageServer interface) FastFileStorageClient-simple interface for project development and integration (StorageServer interface) AppendFileStorageClient-interface to support file continuation operation (StorageServer interface)

two。 Actual test case

Package com.maxbill;import com.github.tobato.fastdfs.domain.fdfs.MetaData;import com.github.tobato.fastdfs.domain.fdfs.StorePath;import com.github.tobato.fastdfs.service.FastFileStorageClient;import com.github.tobato.fastdfs.service.TrackerClient;import lombok.extern.log4j.Log4j2;import org.springframework.beans.factory.annotation.Autowired;import org.springframework.stereotype.Component;import javax.annotation.PostConstruct;import java.io.File;import java.io.FileInputStream;import java.io.InputStream;import java.util.HashSet Import java.util.Map;import java.util.Set;@Log4j2@Componentpublic class FdfsClientUtil {private static FdfsClientUtil fdfsClientUtil; @ Autowired private TrackerClient trackerClient; @ Autowired private FastFileStorageClient storageClient; @ PostConstruct public void init () {fdfsClientUtil = this;} private static Set getMetaData (Map infoMap) {Set metaDataSet = new HashSet (); metaDataSet.add (new MetaData ("createUser", "maxbill")) MetaDataSet.add (new MetaData ("createDate", "2019-11-18"); return metaDataSet;} public static String uploadFile (File file, Map infoMap) {try {String fileName = file.getName (); String fileType = fileName.substring (fileName.lastIndexOf ("\") + 1); log.info ("upload file name: {}", fileName) StorePath path = fdfsClientUtil.storageClient.uploadFile (new FileInputStream (file), file.length (), fileType, getMetaData (infoMap)); log.info ("upload success path: {}", path); Set metaData = fdfsClientUtil.storageClient.getMetadata (path.getGroup (), path.getPath ()); log.info ("upload success meta: {}", metaData); return path.getFullPath () } catch (Exception e) {log.error (e.getMessage ()); return e.getMessage () } Thank you for reading this article carefully. I hope the article "how to install and deploy distributed File system FastDFS" shared by the editor will be helpful to you. At the same time, I also hope that you will support and follow the industry information channel. More related knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.