Shulou(Shulou.com)06/02 Report--

This article introduces the relevant knowledge of "the principle and use of FastDFS". In the operation of actual cases, many people will encounter such a dilemma. Next, let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

The file system is the system software responsible for managing and storing files. It is the bridge between the operating system and the hardware driver. The operating system accesses the files through the interface provided by the file system, and the user accesses the files on the disk through the operating system.

Distributed file system is produced in the face of the needs of the Internet, how to store massive data in the Internet era? Simply increasing the number of hard drives has been unable to meet our requirements, because the hard disk transmission speed is limited, but the data is growing rapidly, in addition, we have to do a good job of data backup, data security and so on.

By using the distributed file system, the file systems of multiple locations can be connected through the network to form a file system grid, the nodes communicate through the network, and the storage and transmission capacity of one file system is limited. we let files be stored on multiple computers and transmitted together through multiple computers.

Benefits:

1. The file system processing capacity of one computer is extended to multiple computers at the same time.

2. If a computer dies, another copy of the computer will be provided to the data.

3. Each computer can be placed in a different region, so that users can access it nearby and provide access speed.

Mainstream distributed file system: NFS,GFS,HDFS

Distributed file service providers: Ali OSS, Qiniuyun, Baidu Cloud

FastDFS is an open source distributed file system written in C language. FastDFS is tailored for the Internet, taking full account of redundant backup, load balancing, linear expansion and other mechanisms, and pays attention to high availability, high performance and other indicators. Using FastDFS, it is easy to build a set of high-performance file server cluster to provide file upload, download and other services. Compared with the above mainstream distributed file systems, FastDFS has a poor development experience, but its system complexity is low and its performance is high. It is very suitable for storing those small files such as pictures. FastDFS does not divide the files into blocks, so it has no block merging overhead. Socket is used in network communication, and the communication speed is very fast.

FastDFS architecture

The FastDFS architecture includes Tracker server and Storage server. The client requests Tracker server to upload and download files, and through Tracker server scheduling, Storage server finally completes the upload and download of files.

Tracker server is used for load balancing and scheduling. When uploading files through Tracker server, you can find Storage server to provide file upload service according to some policies. You can call tracker a tracking server or a scheduling server.

The role of Storage server is to store files, and the files uploaded by the client are finally stored on the Storage server. Storage server does not implement its own file system but uses the file system of the operating system to manage files. You can call Storage a storage server.

Tracker cluster

Tracker server in FastDFS cluster can, Tracker server are equal to each other and provide services at the same time, and there is no single point of failure in Tracker server. Client requests Tracker server

Polling is used to change to another tracker if the requested tracker cannot provide the service.

Storage cluster

Storage cluster adopts packet storage mode. Storage cluster is composed of one or more groups. The total storage capacity of the cluster is only the sum of the storage capacity of all groups in the cluster. A group consists of one or more storage servers. There is an equal relationship between the Storage server in the group. The Storage server in different groups will not communicate with each other, and the Storage server in the same group will be connected to each other for file synchronization to ensure that the files on each Storage in the same group are exactly the same. The storage capacity of a group is the one with the smallest storage server capacity in the group, so it can be seen that the hardware and software configuration of the storage server in the group is best consistent.

The advantage of packet storage is flexibility and strong controllability. For example, when uploading files, the group to be uploaded can be directly specified by the client, or it can be scheduled and selected by Tracker. When the access pressure of the storage server of a packet is high, the storage server can be added to the group to expand the service capacity. When the system capacity is insufficient, you can add groups to expand the storage capacity.

Storage server will connect to all Tracker server in the cluster and report its status to them regularly, including statistics such as disk free space, file synchronization status, number of file uploads and downloads, and so on.

File upload process:

1. Client sends an upload connection request to Tracker server

2. Tracker server queries the available storage and returns the queried information (ip and port of Storage) to Client

3. Client uploads files to Storage server

4. Storage server generates the file ID, stores the file, and returns the file ID to Client

5. Client stores the file ID

Note: file ID contains group name, virtual disk path, data two-level directory, and file name.

File download process

1. Client sends download link request to Tracker server

2. Tracker server queries the available Storage and returns the queried information (ip and port of Storage) to Client

3. Client sends the file ID to Storage server

4. Storage server finds the file according to the file ID and returns the contents of the file to Client

Starter Code:

First, import dependency

4.0.0 lianbang.wu FastDFSDemo 1.0-SNAPSHOT org.springframework.boot spring-boot-starter-parent 2.0.4.RELEASE net.oschina.zcx7878 fastdfs-client-java 1.27.0.0 org.springframework.boot spring-boot-starter-test Org.apache.commons commons-io 1.3.2

Configuration file, create fastdfs-client under the classpath. Properties

Fastdfs.connect_timeout_in_seconds = 5fastdfs.network_timeout_in_seconds = 30fastdfs.charset = UTF-8fastdfs.tracker_servers = 192.168.0.0pur22222

Third, test code

Package lianbang.wu.fastDFS;import org.csource.common.MyException;import org.csource.common.NameValuePair;import org.csource.fastdfs.*;import org.junit.Test;import org.junit.runner.RunWith;import org.springframework.boot.test.context.SpringBootTest;import org.springframework.test.context.junit4.SpringRunner;import java.io.File;import java.io.FileOutputStream;import java.io.IOException @ SpringBootTest@RunWith (SpringRunner.class) public class FastdfsTest {/ * File upload * / @ Test public void testUpload () {try {/ / load global configuration ClientGlobal.initByProperties ("config/fastdfs_client.properties"); / / create client TrackerClient trackerClient = new TrackerClient () / / Connect tracker server TrackerServer trackerServer = trackerClient.getConnection (); if (trackerServer = = null) {return;} / / get a storage server StorageServer storeStorage = trackerClient.getStoreStorage (trackerServer); StorageClient storageClient = new StorageClient (trackerServer, storeStorage); NameValuePair [] list = null String [] fileId = storageClient.upload_file ("C:\ user\\ admin\\ 1.png", "png", list); System.out.println (fileId);} catch (Exception e) {e.printStackTrace () }} / * File query * / @ Test public void testQueryFile () throws IOException, MyException {/ / load global configuration ClientGlobal.initByProperties ("config/fastdfs_client.properties"); TrackerClient trackerClient = new TrackerClient (); TrackerServer trackerServer = trackerClient.getConnection (); StorageServer storeStorage = trackerClient.getStoreStorage (trackerServer); StorageClient storageClient = new StorageClient (trackerServer, storeStorage) FileInfo fileInfo = storageClient.query_file_info ("group1", "M00/00/01/w.png"); System.out.println (fileInfo);} / * * File download * / @ Test public void testDownloadFile () throws IOException, MyException {/ / load global configuration ClientGlobal.initByProperties ("config/fastdfs_client.properties"); TrackerClient trackerClient = new TrackerClient () TrackerServer trackerServer = trackerClient.getConnection (); StorageServer storageServer = trackerClient.getStoreStorage (trackerServer); StorageClient storageClient = new StorageClient (trackerServer, storageServer); byte [] bytes = storageClient.download_file ("group1", "M00/00/01/w.png"); FileOutputStream fileOutputStream = new FileOutputStream (new File ("d:/1.png")); fileOutputStream.write (bytes); fileOutputStream.close () This is the end of the introduction of "the principle and usage of FastDFS". Thank you for your reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.