Shulou(Shulou.com)05/31 Report--

What are the advantages and disadvantages of Nginx+NFS, Redis and distributed file system Moosefs? I believe many inexperienced people don't know what to do about it. Therefore, this paper summarizes the causes and solutions of the problem. Through this article, I hope you can solve this problem.

A large number of small file storage has always been one of the typical application scenarios. Now there is 3KW data, and a single piece of data is very small. According to key-value, key is a 32-bit MD5 string, and value is about 100 bytes on average. What are the advantages and disadvantages of Nginx+NFS, Redis and distributed file system Moosefs?

What are the advantages and disadvantages of Nginx+NFS, Redis and distributed file system Moosefs

1. Generate static files directly and use nginx to cache static files efficiently.

At that time, the hardware resources of the server were limited, so it was adopted in this way and has been used ever since.

Servers are shared through NFS

There are too many small files to manage.

NFS is not convenient for operation and maintenance and expansion.

The content of the file is very small (about 100 bytes), and the 3KW is about 2.5G in size.

However, the file storage is related to the family size of the hard disk partition, where the family size of the disk partition is 8K, so although the file content is only 100 bytes, a single file is actually 8K when it is actually stored on disk.

So the disk space occupied by 3KW's file world is about 200g (a serious waste of space).

2. Redis (V2.2.11) [KV Database]

I heard colleagues say that turning on VM will lead to a sharp decline in performance, so basically ignore VM and store all the data in memory.

Key is a 32-bit MD5 string

Test data: 10W data accounts for about 20m of memory

Test data: 500W data accounts for about 1G of memory, and persistent rdb data files are about 350m.

Calculated 3KW data: 6GB of memory, persistent rdb data file is about 2G (compressed?)

Because Redis doubles its memory when it is persisted, and takes into account the growth of data, it is basically no problem that a machine with 20GB of memory is needed (disaster recovery and so on).

Redis is very fast, if there is no problem with the hardware, it is best to choose this one.

What are the advantages and disadvantages of Nginx+NFS, Redis and distributed file system Moosefs

3. Moosefs (MFS) [distributed File Storage system]

Mfs supports NFS to operate directly from mount to local (if mfs is used, the current architecture is basically unchanged)

The most basic need is a master server (MasterServer) and a data server (ChunkServer).

The indexes of files and directories need to be loaded into the memory of the master server, so there are certain requirements for the memory of the master server.

Write 30W files, and write starts to drop sharply at 20W (my 5400-rpm laptop hard drive)

30W files, 4W directories, master server occupies about 120m of memory

Files seem to be stored in the same way as common files (uncertain), with 30W files taking up about 2.4G of disk space (also 8K a file).

There are so many small files that it is not convenient to maintain and migrate (I wonder if I set it wrong?).

Memory and hard disk take up a lot of time, and the performance is not very outstanding.

So I basically don't think about it.

After reading the above, have you mastered the advantages and disadvantages of Nginx+NFS, Redis and distributed file system Moosefs? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.