Shulou(Shulou.com)06/02 Report--

This article mainly explains "is the data format of MONGODB disk and memory consistent?", interested friends may wish to have a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn "is the data format of MONGODB disk and memory consistent?"

Recently, the amount of data in the MONGODB system has gradually increased, which has contacted the archive and cleanup matters that the developers are ready to start working on the data. Among them, the classmate of the operation and maintenance asked me that the data size you stored was different from the actual data size, and I replied that it was the data compression function of mongodb.

But I am confused about the following questions, such as whether the data on disk and memory are exactly the same, what the compression ratio is, and so on. No one is perfect, and I'm not an DB encyclopedia. But fortunately, he is not a middle-aged greasy uncle, so if he is blurred, then go to learning.

Then go straight to the topic of wiredTiger, wiredTiger as the main storage engine of MONGODB (after 3.0, of course, there is PERCONA, but it is not mainstream right now).

First of all, let's take a look at how wiredTiger stores data. WiredTiger itself stores data in two ways, row mode and column mode. Both of these methods use B + TREE to store data.

In row mode, both KEY and Values are strings-mutable, while if you use the column approach, key records the Values through 64BIT, just like the row mode. So the difference between the mode of rows and columns is the way key is stored.

These two storage methods have their own advantages and disadvantages.

The way to extract the whole row of data is fast, but it requires a lot of storage space, and if some aspects are not well designed, the consumption of IWeiO is relatively large both in terms of performance and size.

Column storage is suitable for extracting the required fields (meaning it is best not to extract more fields at once, otherwise there is no advantage), and the stored value is limited to the size of the limited value of 8-BITS. The concurrency control of related mongodb is mainly in transactions, snapshots and cache.

The basic rows in MONGODB are stored in row mode, and the data compression in mongodb is involved in the data storage.

Currently, four compression methods are supported for the storage of rows.

1 key value prefix compression

2 Field Compression

3 Huffman data coding and compression

4-block compression

For the four specific compression methods, the default is the key prefix compression. This is mainly based on the CPU disk consumption, compression, and algorithm complexity and other factors.

The first thing to make sure is that the data stored on disk must be different from the data in memory, or there is compression.

one

The data in the file system cache is in the same format as on disk, including any compression of the data files, and MONGODB uses the file system cache to reduce disk I and O.

two

Indexes loaded in the WiredTiger internal cache have a different data representation than the disk format, but index prefix compression can still be used to reduce the use of RAM. Index prefix compression removes common prefixes from index fields.

three

The collected data in the WiredTiger internal cache is uncompressed and uses a different representation than the disk format.

On the question of compression ratio, in fact, there is no specific fixed ratio, which is related to the stored data. Here is a diagram, which is a relatively high compression ratio once mentioned in a PPT of percona.

At this point, I believe that everyone on the "MONGODB disk and memory data format consistent" have a deeper understanding, might as well to the actual operation of it! Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.