Shulou(Shulou.com)06/01 Report--

Introduction MongoDB 3.0 was officially released on March 3, 2015 after a revision of the version number (version 2.8 jumped directly to version 3.0) and 11 rc versions. It is no exaggeration to say that the new features of this version indicate that MongoDB, a typical NoSQL database, has entered a new stage of development. The following sections of this article will take an inventory of the new features in version 3.0 one by one. Plug-in storage engine API

MongoDB 3.0 introduces plug-in storage engine API, which provides convenience for third-party storage engine manufacturers to join MongoDB. This change undoubtedly refers to the design concept of MySQL. At present, in addition to the early MMAP storage engine, both WiredTiger and RocksDB have completed their support for MongoDB, and the former has been directly introduced into MongoDB 3.0 after being acquired by MongoDB. The introduction of plug-in storage engine API provides unlimited possibilities for MongoDB to enrich its arsenal to handle more different types of business. Memory storage engine, transaction storage engine and even Hadoop may be accessed in the future.

WiredTiger storage engine

If the plug-in storage engine API has created an arsenal for MongoDB 3.0, then WiredTiger is definitely the first and most important blockbuster in the arsenal. Because of the natural defects of the MMAP storage engine itself (consuming disk space and memory space and being difficult to clean up, library-level locks), MongoDB has brought great pain to database operators, and even some people have begun to turn to TokuMX, although the latter is not very stable at present. Aware of this problem, MongoDB made a rich and wayward decision to buy storage engine manufacturer WiredTiger directly and integrate WiredTiger storage engine into version 3.0 (available only in 64-bit versions). So what are the expected features of this storage engine in the spotlight?

1. Document-level concurrency control

WiredTiger implements document-level concurrency control through MVCC, that is, document-level locking. This allows multiple client requests to update multiple documents in a collection of memory at the same time, eliminating the need to queue for library-level write locks. This not only improves the read and write performance of the database, but also greatly improves the concurrent processing ability of the system. The effect of this can be directly reflected in the monitoring tool mongostat. The old version of the monitoring indicator will have locked db (too high is a major pain point for mongo users), but the new version of mongostat can no longer be seen.

MongoDB version 2.4.12

$/ home/mongodb/mongodb-linux-x86_64-2.4.12/bin/mongostat-port 55060insert query update delete getmore command flushes mapped vsize res faults locked db idx miss% qr | qw ar | aw netIn netOut conn time*0 * 0 * 0 * 0 01 | 0 018g 18.3g 16.1g 0 ycsb:0.0% 0 0 | 0 0 | 0 62b 2k 1 13V 04port 55060insert query update delete getmore command flushes mapped vsize res faults locked db idx miss 01v 01L * 0 * 0 * 0 * 01 | 0 018g 18.3g 16.1g 0 ycsb:0.0% 0 | 0 0 | 0 62b 2k 1 13L 04lane 02* 0 * 0 * 0 * 0 01 | 0 018g 18.3g 161g 0 ycsb:0.0% 00 | 0 62b 2k 1 13:04:03MongoDB 3.0 rc8 version

$/ home/mongodb/mongodb-linux-x86_64-3.0.0-rc8/bin/mongostat-port 55050insert query update delete getmore command% dirty% used flushes vsize res qr | qw ar | aw netIn netOut conn time*0 * 0 * 0 * 0 1 | 0 0.0 42.2 0 30.6 G 30.4 G 0 | 0 0 | 0 79 b 16k 1 13 purge 02port 55050insert query update delete getmore command 38 °0 * 0 * 0 * 0 * 0 | 0 0.0 42.2 0 30.6 G 30.4 G 0 | 0 0 | 0 79 b 16k 1 13 purge 02vy 39 * 0 * 0 * 0 * 0 0 1 | 0 0.0 42.2 0 30.6 G 30.4 G 0 | 0 0 | 0 79b 16k 1 13:02:40

2. Disk data compression

WiredTiger supports Block compression and prefix compression for all collections and indexes (the same compression for journal,journal files if the database is enabled). Supported compression options include: no compression, Snappy compression, and Zlib compression. This brings another good news for Mongo users, because many Mongo databases have no choice but to expand because the MMAP storage engine consumes too much disk space. Snappy compression is the default compression mode of the database, and users can choose the appropriate compression method according to their business needs. In theory, Snappy compression speed is fast, compression ratio OK, while Zlib compression ratio is high, CPU consumption is high and the speed is slightly slower. Of course, as long as you choose to use compression, Mongo will certainly take up more CPU usage, but considering that Mongo itself is not very CPU-intensive, it is worthwhile to enable compression.

In addition, the storage mode of WiredTiger has also been greatly improved. The old version of Mongo allocates files at the database level, and all collections and indexes in the database are mixed and stored in the database file, so even if a collection or index is deleted, the disk space occupied is difficult to be automatically reclaimed in time. WiredTiger allocates files at the collection and index level, and all collections and indexes in the database are stored in separate files. After the collection or index is deleted, the corresponding storage file is deleted. Of course, because of the different storage methods, the lower version of the database can not be directly upgraded to the WiredTiger storage engine, but can only be achieved by exporting and importing data.

MongoDB version 2.4.12

[mongodb@mongo-data-emergency-001.m6.momo.com mongodb_2_4_12] $lldrwxrwxr-x 3 mongodb mongodb 4096 February 25 19:03 local-rwxrwxr-x 1 mongodb mongodb 6 February 25 19:04 mongod.lockdrwxrwxr-x 2 mongodb mongodb 4096 February 27 18:30 _ tmpdrwxrwxr-x 3 mongodb mongodb 4096 February 27 18:39 ycsb [mongodb@mongo-data-emergency-001.m6.momo.com mongodb_2_4_12] $ll ycsb/drwxrwxr-x 2 mongodb mongodb 4096 February 27 27 18:39 _ tmp-rw--- 1 mongodb mongodb 67108864 February 27 18:57 ycsb.0-rw--- 1 mongodb mongodb 134217728 February 27 18:57 ycsb.1-rw--- 1 mongodb mongodb 2146435072 February 27 18:57 ycsb.10-rw--- 1 mongodb mongodb 2146435072 February 27 18:57 ycsb.11-rw--- 1 mongodb mongodb 2146435072 February 27 18:57 ycsb.12-rw--- 1 mongodb mongodb 2146435072 February 27 18:39 ycsb.13-rw--- 1 Mongodb mongodb 268435456 February 27 18:57 ycsb.2-rw--- 1 mongodb mongodb 536870912 February 27 18:57 ycsb.3-rw--- 1 mongodb mongodb 1073741824 February 27 18:57 ycsb.4-rw--- 1 mongodb mongodb 2146435072 February 27 18:57 ycsb.5-rw--- 1 mongodb mongodb 2146435072 February 27 18:57 ycsb.6-rw--- 1 mongodb mongodb 2146435072 February 27 18:57 ycsb.7-rw--- 1 mongodb mongodb 2146435072 February 27 18:57 ycsb.8 -rw--- 1 mongodb mongodb 2146435072 February 27 18:57 ycsb.9-rw--- 1 mongodb mongodb 16777216 February 27 18:40 ycsb.ns

Mongo 3.0 rc8 version

[mongodb@mongo-data-emergency-001.m6.momo.com mongodb_3_0_0] $lldrwxrwxr-x 2 mongodb mongodb 4096 February 28 18:32 local-rw-rw-r- 1 mongodb mongodb 36864 March 21 13:41 _ mdb_catalog.wt-rwxrwxr-x 1 mongodb mongodb 6 February 28 18:32 mongod.lock-rw-rw-r- 1 mongodb mongodb 36864 March 21 13:42 sizeStorer.wt-rw-rw-r- 1 mongodb mongodb 95 February 28 18:32 storage.bson- Rw-rw-r- 1 mongodb mongodb 49 February 28 18:32 WiredTiger-rw-rw-r- 1 mongodb mongodb 433 February 28 18:32 WiredTiger.basecfg-rw-rw-r- 1 mongodb mongodb 21 February 28 18:32 WiredTiger.lock-rw-rw-r- 1 mongodb mongodb 921 March 21 13:41 WiredTiger.turtle-rw-rw-r- 1 mongodb mongodb 53248 March 21 13:41 WiredTiger.wtdrwxrwxr-x 2 mongodb mongodb 4096 March 21 13:41 ycsb [mongodb@mongo-data-emergency-001 .m6.momo.com mongodb_3_0_0] $ll ycsb/-rw-rw-r- 1 mongodb mongodb 19314257920 February 28 19:16 collection-4-1318477584648278106.wtAfter RWLY RWLY-1 mongodb mongodb 602112 March 21 13:40 collection-6-1318477584648278106.wtMurr RWLY r-1 mongodb mongodb 262598656 February 28 18:53 index-5-1318477584648278106.wt RWLY rwLY-1 mongodb mongodb 827392 March 21 13:40 index-7-1318477584648278106.wt-rw-rw-r -1 mongodb mongodb 1085440 March 21 13:41 index-8-1318477584648278106.wt3, Upper limit of configurable memory usage

WiredTiger supports memory usage capacity configuration, and users can control the maximum memory that MongoDB can use through the storage.wiredTiger.engineConfig.cacheSizeGB parameter, which defaults to half of the physical memory size. This also brings another good news for the majority of Mongo users. MMAP storage engine is famous for consuming memory, as long as the amount of data is large enough.

MMAPv1 storage engine upgrade

In addition to the introduction of WiredTiger, MongoDB 3.0 has also made some improvements to the original storage engine MMAP, which is still the default storage engine of version 3.0. Unfortunately, the improved MMAP storage engine still allocates files at the database level, and all collections and indexes in the database are mixed stored in the database file, so the disk space can not be automatically reclaimed in time.

1. Lock granularity is increased from library-level lock to collection-level lock.

To some extent, this can also improve the concurrent processing ability of the database.

2. The allocation of document space has changed.

In the MMAP storage engine, documents are stored in write order. If the length of the document becomes longer after the update and there is not enough space behind the original storage location to hold the growing part of the data, then the document will be moved to another location in the file. This kind of document location shift caused by updates can seriously reduce write performance, because once the document is moved, all indexes in the collection have to synchronously modify the new storage location of the document.

In order to reduce the occurrence of this situation, MMAP storage engine provides two ways of document space allocation: adaptive allocation based on paddingFactor (fill factor) and pre-allocation based on usePowerOf2Sizes, of which the former is the default. The first method will calculate the average growth length of document updates based on the history of document updates in each collection, and then fill a part of the space when the new document is inserted or the old document is moved, such as the current collection paddingFactor value is 1.5, then a document with a size of 200 bytes will automatically fill 100 bytes of space after the document is inserted. The second method does not take into account the update history and directly allocates storage space of 2 to the N power of the document, such as 256 bytes when a document with the same size of 200 bytes is inserted.

MMAPv1 in MongoDB version 3.0 abandons the paddingFactor-based adaptive allocation because it looks smart, but because the documents in a collection vary in size, the size of the filled space is not the same. If there are a lot of update operations on the collection, the free space caused by the move of the record can be difficult to reuse because of different sizes. At present, the pre-allocation method based on usePowerOf2Sizes has become the default document space allocation method, because the space allocated and reclaimed is 2 to the N power (when the size exceeds 2MB, it becomes a multiple growth of 2MB), so it is easier to maintain and use. If there is only insert or in-place update on a collection, the user can turn off space pre-allocation by setting the noPadding flag bit for the collection.

Replication set improvement

1. Growth of replica set membership

The maximum number of replica set members in MongoDB 3.0 has increased from 12 to 50, but the maximum number of members who can vote is still 7, and the w: "majority" item in the corresponding getLastError only represents the majority of voting nodes.

2. Changes in StepDown processing methods of Primary nodes

Through the replSetStepDown command in the replication set, the current Primary node can be abdicated and the new Primay node can be re-elected. MongoDB 3.0 has made the following changes in the way StepDown is handled: 1) before Primary abdicates, it will first interrupt some time-consuming user operations such as indexing, writing, Mapreduce tasks, etc. 2) in order to prevent data rollback, Primary nodes wait for an electable Secondary node to synchronize to the latest data before abdication, while in the old version, Primary nodes abdicated as long as the data of Secondary nodes were synchronized to less than 10 seconds. 3) at the same time, a secondaryCatchUpPeriodSecs parameter has been added to the replSetStepDown command. You can specify that the Primary node waits for the data with the Secondary node to be synchronized to the specified number of seconds before abdicating.

Slicing cluster improvement

1. Add tool function sh.removeTagRange ()

There is only sh.addTagRange () in the old version, and if you want to delete the tagRange, you can only delete it manually in the config.tags collection.

2. Provide more predictable Read Preference processing

In the new version, the mongos instance no longer pins the connection to the replication set member when performing a read operation, but reevaluates the Read Preference for each read operation. This makes the behavior of the Read Preference easier to predict when it is modified.

3. Provide writeConcern settings for chunk migration

The new version provides writeConcern parameters for equalizers for moveChunk and cleanupOrphaned, two commands that involve chunk migration.

4. Add equalizer status display

In the new version, the state information of the equalizer can be seen through sh.status ().

Other changes

1. Optimize explan function

The new version of the explain function can support query plan display for operations such as count,find,group,aggregate,update,remove, and the results are more comprehensive and refined.

2. Rewrite the mongodb tool

All mongodb native tools in the new version are rewritten in the GE language, especially with the addition of parallel mechanisms in mongodump and mongorestore, which can greatly speed up the export and import of data.

3. Log output control

In the new version, logs are divided into different modules, including ACCESS, COMMAND, CONTROL, GEO, INDEX, NETWORK, QUERY, REPL, SHARDING, STORAGE, JOURNAL and WRITE. Users can dynamically adjust the log level of each module, which is undoubtedly more conducive to system problem diagnosis.

4. Optimization of index construction

In the process of background index establishment, the operation of deleting database, deleting table and deleting index cannot be carried out, and the process of background index establishment will not be interrupted automatically. In addition, using the createIndexes command, multiple indexes can be built at the same time, and the data can be scanned only once, which improves the efficiency of indexing.

Summary

The above lists only some of the main features and modifications of MongoDB 3.0. if you want to know more, you can check out the Release-Notes of MongoDB 3.0. Overall, MongoDB 3.0 provides more surprising new features and makes people more optimistic about its future development.

Reference link MongoDB 3.0official Release-Notes: http://docs.mongodb.org/manual/release-notes/3.0/ Beijing Mongo user group second offline activity PPT: http://www.mongoing.com/archives/543 http://blog.chinaunix.net/xmlrpc.php?r=blog/article&id=4865696&uid=15795819

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.