Shulou(Shulou.com)06/01 Report--

This article shows you how to achieve slicing management in MongoDB, the content is concise and easy to understand, it can definitely brighten your eyes. I hope you can get something through the detailed introduction of this article.

First, the concept of slicing. Data block

A block is also called an interval, and there may be two cases: one segment, one interval and one fragment, multiple intervals.

One slice and one interval: the data does not automatically move between slices to maintain the uniformity of the data. It is necessary to split the slices manually to move the data.

In the case of a shard with multiple intervals: a data block defaults to 64MB. When a data block reaches 64MB, a new block will be created, of course, provided that the current granularity allows for further splitting, and the balancer will ensure the uniformity of each shard. However, moving blocks also follow the principle of slicing, and the data sets between blocks can not be intersected.

For example, if a block [50-100) is now split into two chunks, it will be split into [50-75) [75-100) by default. If the current shard is larger than 9 of other shards, then it is possible that [75-100) will be moved to the new shard.

two。 Balancer

The balancer (balancer) is responsible for data migration. It periodically checks whether there is an imbalance between shards and, if so, turns on the migration of blocks. Unbalanced performance means that one slice has significantly more blocks than other fragments. If some sets reach the threshold, the balancer starts to do block migration. It selects a block from a shard with a relatively heavy load and asks if the shard needs to be split before migration. After the necessary split is completed, the blocks are migrated to a relatively small number of machines.

Second, multipart query 1. Query cluster status

Need to show hidden fragment information execution

two。 Check configuration information

All configuration information is stored in the config database of the configuration server.

Actionlog

Record the operation log of the balancer.

Changelog

Track and record the operations of the cluster, including collection slicing operations, block splitting and migration, adding and deleting shards, etc. For example, block split information:

Db.getCollection ('changelog') .find ({"what": / split/}) .sort ({"time":-1}) .limit (2)

Each time a database block is moved, four records are created and inserted into the total changelog document, namely start, commit, from, and to.

{"_ id": "backup-2018-04-09T15:52:26.656+0800-5acb1bbaebfa528b3521327c", "server": "backup", "clientAddr": "192.168.137.30 5acb1bbaebfa528b3521327c 53996", "time": ISODate ("2018-04-09T07:52:26.656Z"), "what": "moveChunk.start", "ns": "test.person" "details": {"min": {"_ id": {"$minKey": 1}}, "max": {"_ id": 1.0}, "from": "rs-b" "to": "rs-a"}} / * 2 * / {"_ id": "backup-2018-04-09T15:52:29.289+0800-5acb1bbdebfa528b35213335", "server": "backup", "clientAddr": "192.168.137.30 ISODate", "time": ISODate ("2018-04-09T07:52:29.289Z"), "what": "moveChunk.commit" "ns": "test.person", "details": {"min": {"_ id": {"$minKey": 1}}, "max": {"_ id": 1.0}, "from": "rs-b" "to": "rs-a"}} / * 3 * / {"_ id": "backup-2018-04-09T15:52:29.297+0800-5acb1bbdebfa528b3521333a", "server": "backup", "clientAddr": "192.168.137.30 ISODate", "time": ISODate ("2018-04-09T07:52:29.297Z"), "what": "moveChunk.from" "ns": "test.person", "details": {"min": {"_ id": {"$minKey": 1}}, "max": {"_ id": 1.0}, "step 1 of 6": 0, "step 2 of 6": 10 "step 3 of 6": 153, "step 4 of 6": 2061, "step 5 of 6": 402, "step 6 of 6": 24, "to": "rs-a", "from": "rs-b" "note": "success"} / * 4 * / {"_ id": "master-2018-04-09T15:52:29.307+0800-5acb1bbd7bc60438ea626411", "server": "master", "clientAddr": "", "time": ISODate ("2018-04-09T07:52:29.307Z"), "what": "moveChunk.to", "ns": "test.person" "details": {"min": {"_ id": {"$minKey": 1}}, "max": {"_ id": 1.0}, "step 1 of 6": 22, "step 2 of 6": 11, "step 3 of 6": 4 "step 4 of 6": 0, "step 5 of 6": 2042, "step 6 of 6": 373, "note": "success"}}

View Code

Each step in the details field represents the time, and the "step N of 6" message shows the length of the step in milliseconds.

When from shards receive a moveChunks command from mongos, it will do the following:

(1)。 Check command parameters

(2) apply to the configuration server for a distribution lock in order to enter the migration process

(3) try to connect to to shards

(4) copy data

(5) work with to sharding and configuration server to confirm whether the migration is completed successfully.

When to shard receives a command from from shard, it performs the following actions:

(1) migrate the index

(2) Delete any data that already exists within the block range

(3) copy all documents in the block to to sharding

(4) actions performed on these documents during running replication on to shards

(5) wait for to shards to copy the migrated data to most of the servers in the replica set.

(6) indicates whether the migration is successful.

Chunks

Store all block information of collection fragments

Collections

Record all fragment collection information, and the records in the record will not be erased because the fragment collection is deleted.

Databases

Record information about the database in the cluster, regardless of whether the database is fragmented or not. If sharding is enabled in the database, the value of the "partitioned": field is true. "primary" records the main shard to which the database belongs. All new collections are created on the database master slice by default. For example, if the current collection does not have data on a shard, it will not create a collection on that shard. So if a shard has a collection of shards, the data of the collection must be removed or deleted, otherwise the shard cannot be deleted.

Lockpings

Record the ping record information of whether the shard is running normally.

Locks

Record and store distributed lock operation information.

Migrations

Mongos

Record the relevant information of mongos, record the information of each mongos instance.

Settings

Contains setting information for balancers and blocks, etc.

Shards

Cluster shard information

Tags

Record fragment label information

Transactions

Version

Cluster version Information

Note: if you need to modify the configuration information, you need to switch to the config database operation by connecting to the mongos instead of connecting directly to the configuration server.

Reference: https://docs.mongodb.com/manual/reference/config-database/

3. View the network connection db.adminCommand ({"connPoolStats": 1}) 4. Limit the number of connections

The maximum number of concurrent connections that mongos can accept. If this setting is higher than the maximum connection tracking threshold configured by the operating system, this setting has no effect.

Note: MongoDB removes the upper limit of the maxIncomingConnections setting in version 2.6.

Third, slicing management 1. Add sharded use admindb.auth ("dba", "dba") sh.addShard ("rs-a/192.168.137.10:27010192.168.137.10:27011192.168.137.10:27012"), sh.addShard ("rs-b/192.168.137.20:28010192.168.137.20:28011192.168.137.20:28012"), sh.addShard ("rs-c/192.168.137.30:26010192.168.137.30:26011192.168.137.30:26012") Sh.status (); 2. Delete shard

First, the database corresponding to the current shard is moved to other shards. The "products" here refers to the database name.

Db.runCommand ({movePrimary: "products", to: "rs-b"})

And then delete the fragment.

Db.runCommand ({"removeShard": "rs-c"})

Note: the db.runCommand ({"removeShard": "rs-c"}) command needs to be executed multiple times to delete the shard. First, the balancer will migrate the data on the shard. You can query whether the migration is completed through the sh.isBalancerRunning () command. After the migration is completed, execute the delete shard command to remove the shard completely.

3. Balancer management

3.1 turn on the balancer

Use adminsh.startBalancer () or sh.setBalancerState (true)

3.2 turn off the balancer

Use adminsh.stopBalancer () or sh.setBalancerState (false)

Check whether the equalizer is closed, return that the flase identity balancer is closed, and query whether the equalizer is running

Sh.getBalancerState (); while (sh.isBalancerRunning ()) {print ("waiting..."); sleep (1000);}

The equalizer should be closed before the database management operation is performed. After closing the equalizer, the system will not enter the balancing process, but the equalizer is not closed immediately, so it is also necessary to query whether the equalizer is running.

3.3 View balancer on statu

Db.settings.find ({"_ id": "balancer"}) or sh.getBalancerState ()

3.4 check if the balancer is running

Returning ture indicates that it is running, while false represents that it is not currently running.

3.5 specify balance time

You must first make sure that the balancer is on.

Use configsh.setBalancerState (true) db.settings.update ({"_ id": "balancer"}, {"$set": {"activeWindows": {"start": "13:00", "stop": "16:00"}}, {upsert:true})

For HH values, use hour values ranging from 00-23.

For MM value, use minute values ranging from 00-59.

3.6 turn off balance time

Use configdb.settings.update ({_ id: "balancer"}, {$unset: {activeWindows: true}})

3.7 turn off the balancer that opens the specified document

Close

Sh.disableBalancing ("test.aa")

open

Sh.enableBalancing ("test.aa")

Query whether to close the balancer of the document. If you return true, it will be closed. If no result is returned, it is not closed. The return error indicates that the document does not exist or the document does not have sharding open.

Db.getSiblingDB ("config") .collections.findOne ({_ id: "test.aa"}) .noBalance

3.8. Matters needing attention during backup

You need to turn off the equalizer before performing a backup, but the equalizer is not closed immediately, so you also need to query whether the equalizer is running. Do not back up while the equalizer is active. You can perform the following query before the backup operation:

Sh.getBalancerState () sh.isBalancerRunning ()

Note: make sure that the balancer is off and that sh.isBalancerRunning () does not return ture; you can also maintain the backup by setting the balancing time of the balancer.

Reference: https://docs.mongodb.com/manual/tutorial/manage-sharded-cluster-balancer/

4. Block management 1. Modify block size

In MB, the default block size is 64MB. The larger the block, the longer it takes to migrate to the shard.

Use config; query current block size db.settings.find ({"_ id": "chunksize"}) modify block size db.settings.save ({_ id: "chunksize", value: 64}); 2. Move data blocks manually

2.1 query document block information

Db.getCollection ('chunks') .find ({"ns": "test.xxxx"})

2.2 move the block "$minKey" to the sharded rs-c

You only need to specify any value in the return of the block to move the block. Note that the range of the block does not contain an upper limit.

Sh.moveChunk ("test.xxxx", {"username": "$minKey"}, "rs-c")

2.3 split block

If the block size exceeds the maximum block size set by setting, the system forbids moving the block, and the block needs to be split. Here, you need to specify a new block range and specify a lower limit.

Sh.splitAt ("test.xxxx", {"username": "p"})

2.4 query block size

Db.runCommand ({dataSize: "test.xxxx", keyPattern: {"username": 1}, min: {"username": "b"}, max: {"username": "c"}})

Unit of bytes, you need to specify the range of the block.

2.5 extra large chunks that cannot be split

Suppose you use the year/month/day field as shard, and one day the business is attacked, resulting in a sharp increase in the amount of data on that day. However, the shard value is already the smallest unit and can no longer be split. At this time, the problem cannot be solved by shredding. You can manually move the chunk to the shard that is not a hot spot.

5. Refresh configuration information

Mongos sometimes fails to update the configuration information correctly from the configuration server. You can use the flushRouterConfig command to manually flush the cache, and if the refresh does not solve the need to restart the mongos process.

Db.adminCommand ({"flushRouterConfig": 1}) 6. Delete sharding database use newsdb.dropDatabase () the above is how to implement sharding management in MongoDB. Have you learned any knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.