Shulou(Shulou.com)06/01 Report--

Background

When MongoDB stores large amounts of data, one machine may not be enough to store data, or it may not be enough to provide acceptable read and write throughput. At this time, we can split the data on multiple machines, so that the database system can store and process more data.

1. Introduction to MongoDB sharding

There are three roles:

Configuration server (config): a separate mongod process that stores cluster and shard metadata, that is, information about what data each shard contains.

Routing server (mongos): acts as a routing function for programs to connect. The data is not saved itself, and the cluster information is loaded from the configuration server at startup.

Sharding server (sharding): a stand-alone mongod process that holds data information. It can be a server, or it can be configured as a replica set if you want high availability.

2. Experimental environment

IP of two machines:

172.16.101.54 sht-sgmhadoopcm-01

172.16.101.55 sht-sgmhadoopnn-01

Config server:

172.16.101.55:27017

Mongos:

172.16.101.55:27018

Sharding:

172.16.101.54:27017

172.16.101.54:27018

172.16.101.54:27019

2. Start the config service

Modify the configuration file of the configuration server. The parameter clusterRole specifies the role as configsvr.

[root@sht-sgmhadoopnn-01 mongodb] # cat / etc/mongod27017.confsystemLog: destination: file path: "/ usr/local/mongodb/log/mongod27017.log" logAppend: true storage: dbPath: / usr/local/mongodb/data/db27017 journal: enabled: true processManagement: fork: true pidFilePath: / usr/local/mongodb/data/db27017/mongod27017.pidnet: port: 27017 bindIp: 0.0.0.0 setParameter: enableLocalhostAuthBypass: false sharding: clusterRole: configsvr archiveMovedChunks: true

[root@sht-sgmhadoopnn-01 mongodb] # bin/mongod-- config / etc/mongod27017.conf

Warning: bind_ip of 0.0.0.0 is unnecessary; listens on all ips by default

About to fork child process, waiting until server is ready for connections.

Forked process: 31033

Child process started successfully, parent exiting

3. Start the mongos service

Modify the configuration file of the routing server. The parameter configDB specifies the IP and port of the config server. There is no need to configure information about the data file, because the routing server does not store data.

[root@sht-sgmhadoopnn-01 mongodb] # cat / etc/mongod27018.confsystemLog: destination: file path: "/ usr/local/mongodb/log/mongod27018.log" logAppend: true processManagement: fork: true pidFilePath: / usr/local/mongodb/data/db27018/mongod27018.pid net: port: 27018 bindIp: 0.0.0.0 setParameter: enableLocalhostAuthBypass: false sharding: autoSplit: true configDB: 172.16.101.55 virtual 27017 chunkSize: 64

[root@sht-sgmhadoopnn-01 mongodb] # bin/mongos-- config / etc/mongod27018.conf

Warning: bind_ip of 0.0.0.0 is unnecessary; listens on all ips by default

2018-11-10T18:57:13.705+0800 W SHARDING running with 1 config server should be done only for testing purposes and is not recommended for production

About to fork child process, waiting until server is ready for connections.

Forked process: 31167

Child process started successfully, parent exiting

4. Start the sharding service

Just an ordinary mongodb process, a common configuration file

[root@sht-sgmhadoopcm-01 mongodb] # cat / etc/mongod27017.confsystemLog: destination: file path: "/ usr/local/mongodb/log/mongod27017.log" logAppend: true storage: dbPath: / usr/local/mongodb/data/db27017 journal: enabled: true processManagement: fork: true pidFilePath: / usr/local/mongodb/data/db27017/mongod27017.pid net: port: 27017 bindIp: 0.0.0.0 setParameter: enableLocalhostAuthBypass: false

[root@sht-sgmhadoopcm-01 mongodb] # bin/mongod-- config / etc/mongod27017.conf

[root@sht-sgmhadoopcm-01 mongodb] # bin/mongod-- config / etc/mongod27018.conf

[root@sht-sgmhadoopcm-01 mongodb] # bin/mongod-- config / etc/mongod27019.conf

5. Log in to Mongos service and add sharding information

[root@sht-sgmhadoopnn-01 mongodb] # bin/mongo-- port=27018

Mongos > sh.addShard ("172.16.101.54 virtual 27017")

{"shardAdded": "shard0000", "ok": 1}

Mongos > sh.addShard ("172.16.101.54 virtual 27018")

{"shardAdded": "shard0001", "ok": 1}

Mongos > sh.addShard ("172.16.101.54 virtual 27019")

{"shardAdded": "shard0002", "ok": 1}

View cluster sharding information

Mongos > sh.status ()-Sharding Status-sharding version: {"_ id": 1, "minCompatibleVersion": 5, "currentVersion": 6, "clusterId": ObjectId ("5be6b98a507b3e0370eb36b4")} shards: {"_ id": "shard0000", "host": "172.16.101.54 Soviet 27017"} {"_ id": "shard0001" "host": "172.16.101.54 host 27018"} {"_ id": "shard0002", "host": "172.16.101.54 host"} balancer: Currently enabled: yes Currently running: no Failed balancer rounds in last 5 attempts: 0 Migration Results for the last 24 hours: No recent migrations databases: {"_ id": "admin", "partitioned": false "primary": "config"} mongos > db.runCommand ({listshards:1}) {"shards": [{"_ id": "shard0000", "host": "172.16.101.54 mongos 27017"}, {"_ id": "shard0001" "host": "172.16.101.54 shard0002 27018"}, {"_ id": "shard0002", "host": "172.16.101.54 host"}], "ok": 1}

6. Open slicing

Libraries and collections of sharding need to be executed, as well as sharding mode, which is divided into two types: hash and range

[root@sht-sgmhadoopnn-01 mongodb] # bin/mongo-- port=27018

The fragment library is testdb.

Mongos > sh.enableSharding ("testdb")

{"ok": 1}

(1) hash sharding mode test

The collection of slicing is collection1, and hash slicing is performed according to id.

Mongos > sh.shardCollection ("testdb.collection1", {"_ id": "hashed"})

{"collectionsharded": "testdb.collection1", "ok": 1}

Insert a total of 10 test document

Mongos > use testdb

Switched to db testdb

Mongos > for (var iTunes 0 I sh.status ()-- Sharding Status-sharding version: {"_ id": 1, "minCompatibleVersion": 5, "currentVersion": 6, "clusterId": ObjectId ("5be6b98a507b3e0370eb36b4")} shards: {"_ id": "shard0000", "host": "172.16.101.54 mosaic 27017"} {"_ id": "shard0001" "host": "172.16.101.54 host 27018"} {"_ id": "shard0002", "host": "172.16.101.54 host"} balancer: Currently enabled: yes Currently running: no Failed balancer rounds in last 5 attempts: 0 Migration Results for the last 24 hours: 2: Success databases: {"_ id": "admin", "partitioned": false "primary": "config"} {"_ id": "test", "partitioned": false, "primary": "shard0000"} {"_ id": "testdb", "partitioned": true "primary": "shard0000"} testdb.collection1 shard key: {"_ id": "hashed"} chunks: shard0000 2 shard0001 2 shard0002 2 {"_ id": {"$minKey": 1}}-- > {"_ id": NumberLong ("- 6148914691236517204")} on: shard0000 Timestamp (3 2) {"_ id": NumberLong ("- 6148914691236517204")}-> {"_ id": NumberLong ("- 3074457345618258602")} on: shard0000 Timestamp (3,3) {"_ id": NumberLong ("- 3074457345618258602")}-> {"_ id": NumberLong (0)} on: shard0001 Timestamp (3) 4) {"_ id": NumberLong (0)}-> {"_ id": NumberLong ("3074457345618258602")} on: shard0001 Timestamp (3,5) {"_ id": NumberLong ("3074457345618258602")}-> {"_ id": NumberLong ("6148914691236517204")} on: shard0002 Timestamp (3) 6) {"_ id": NumberLong ("6148914691236517204")}-> > {"_ id": {"$maxKey": 1}} on: shard0002 Timestamp (3,7)

View the data distribution on each sharding db.collection.stats ()

Mongos > db.collection1.stats () {"sharded": true, "paddingFactorNote": "paddingFactor is unused and unmaintained in 3.0. It remains hard coded to 1.0 for compatibility only., "userFlags": 1, "capped": false, "ns": "testdb.collection1", "count": 10, # Ten document data "numExtents": 3, "size": 480, "storageSize": 24576, "totalIndexSize": 49056, "indexSizes": {"_ id_": 24528 "_ id_hashed": 24528}, "avgObjSize": 48, "nindexes": 2, "nchunks": 6, "shards": {"shard0000": {"ns": "testdb.collection1", "count": 0, "size": 0, "numExtents": 1 "storageSize": 8192, "lastExtentSize": 8192, "paddingFactor": 1, "paddingFactorNote": "paddingFactor is unused and unmaintained in It remains hard coded to 1.0 for compatibility only., "userFlags": 1, "capped": false, "nindexes": 2, "totalIndexSize": 16352, "indexSizes": {"_ id_": 8176, "_ id_hashed": 8176} "ok": 1}, "shard0001": {"ns": "testdb.collection1", "count": 6, "size": 288, "avgObjSize": 48, "numExtents": 1, "storageSize": 8192, "lastExtentSize": 8192 "paddingFactor": 1, "paddingFactorNote": "paddingFactor is unused and unmaintained in 3.0 It remains hard coded to 1.0 for compatibility only., "userFlags": 1, "capped": false, "nindexes": 2, "totalIndexSize": 16352, "indexSizes": {"_ id_": 8176, "_ id_hashed": 8176} "ok": 1}, "shard0002": {"ns": "testdb.collection1", "count": 4, "size": 192, "avgObjSize": 48, "numExtents": 1, "storageSize": 8192, "lastExtentSize": 8192 "paddingFactor": 1, "paddingFactorNote": "paddingFactor is unused and unmaintained in 3.0 It remains hard coded to 1.0 for compatibility only., "userFlags": 1, "capped": false, "nindexes": 2, "totalIndexSize": 16352, "indexSizes": {"_ id_": 8176, "_ id_hashed": 8176} "ok": 1}}, "ok": 1}

Log in to the sharding node to view the data distribution, which is consistent with the result seen by the command db.collection.stats ().

It can be found that there is no data on node 27017, 6 document on node 27018 and 4 document on node 27019. The reason for this may be that the amount of data inserted is too small and not evenly distributed. The larger the amount of data, the more uniform the distribution.

[root@sht-sgmhadoopcm-01 mongodb] # bin/mongo 172.16.101.54:27017/testdb

> db.collection1.find ()

[root@sht-sgmhadoopcm-01 mongodb] # bin/mongo 172.16.101.54:27018/testdb

> db.collection1.find ()

{"_ id": ObjectId ("5be6c467e6467cc8077da816"), "name": "jack1"}

{"_ id": ObjectId ("5be6c467e6467cc8077da817"), "name": "jack2"}

{"_ id": ObjectId ("5be6c467e6467cc8077da818"), "name": "jack3"}

{"_ id": ObjectId ("5be6c467e6467cc8077da81a"), "name": "jack5"}

{"_ id": ObjectId ("5be6c467e6467cc8077da81c"), "name": "jack7"}

{"_ id": ObjectId ("5be6c467e6467cc8077da81e"), "name": "jack9"}

[root@sht-sgmhadoopcm-01 mongodb] # bin/mongo 172.16.101.54:27019/testdb

> db.collection1.find ()

{"_ id": ObjectId ("5be6c467e6467cc8077da815"), "name": "jack0"}

{"_ id": ObjectId ("5be6c467e6467cc8077da819"), "name": "jack4"}

{"_ id": ObjectId ("5be6c467e6467cc8077da81b"), "name": "jack6"}

{"_ id": ObjectId ("5be6c467e6467cc8077da81d"), "name": "jack8"}

(2) range sharding mode test

The collection of slicing is collection2, and range slicing is performed according to name.

Mongos > sh.shardCollection ("testdb.collection2", {"name": 1})

{"collectionsharded": "testdb.collection2", "ok": 1}

Mongos > for (var iTunes 0 I sh.status ()-- Sharding Status-sharding version: {"_ id": 1, "minCompatibleVersion": 5, "currentVersion": 6, "clusterId": ObjectId ("5be6b98a507b3e0370eb36b4")} shards: {"_ id": "shard0000", "host": "172.16.101.54 mosaic 27017"} {"_ id": "shard0001" "host": "172.16.101.54 host 27018"} {"_ id": "shard0002", "host": "172.16.101.54 host"} balancer: Currently enabled: yes Currently running: no Failed balancer rounds in last 5 attempts: 0 Migration Results for the last 24 hours: 4: Success databases: {"_ id": "admin", "partitioned": false "primary": "config"} {"_ id": "test", "partitioned": false, "primary": "shard0000"} {"_ id": "testdb", "partitioned": true "primary": "shard0000"} testdb.collection1 shard key: {"_ id": "hashed"} chunks: shard0000 2 shard0001 2 shard0002 2 {"_ id": {"$minKey": 1}}-- > {"_ id": NumberLong ("- 6148914691236517204")} on: shard0000 Timestamp (3 2) {"_ id": NumberLong ("- 6148914691236517204")}-> {"_ id": NumberLong ("- 3074457345618258602")} on: shard0000 Timestamp (3,3) {"_ id": NumberLong ("- 3074457345618258602")}-> {"_ id": NumberLong (0)} on: shard0001 Timestamp (3) 4) {"_ id": NumberLong (0)}-> {"_ id": NumberLong ("3074457345618258602")} on: shard0001 Timestamp (3,5) {"_ id": NumberLong ("3074457345618258602")}-> {"_ id": NumberLong ("6148914691236517204")} on: shard0002 Timestamp (3) 6) {"_ id": NumberLong ("6148914691236517204")}-- > > {"_ id": {"$maxKey": 1}} on: shard0002 Timestamp (3 7) testdb.collection2 shard key: {"name": 1} chunks: shard0000 1 shard0001 1 shard0002 1 {"name": {"$minKey": 1}}-- > {"name": "jack1"} on: shard0001 Timestamp (2 0) {"name": "jack1"}-- > > {"name": "jack5"} on: shard0002 Timestamp (3,0) {"name": "jack5"}-- > > {"name": {"$maxKey": 1}} on: shard0000 Timestamp (3,1)

Mongos > db.collection2.stats () {"sharded": true, "paddingFactorNote": "paddingFactor is unused and unmaintained in 3.0. It remains hard coded to 1.0 for compatibility only., "userFlags": 1, "capped": false, "ns": "testdb.collection2", "count": 1000, "numExtents": 6, "size": 48032, "storageSize": 221184, "totalIndexSize": 130816, "indexSizes": {"_ id_": 65408, "name_1": 65408} "avgObjSize": 48.032, "nindexes": 2, "nchunks": 3, "shards": {"shard0000": {"ns": "testdb.collection2", "count": 555, "size": 26656, "avgObjSize": 48, "numExtents": 3, "storageSize": 172032 LastExtentSize: 131072, paddingFactor: 1, paddingFactorNote: paddingFactor is unused and unmaintained in 3.0. It remains hard coded to 1.0 for compatibility only., "userFlags": 1, "capped": false, "nindexes": 2, "totalIndexSize": 65408, "indexSizes": {"_ id_": 32704, "name_1": 32704} "ok": 1}, "shard0001": {"ns": "testdb.collection2", "count": 1, "size": 48, "avgObjSize": 48, "numExtents": 1, "storageSize": 8192, "lastExtentSize": 8192 "paddingFactor": 1, "paddingFactorNote": "paddingFactor is unused and unmaintained in 3.0 It remains hard coded to 1.0 for compatibility only., "userFlags": 1, "capped": false, "nindexes": 2, "totalIndexSize": 16352, "indexSizes": {"_ id_": 8176, "name_1": 8176} "ok": 1}, "shard0002": {"ns": "testdb.collection2", "count": 444, "size": 21328, "avgObjSize": 48, "numExtents": 2, "storageSize": 40960, "lastExtentSize": 32768 "paddingFactor": 1, "paddingFactorNote": "paddingFactor is unused and unmaintained in 3.0 It remains hard coded to 1.0 for compatibility only., "userFlags": 1, "capped": false, "nindexes": 2, "totalIndexSize": 49056, "indexSizes": {"_ id_": 24528, "name_1": 24528} "ok": 1}}, "ok": 1}

Reference link

Sharding

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.