Shulou(Shulou.com)06/01 Report--

MongoDB sharding introduces the purpose of slicing

The database application with high data volume and throughput will cause great pressure on the performance of the stand-alone machine, the large amount of query will exhaust the CPU of the single machine, and the large amount of data will exert great pressure on the storage of the single machine, which will eventually exhaust the memory of the system and transfer the pressure to the disk IO.

In order to solve these problems, there are two basic methods: vertical expansion and horizontal expansion.

Vertical scaling: add more CPU and storage resources to expand capacity.

Scale horizontally: distribute datasets across multiple servers. Horizontal expansion is slicing.

Sharding provides a method to cope with high throughput and large amount of data. The use of sharding reduces the number of requests to be processed per shard, so by scaling horizontally, the cluster can increase its own storage capacity and throughput. For example, when inserting a piece of data, the application only needs to access the shards that store the data, and the use of shards reduces the data stored in each shard.

Sharding cluster architecture

(1) mongos: a module for data routing and dealing with clients. Mongos doesn't have any data, and he doesn't know what to do with it. He goes to config server.

(2) config server: all the ways of storing and fetching data, the information of all shard nodes, and some configuration information of the sharding function. Metadata that can be understood as real data.

(3) shard: the real data storage location, which stores data in chunk units.

Architecture diagram

Environment description

System version: CentOS Linux release 7.5.1804 (Core)

Firewall SE is off

Mongodb version: mongodb-linux-x86_64-3.2.1.tgz

Prepare in advance and operate under the root user

Using multiple instances of Mongodb.

Extract the Mobile mongodb installation package

Tar zxvf mongodb-linux-x86_64-3.2.1.tgz-C / opt/

Cd / opt/

Mv mongodb-linux-x86_64-3.2.1 / / usr/local/mongodb

Establish a soft connection to facilitate management

Cd / usr/local/mongodb/

Cd bin/

Ln-s / usr/local/mongodb/bin/mongo / usr/bin/mongo

Ln-s / usr/local/mongodb/bin/mongod / usr/bin/mongod

Create the required directory

Mkdir-p / data/mongodb/mongodb {1, 2, 3, 4}

Cd / data/mongodb/

Mkdir logs

Cd logs/

Touch mongodb {1,2,3,4} .log

Chmod 777 * .log

Optimize the process and open the number of files

Ulimit-u 25000

Ulimit-n 25000

Edit configuration file (modify bold item)

Cd / usr/local/mongodb/bin/

Vim mongodb1.conf

Port=37017

Dbpath=/data/mongodb/mongodb1

Logpath=/data/mongodb/logs/mongodb1.log

Logappend=true

Fork=true

MaxConns=5000

StorageEngine=mmapv1

Configsvr=true / / designated as the configuration server

Memory allocation: when one node runs out of memory, allocate memory from other nodes

Sysctl-w vm.zone_reclaim_mode=0

Echo never > / sys/kernel/mm/transparent_hugepage/enabled

Echo never > / sys/kernel/mm/transparent_hugepage/defrag

Copy and modify the configuration file of 2prime3 (modify bold)

Cp-p mongodb1.conf mongodb2.conf

Vim mongodb2.conf

Port=47017

Dbpath=/data/mongodb/mongodb2

Logpath=/data/mongodb/logs/mongodb2.log

Logappend=true

Fork=true

MaxConns=5000

StorageEngine=mmapv1

Shardsvr=true

Cp-p mongodb2.conf mongodb3.conf

Vim mongodb3.conf

Port=47018

Dbpath=/data/mongodb/mongodb3

Logpath=/data/mongodb/logs/mongodb3.log

Logappend=true

Fork=true

MaxConns=5000

StorageEngine=mmapv1

Shardsvr=true

Start the service

Mongod-f mongodb1.conf

Mongod-f mongodb2.conf

Mongod-f mongodb3.conf

Turn on route mode

. / mongos-- port 27017-- fork-- logpath=/usr/local/mongodb/bin/route.log-- configdb 192.168.137.13 fork 37017-- chunkSize 1

To this replication set configuration is complete

test

Landing

[root@cent bin] # mongo

MongoDB shell version: 3.2.1

Connecting to: test

Welcome to the MongoDB shell.

...

Mongos > / / this shows that you have successfully logged in with mongs (configure server)

Check to see if there is another sharding server

Mongos > sh.status ()

]-Sharding Status

Sharding version: {

"_ id": 1

"minCompatibleVersion": 5

"currentVersion": 6

"clusterId": ObjectId ("5b9e394051c099916cc7b236")

}

Shards: / / there is nothing here, that is, no sharding server has been created

Active mongoses:

"3.2.1": 1

Balancer:

Currently enabled: yes

Currently running: no

Failed balancer rounds in last 5 attempts: 0

Migration Results for the last 24 hours:

No recent migrations

Databases:

Add a sharding server

Mongos > sh.addShard ("192.168.137.13)) / / add 47017

{"shardAdded": "shard0000", "ok": 1}

Mongos > sh.addShard ("192.168.137.13)) / / add 47018

{"shardAdded": "shard0001", "ok": 1}

Mongos > sh.status ()

-Sharding Status

Sharding version: {

"_ id": 1

"minCompatibleVersion": 5

"currentVersion": 6

"clusterId": ObjectId ("5b9e394051c099916cc7b236")

}

Shards:

{"_ id": "shard0000", "host": "192.168.137.13 47017"} / / see the addition succeeded

{"_ id": "shard0001", "host": "192.168.137.13 virtual 47018"}

Active mongoses:

"3.2.1": 1

Balancer:

Currently enabled: yes

Currently running: no

Failed balancer rounds in last 5 attempts: 0

Migration Results for the last 24 hours:

No recent migrations

Databases:

Add something to the database to see if it will be fragmented.

Mongos > use mood

Switched to db mood

Mongos > for (var iTune1 witch I sh.status ()

-Sharding Status

Sharding version: {

"_ id": 1

"minCompatibleVersion": 5

"currentVersion": 6

"clusterId": ObjectId ("5ba24eeef6c5933feb7b6cf5")

}

Shards:

Active mongoses:

"3.2.1": 1

Balancer:

Currently enabled: yes

Currently running: no

Failed balancer rounds in last 5 attempts: 0

Migration Results for the last 24 hours:

No recent migrations

Databases: / / not at all

Because we need to start sharding, let's start it.

Mongos > sh.enableSharding ("mood") / / enable database sharding

{"ok": 1}

Mongos > db.info.createIndex ({"id": 1}) / / create a table index

{

"raw": {

"192.168.137.13 purl 47017": {

"createdCollectionAutomatically": false

"numIndexesBefore": 1

"numIndexesAfter": 2

"ok": 1

}

}

"ok": 1

}

Mongos > sh.shardCollection ("mood.info", {"id": 1}) / / Table fragmentation

{"collectionsharded": "mood.info", "ok": 1}

Mongos > sh.status () / / View the result

-Sharding Status

Sharding version: {

"_ id": 1

"minCompatibleVersion": 5

"currentVersion": 6

"clusterId": ObjectId ("5ba3b64dcc3cc1318414a128")

}

Shards:

{"_ id": "shard0000", "host": "192.168.137.13 virtual 47017"}

{"_ id": "shard0001", "host": "192.168.137.13 virtual 47018"}

Active mongoses:

"3.2.1": 1

Balancer:

Currently enabled: yes

Currently running: no

Failed balancer rounds in last 5 attempts: 0

Migration Results for the last 24 hours:

1: Success

Databases:

{"_ id": "mood", "primary": "shard0000", "partitioned": true}

Mood.info

Shard key: {"id": 1}

Unique: false

Balancing: true

Chunks:

Shard0000 2

Shard0001 1

{"id": {"$minKey": 1}}-- > {"id": 4682} on: shard0001 Timestamp (2,0) / / has started slicing

{"id": 4682}-- > {"id": 9364} on: shard0000 Timestamp (2,1)

{"id": 9364}-- > {"id": {"$maxKey": 1}} on: shard0000 Timestamp (1,2)

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.