Shulou(Shulou.com)06/01 Report--

Demand pain points solved by sharding technology: (1) database applications with high data volume and throughput will cause great pressure on the performance of the stand-alone machine; (2) a large amount of query will exhaust the CPU of the stand-alone machine; (3) a large amount of data will cause great pressure on the storage of the stand-alone machine, which will eventually exhaust the memory of the system and transfer the pressure to the keyboard IO. MongoDB slicing technology is a method of using multiple servers to store data to support huge data storage and data operations. When MongoDB stores large amounts of data, one machine may not be enough to store data, or it may not be enough to provide acceptable read and write throughput. At this time, we can split the data on multiple machines, so that the database system can store and process more data. MongoDB sharding structure the following figure shows the composition and distribution of MongoDB sharding cluster structure:

In the above figure, there are three main components as follows: Shard: sharding server, which is used to store actual data blocks. In the actual production environment, a shard server role can be undertaken by several machines grouping a replica set to prevent host from single point of failure Config Server: configuration server, mongod instance, stores the configuration information of the entire sharding cluster, including chunk information. Query Routers: front-end routing, client access, and make the entire cluster look like a single database. Front-end applications can transparently use MongoDB sharding advantages: reduce the number of requests to be processed by a single shard, and improve the storage capacity and throughput of the cluster. For example, when inserting a piece of data, the application only needs to access the shards that store this data to reduce the data stored in a single shard and improve data availability. Improve the performance of large database query services. When MongoDB single point database server storage and performance become a bottleneck, or large applications need to be deployed to make full use of memory, sharding clustering can be used for simple deployment: the following describes how to deploy a simple MongoDB sharding cluster on a physical server of a CentOS7 system. 1 routing instance (port 27017) 1 configuration instance (port 37017) 2 sharding instances (port 47017, 47018) this experiment uses manual compilation to install mongodb, Baidu network disk provides free installation package: https://pan.baidu.com/s/1bZXJDBrn9qxBF6r6IF_aLQ specific steps are as follows: install the supported software, decompress the prepared MongoDB software package, note: this installation is on 64-bit Linux system

Yum install openssl-devel-y

Tar zxvf mongodb-linux-x86_64-3.2.1.tgz-C / opt/

Mv mongodb-linux-x86_64-3.2.1 / / usr/local/mongodb

Create a data storage directory for four instances of MongoDB, log storage directory / data/logs, log files, and modify permissions

Mkdir-p / data/mongodb/mongodb {1, 2, 3, 4}

Mkdir / data/logs

Touch / data/logs/mongodb {1,2,3,4} .log

Chmod-R 777 / data/logs/*.log

Kernel tuning of Linux system for MongoDB service

Ulimit-n 25000 # maximum number of files

Ulimit-u 25000 # maximum number of processes

When one node runs out of memory, allocate memory from other nodes

Sysctl-w vm.zone_reclaim_mode=0 # permanent setting

Echo never > / sys/kernel/mm/transparent_hugepage/enabled

Echo never > / sys/kernel/mm/transparent_hugepage/defrag

Ln-s / usr/local/mongodb/bin/* / usr/bin/ # facilitates service management

The deployment configuration server edits the mongodb1.conf configuration file, port number is 37017, sets configsvr=true, and starts the configuration server

Cd / usr/local/mongodb/bin/

Vim mongodb1.conf

Port=37017 # specify service port

Dbpath=/data/mongodb/mongodb1 # data storage directory

Logpath=/data/logs/mongodb1.log # Log File

Logappend=true # writes logs by appending

Fork=true # running in the background

MaxConns=5000 # sets the maximum number of simultaneous connections. Default is 2000.

StorageEngine=mmapv1 # specifies the storage engine as a memory-mapped file

Configsvr=true # configuration Service

Start the configuration service and enter it. You can see that the configuration service is prefixed with "configsvr >".

[root@localhost bin] # mongod-f mongodb1.conf # Startup configuration Server

About to fork child process, waiting until server is ready for connections.

Forked process: 12594

Child process started successfully, parent exiting

[root@localhost bin] # mongo-- port 37017 # enter the service

MongoDB shell version: 3.2.1

Connecting to: 127.0.0.1:37017/test

Server has startup warnings:

2018-09-15T04:39:24.967+0800 I STORAGE [initandlisten]

2018-09-15T04:39:24.967+0800 I STORAGE [initandlisten] * * WARNING: Readahead for / data/mongodb1 is set to 4096KB

2018-09-15T04:39:24.967+0800 I STORAGE [initandlisten] * * We suggest setting it to 256KB (512 sectors) or less

2018-09-15T04:39:24.967+0800 I STORAGE [initandlisten] * * http://dochub.mongodb.org/core/readahead

2018-09-15T04:39:25.828+0800 I CONTROL [initandlisten] * * WARNING: You are running this process as the root user, which is not recommended.

2018-09-15T04:39:25.828+0800 I CONTROL [initandlisten]

2018-09-15T04:39:25.829+0800 I CONTROL [initandlisten]

2018-09-15T04:39:25.829+0800 I CONTROL [initandlisten] * * WARNING: / sys/kernel/mm/transparent_hugepage/enabled is' always'.

2018-09-15T04:39:25.829+0800 I CONTROL [initandlisten] * * We suggest setting it to 'never'

2018-09-15T04:39:25.829+0800 I CONTROL [initandlisten]

2018-09-15T04:39:25.829+0800 I CONTROL [initandlisten] * * WARNING: / sys/kernel/mm/transparent_hugepage/defrag is' always'.

2018-09-15T04:39:25.829+0800 I CONTROL [initandlisten] * * We suggest setting it to 'never'

2018-09-15T04:39:25.829+0800 I CONTROL [initandlisten]

Configsvr >

Deploy sharding server to edit mongodb2.conf configuration file, port number is 47017, set shardsvr=true

Vim mongodb2.conf

Port=47017

Dbpath=/data/mongodb/mongodb2

Logpath=/data/logs/mongodb2.log

Logappend=true

Fork=true

MaxConns=5000

StorageEngine=mmapv1

Shardsvr=true

Also edit the mongodb3.conf configuration file with port number 47018

Vim mongodb3.conf

Port=47018

Dbpath=/data/mongodb/mongodb3

Logpath=/data/logs/mongodb3.log

Logappend=true

Fork=true

MaxConns=5000

StorageEngine=mmapv1

Shardsvr=true

Start two sharding servers

[root@localhost bin] # mongod-f mongodb2.conf

About to fork child process, waiting until server is ready for connections.

Forked process: 12810

Child process started successfully, parent exiting

[root@localhost bin] # mongod-f mongodb3.conf

About to fork child process, waiting until server is ready for connections.

Forked process: 12825

Child process started successfully, parent exiting

Start the routing server through the. / mongos-- help command to view the parameter information related to starting the route. ChunkSize is the block size, and the default is 200MB. For testing purposes, the value is set to 1.

[root@localhost bin] #. / mongos-- port 27017-- fork-- logpath=/usr/local/mongodb/bin/route.log-- configdb 192.168.30.55 root@localhost bin 37017-- chunkSize 1

2018-09-15T05:03:06.217+0800 W SHARDING [main] Running a sharded cluster with fewer than 3 config servers should only be done for testing purposes and is not recommended for production.

About to fork child process, waiting until server is ready for connections.

Forked process: 12934

Child process started successfully, parent exiting

After enabling the sharding server to connect to the routing instance, you can view the sharding status information through the sh.status () command

[root@localhost bin] # mongo # enter the routing instance

MongoDB shell version: 3.2.1

Connecting to: test

Server has startup warnings:

2018-09-15T16:00:19.746+0800 I CONTROL [main] * * WARNING: You are running this process as the root user, which is not recommended.

2018-09-15T16:00:19.746+0800 I CONTROL [main]

Mongos > sh.status ()

-Sharding Status

Sharding version: {

"_ id": 1

"minCompatibleVersion": 5

"currentVersion": 6

"clusterId": ObjectId ("5b9cbc14b4c77895df796bac")

}

Shards: empty under # shards, no sharding server

Active mongoses:

"3.2.1": 1

Balancer:

Currently enabled: yes

Currently running: no

Failed balancer rounds in last 5 attempts: 0

Migration Results for the last 24 hours:

No recent migrations

Databases:

Add two sharding servers through the sh.addShard command

Mongos > sh.addShard ("192.168.30.55 purl 47017")

{"shardAdded": "shard0000", "ok": 1}

Mongos > sh.addShard ("192.168.30.55 purl 47018")

{"shardAdded": "shard0001", "ok": 1}

If you look at the sharding information again, you can see that the sharding server just added has been displayed under the shards: option.

Mongos > sh.status ()

-Sharding Status

Sharding version: {

"_ id": 1

"minCompatibleVersion": 5

"currentVersion": 6

"clusterId": ObjectId ("5b9cbc14b4c77895df796bac")

}

Shards:

{"_ id": "shard0000", "host": "192.168.30.55 virtual 47017"}

{"_ id": "shard0001", "host": "192.168.30.55 virtual 47018"}

Active mongoses:

"3.2.1": 1

Balancer:

Currently enabled: yes

Currently running: no

Failed balancer rounds in last 5 attempts: 0

Migration Results for the last 24 hours:

No recent migrations

Databases:

After two sharding servers are added to implement the sharding function, sharding is not enabled in the database and collection.

Mongos > show dbs

Config 0.031GB

Mongos > use school # enter and create the database school

Switched to db school

Mongos > for (var iTune1 witch I sh.enableSharding ("school")

{"ok": 1}

Create an index on the info collection

Mongos > db.info.createIndex ({"id": 1})

{

"raw": {

"192.168.30.55 purl 47017": {

"createdCollectionAutomatically": false

"numIndexesBefore": 1

"numIndexesAfter": 2

"ok": 1

}

}

"ok": 1

}

Use the sh.shardCollection ("school.info", {"id": 1}) command to fragment the collection info

Mongos > sh.shardCollection ("school.info", {"id": 1})

{"collectionsharded": "school.info", "ok": 1}

View sharding information

Mongos > sh.status ()

-Sharding Status

Sharding version: {

"_ id": 1

"minCompatibleVersion": 5

"currentVersion": 6

"clusterId": ObjectId ("5b9cbc14b4c77895df796bac")

}

Shards:

{"_ id": "shard0000", "host": "192.168.30.55 host"} # two sharding server information

{"_ id": "shard0001", "host": "192.168.30.55 virtual 47018"}

Active mongoses:

"3.2.1": 1

Balancer:

Currently enabled: yes

Currently running: no

Failed balancer rounds in last 5 attempts: 0

Migration Results for the last 24 hours:

5: Success

Databases:

{"_ id": "school", "primary": "shard0000", "partitioned": true} # Database school sharding information

School.info

Shard key: {"id": 1}

Unique: false

Balancing: true

Chunks: # you can see that chunks is evenly distributed over the two shards

Shard0000 6

Shard0001 5

{"id": {"$minKey": 1}}-- > {"id": 4682} on: shard0001 Timestamp (2,0)

{"id": 4682}-- > {"id": 9364} on: shard0001 Timestamp (3,0)

{"id": 9364}-- > {"id": 14046} on: shard0001 Timestamp (4,0)

{"id": 14046}-- > {"id": 18728} on: shard0001 Timestamp (5,0)

{"id": 18728}-- > {"id": 23410} on: shard0001 Timestamp (6,0)

{"id": 23410}-- > {"id": 28092} on: shard0000 Timestamp (6,1)

{"id": 28092}-- > {"id": 32774} on: shard0000 Timestamp (1,6)

{"id": 32774}-- > {"id": 37456} on: shard0000 Timestamp (1,7)

{"id": 37456}-- > {"id": 42138} on: shard0000 Timestamp (1,8)

{"id": 42138}-- > {"id": 46820} on: shard0000 Timestamp (1,9)

{"id": 46820}-- > {"id": {"$maxKey": 1}} on: shard0000 Timestamp (1,10)

Add tags

Mongos > sh.addShardTag ("shard0000", "abc01")

Mongos > sh.addShardTag ("shard0001", "abc02")

Mongos > sh.status ()

-Sharding Status

Sharding version: {

"_ id": 1

"minCompatibleVersion": 5

"currentVersion": 6

"clusterId": ObjectId ("5b9cbc14b4c77895df796bac")

}

Shards:

{"_ id": "shard0000", "host": "192.168.30.55 host", "tags": ["abc01"]}

{"_ id": "shard0001", "host": "192.168.30.55 host", "tags": ["abc02"]}

MongoDB sharding server management can add or remove sharding server as needed.

[root@localhost bin] # cp mongodb3.conf mongodb4.conf

[root@localhost bin] # vim mongodb4.conf

Port=47019

Logpath=/data/logs/mongodb4.log

Dbpath=/data/mongodb4

Fork=true

Logappend=true

MaxConns=5000

StorageEngine=mmapv1

Shardsvr=true

Start instance 4, enter the routing instance and add a new sharding server

[root@localhost bin] # mongod-f mongodb4.conf

About to fork child process, waiting until server is ready for connections.

Forked process: 52634

Child process started successfully, parent exiting

[root@localhost bin] # mongo

MongoDB shell version: 3.2.1

Connecting to: test

Server has startup warnings:

2018-09-15T16:00:19.746+0800 I CONTROL [main] * * WARNING: You are running this process as the root user, which is not recommended.

2018-09-15T16:00:19.746+0800 I CONTROL [main]

Add a new sharding server using the sh.addShard ("192.168.30.55 virtual 47019") command, check the sharding information again, and find that the shards are evenly distributed among the three sharding servers.

Mongos > sh.addShard ("192.168.30.55 virtual 47019") # add sharding server

{"shardAdded": "shard0002", "ok": 1}

Mongos > sh.status () # View sharding service information

-Sharding Status

Sharding version: {

"_ id": 1

"minCompatibleVersion": 5

"currentVersion": 6

"clusterId": ObjectId ("5b9cbc14b4c77895df796bac")

}

Shards:

{"_ id": "shard0000", "host": "192.168.30.55 virtual 47017"}

{"_ id": "shard0001", "host": "192.168.30.55 virtual 47018"}

{"_ id": "shard0002", "host": "192.168.30.55 virtual 47019"}

Active mongoses:

"3.2.1": 1

Balancer:

Currently enabled: yes

Currently running: no

Failed balancer rounds in last 5 attempts: 0

Migration Results for the last 24 hours:

8: Success

Databases:

{"_ id": "school", "primary": "shard0000", "partitioned": true}

School.info

Shard key: {"id": 1}

Unique: false

Balancing: true

Chunks:

Shard0000 4

Shard0001 4

Shard0002 3

{"id": {"$minKey": 1}}-- > {"id": 4682} on: shard0002 Timestamp (9,0)

{"id": 4682}-- > {"id": 9364} on: shard0001 Timestamp (9,1)

{"id": 9364}-- > {"id": 14046} on: shard0001 Timestamp (4,0)

{"id": 14046}-- > {"id": 18728} on: shard0001 Timestamp (5,0)

{"id": 18728}-- > {"id": 23410} on: shard0001 Timestamp (6,0)

{"id": 23410}-- > {"id": 28092} on: shard0002 Timestamp (7,0)

{"id": 28092}-- > {"id": 32774} on: shard0002 Timestamp (8,0)

{"id": 32774}-- > {"id": 37456} on: shard0000 Timestamp (8,1)

{"id": 37456}-- > {"id": 42138} on: shard0000 Timestamp (1,8)

{"id": 42138}-- > {"id": 46820} on: shard0000 Timestamp (1,9)

{"id": 46820}-- > {"id": {"$maxKey": 1}} on: shard0000 Timestamp (1,10)

You can delete a newly added sharding server using the db.runCommand ({"removeshard": "192.168.30.55 virtual 47019"}) command.

Mongos > use admin # Note: execute the command under admin db.

Switched to db admin

Mongos > db.runCommand ({"removeshard": "192.168.30.55 virtual 47019"})

{

"msg": "draining started successfully"

"state": "started"

"shard": "shard0002"

"note": "you need to drop or movePrimary these databases"

"dbsToMove": []

"ok": 1

}

Mongos > db.runCommand ({"removeshard": "192.168.30.55 virtual 47019"})

{

"msg": "removeshard completed successfully"

"state": "completed"

"shard": "shard0002"

"ok": 1

}

"Note: the command cannot be deleted successfully until state is completed, otherwise it will not be deleted successfully. The shard is in the state of" draining ": true. In this state, not only the shard is not deleted successfully, but also other shard operations are deleted. When you encounter this state, you can execute removeshard again. It is best to repeat the deletion command when you delete the shard. Until state checks the shard information for completed again, you can see that the shard has been deleted.

Mongos > sh.status ()

-Sharding Status

Sharding version: {

"_ id": 1

"minCompatibleVersion": 5

"currentVersion": 6

"clusterId": ObjectId ("5b9cbc14b4c77895df796bac")

}

Shards:

{"_ id": "shard0000", "host": "192.168.30.55 virtual 47017"}

{"_ id": "shard0001", "host": "192.168.30.55 virtual 47018"}

Active mongoses:

"3.2.1": 1

Balancer:

Currently enabled: yes

Currently running: no

Failed balancer rounds in last 5 attempts: 0

Migration Results for the last 24 hours:

11: Success

Databases:

{"_ id": "school", "primary": "shard0000", "partitioned": true}

School.info

Shard key: {"id": 1}

Unique: false

Balancing: true

Chunks:

Shard0000 6

Shard0001 5

{"id": {"$minKey": 1}}-- > {"id": 4682} on: shard0000 Timestamp (10,0)

{"id": 4682}-- > {"id": 9364} on: shard0001 Timestamp (9,1)

{"id": 9364}-- > {"id": 14046} on: shard0001 Timestamp (4,0)

{"id": 14046}-- > {"id": 18728} on: shard0001 Timestamp (5,0)

{"id": 18728}-- > {"id": 23410} on: shard0001 Timestamp (6,0)

{"id": 23410}-- > {"id": 28092} on: shard0001 Timestamp (11,0)

{"id": 28092}-- > {"id": 32774} on: shard0000 Timestamp (12,0)

{"id": 32774}-- > {"id": 37456} on: shard0000 Timestamp (8,1)

{"id": 37456}-- > {"id": 42138} on: shard0000 Timestamp (1,8)

{"id": 42138}-- > {"id": 46820} on: shard0000 Timestamp (1,9)

{"id": 46820}-- > {"id": {"$maxKey": 1}} on: shard0000 Timestamp (1,10)

Connection configuration server view information the configuration server stores the details of the MongoDB database collection shards, which can be viewed with the following command

Mongo-port 37017

Configsvr > use config

Configsvr > show collections

....

Collections

Chunks

Databases

....

Configsvr > db.chunks.findOne ()

Configsvr > db.collections.find ()

Configsvr > db.databases.find ()

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.