Shulou(Shulou.com)06/01 Report--

Overview of deploying MongoDB sharding cluster sharding

The database application with high data volume and throughput will cause great pressure on the performance of the stand-alone machine, the large amount of query will exhaust the CPU of the single machine, and the large amount of data will cause greater pressure on the storage of the single machine, which will eventually deplete the memory of the system and transfer the pressure to the disk IO.

MongoDB sharding is a method of using multiple servers to store data to support huge data storage and data manipulation. Slicing technology can meet the needs of the massive growth of MongoDB data. When one MongoDB server is not enough to store massive data or provide acceptable read and write throughput, we can split the data on multiple servers so that the database system can store and process more data.

MongoDB fragmentation advantage

Sharding provides a way to cope with throughput and large amounts of data.

1. The shard reduces the number of requests to be processed per shard, so through horizontal scaling, the cluster can improve its storage capacity and throughput. For example, when inserting a piece of data, the application only needs to access the shard that stores the data.

2. The use of sharding reduces the data stored in each shard.

As shown in the figure below, if there is 1TB data in a collection, the 1TB data can be shredded into four shard by sharding. When querying the corresponding data, it will go to the corresponding sharding server to query, which can improve the data availability.

Composition of MongoDB shard cluster

MongoDB sharding cluster has three main components:

1. Shard: sharding server, which is used to store actual data blocks. In the actual production environment, a shard server role can be assumed by several servers forming a replication set (Replica Set) to prevent a single point of failure of the host.

2. Config Server: configuration server, which stores the configuration information of the entire sharding cluster, including chunk information.

3. Routers: front-end routing, from which the client is accessed, and makes the whole cluster look like a single database, and the front-end applications can be used transparently.

Experimental environment

The experiment here is to build MongoDB sharding on the same server, but after MongoDB3.4, Shard defaults to build MongoDB service set. For convenience, we use version 3.2 of MongoDB.

Component port number routing instance 27017 configure instance 37017Shard1 instance 47017Shard2 instance 47018 shard cluster deployment 1, deployment configuration server

Install related dependency packages and unzip software to the specified directory.

[root@localhost ~] # yum install openssl-devel-y # installation dependency package [root@localhost ~] # tar xfvz mongodb-linux-x86_64-3.2.1.tgz-C / opt/ [root@localhost ~] # cd / opt/ [root@localhost opt] # mv mongodb-linux-x86_64-3.2.1 / / usr/local/mongodb# rename the package to facilitate the following use

Create the data directory, log directory and log files required by the instance to be created.

[root@localhost opt] # mkdir-p / data/mongodb/mongodb {1meme 2diem 3pm 4} # create instance data storage directory [root@localhost opt] # mkdir / data/mongodb/logs# create instance log directory [root@localhost opt] # touch / data/mongodb/logs/mongodb {1meme2 3 root@localhost opt 4} .log # create instance log file [root@localhost opt] # chmod-R 777 / data/mongodb/logs/*.log# modify log permissions [root@localhost opt] # ulimit-n 2500processes change the maximum number of processes that can be opened to 25000 [root@localhost opt] # ulimit-u 2500log change the maximum number of files that can be opened to 25000 [root@localhost opt] # sysctl-w vm.zone_reclaim_mode=0# when a node runs out of available memory If 0, the system tends to allocate memory from other nodes. If it is 1, then the system tends to reclaim Cache memory from the local node most of the time. [root@localhost opt] # echo never > / sys/kernel/mm/transparent_hugepage/ enabled [root @ localhost opt] # echo never > / sys/kernel/mm/transparent_hugepage/defrag# when the node is running out of available memory Choose to close the large memory page [root@localhost opt] # export PATH=$PATH:/usr/local/mongodb/bin/# and add the MongoDB path to the environment variable [root@localhost bin] # vim mongodb1.conf# create the configuration server configuration file port=37017dbpath=/data/mongodb/mongodb1logpath=/data/mongodb/logs/mongodb1.loglogappend=truefork=truemaxConns=5000storageEngine=mmapv1configsvr=true # set to the configuration server [root@localhost mongodb] # mongod-f / usr/local/mongodb/bin/mongodb1.conf # start the configuration server process

2. Configure sharding server

Edit the mongodb2.conf configuration file, port 47017, set to shardsvr=true, also edit mongodb3.conf, port 47018 dsvrue true, and then start at the same time.

[root@localhost mongodb] # cd / usr/local/mongodb/bin/ [root@localhost bin] # cp-p mongodb1.conf mongodb2.conf [root@localhost bin] # vim mongodb2.conf port=47017 # set port dbpath=/data/mongodb/mongodb2 # data storage directory to modify logpath=/data/mongodb/logs/mongodb2.log # log file also modify logappend=truefork=truemaxConns=5000storageEngine=mmapv1shardsvr=true # to set the sharding server [root@localhost bin] # cp-p mongodb2.conf mongodb3. Conf [root@localhost bin] # vim mongodb3.conf port=47018 # set port dbpath=/data/mongodb/mongodb3 # data storage directory to modify logpath=/data/mongodb/logs/mongodb3.log # log files also modify logappend=truefork=truemaxConns=5000storageEngine=mmapv1shardsvr=true

After all the modifications are made, start the two sharding servers.

3. Start the routing server [root@localhost bin] #. / mongos-- port 27017-- fork-- logpath=/usr/local/mongodb/bin/route.log-- configdb 192.168.58.131 logpath=/usr/local/mongodb/bin/route.log 37017-- chunkSize "this command means to access port 27017 of mongodb. The city will go to 192.168.58.131 Running a sharded cluster with fewer than 37017 [root@localhost bin] #. / mongos-- port 27017-- fork-- logpath=/usr/local/mongodb/bin/route.log-- configdb 192.168.58.131 port 37017-- chunkSize 12018-07-17T17:20:58.990+0800 W SHARDING [main] Running a sharded cluster with fewer than 3 config servers should only be done for testing purposes and is not recommended for production.about to fork child process, waiting until server is ready for connections.forked process: 6430child process started successfully, parent exiting

[root@localhost ~] # mongoMongoDB shell version: 3.2.1connecting to: testServer has startup warnings: 2018-07-17T17:20:59.120+0800 I CONTROL [main] * * WARNING: You are running this process as the root user, which is not recommended.2018-07-17T17:20:59.120+0800 I CONTROL [main] mongos > sh.status ()-- Sharding Status-sharding version: {"_ id": 1, "minCompatibleVersion": 5, "currentVersion": 6 "clusterId": ObjectId ("5b4db4fdf19287c74d1a4081")} shards: active mongoses: "3.2.1": 1 balancer: Currently enabled: yes Currently running: no Failed balancer rounds in last 5 attempts: 0 Migration Results for the last 24 hours: No recent migrations databases:# can see the empty space under shards So we need to add two sharding servers.

Mongos > sh.addShard ("192.168.58.131shardAdded 47017") # add sharding servers on port 47017 {"shardAdded": "shard0000", "ok": 1} mongos > sh.addShard ("192.168.58.131ve47018") # add sharding servers on port 47018 {"shardAdded": "shard0001", "ok": 1}

To test to see if sharding can be done, we create a collection and insert 50000 pieces of data into it.

Mongos > use testswitched to db testmongos > for (var iSuppli db.users.find (). Limit (10) # View the first 10 pieces of data in collection, indicating that the insertion was successful.

Mongos > sh.status ()-Sharding Status-sharding version: {"_ id": 1, "minCompatibleVersion": 5, "currentVersion": 6, "clusterId": ObjectId ("5b4db4fdf19287c74d1a4081")} shards: {"_ id": "shard0000", "host": "192.168.58.131v 47017"} {"_ id": "shard0001" "host": "192.168.58.131 yes Currently running 47018"} active mongoses: "3.2.1": 1 balancer: Currently enabled: yes Currently running: no Failed balancer rounds in last 5 attempts: 0 Migration Results for the last 24 hours: No recent migrations databases: {"_ id": "test", "primary": "shard0000", "partitioned": false} # you can see partitioned:false It shows that there is no slicing. Mongos > sh.enableSharding ("test") {"ok": 1} # enable sharding function mongos > sh.status () {"_ id": "test", "primary": "shard0000", "partitioned": true} mongos > db.users.createIndex ({"id": 1}) {"raw": {"192.168.58.131test": {"createdCollectionAutomatically": false "numIndexesBefore": 1, "numIndexesAfter": 2, "ok": 1}, "ok": 1} # create index Convenient sharding mongos > sh.shardCollection ("test.users", {"id": 1}) {"collectionsharded": "test.users", "ok": 1} # perform sharding. You can see that the sharding succeeded by 1-20165, written to shard0001, and written to shard0000 by 20166-50000.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.