Shulou(Shulou.com)06/01 Report--

Contents Concept Basic idea Applicable scene Role Introduction Experimental environment Experimental process

I. Concept

Sharding is the process of splitting a database and distributing it across different machines. Spreading data across different machines allows you to store more data and handle larger loads without requiring powerful servers.

II. Basic ideas

The collection is divided into small pieces, which are distributed into several slices, each slice responsible for only a portion of the total data, and finally an equalizer is used to equalize the slices (data migration). It operates through a routing process called mongos, which knows the correspondence between data and slices (through the configuration server). Most usage scenarios are to solve the problem of disk space, which may deteriorate for writing, and queries should try to avoid cross-fragment queries.

III. Applicable scenarios

1 Use fragmentation to solve disk space problems.

② Distribute the write pressure to each shard through fragmentation, using the resources of the shard server itself.

③ Want to put a lot of data in memory to improve performance. As above, sharding uses the sharding server's own resources.

IV. Introduction to the role

1. Configure the server. Is a separate mongod process that holds metadata about clusters and shards, i.e. information about what data each shard contains. Start building first and enable logging. Start the configuration server as you would a normal mongod, specifying configsvr. When the service is unavailable, it becomes read-only, and data cannot be segmented and migrated.

2. Routing server. Mongos, or mongos, function as a route for programs to connect to. It does not save data itself. It loads cluster information from the configuration server at startup. To start the mongos process, you need to know the address of the configuration server and specify the configdb option.

③ Fragment server. Is a stand-alone ordinary mongod process, save data information. This can be a replica set or a single server.

principle topology diagram

Experimental environment:

Version: MongoDB 3.2.1

Installation package: Link: pan.baidu.com/s/1Dfy6Ria49Dc_oMKOBUNY1g Password: fnwv

host virtual machine IP address port configuration server CentOS-7-x86_64192.168.37.128 instance 137017 routing server CentOS-7-x86_64192.168.37.128 instance 137017 sharding server 1CentOS-7-x86_64192.168.37.128 instance 247017 sharding server 2CentOS-7-x86_64192.168.37.128 instance 64192.168.37.128 Example 347018

Experimental process:

1. Unzip the mongodb installation package

mkdir /opt/abc

mount.cifs //192.168.37.1/SHARE /opt/abc

cd /opt/abc

tar zxvf mongodb-linux-x86_64-3.2.1.tgz -C /opt

Move the mongodb file to/usr/local

cd /opt

mv mongodb-linux-x86_64-3.2.1/ /usr/local/mongodb

3. Make mongo and mongod quick references

ln -s /usr/local/mongodb/bin/mongo /usr/bin/mongo

ln -s /usr/local/mongodb/bin/mongod /usr/bin/mongod

4. Create data storage directory and log storage directory of instances (4 instances: 3 for verification and 1 for standby)

mkdir -p /data/mongodb/mongodb{1,2,3,4} #Data Storage Directory

cd /data/mongodb/

mkdir logs

touch logs/mongodb{1,2,3,4}.log #Log storage file

chmod 777 logs/*.log

5. Optimization

ulimit -u 25000 #zoom in number of processes

ulimit -n 25000 #Zoom in Number of open files

configuration server

6. Edit the configuration server file

cd /usr/local/mongodb/bin

vim mongodb1.conf

port=37017dbpath=/data/mongodb/mongodb1logpath=/data/mongodb/logs/mongodb1.loglogappend=truefork=truemaxConns=5000storageEngine=mmapv1configsvr=true ** #Configure Server Mode **

7. When the node memory is insufficient, allocate memory from other nodes.

sysctl -w vm.zone_reclaim_mode=0

echo never > /sys/kernel/mm/transparent_hugepage/enabled #Disable large page memory

echo never > /sys/kernel/mm/transparent_hugepage/defrag

8. Open the configuration server

mongod -f mongodb.conf

mongo --port 37017

slice server

9. Copy Instance 1 Configuration File (Instance 2)

cp -p mongodb.conf mongodb2.conf

10. Modify the configuration file (Example 2)

vim mongodb2.conf

port=47017dbpath=/data/mongodb/mongodb2logpath=/data/mongodb/logs/mongodb2.logshardsvr=true #Fragment Server Mode

11. Copy Instance 2 Configuration File (Instance 3)

cp -p mongodb2.conf mongodb3.conf

10. Modify the configuration file (Example 3)

vim mongodb3.conf

port=47018dbpath=/data/mongodb/mongodb3logpath=/data/mongodb/logs/mongodb3.log

11. Open the fragmentation server

mongod -f mongodb2.conf

mongo --port 47017

mongod -f mongodb3.conf

mongo --port 47018

routing server

12. Start routing service

./ mongos --port 27017 --fork --logpath=/usr/local/mongodb/bin/route.log --configdb 192.168.37.128:37017 --chunkSize 1

--port 27017: route configuration server entry--fork: background running logpath: log file storage path--configdb: specify the server that actually processes the request (configuration server)--chunkSize 1: no addition, default 200MB

Enable fragmentation services

13. Login to database

mongo

14. Check whether the shard server exists

sh.status() #Empty under shards, no shard server

15. Add a fragmentation server

sh.addShard("192.168.37.128:47017")

sh.addShard("192.168.37.128:47018")

Verify fragmentation service functionality

16. Create database kgc---Set users---Add data

use kgc

for(var i=1;i show dbs

config 0.031GBkgc 0.078GB

17. Check whether fragmentation is performed

sh.status()

18. Enable database fragmentation

sh.enableSharding("kgc")

sh.status()

19. Create an index and fragment according to the index

db.users.createIndex({"id":1}) #Create index on user table

sh.shardCollection("kgc.users",{"id":1}) #Table fragmentation

sh.status()

20, the experiment is complete. In addition, you can add instance 4 according to the above steps, and the fragmentation process will be repeated.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.