Shulou(Shulou.com)06/01 Report--

MongoDB slicing briefly describes that database applications with high data volume and throughput will cause great pressure on the performance of a single machine, a large amount of query will exhaust the CPU of a single machine, and a large amount of data will exert greater pressure on the storage of a single machine, which will eventually deplete the memory of the system and transfer the pressure to the disk IO. MongoDB sharding is a method of using multiple servers to store data to support huge data storage and data manipulation. Slicing technology can meet the needs of the massive growth of MongoDB data. When one MongoDB server is not enough to store massive data or provide acceptable read and write throughput, we can split the data on multiple servers so that the database system can store and process more data. MongoDB sharding advantage sharding provides a way to deal with high throughput and large amount of data: the use of sharding reduces the number of requests to be processed per shard, so through horizontal scaling, the cluster can increase its own storage capacity. For example, when inserting a piece of data, the application only needs to access the shard that stores the data. The use of sharding reduces the data stored in each shard village.

The advantage of sharding is to provide an architecture similar to linear growth, improve data availability and improve the performance of large database query servers. Sharding can be used when MongoDB single point database server storage becomes a bottleneck, when the performance of a single point database server becomes a bottleneck, or when large applications need to be deployed to make full use of memory.

Composition of MongoDB shard cluster

The MongoDB sharding cluster has the following three main components:

Shard: sharding server, which is used to store actual data blocks. In the actual production environment, a shard server role can be assumed by a Peplica Set composed of several servers to prevent a single point of failure of the host. Config Server: configuration server that stores configuration information for the entire sharded cluster, including chunk information. Routers: front-end routing, which allows clients to access, and makes the entire cluster look like a single database, so that front-end applications can be used transparently. System environment system: CentOS 7.4x86x64 software version: 4.0 turn off firewall and selinuxIP address routing server (Routers) configuration server (Config Server) Shard1Shard2Shard3192.168.125.1192701727018270012700227003192.168.125.1202701727018270012700227003192.168.125.1212701727018270012700227003 deploy sharding cluster

Installation and configuration of three physical servers download and decompress MongoDBwget https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-4.0.0.tgztar zxvf mongodb-linux-x86_64-4.0.0.tgz-C / optmv / opt/mongodb-linux-x86_64-4.0.0 / / usr/local/mongodb create routing, configuration, data storage directory and log management of sharding servers

The routing server does not store data, so there is no need to create a data storage directory, and permissions are required for log files to be created. Mkdir-p / data/mongodb/configmkdir-p / data/mongodb/shard {1Magne 2jue 3} mkdir-p / data/mongodb/logstouch / data/mongodb/logs/shard {1ghe2jue 3} .logtouch / data/mongodb/logs/mongos.logtouch / data/mongodb/logs/config.logchmod 777 / data/mongodb/logs/*.log create an administrative user, modify directory permissions useradd-M-s / sbin/nologin mongochown-R mongo:mongo / usr/local/mongodbchown-R mongo:mongo / data/mongodb to add environment variables It is easy to use echo 'export MONGODB_HOME=/usr/local/mongodb' > > / etc/profileecho' export PATH=$PATH:$MONGODB_HOME/bin' > > / etc/profilesource / etc/profile system parameters to optimize ulimit-n 25000 / / maximum number of files that can be opened ulimit-u 25000 / / maximum number of processes available to users sysctl-w vm.zone_reclaim_mode=0 / / when memory is insufficient, memory is allocated from other nodes # starting from CentOS7, MongoDB will recommend turning off the THP feature of the system Otherwise, it may lead to performance degradation echo never > / sys/kernel/mm/transparent_hugepage/enabledecho never > / sys/kernel/mm/transparent_hugepage/defrag / / * Note * these optimizations are temporary. Restart the failed deployment configuration server (the configuration steps of the three physical servers are the same) and write to the configuration file. We can use the scp command to send the configuration file to the other two physical servers # vim config.confdbpath=/data/mongodb/config / / data file storage location logpath=/data/logs/config.log / / log file port=27018 / / port number logappend=truefork=truemaxConns=5000 storageEngine=mmapv1replSet=configs / / replication set name configsvr=true / / set parameter to true# mongod-f config.conf / / start the config instance scp / usr/local/mongodb/bin/config.conf Root@192.168.125.120:/usr/local/mongodb/binscp / usr/local/mongodb/bin/config.conf root@192.168.125.121:/usr/local/mongodb/bin configuration replication set (can be operated on any physical machine) mongo-- port 27018config= {_ id: "configs" Members: [{_ id:0,host: "192.168.125.119mov 27018"}, {_ id:1,host: "192.168.125.120 id:1,host 27018"}, {_ id:2,host: "192.168.125.121 id:1,host 27018"}]} / / create a replication set rs.initiate (config) / / initialize the replication set deployment sharding server to edit the shard {1cue 2jue 3} .conf configuration file The port is 27001, 27002, respectively, set shardsvr=true, and start the sharding server # vim shard1.confdbpath=/data/mongodb/shard1logpath=/data/logs/shard1.logport=27001logappend=truefork=truemaxConns=5000 storageEngine=mmapv1shardsvr=true# mongod-f shard1.conf#, which is the same as the configuration files of the other two instances, only the port, data file storage and log files need to be changed. You only need to start after configuration to configure shard1 as a replication set (note here is that Create a replication set error on the server that is pre-set as the quorum node.) Mongo-- port 27001use adminconfig= {_ id: "shard1", members: [{_ id:0,host: "192.168.125.119VR 27001"}, {_ id:1,host: "192.168.125.120 id 27001"}, {_ id:2,host: "192.168.125.12127001"}]} / / create replication set rs.initiate (config) / / initialize the replication set the remaining two sharding servers have the same shard2 and shard3 settings If you pay attention to the problems with the port and the quorum node, you can deploy the routing server to create a profile and send the profile to other physical servers. Note that the routing server does not need to store the data directory # vim mongos.conflogpath=/data/mongodb/logs/mongos.loglogappend = trueport = 27017fork = trueconfigdb = configs/192.168.125.119:27018192.168.125.120:27018192.168.125.121:27018maxConns=20000 to start the mongos instance mongs-f / usr/local/mongodb/bin/mongos.conf# Note Here, the mongos command is used to start the sharding function mongo / / enter port 27017 mongos > use adminmongos > sh.addShard ("shard1/192.168.125.119:27001192.168.125.120:27001172.16.10.29:27001") mongos > sh.addShard ("shard2/192.168.125.119:27002192.168.125.120:27002172.16.10.29:27002") mongos > sh.status () / / View the cluster by default Status # here to add two sharding servers Subsequent additions will also evenly distribute sharding data to achieve sharding function setting chunk size mongos > use configmongos > db.settings.save ({"_ id": "chunksize", "value": 1}) # setting the block size to 1m is convenient for the experiment, otherwise you need to insert massive data to simulate writing data mongos > use schoolmongos > show collectionsmongos > for (iTun1) Ish.enableSharding ("school") # We can customize libraries or tables that need sharding to create indexes for user collections in the school library Then fragment the table mongos > db.user.createIndex ({"id": 1}) # use "id" as the index mongos > sh.shardCollection ("school.user", {"id": 1}) # fragment the user table according to "id" mongos > sh.status () # View sharding mongos > sh.help () # View sharding related commands

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.