Shulou(Shulou.com)06/01 Report--

Brief introduction of MongoDB fragmentation

The database application with high data volume and throughput will cause great pressure on the performance of the stand-alone machine, the large amount of query will exhaust the CPU of the single machine, and the large amount of data will exert great pressure on the storage of the single machine, which will eventually exhaust the memory of the system and transfer the pressure to the disk IO.

MongoDB sharding is a method of using multiple servers to store data to support huge data storage and data manipulation. Slicing technology can meet the needs of the massive growth of MongoDB data. When one MongoDB server is not enough to store massive data or provide acceptable read and write throughput, we can split the data on multiple servers so that the database system can store and process more data.

MongoDB fragmentation advantage

Sharding provides a way to cope with high throughput and large amounts of data:

The use of shards reduces the number of requests to be processed per shard, so by scaling horizontally, the cluster can increase its own storage capacity. For example, when inserting a piece of data, the application only needs to access the shard that stores the data. The use of sharding reduces the data stored in each shard village.

The advantage of sharding is to provide an architecture similar to linear growth, improve data availability and improve the performance of large database query servers. Sharding can be used when MongoDB single point database server storage becomes a bottleneck, when the performance of a single point database server becomes a bottleneck, or when large applications need to be deployed to make full use of memory.

Composition of MongoDB shard cluster

The MongoDB sharding cluster has the following three main components:

Shard: sharding server, which is used to store actual data blocks. In the actual production environment, a shard server role can be assumed by a Peplica Set composed of several servers to prevent a single point of failure of the host. Config Server: configuration server that stores configuration information for the entire sharded cluster, including chunk information. Routers: front-end routing, which allows clients to access, and makes the entire cluster look like a single database, so that front-end applications can be used transparently.

System environment

System: CentOS 7.4x86x64

Software version: 4.0

Turn off the firewall and selinux

IP address routing server (Routers) configuration server (Config Server) Shard1Shard2Shard3192.168.125.1192701727018270012700227003192.168.125.1202701727018270012700227003192.168.125.1212701727018270012700227003

Deploy sharded clusters

Installation and configuration of three physical servers

1. Download and decompress MongoDB

Wget https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-4.0.0.tgztar zxvf mongodb-linux-x86_64-4.0.0.tgz-C / optmv / opt/mongodb-linux-x86_64-4.0.0 / / usr/local/mongodb

2. Create the data storage directory and log management of routing, configuration and sharding server.

The routing server does not store data, so there is no need to create a data storage directory, and permissions are required for log files to be created.

Mkdir-p / data/mongodb/configmkdir-p / data/mongodb/shard {1 data/mongodb/logs/mongos.logtouch / data/mongodb/logs/config.logchmod 3} mkdir-p / data/mongodb/logstouch / data/mongodb/logs/shard. Logtouch / data/mongodb/logs/mongos.logtouch / data/mongodb/logs/config.logchmod 777 / data/mongodb/logs/*.log

3. Create administrative users and modify directory permissions

Useradd-M-s / sbin/nologin mongochown-R mongo:mongo / usr/local/mongodbchown-R mongo:mongo / data/mongodb

4. add environment variables to make it easy to use

Echo 'export MONGODB_HOME=/usr/local/mongodb' > > / etc/profileecho' export PATH=$PATH:$MONGODB_HOME/bin' > > / etc/profilesource / etc/profile

5. Optimization of system parameters

Ulimit-n 25000 / / maximum number of files that can be opened ulimit-u 25000 / / maximum number of processes available to users sysctl-w vm.zone_reclaim_mode=0 / / when memory is insufficient, memory is allocated from other nodes # starting from CentOS7, MongoDB recommends that the THP feature of the system be turned off Otherwise, it may lead to performance degradation echo never > / sys/kernel/mm/transparent_hugepage/enabledecho never > / sys/kernel/mm/transparent_hugepage/defrag / / * Note * these optimizations are temporary and restart fails

Deploy the configuration server (three physical servers have the same configuration steps)

1. Write the configuration file. We can use the scp command to send the configuration file to the other two physical servers

# vim config.confdbpath=/data/mongodb/config / / location of data files logpath=/data/logs/config.log / / log file port=27018 / / port number logappend=truefork=truemaxConns=5000 storageEngine=mmapv1replSet=configs / / replication set name configsvr=true / / set parameter to true# mongod-f config.conf / / start config instance scp / usr/local/mongodb/bin/config.conf root@192.168.125.120:/usr/local/mongodb/binscp / usr/local/mongodb/bin/config.conf Root@192.168.125.121:/usr/local/mongodb/bin

2. Configure the replication set (you can operate on any physical machine)

Mongo-- port 27018config= {_ id: "configs", members: [{_ id:0,host: "192.168.125.119 configs 27018"}, {_ id:1,host: "192.168.125.120 id 27018"}, {_ id:2,host: "192.168.125.121purl 27018"}} / / create replication set rs.initiate (config) / initialize replication set

Deploy sharding server

Edit the configuration file shard {1JERT 2JI 3} .conf, port 27001Magi 27002JR 27003, set shardsvr=true, and start the sharding server.

# vim shard1.confdbpath=/data/mongodb/shard1logpath=/data/logs/shard1.logport=27001logappend=truefork=truemaxConns=5000 storageEngine=mmapv1shardsvr=true# mongod-f shard1.conf# is the same as the configuration files of the other two instances, only the port, data file storage and log files need to be changed. You only need to start after the configuration is completed.

Configure shard1 as a replication set (note here that an error was reported when creating a replication set on a server that was pre-set as a quorum node.)

Mongo-- port 27001use adminconfig= {_ id: "shard1", members: [{_ id:0,host: "192.168.125.119id 27001"}, {_ id:1,host: "192.168.125.120 id 27001"}, {_ id:2,host: "192.168.125.12127001"}]} / / create a replication set rs.initiate (config) / / initialize a replication set

The shard2 and shard3 settings of the other two sharding servers are the same. Just pay attention to the port and arbitration node problems.

Deploy routing server

Create a profile and send the profile to another physical server. Note that the routing server does not need to store data directories

# vim mongos.conflogpath=/data/mongodb/logs/mongos.loglogappend = trueport = 27017fork = trueconfigdb = configs/192.168.125.119:27018192.168.125.120:27018192.168.125.121:27018maxConns=20000

Start the mongos instance

Mongs-f / usr/local/mongodb/bin/mongos.conf# Note: the mongos command is used to start the mongos instance.

Start the sharding function

Mongo / / enter port 27017 by default mongos > use adminmongos > sh.addShard ("shard1/192.168.125.119:27001192.168.125.120:27001172.16.10.29:27001") mongos > sh.addShard ("shard2/192.168.125.119:27002192.168.125.120:27002172.16.10.29:27002") mongos > sh.status () / / View the cluster status # here to add two sharding servers, the subsequent sharding data will be evenly distributed

Realize the function of slicing

Set the size of the sharded chunk

Mongos > use configmongos > db.settings.save ({"_ id": "chunksize", "value": 1}) # setting the block size to 1m is convenient for experiment, otherwise massive data need to be inserted

Analog write data

Mongos > use schoolmongos > show collectionsmongos > for (iTun1transferish.enableSharding ("school") # We can customize the libraries or tables that need sharding

Create an index for the user collection in the school library, and then fragment the table

Mongos > db.user.createIndex ({"id": 1}) # use "id" as the index mongos > sh.shardCollection ("school.user", {"id": 1}) # fragment the user table according to "id" mongos > sh.status () # View sharding mongos > sh.help () # View sharding related commands

The above is the whole content of this article, I hope it will be helpful to your study, and I also hope that you will support it.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.