Shulou(Shulou.com)06/01 Report--

Brief introduction of mongodb fragmentation Cluster of CentOS7

The database application with high data volume and throughput will cause great pressure on the performance of the stand-alone machine, the large amount of query will exhaust the CPU of the single machine, and the large amount of data will cause greater pressure on the storage of the single machine, which will eventually deplete the memory of the system and transfer the pressure to the disk IO.

Mongodb sharding is a method of using multiple servers to store data to support huge data storage and data manipulation. Slicing technology can meet the needs of the massive growth of Mongodb data. When one mongodb server is insufficient to store large amounts of data or provide acceptable read and write throughput, the database system can store and process more data by splitting the data on multiple servers.

Advantages of slicing

The use of shards reduces the number of requests to be processed per shard, and by scaling horizontally, the cluster can improve its storage capacity and throughput.

Using sharding reduces the composition of each sharded cluster: sharding server for storing actual data blocks configserver: configuration server, storing configuration information for the entire sharding cluster routers: routing server sharding cluster management experiment

The experimental configuration diagram is as above

Let's start the experiment, where I open multiple instances on one machine instead of multiple servers.

Install the mongodb3.2 version (compile and install manually) install the software environment package

Yum-y install openssl-devel

Extract the mongodb software package

Tar zxvf mongodb-linux-x86_64-3.2.1.tgz-C / opt

Move the software package to the directory identified by the system

Mv mongodb-linux-x86_64-3.2.1 / / usr/local/mongodb

Create a data store (/ data/mongodb1, 2, 3, 4,) and log storage directory (/ data/logs) for mongodb

Mkdir-p / data/mongodb/mongodb {1, 2, 3, 4}

Mkdir / data/mongodb/logs

Touch / data/mongodb/logs/mongodb {1,2,3,4} .log

Chmod-R 777 / data/mongodb/logs/*.log

Set the values of ulimit-n and ulimit-u

When mongodb is frequently accessed, if the resource consumed by shell to start the process is set too low, an error will occur and the mongodb instance cannot be connected. Therefore, you need to set the values of ulimit-n and ulimit-u to be greater than 20000.

Ulimit-n 25000

Ulimit-u 25000

Create a configuration file for the configuration server

Cd / usr/local/mongodb/bin/

Vim mongodb1.conf

Port=37017 # Port dbpath=/data/mongodb/mongodb1 # data storage location logpath=/data/mongodb/logs/mongodb1.log # log storage location logappend=true # error log is appended. After configuring this option, mongodb logs will be appended to the existing log file Instead of creating a new file fork=true # background running maxConns=5000 # maximum number of simultaneous connections storageEngine=mmapv1 # specify the storage engine for the memory-mapped file configsvr=true # specify the mode of the configuration server to set kernel parameters

When one node runs out of memory, the system allocates memory from other nodes

Sysctl-w vm.zone_reclaim_mode=0 # permanent setting

Echo never > / sys/kernel/mm/transparent_hugepage/enabled

Echo never > / sys/kernel/mm/transparent_hugepage/defrag

Create soft links for easy management

Ln-s / usr/local/mongodb/bin/mongo / usr/bin/mongo

Ln-s / usr/local/mongodb/bin/mongod / usr/bin/mongod

Open the first instance (configure server)

Configure an instance of a sharding server

Copy build configuration file

Cp-p mongodb1.conf mongodb2.conf # copy the configuration file of the configuration server to generate the configuration file of instance 2

Cp-p mongodb1.conf mongodb3.conf # copy the configuration file of the configuration server to generate the configuration file of instance 2

Modify the configuration files of instances 2 and 3

Vim mongodb2.conf

Port=47017 # modify port number dbpath=/data/mongodb/mongodb2 # modify data storage directory logpath=/data/mongodb/logs/mongodb2.log # modify log storage directory logappend=true fork=truemaxConns=5000storageEngine=mmapv1shardsvr=true # specify the mode of the sharding server

Vim mongodb3.conf

Port=47018dbpath=/data/mongodb/mongodb3logpath=/data/mongodb/logs/mongodb3.loglogappend=truefork=truemaxConns=5000storageEngine=mmapv1shardsvr=true starts two instances of sharding servers.

Mongod-f mongodb2.conf

Mongod-f mongodb3.conf

Start the routing server

. / mongos-- port 27017-- fork-- logpath=/usr/local/mongodb/bin/route.log-- configdb 192.168.234.177VR 37017-- chunkSize 1

I don't know how to use the mongos command here. You can check the help information.

Enable sharding server

Mongo

Add a sharding server

Sh.addShard ("192.168.234.177purl 47017")

Sh.addShard ("192.168.234.177VR 47018")

View it again after adding a sharding server

Test sharding function

Mongos > show dbs

Config 0.031GB

Mongos > use kgc # enter and create a collection of use

Switched to db kgc

Mongos > db.users.insert ({"id": 1, "name": "zhangsan"}) # add a piece of data

WriteResult ({"nInserted": 1})

Mongos > for (var item2bot / I show dbs # look at the tablespace and there will be a kgc

Config 0.031GB

Kgc 0.078GB

Mongos > show tables # you can also see the users table by viewing the table information

System.indexes

Users

Enable the sharding function

Sh.enableSharding ("kgc")

Turn on sharding of collections in the database

Db.users.createIndex ({"id": 1}) # create an index on the users table

Sh.shardCollection ("kgc.users", {"id": 1}) # Table fragmentation

Sh.status () # check it again at this time, and you can already do the sharding processing.

Here, the introduction of mongodb fragmentation cluster management is all completed. If you think it is helpful, remember to like and reward the younger brother.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.