Shulou(Shulou.com)06/01 Report--

MongoDB fragmentation

Slice

There is another kind of cluster in Mongodb, that is, sharding technology, which can meet the needs of the massive growth of MongoDB data.

When MongoDB stores large amounts of data, one machine may not be enough to store data, or it may not be enough to provide acceptable read and write throughput. At this time, we can split the data on multiple machines, so that the database system can store and process more data.

Why do you use fragmentation?

Copy all writes to the primary node

Delay-sensitive data will be queried at the primary node

A single replica set is limited to 12 nodes

Running out of memory occurs when the number of requests is large.

Insufficient local disk

Vertical expansion is expensive

MongoDB fragmentation

The following figure shows the distribution using the sharding cluster structure in MongoDB:

There are three main components in the above figure as follows:

Shard:

It is used to store actual data blocks. In the actual production environment, a shard server role can be assumed by several machines grouped into a replica set to prevent a single point of failure of the host.

Config Server:

Mongod instance, which stores the entire ClusterMetadata, including chunk information.

Query Routers:

The front-end routing allows the client to access, and makes the whole cluster look like a single database, and the front-end applications can be used transparently.

Sharding instance

The port distribution of the sharding structure is as follows:

Shard Server 1:27020

Shard Server 2:27021

Shard Server 3:27022

Shard Server 4:27023

Config Server: 27100

Route Process:40000

Step 1: start Shard Server

[root@100 /] # mkdir-p / www/mongoDB/shard/s0

[root@100 /] # mkdir-p / www/mongoDB/shard/s1

[root@100 /] # mkdir-p / www/mongoDB/shard/s2

[root@100 /] # mkdir-p / www/mongoDB/shard/s3

[root@100 /] # mkdir-p / www/mongoDB/shard/log

[root@100 /] # / usr/local/mongoDB/bin/mongod-port 27020-dbpath=/www/mongoDB/shard/s0-logpath=/www/mongoDB/shard/log/s0.log-logappend-fork

....

[root@100 /] # / usr/local/mongoDB/bin/mongod-port 27023-dbpath=/www/mongoDB/shard/s3-logpath=/www/mongoDB/shard/log/s3.log-logappend-fork

Step 2: start Config Server

[root@100 /] # mkdir-p / www/mongoDB/shard/config

[root@100 /] # / usr/local/mongoDB/bin/mongod-port 27100-dbpath=/www/mongoDB/shard/config-logpath=/www/mongoDB/shard/log/config.log-logappend-fork

Note: here we can start just like starting a normal mongodb service without adding the-shardsvr and configsvr parameters. Because the function of these two parameters is to change the startup port, we can specify the port ourselves.

Step 3: start Route Process

/ usr/local/mongoDB/bin/mongos-port 40000-configdb localhost:27100-fork-logpath=/www/mongoDB/shard/log/route.log-chunkSize

Among the mongos startup parameters, chunkSize is used to specify the size of the chunk (in MB). The default size is 200MB.

Step 4: configure Sharding

Next, we use MongoDB Shell to log in to mongos and add the Shard node

[root@100 shard] # / usr/local/mongoDB/bin/mongo admin-- port 40000

MongoDB shell version: 2.0.7

Connecting to: 127.0.0.1:40000/admin

Mongos > db.runCommand ({addshard: "localhost:27020"})

{"shardAdded": "shard0000", "ok": 1}

.

Mongos > db.runCommand ({addshard: "localhost:27029"})

{"shardAdded": "shard0009", "ok": 1}

Mongos > db.runCommand ({enablesharding: "test"}) # set the database for multipart storage

{"ok": 1}

Mongos > db.runCommand ({shardcollection: "test.log", key: {id:1,time:1}})

{"collectionsharded": "test.log", "ok": 1}

Step 5: connect the database to interface 40000 without much change in the program code, just like connecting to an ordinary mongo database.

Add:

1. Create Sharding replication set rs0

# mkdir / data/log

# mkdir / data/db1

# nohup mongod-port 27020-dbpath=/data/db1-logpath=/data/log/rs0-1.log-logappend-fork-shardsvr-replSet=rs0 &

# mkdir / data/db2

# nohup mongod-port 27021-dbpath=/data/db2-logpath=/data/log/rs0-2.log-logappend-fork-shardsvr-replSet=rs0 &

1.1 replication set rs0 configuration

# mongo localhost:27020 > rs.initiate ({_ id: 'rs0', members: [{_ id: 0, host:' localhost:27020'}, {_ id: 1, host: 'localhost:27021'}]}) > rs.isMaster () # View master-slave relationship

two。 Create Sharding replication set rs1

# mkdir / data/db3

# nohup mongod-port 27030-dbpath=/data/db3-logpath=/data/log/rs1-1.log-logappend-fork-shardsvr-replSet=rs1 &

# mkdir / data/db4

# nohup mongod-port 27031-dbpath=/data/db4-logpath=/data/log/rs1-2.log-logappend-fork-shardsvr-replSet=rs1 &

2.1 replication set rs1 configuration

# mongo localhost:27030

> rs.initiate ({_ id: 'rs1', members: [{_ id: 0, host:' localhost:27030'}, {_ id: 1, host: 'localhost:27031'}]})

> rs.isMaster () # View master-slave relationship

3. Create Config replication set conf

# mkdir / data/conf1

# nohup mongod-port 27100-dbpath=/data/conf1-logpath=/data/log/conf-1.log-logappend-fork-configsvr-replSet=conf &

# mkdir / data/conf2

# nohup mongod-port 27101-dbpath=/data/conf2-logpath=/data/log/conf-2.log-logappend-fork-configsvr-replSet=conf &

3.1 replication set conf configuration

# mongo localhost:27100

> rs.initiate ({_ id: 'conf', members: [{_ id: 0, host:' localhost:27100'}, {_ id: 1, host: 'localhost:27101'}]})

> rs.isMaster () # View master-slave relationship

4. Create Route

# nohup mongos-port 40000-configdb conf/localhost:27100,localhost:27101-fork-logpath=/data/log/route.log-logappend &

4.1 set shards

# mongo localhost:40000

> use admin

> db.runCommand ({addshard: 'rs0/localhost:27020,localhost:27021'})

> db.runCommand ({addshard: 'rs1/localhost:27030,localhost:27031'})

> db.runCommand ({enablesharding: 'test'})

> db.runCommand ({shardcollection: 'test.user', key: {name: 1}})

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.