Shulou(Shulou.com)06/03 Report--

Configure mongodb sharding cluster

Sharding cluster introduction

This is a mode that can be scaled horizontally, especially when there is a large amount of data, and this architecture is generally used to build a monodb system when the amount of data is large.

To build a MongoDB Sharding Cluster, you need three roles:

Shard Server: mongod instance, which is used to store actual data blocks. In the actual production environment, a shard server role can be assumed by several machines grouped into a relica set to prevent a single point of failure of the host.

Config Server: mongod instance, which stores the entire Cluster Metadata, including chunk information.

Route Server: mongos instance, front-end routing, client access, and make the whole cluster look like a single database, front-end applications can be used transparently.

Sharding architecture diagram:

Actual environment architecture

This example architecture example diagram:

1. The machine runs an instance of mongod (called mongod shard11,mongod shard12,mongod shard13) to organize replica set1 as the shard1 of cluster.

two。 The machine runs an instance of mongod (called mongod shard21,mongod shard22,mongod shard23) to organize replica set2 as the shard2 of cluster.

3. The machine runs an instance of mongod (called mongod shard31,mongod shard32,mongod shard33) to organize replica set3 as the shard3 of cluster.

4. The machine runs an instance of mongod (called mongod shard41,mongod shard42,mongod shard43) to organize replica set4 as the shard4 of cluster.

5.1.107, 1.108, 2.89 machines run one mongod instance as three config server

6. Each machine runs a mongs process for client connections

Host IP Port Information

Server1 10.10.1.107 mongod shard11:27017

Mongod shard21:27018

Mongod config1:20000

Mongs1:30000

Server2 10.10.1.108 mongod shard12:27017

Mongod shard22:27018

Mongod config2:20000

Mongs2:30000

Server3 10.10.1.109 mongod shard13:27017

Mongod shard23:27018

Mongs3:30000

Server4 10.10.2.89 mongod shard31:27017

Mongod shard41:27018

Mongod config3:20000

Mongs4:30000

Server5 10.10.2.90 mongod shard32:27017

Mongod shard42:27018

Mongs5:30000

Server6 10.10.2.91 mongod shard33:27017

Mongod shard43:27018

Mongs6:30000

Add hosts files for each node

# shard1

10.10.1.107 MongoDB_shard11.itings.com

10.10.1.108 MongoDB_shard12.itings.com

10.10.1.109 MongoDB_shard13.itings.com

# shard210.10.1.107 MongoDB_shard21.itings.com 10.10.1.108 MongoDB_shard22.itings.com 10.10.1.109 MongoDB_shard23.itings.com # shard310.10.2.89 MongoDB_shard31.itings.com 10.10.2.90 MongoDB_shard32.itings.com 10.10.2.91 MongoDB_shard33.itings.com # shard410.10.2.89 MongoDB_shard41.itings.com 10.10.2.90 MongoDB_shard42.itings.com 10.10.2 . 91 MongoDB_shard43.itings.com # config1 10.10.1.107 MongoDB_config1.itings.com 10.10.1.108 MongoDB_config2.itings.com 10.10.2.89 MongoDB_config3.itings.com # mongos10.10.1.107 MongoDB_mongos1.itings.com 10.10.1.108 MongoDB_mongos2.itings.com 10.10.1.109 MongoDB_mongos3.itings.com 10.10.2.89 MongoDB_mongos4.itings.com 10.10.2.90 MongoDB_mongos5 .itings.com 10.10.2.91 MongoDB_mongos6.itings.com

Software preparation

1. Install monodb softwar

Wget http://fastdl.mongodb.org/linux/mongodb-linux-x86_64-2.4.9.tgz

Tar xvzf mongodb-linux-x86_64-2.4.9.tgz

Mv mongodb-linux-x86_64-2.4.9 / usr/local/mongodb

After installation, the directory structure is as follows:

Tree mongodb-linux-x86_64-2.4.9

Mongodb-linux-x86_64-1.6.2

|-- GNU-AGPL-3.0

|-- README

|-- THIRD-PARTY-NOTICES

-- bin |-- bsondump |-- mongo |-- mongod |-- mongodump |-- mongoexport |-- mongofiles |-- mongoimport |-- mongorestore |-- mongos |-- mongosniff-- mongostat

1 directory, 14 files

two。 Create a data directory

Create a directory of shard data files on each sever, as shown in this example sharding architecture diagram

Server1:

Mkdir-p / data/mongodb/data/shard11

Mkdir-p / data/mongodb/data/shard21

Server2:

Mkdir-p / data/mongodb/data/shard12

Mkdir-p / data/mongodb/data/shard22

Server3:

Mkdir-p / data/mongodb/data/shard13

Mkdir-p / data/mongodb/data/shard23

Server4:

Mkdir-p / data/mongodb/data/shard31

Mkdir-p / data/mongodb/data/shard41

Server5:

Mkdir-p / data/mongodb/data/shard32

Mkdir-p / data/mongodb/data/shard42

Server6:

Mkdir-p / data/mongodb/data/shard33

Mkdir-p / data/mongodb/data/shard43

Configure relica sets

1. Configure replica sets:

Server1:

/ usr/local/mongodb/bin/mongod-shardsvr-replSet shard1-port 27017-dbpath / data/mongodb/data/shard11-oplogSize 8192-logpath / data/mongodb/data/shard11.log-logappend-fork

/ usr/local/mongodb/bin/mongod-shardsvr-replSet shard2-port 27018-dbpath / data/mongodb/data/shard21-oplogSize 8192-logpath / data/mongodb/data/shard21.log-logappend-fork

Server2:

/ usr/local/mongodb/bin/mongod-shardsvr-replSet shard1-port 27017-dbpath / data/mongodb/data/shard12-oplogSize 8192-logpath / data/mongodb/data/shard12.log-logappend-fork

/ usr/local/mongodb/bin/mongod-shardsvr-replSet shard2-port 27018-dbpath / data/mongodb/data/shard22-oplogSize 8192-logpath / data/mongodb/data/shard22.log-logappend-fork

Server3:

/ usr/local/mongodb/bin/mongod-shardsvr-replSet shard1-port 27017-dbpath / data/mongodb/data/shard13-oplogSize 8192-logpath / data/mongodb/data/shard13.log-logappend-fork

/ usr/local/mongodb/bin/mongod-shardsvr-replSet shard2-port 27018-dbpath / data/mongodb/data/shard23-oplogSize 8192-logpath / data/mongodb/data/shard23.log-logappend-fork

Server4:

/ usr/local/mongodb/bin/mongod-shardsvr-replSet shard3-port 27017-dbpath / data/mongodb/data/shard31-oplogSize 8192-logpath / data/mongodb/data/shard31.log-logappend-fork

/ usr/local/mongodb/bin/mongod-shardsvr-replSet shard4-port 27018-dbpath / data/mongodb/data/shard41-oplogSize 8192-logpath / data/mongodb/data/shard41.log-logappend-fork

Server5:

/ usr/local/mongodb/bin/mongod-shardsvr-replSet shard3-port 27017-dbpath / data/mongodb/data/shard32-oplogSize 8192-logpath / data/mongodb/data/shard32.log-logappend-fork

/ usr/local/mongodb/bin/mongod-shardsvr-replSet shard4-port 27018-dbpath / data/mongodb/data/shard42-oplogSize 8192-logpath / data/mongodb/data/shard42.log-logappend-fork

Server6:

/ usr/local/mongodb/bin/mongod-shardsvr-replSet shard3-port 27017-dbpath / data/mongodb/data/shard33-oplogSize 8192-logpath / data/mongodb/data/shard33.log-logappend-fork

/ usr/local/mongodb/bin/mongod-shardsvr-replSet shard4-port 27018-dbpath / data/mongodb/data/shard43-oplogSize 8192-logpath / data/mongodb/data/shard43.log-logappend-fork

Initialize replica set

Connect one of the mongod with mongo, execute:

/ usr/local/mongodb/bin/mongo-host 10.10.1.107-port 27017

Config = {_ id: 'shard1', members: [

{_ id: 0, host: 'MongoDB_shard11.itings.com:27017'}

{_ id: 1, host: 'MongoDB_shard12.itings.com:27017'}

{_ id: 2, host: 'MongoDB_shard13.itings.com:27017'}]

}

Rs.initiate (config)

/ usr/local/mongodb/bin/mongo-host 10.10.1.107-port 27018

Config = {_ id: 'shard2', members: [

{_ id: 0, host: 'MongoDB_shard21.itings.com:27018'}

{_ id: 1, host: 'MongoDB_shard22.itings.com:27018'}

{_ id: 2, host: 'MongoDB_shard23.itings.com:27018'}]

}

Rs.initiate (config)

Initialize node

/ usr/local/mongodb/bin/mongo-host 10.10.2.89-port 27017

Config = {_ id: 'shard3', members: [

{_ id: 0, host: 'MongoDB_shard31.itings.com:27017'}

{_ id: 1, host: 'MongoDB_shard32.itings.com:27017'}

{_ id: 2, host: 'MongoDB_shard33.itings.com:27017'}]

}

Rs.initiate (config)

/ usr/local/mongodb/bin/mongo-host 10.10.2.89-port 27018

Initialize node

Config = {_ id: 'shard4', members: [

{_ id: 0, host: 'MongoDB_shard41.itings.com:27018'}

{_ id: 1, host: 'MongoDB_shard42.itings.com:27018'}

{_ id: 2, host: 'MongoDB_shard43.itings.com:27018'}]

}

Rs.initiate (config)

Configure three config server

Server1 machine

Mkdir-p / data/mongodb/data/config

Start configserver1

/ usr/local/mongodb/bin/mongod-configsvr-dbpath / data/mongodb/data/config-port 20000-logpath / data/mongodb/data/config.log-logappend-fork

Server2 machine

Mkdir-p / data/mongodb/data/config

Start configserver2

/ usr/local/mongodb/bin/mongod-configsvr-dbpath / data/mongodb/data/config-port 20000-logpath / data/mongodb/data/config.log-logappend-fork

Server4 machine

Mkdir-p / data/mongodb/data/config

Start configserver3

/ usr/local/mongodb/bin/mongod-configsvr-dbpath / data/mongodb/data/config-port 20000-logpath / data/mongodb/data/config.log-logappend-fork

Configure mongs

All machines start mongos

/ usr/local/mongodb/bin/mongos-configdb MongoDB_config1.itings.com:20000,MongoDB_config2.itings.com:20000,MongoDB_config3.itings.com:20000-port 30000-chunkSize 5-logpath / data/mongodb/data/mongos.log-logappend-fork

# dbpath is not required for mongs

Configuring the Shard Cluster

Connect to one of the mongos processes and switch to the admin database for the following configuration

1. Connect to mongs and switch to admin

/ usr/local/mongodb/bin/mongo MongoDB_mongos1.itings.com:30000/admin

two。 Join shards

Db.runCommand ({addshard: "shard1/MongoDB_shard11.itings.com:27017,MongoDB_shard12.itings.com:27017,MongoDB_shard13.itings.com:27017", name: "S1"})

Db.runCommand ({addshard: "shard2/MongoDB_shard21.itings.com:27018,MongoDB_shard22.itings.com:27018,MongoDB_shard23.itings.com:27018", name: "S2"})

Db.runCommand ({addshard: "shard3/MongoDB_shard31.itings.com:27017,MongoDB_shard32.itings.com:27017,MongoDB_shard33.itings.com:27017", name: "S3"})

Db.runCommand ({addshard: "shard4/MongoDB_shard41.itings.com:27018,MongoDB_shard42.itings.com:27018,MongoDB_shard43.itings.com:27018", name: "S4"})

Get shard summary information:

Db.printShardingStatus ()

3. Optional parameter

Name: used to specify the name of each shard. If not, the system will automatically assign it.

MaxSize: specifies the maximum disk space that can be used by each shard, in megabytes

Example:

Db.runCommand ({addshard: "shard1/MongoDB_shard11.itings.com:27017,MongoDB_shard12.itings.com:27017,MongoDB_shard13.itings.com:27017", name: "S1", maxSize: 512000})

Get shard summary information:

Db.printShardingStatus ()

Db.user.getShardDistribution ()

Db.runCommand ({listshards: 1})

Check the status of each collection

Db.printCollectionStats ()

Activate shard exampl

Db.runCommand ({"enablesharding": "kaola_play"})

Db.runCommand ({shardcollection: "kaola_play.user", key: {_ id: 1}})

Appendix: introduction to mongodb operation mode

 stand-alone mode

 Replication mode, which can be divided into Master-Slave Replication and Replica Sets

 Sharding mode

1. Stand-alone mode

Here is an introduction to the current version of mongodb 1.6.2. The directory structure after mongodb installation is very simple as shown in the following figure, and occupies less than 100m of space.

A mongodb instance (also known as an instance) can be started by executing bin/mongod-dbpath, and similar to mysql, multiple databases can be created in a mongodb instance. Multiple mongodb instances can also be created on the same host

1.Replication mode

MongoDB provides asynchronous data replication between sever for failover and data redundancy. Replica sets function has been added since version 1.6, and only Master/Slave replication structure was available before version 1.6.

The structure of  Mster-Slave Relication is similar to that of mysql.

New features introduced in  Relica Sets version 1.6 that automatically fail over

About Mster-Slave Relication and Relica Sets, how to choose?

Which should I use?

 if using

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.