Shulou(Shulou.com)06/01 Report--

MongoDB fragmentation (Sharding) technology

Sharding is the method used by MongoDB to split large collections into different servers (or a cluster). Although sharding originated from relational database partitioning, MongoDB sharding is quite another matter.

Compared with the MySQL partitioning scheme, the biggest difference of MongoDB is that it can do almost everything automatically, as long as it tells MongoDB to allocate data, it can automatically maintain the balance of data between different servers.

1 purpose of slicing

The database application with high data volume and throughput will cause great pressure on the performance of the stand-alone machine, the large amount of query will exhaust the CPU of the single machine, and the large amount of data will exert great pressure on the storage of the single machine, which will eventually exhaust the memory of the system and transfer the pressure to the disk IO.

In order to solve these problems, there are two basic methods: vertical expansion and horizontal expansion.

Vertical scaling: add more CPU and storage resources to expand capacity.

Scale horizontally: distribute datasets across multiple servers. Horizontal expansion is slicing.

2 slicing design idea

Sharding provides a method to cope with high throughput and large amount of data. The use of sharding reduces the number of requests to be processed per shard, so by scaling horizontally, the cluster can increase its own storage capacity and throughput. For example, when inserting a piece of data, the application only needs to access the shard in which the data is stored.

The use of sharding reduces the data stored in each shard.

For example, if the database 1tb has a dataset and has 4 shards, then each shard may hold only 256 GB of data. If there are 40 shards, then each shard may have only 25GB data.

Docker deployment

Architectural design:

Three hosts start the same process, mongos, config server, shard server

Rancher deployment

Each host is tagged as planned:

Mongos=true

Shard2=true

Shard3=true

Shard1=true

Config=true

In host network mode, ports must not be occupied.

New service:

Paste the following:

Composeversion: '2'services: shard2: image: mongo:4.0.1-xenial stdin_open: true network_mode: host volumes:-/ data/shard2:/data/db tty: true command:-/ usr/bin/mongod-shardsvr-replSet-shard2-port -' 27002'--bind_ip_all-dbpath -/ data/db-logpath-/ data/db/shard2.log-- oplogSize-'10' labels: io.rancher.scheduler.affinity:host_label: shard2=true io.rancher.container.pull_image: always io.rancher.scheduler.global:' true' mongos: image: mongo:4.0.1-xenial stdin_open: true network_mode: host volumes: -/ data/mongos:/data/db tty: true command:-mongos-configdb-myset/172.20.101.132:27000172.20.101.133:27000172.20.101.134:27000-port-'27017'-bind_ip_all labels: io.rancher.scheduler.affinity:host_label: mongos=true io.rancher.container.pull_image: always io. Rancher.scheduler.global: 'true' shard3: image: mongo:4.0.1-xenial stdin_open: true network_mode: host volumes:-/ data/shard3:/data/db tty: true command:-/ usr/bin/mongod-shardsvr-replSet-shard3-port -' 27003'-bind_ip_all Dbpath-/ data/db-logpath-/ data/db/shard3.log-oplogSize-'10' labels: io.rancher.scheduler.affinity:host_label: shard3=true io.rancher.container.pull_image: always io.rancher.scheduler.global:' true' shard1: image: mongo:4.0.1-xenial stdin_open: true network_mode: host volumes: -/ data/shard1:/data/db tty: true command:-/ usr/bin/mongod-shardsvr-replSet-shard1-port-- '27001'-bind_ip_all-dbpath-/ data/db-logpath-/ data/db/shard1.log-oplogSize -'10 'labels: io.rancher.scheduler.affinity:host_label: shard1=true io.rancher.container.pull_image: always io.rancher.scheduler.global:' true' config: image: mongo:4.0.1-xenial stdin_open: true network_mode: host volumes:-/ data/config:/data/db tty: true command:-/ usr/bin/mongod-configsvr- -- replSet-myset-bind_ip_all-- dbpath-/ data/db-port-'27000'-- logpath-/ data/db/config.log labels: io.rancher.scheduler.affinity:host_label: config=true io.rancher.container.pull_image: always io.rancher.scheduler.global: 'true'rancher-compose:rancher-composeversion: '2'services: mongos: start_on_create: true shard2: start_on_create: true shard3: start_on_create: true shard1: start_on_create: true config: start_on_create: true

1. Log in to a host and link to shard1:

127.0.0.1:27001/admin

Config = {_ id: "shard1", members: [{_ id:0,host: "172.20.101.132shard1"}, {_ id:1,host: "172.20.101.133config 27001"}, {_ id:2,host: "172.20.101.134shard1"}]}

Rs.initiate (config)

2. Link shard2

127.0.0.1:27002/admin

Config = {_ id: "shard2", members: [{_ id:0,host: "172.20.101.132config 27002"}, {_ id:1,host: "172.20.101.133purr 27002"}, {_ id:2,host: "172.20.101.134pur27002"}]}

Rs.initiate (config)

3. Link shard3

127.0.0.1:27003/admin

Config = {_ id: "shard3", members: [{_ id:0,host: "172.20.101.132purl 27003"}, {_ id:1,host: "172.20.101.133purr 27003"}, {_ id:2,host: "172.20.101.134shard3"}]}

Rs.initiate (config)

4. Link the config service:

Config = {_ id: 'myset', members: [{_ id: 0, host:' 172.20.101.132myset', members 27000'}, {_ id: 1, host: '172.20.101.133 config 27000'}, {_ id: 2, host:' 172.20.101.134Suzhou 27000'}]}

Rs.initiate (config)

5. Log in to mongos:

Mongo 127.0.0.1:27017/admin # Log in to mongos and add node db.runCommand ({addshard: "shard1/172.20.101.132:27001172.20.101.133:27001172.20.101.134:27001", name: "shard1"}); db.runCommand ({addshard: "shard2/172.20.101.132:27002172.20.101.133:27002172.20.101.134:27002", name: "shard2"}) Db.runCommand ({addshard: "shard3/172.20.101.132:27003172.20.101.133:27003172.20.101.134:27003", name: "shard3"})

6. Check the cluster status

Sh.status ()

Host installation

I. installation

1 、

Cat > / etc/yum.repos.d/mongodb-org-4.0.repo db.bike_bak.find () {"_ id": ObjectId ("59e8c27804390e04a063159d"), "lat": 39.9571954199, "bikeId": "pgdAVg", "current_time": "2017-10-19 23:19:19", "source": "ofo" "lng": 116.3926501736} # delete data from the collection > db.bike_bak.remove ({"bikeId": "pgdAVg"}) WriteResult ({"nRemoved": 1}) # check whether the collection contains data > db.bike_bak.find () > # Import data [root@iZ2ze4b308vd83fulq9n7iZ ~] # mongoimport-- port 27030-u sa-p Expressin@0618-d mapdb-c bike_bak-- type=json-- file bike.csv 2017 -10-25T11:59:51.020+0800 connected to: localhost:27030

2017-10-25T11:59:51.030+0800 imported 1 document # check whether the data is imported successfully > db.bike_bak.find () {"_ id": ObjectId ("59e8c27804390e04a063159d"), "bikeId": "pgdAVg", "current_time": "2017-10-19 23:19:19", "lat": 39.9571954199, "lng": 116.3926501736, "source": "ofo"}

[root@iZ2ze4b308vd83fulq9n7iZ ~] # mongoimport-- help Usage: mongoimport

Note:

When querying, use sort,skip,limit at the same time, regardless of location, first execute sequential sort, then skip, and then limit.

Example:

First of all, take a look at the total number of pieces of data in the database, and check the data information we want to export through the command.

Db.bike.find () .count () 16878865

Db.bike.find ({"source": "ofo"}) .limit (1) {"_ id": ObjectId ("59e8c27804390e04a063159d"), "lat": 39.9571954199, "bikeId": "pgdAVg", "current_time": "2017-10-19 23:19:19", "source": "ofo", "lng": 116.3926501736} >

How to export the data of "bikeId": "pgdAVg" through mongoexport? I have exported two types of data: json and csv.

# Export type is json, database: mapdb, collection: bike field: bikeId,lat,lng,current_time,source, condition is that source field is ofo first data mongoexport-- port 27030-u sa-p Expressin@0618-d mapdb-c bike-f bikeId,lat,lng,current_time,source-- type=json-o bike.csv-- query=' {"source": "ofo"}'--limit=1 # Export type is csv Database: mapdb, collection: bike field: bikeId,lat,lng,current_time,source, if the source field is ofo first data mongoexport-- port 27030-u sa-p Expressin@0618-d mapdb-c bike-f bikeId,lat,lng,current_time,source-- type=csv-o bike.csv-- query=' {"source": "ofo"}'--limit=1

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.