In addition to Weibo, there is also WeChat
Please pay attention
WeChat public account
Shulou
2025-02-24 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > Database >
Share
Shulou(Shulou.com)06/01 Report--
Preface
Recently, due to work reasons, in learning to use mongodb database, mongodb is the most commonly used nodql database, has risen to the top six in the database ranking. This article describes how to build a highly available mongodb (shard + replica) cluster and share it for your reference and learning. Let's take a look at the detailed introduction.
Before building a cluster, you need to understand several concepts: routing, sharding, replica set, configuration server, and so on.
Related concepts
Let's take a look at a picture:
You can see from the figure that there are four components: mongos, config server, shard, and replica set.
Mongos, the entry of database cluster requests, all requests are coordinated through mongos. There is no need to add a route selector in the application. Mongos itself is a request distribution center, which is responsible for forwarding the corresponding data requests to the corresponding shard server. In a production environment, there is usually multiple mongos as the entry point for requests, and there is no way to prevent one of them from hanging up all mongodb requests.
Config server, as its name implies, is the configuration server that stores all database meta-information (routing, sharding) configuration. Mongos itself does not physically store sharding server and data routing information, but caches it in memory, while the configuration server actually stores the data. The first time mongos starts or shuts down and restarts, the configuration information will be loaded from config server, and then all mongos will be notified to update their status if the configuration server information changes, so that mongos can continue to route accurately. There are usually multiple config server configuration servers in a production environment because it stores fragmented routing metadata to prevent data loss!
Shard, sharding refers to the process of splitting a database and distributing it on different machines. By distributing the data to different machines, you don't need a powerful server to store more data and handle larger loads. The basic idea is to cut the set into small pieces, which are scattered into several pieces, each of which is only responsible for a part of the total data, and finally equalizes each slice (data migration) through an equalizer.
Replica set, a Chinese translation copy set, is actually a backup of shard to prevent data loss after shard is hung up. Replication provides redundant backup of data, and stores copies of data on multiple servers, which improves the availability of data and ensures the security of data.
The Arbiter, which is an instance of MongoDB in the replication set, does not hold data. The quorum node uses minimal resources and does not require hardware equipment, so Arbiter cannot be deployed in the same dataset node. It can be deployed in other application servers or monitoring servers, or in separate virtual machines. To ensure that there are an odd number of voting members (including primary) in the replication set, you need to add an arbitration node as a vote, otherwise the primary will not be automatically switched when primary cannot run.
After a brief understanding, we can sum up that the application requests mongos to operate the addition, deletion, modification and query of mongodb, configure the server to store database meta-information, and synchronize with mongos, and the data is finally stored on shard (shard). In order to prevent data loss and synchronization, a copy is stored in the replica set, and arbitration decides which node to store the data in the shard.
Environmental preparation
System system centos6.5 three servers: 192.168.0.75 Universe 84App 86 installation package: mongodb-linux-x86_64-3.4.6.tgz
Server planning
Server 75 Server 84 Server 86mongosmongosmongosconfig serverconfig serverconfig servershard server1 Primary Node shard server1 Secondary Node shard server1 Arbitration shard server2 Arbitration shard server2 Primary Node shard server2 Auxiliary Node shard server3 Auxiliary Node shard server3 Arbitration shard server3 Primary Node
Port assignment:
Mongos:20000config:21000shard1:27001shard2:27002shard3:27003
Cluster building
1. Install mongodb
# decompress tar-xzvf mongodb-linux-x86_64-3.4.6.tgz-C / usr/local/# and rename it mv mongodb-linux-x86_64-3.4.6 mongodb
Create six directories: conf, mongos, config, shard1, shard2 and shard3 on each machine, because mongos does not store data, you only need to establish a log file directory.
Mkdir-p / usr/local/mongodb/confmkdir-p / usr/local/mongodb/mongos/logmkdir-p / usr/local/mongodb/config/datamkdir-p / usr/local/mongodb/config/logmkdir-p / usr/local/mongodb/shard1/datamkdir-p / usr/local/mongodb/shard1/logmkdir-p / usr/local/mongodb/shard2/datamkdir-p / usr/local/mongodb/shard2/logmkdir-p / usr/local/mongodb/shard3/datamkdir-p / usr/local/mongodb/shard3/log
Configure environment variables
Vim / etc/profile# content export MONGODB_HOME=/usr/local/mongodbexport PATH=$MONGODB_HOME/bin:$PATH# makes it effective immediately source / etc/profile
2. Config server configuration server
In the future, mongodb3.4 requires the configuration server to also create a replica set, otherwise the cluster will not be built successfully.
Add Profil
Contents of vi / usr/local/mongodb/conf/config.conf## configuration file pidfilepath = / usr/local/mongodb/config/log/configsrv.piddbpath = / usr/local/mongodb/config/datalogpath = / usr/local/mongodb/config/log/congigsrv.loglogappend = true bind_ip = 0.0.0.0port = 21000fork = true# declare this is a config db of a cluster;configsvr = true# replica set name replSet=configs # set maximum number of connections maxConns=20000
Start the config server of three servers
Mongod-f / usr/local/mongodb/conf/config.conf
Log in to any configuration server and initialize the configuration copy set
# Connect mongo-- port 21000#config variable config = {... _ id: "configs",... Members: [. {_ id: 0, host: "192.168.0.75 id 21000"},... {_ id: 1, host: "192.168.0.84 id 21000"},... {_ id: 2, host: "192.168.0.86 id 21000"}. } # initialize replica set rs.initiate (config)
Where "_ id": "configs" should be consistent with the replicaction.replSetName configured in the configuration file, and "host" in "members" is the ip and port of three nodes.
3. Configure sharded copy set (three machines)
Set the first fragmented copy set
Configuration file
Contents of vi / usr/local/mongodb/conf/shard1.conf# configuration file #-pidfilepath = / usr/local/mongodb/shard1/log/shard1.piddbpath = / usr/local/mongodb/shard1/datalogpath = / usr/local/mongodb/shard1/log/shard1.loglogappend = truebind_ip = 0.0.0.0port = 27001fork = true # Open web Monitoring httpinterface=truerest=true # replica set name replSet=shard1 # declare this is a shard db of a cluster;shardsvr = true # set maximum number of connections maxConns=20000
Start the shard1 server of three servers
Mongod-f / usr/local/mongodb/conf/shard1.conf
Log in to any server and initialize the replica set
Mongo-port 2700 defines the replica set configuration using the admin database use admin#, and the "arbiterOnly": true of the third node represents it as the quorum node. Config = {... _ id: "shard1",... Members: [. {_ id: 0, host: "192.168.0.75 host 27001"},... {_ id: 1, host: "192.168.0.84 host 27001"},... {_ id: 2, host: "192.168.0.86 host 27001", arbiterOnly: true}.]. } # initialize replica set configuration rs.initiate (config)
Set the second fragmented copy set
Configuration file
Contents of vi / usr/local/mongodb/conf/shard2.conf# configuration file #-pidfilepath = / usr/local/mongodb/shard2/log/shard2.piddbpath = / usr/local/mongodb/shard2/datalogpath = / usr/local/mongodb/shard2/log/shard2.loglogappend = truebind_ip = 0.0.0.0port = 27002fork = true # Open web Monitoring httpinterface=truerest=true # replica set name replSet=shard2 # declare this is a shard db of a cluster;shardsvr = true # set maximum number of connections maxConns=20000
Start the shard2 server of three servers
Mongod-f / usr/local/mongodb/conf/shard2.conf
Log in to any server and initialize the replica set
Mongo-- port 2700 uses admin database use admin# to define replica set configuration config = {... _ id: "shard2",... Members: [. {_ id: 0, host: "192.168.0.75 host 27002", arbiterOnly: true},... {_ id: 1, host: "192.168.0.84 members 27002"},... {_ id: 2, host: "192.168.0.86 host 27002"}.]. } # initialize replica set configuration rs.initiate (config)
Set the third fragmented copy set
Configuration file
Contents of vi / usr/local/mongodb/conf/shard3.conf # configuration file #-pidfilepath = / usr/local/mongodb/shard3/log/shard3.piddbpath = / usr/local/mongodb/shard3/datalogpath = / usr/local/mongodb/shard3/log/shard3.loglogappend = truebind_ip = 0.0.0.0port = 27003fork = true # Open web Monitoring httpinterface=truerest=true # replica set name replSet=shard3 # declare this is a shard db of a cluster;shardsvr = true # set maximum number of connections maxConns=20000
Start the shard3 server of three servers
Mongod-f / usr/local/mongodb/conf/shard3.conf
Log in to any server and initialize the replica set
Mongo-- port 2700 uses admin database use admin# to define replica set configuration config = {... _ id: "shard3",... Members: [. {_ id: 0, host: "192.168.0.75 host 27003"},... {_ id: 1, host: "192.168.0.84 host 27003", arbiterOnly: true},... {_ id: 2, host: "192.168.0.86 host 27003"}.]. } # initialize replica set configuration rs.initiate (config)
4. Configure the routing server mongos
Start the configuration server and sharding server first, and then start the routing instance to start the routing instance: (three machines)
Vi / usr/local/mongodb/conf/mongos.conf# content pidfilepath = / usr/local/mongodb/mongos/log/mongos.pidlogpath = / usr/local/mongodb/mongos/log/mongos.loglogappend = truebind_ip = 0.0.0.0port = 20000fork = configuration server that true# listens to Only one or three configs can set the maximum number of connections maxConns=20000 for the replica set name configdb = configs/192.168.0.75:21000192.168.0.84:21000192.168.0.86:21000 # of the configuration server
Start the mongos server of three servers
Mongod-f / usr/local/mongodb/conf/mongos.conf
5. Enable sharding
Currently, mongodb configuration server, routing server and sharding server are built. However, sharding mechanism cannot be used when the application connects to mongos routing server. Sharding configuration needs to be set in the program to allow sharding to take effect.
Log in to any mongos
Mongo-port 2000 routing server and allocation replica set sh.addShard ("shard1/192.168.0.75:27001192.168.0.84:27001192.168.0.86:27001") sh.addShard ("shard2/192.168.0.75:27002192.168.0.84:27002192.168.0.86:27002") sh.addShard ("shard3/192.168.0.75:27003192.168.0.84:27003192") using the admin database user admin# in tandem .168.0.86: 27003 ") # View cluster status sh.status ()
6. Testing
At present, the configuration service, routing service, sharding service and replica set service have been connected in series, but our purpose is to insert data so that the data can be sliced automatically. Connect to the mongos and prepare to make the specified database and the specified collection shard effective.
# specify the effective db.runCommand of testdb sharding ({enablesharding: "testdb"}); # specify the collection and key db.runCommand in the database that need sharding ({shardcollection: "testdb.table1", key: {id: 1}})
We need to fragment the table1 table of testdb, which is automatically sliced to shard1 and shard2,shard3 according to id. This is set because not all mongodb databases and tables need sharding!
Test shard configuration results
Mongo 127.0.0.1 for 2000 insert test data for (var I = 1; I) using testdbuse testdb;#
Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.
Views: 0
*The comments in the above article only represent the author's personal views and do not represent the views and positions of this website. If you have more insights, please feel free to contribute and share.
Continue with the installation of the previous hadoop.First, install zookooper1. Decompress zookoope
"Every 5-10 years, there's a rare product, a really special, very unusual product that's the most un
© 2024 shulou.com SLNews company. All rights reserved.