Shulou(Shulou.com)06/01 Report--

This article mainly introduces how to build the high availability of OpenStack, the article is very detailed, has a certain reference value, interested friends must read it!

1. CAP theory

1) CAP theory gives three basic elements:

Consistency: any read operation can always read the result of a previously completed write operation

Availability (Availability): every operation can always be returned within a certain amount of time

Partition tolerance (Tolerance of network Partition): in the case of network partitions, consistency and availability can still be met

CAP theory points out that the three can not be satisfied at the same time. There are many objections to this theory, but its reference value is still huge.

This theory does not provide an excuse for the design that does not meet these three basic requirements, but only shows that the three cannot be absolutely satisfied in theory, and absolute consistency or availability is never required in engineering, but a balance and * must be sought.

For distributed data systems, partition tolerance is a basic requirement. Therefore, the design of distributed data system is often to find a balance between consistency and availability (reliability). More discussions about system performance and architecture also focus on consistency and availability.

2) Engineering practice of OpenStack, Swift and CAP

Compared with CAP theory, OpenStack's distributed object storage system Swift satisfies availability and partition tolerance, and does not guarantee consistency (optional), but only achieves the final consistency. Swift if the GET operation does not include the 'XmurNeast' header in the request header, then the read may not read the * object, and the object is not updated within the consistency window time, then the object read by the subsequent GET operation will be *, ensuring the final consistency; otherwise, the GET operation can always read the * obejct, which is consistent.

In OpenStack architecture, a lot of work needs to be done to ensure high availability. Therefore, usability in the OpenStack structure is discussed below:

2. High availability of OpenStack (OpenStack HA)

To figure out how to achieve high availability, you need to know which services are prone to unreliability. First, get a general idea of the structure of OpenStack.

OpenStack consists of five components (computing nova, identity management keystone, image management glance, front-end management dashboard and object storage swift).

Nova is the core component of computing and control, and it also includes services such as nova-compute, nova-scheduler, nova-volume, nova-network and nova-api. Borrow the following picture from http://ken.people.info to learn about the five components and functions of OpenStack:

The following figure describes the function and service structure of each component:

Like most other distributed systems, OpenStack is also divided into control nodes and computing nodes with two different functions. The control node provides services other than nova-compute. These components and services can be installed independently and can be combined optionally.

Nova-compute runs on each compute node, assuming that it can be trusted, or use a backup machine for failover (although the cost of configuring a backup per compute node seems too high compared to the benefits).

The high reliability of the control node is the main problem, and it has its own high reliability requirements and schemes for different components.

(1) since there is only one CotrolNode and is responsible for the management and control of the whole system, what if the CotrolNode cannot provide normal service? This is the common single node failure (SPoF,single point of failure) problem.

There is basically no way to achieve the goal of high availability through one, and more often it is the design to ensure that the malfunctioning machine is taken over as soon as possible in the event of a problem, at a higher cost.

For the single point problem, the solution is generally to use redundant equipment or hot standby, because of hardware errors or human reasons, it is always possible to cause the failure of single or multiple nodes, sometimes do node maintenance or upgrade, also need to temporarily stop some nodes, so a reliable system must be able to withstand the stop of single or multiple nodes.

The common deployment modes are: Active-passive active / standby mode, Active-active dual active mode, cluster mode.

(2) so how to build redundant control nodes? Or what other ways to achieve highly reliable control?

Many people may want to implement the active-passive pattern, use a heartbeat mechanism or similar method for backup, and achieve high reliability through failover. Openstack does not have multiple control nodes, and Pacemaker requires a variety of services to implement this backup, monitoring, and switching.

Careful analysis of the services provided by the control node, mainly nova-api, nova-network, nova-schedule, nova-volume, as well as glance, keysonte and database mysql, these services are provided separately. Nova-api, nova-network, glance, etc., can work on each computing node, RabbitMQ can work in active / standby mode, and mysql can use redundant high availability clusters.

The following are described separately:

1) High reliability of nova-api and nova-scheduler

Each compute node can run its own nova-api and nova-scheduler, providing load balancing to ensure that this works correctly.

In this way, when the control node fails, the nova-api and other services of the computing node are carried out as usual.

2) High reliability of nova-volume

There is no perfect HA (high availability) method for nova-volume at present, and a lot of work needs to be done.

However, nova-volume is driven by iSCSI, and this protocol, combined with DRBD, or a highly reliable hardware solution based on iSCSI, can achieve high reliability.

3) High reliability of network service nova-network

OpenStack's network already has a variety of highly reliable solutions, often you only need to use the-multi_host option to make the web service in high availability mode (high availability mode), see Existing High Availability Options for Networking for details.

Option 1: Multi-host

Multi-mainframe. Nova-network is configured on each compute node. In this way, each computing node will achieve the functions of NAT, DHCP and gateway, which inevitably requires a certain amount of overhead, which can be combined with hardware gateway to avoid the gateway function of each computing node. In this way, each computing node needs to install nova-network and nova-api in addition to nova-compute, and needs to be able to connect to the external network. For more information, please see Nova Multi-host Mode against SPoF.

Option 2: Failover

Failover. Can be transferred to hot backup in 4 seconds, see https://lists.launchpad.net/openstack/msg02099.html for details. The disadvantage is that a backup machine is required and there is a 4-second delay.

Option 3: Multi-nic

Multi-network card technology. By bridging the VM to multiple networks, the VM has two outgoing routes for switching in the event of a failure. But this requires monitoring multiple networks and designing handover strategies.

Option 4: Hardware gateway

Hardware gateway. An external gateway needs to be configured. Because VLAN mode requires one gateway for each network, while hardware gateway mode can only use one gateway for all instances, it cannot be used in VLAN mode.

Scenario 5: Quantum (in the next version of OpenStack Folsom)

The goal of Quantum is to gradually realize virtual network services with complete functions. For the time being, it will continue to be compatible with old nova-network functions such as Flat, Flatdhcp, etc. However, it implements a function similar to multi_host and supports OpenStack to work in the active / standby mode (active-backup, a high availability mode).

Quantum requires only one instance of nova-network to run, so it does not work with the multi_host schema.

Quantum allows a single tenant to have multiple private L2 networks, and by strengthening QoS, hadoop clusters should work well on nova nodes in the future.

For the installation and use of Quantum, this article Quantum Setup has an introduction.

4) High reliability of glance and keystone

OpenStack images can be stored using swift, and glance can be run on multiple hosts. Integrating OpenStack ImageService (Glance) with Swift introduces glance's use of swift storage.

Cluster management tool Pacemaker is a powerful high availability solution, which can manage multi-node clusters, achieve service switching and transfer, and can be used with Corosync and Heartbeat. Pacemaker can flexibly implement a variety of modes, such as active / standby, Numbl, NMUN and so on.

Bringing-high-availability-openstack-keystone-and-glance describes how to achieve the high reliability of keystone and glance through Pacemaker. After each node installs the OCF agent, it can tell one node whether the other node is running glance and keysone services properly, thus helping Pacemaker to start, stop, and monitor these services.

5) High reliability of Swift object storage

In general, the HA of OpenStack's distributed object storage system Swift does not need to be added on its own. Because Swift is designed to be distributed (without master nodes), fault tolerant, redundant, data recovery, scalable and highly reliable. Here are some of the advantages of Swift, which also illustrates this point.

Built-in Replication (N copies of accounts, container, objects)

3x + data redundancy compared to 2x on RAID

Built-in redundancy mechanism

RAID technology only makes two backups, while Swift has at least three backups

High Availability

High reliability

Easily add capacity unlike RAID resize

It is convenient to expand the storage capacity.

Elastic data scaling with ease

Convenient capacity expansion

No central database

No central node

Higher performance, No bottlenecks

High performance, no bottleneck limit

6) High reliability of message queuing service RabbitMQ

The failure of RabbitMQ results in the loss of messages, and there can be a variety of HA mechanisms:

The publisher confirms method informs you what has been written to disk in the event of a failure.

Multi-machine cluster mechanism, but node failure can easily lead to queue failure.

Active / standby mode (active-passive) can achieve failover in case of failure, but starting the backup machine may require delay or even failure.

In terms of disaster recovery and availability, RabbitMQ provides persistent queues. The ability to persist outstanding messages to disk when the queue service crashes. To avoid information loss due to the delay between sending and writing messages, RabbitMQ introduces a Publisher Confirm mechanism to ensure that messages are actually written to disk. Its support for Cluster provides both Active/Passive and Active/Active modes. For example, in Active/Passive mode, once a node fails, the Passive node is immediately activated and quickly replaces the failed Active node to assume the responsibility of message delivery. As shown in the figure:

Photo Active/Passive Cluster (pictures from the official website of RabbitMQ)

There are some problems in active-passive mode, so a dual active clustering mechanism (active-active) based on RabbitMQ cluster is introduced to solve these problems. Http://www.rabbitmq.com/ha.html this article details the highly reliable deployment and principles of RabbitMQ.

7) High reliability of database mysql

Clusters are not highly reliable. The common methods to build highly reliable mysql are Active-passive active / standby mode: using DRBD to achieve disaster capacity of master and standby, Heartbeat or Corosync for heartbeat monitoring, service switching and even failover,Pacemaker for service (resource) switching and control, or similar mechanisms. Among them, Pacemaker is mainly used to implement mysql's active-passive high availability cluster.

An important technology is DRBD: (distributed replication block device), that is, distributed replication block devices, which are often used instead of shared disks.

Its working principle is: when there is a write request for a specified disk device on host A, the data is sent to the kernel of host A, and then through a module in kernel, the same data is transmitted to a copy of the kernel of host B, and then host B is written to its own designated disk device, thus realizing the synchronization of the data of the two hosts, thus realizing the high availability of write operations. DRBD is generally one master and one slave, and all read and write operations can only be mounted on the master node server, but the master and slave DRBD servers can be exchanged. Here is an introduction to DRBD.

HAforNovaDB-OpenStack introduces the high reliability of OpenStack through Pacemaker by using only shared disks instead of DRBD.

NovaZooKeeperHeartbeat introduces the use of ZooKeeper for heartbeat detection.

MySQL HA with Pacemaker introduces the use of Pacemaker to provide highly reliable services, which is also a common solution.

Galera is a synchronous multi-master cluster open source project for Mysql/InnoDB, which provides many advantages (such as synchronous replication, reading and writing to any node, automatic membership control, automatic node joining, small latency, etc.).

For the working modes of Pacemaker, DRBD and Mysql, please refer to the following figure:

For other scenarios, according to MySQLPerformance Blog, the availability of several high-availability solutions for MySQL is as follows:

3. Build high availability OpenStack (High-availability OpenStack)

Generally speaking, high availability means the establishment of redundant backups. Common strategies are:

Cluster working mode. Multi-machine backup each other, so that each instance is backed up in multiple copies without a central node, such as distributed object storage system Swift and nova-network multi-host mode.

Autonomous mode. Sometimes, to solve the single point of failure (SPoF), we can simply use each node to work independently, by removing the master-slave relationship to reduce the problems caused by the failure of the master node, for example, nova-api is only responsible for its own node.

Active and standby mode. In the common mode of active-passive, passive nodes are in monitoring and backup mode and switch in time in case of failure, such as mysql high availability cluster, glance, keystone using Pacemaker and Heartbeat, and so on.

Dual master mode. This mode provides mutual backup and mutual assistance. RabbitMQ means the high availability of active-active clusters, and the nodes in the cluster can replicate queues. Architecturally speaking, so you don't have to worry about the passive node not being able to start or having too much delay?

The above is all the content of the article "how to build the High availability of OpenStack". Thank you for reading! Hope to share the content to help you, more related knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.