Shulou(Shulou.com)06/01 Report--

When I participated in the technology sharing event last year, a technology of Qiniu briefly introduced some highly available and scalable experiences. After listening to it, I benefited a lot. Sort it out, it is mainly divided into the following parts:

Entry level height available

Business tier high availability

Cache layer height available

Database is highly available

The entrance layer is scalable.

Business layer is scalable

Cache layer is scalable

Database scalability

Let's introduce the practice method in layers.

Entry level height available

Nigix two keeplive to keep the heartbeat alive.

Use heartbeat technology: keeplive provides this technology

For example, if machine An IP is 1.2.3.4 and machine B IP is 1.2.3.5, then apply for another IP (1.2.3.6) we call it heartbeat IP, which is usually bound to A. If An is down, then IP will be automatically bound to B.

Just bind the DNS layer to the heartbeat IP.

Two machines must be on the same network segment

Service listeners must listen to all IP. If only the heartbeat IP is monitored, the slave service (the machine that does not hold the heartbeat IP) will fail to start.

Server utilization is down (hybrid deployment can improve this)

Consider a question: is it highly available that two machines and two public network IP,DNS locate the domain name to two IP at the same time?

No, clients (such as browsers) will randomly choose an IP to access after parsing, rather than going to the other after a failure. So if a machine crashes, then about half of the users cannot access it.

Business tier high availability

There is no state in the business tier, which is distributed to the cache tier and the database tier. As long as there is no state, after the service of the business layer dies, the front nginx will automatically call the traffic to the rest of the service. Therefore, the business layer statelessness is a key point.

Friendly reminder: do not use cookie session directly because you want to make the service stateless, the pit inside is a little big, it is better to use it after you examine it clearly. For example, replay *.

Cache layer height available

The cache layer is divided into more details to ensure that the database can survive a single cache downtime.

The cache layer and business layer under small and medium-sized models can be mixed and deployed, which can save machines.

Large-scale websites, business layer and cache layer are deployed separately.

The cache layer is highly available, and the cache can enable both master and slave. When the master cache is alive, the master cache reads and both master and slave caches are written. After the master cache downtime, the master cache changes to master, and after master recovery, it becomes a new slave. This ensures data integrity and high availability

Database is highly available

MySQL has master-slave mode and master-master mode can meet your needs.

MongoDB also has the concept of ReplicaSet, which can basically meet everyone's needs.

High availability summary

The entrance layer is scalable.

How does the entry layer provide scalability? Just lay the machine? Then DNS and IP will be all right?

Yes, but there are also some points to pay attention to.

Although there is no problem in resolving a domain name to dozens of IP, browser clients will only use the first few IP, and some domain name vendors have optimized this (for example, the order of IP returned each time is random), but the optimization effect is unstable. The recommended practice is to use a small number of nginx machines as the input, and the business server hides the intranet (most of the HTTP-type business)

Business layer is scalable

Like dealing with high availability, statelessness is a good way to deal with it. Add machines and continue to deploy horizontally.

Cache layer is scalable

You can use codis or redis 3.0 directly.

If the database can withstand the low peak, then directly offline storage and then on the new cache is the most simple and effective way.

Cache Typ

Strongly consistent cache: unable to accept incorrect data read from the cache (such as user balance)

Weak consistency cache: can accept incorrect data over a period of time, such as the number of Weibo retweets

Immutable cache: the value corresponding to the cache key will not change

Weak consistency and immutable cache expansion is very convenient. You can use consistent HASH.

Consistent HASH

In a distributed cluster, adding and deleting machines, or automatically leaving the cluster after a machine failure is the most basic function of distributed cluster management. If the commonly used hash (object)% N algorithm is adopted, many of the original data cannot be found after machines are added or deleted, which seriously violates the principle of monotonicity.

Consistent HASH can effectively solve this problem. Consistent hashing algorithm not only keeps monotonicity, but also minimizes data migration. This algorithm is very suitable for distributed clusters, avoiding a large number of data migration and reducing the pressure on the server.

For instance

If the cache capacity is expanded from 9 to 10, 90% of the cache will expire immediately in the case of simple hash, while only 10% of the cache will expire in the case of consistent Hash.

Strong consistent caching problem

1. There is a slight difference in the configuration update time of the cache client, and it is possible to get expired data within this time window.

two。 If you abolish the node after expanding the node, you will get dirty data. For example, before a, the key is in machine 1, and after expansion, in machine 2, the data is updated, but after the nodes are eliminated, the key returns to machine 1, so there will be dirty data problems.

The second solution to the problem: either keep never reducing nodes, or the node adjustment interval is greater than the effective time of the data.

Problem one solution:

-both hash configurations are updated to the client, but still use the old configuration

-two clients will use cache only if the results of two sets of hash are the same, and the rest will be read from the database but written to the cache.

-use the new configuration on a client-by-client basis.

Database scalability

Horizontal split

Vertical split

Periodic scrolling

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.