Shulou(Shulou.com)06/03 Report--

This article focuses on "what are the advantages of the 12306 architecture". Interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Now let the editor take you to learn "what are the advantages of the 12306 architecture"?

How to provide normal and stable service when 1 million people grab 10,000 train tickets at the same time.

Github code address:

Https://github.com/GuoZhaoran/spikeSystem large-scale high concurrency system architecture

Highly concurrent system architectures are deployed in distributed clusters, and there are layers of load balancing in the upper layer of the service, and various disaster recovery methods (double fire room, node fault tolerance, server disaster recovery, etc.) are provided to ensure the high availability of the system. Traffic will also be balanced to different servers according to different load capacity and configuration policies.

Here is a simple diagram:

Brief introduction of load balancing

The above figure shows that the user request to the server has experienced three layers of load balancing. The following is a brief introduction to these three kinds of load balancing.

① OSPF (Open shortest Link first) is an interior gateway protocol (Interior Gateway Protocol, IGP).

OSPF establishes a link-state database by advertising the status of network interfaces between routers, generates the shortest path tree, and OSPF automatically calculates the Cost value on the routing interface.

However, you can also specify the Cost value of the interface manually, which takes precedence over the automatically calculated value.

The Cost calculated by OSPF is also inversely proportional to the interface bandwidth. The higher the bandwidth, the smaller the Cost.

The path that reaches the same Cost value of the target can perform load balancing, and up to 6 links can perform load balancing at the same time.

② LVS (Linux Virtual Server)

It is a Cluster technology, which adopts IP load balancing technology and content-based request distribution technology.

The scheduler has a good throughput, it transfers requests to different servers evenly, and the scheduler automatically shields the failure of the server, thus forming a group of servers into a high-performance and highly available virtual server.

③ Nginx

As we all know, it is a very high-performance HTTP proxy / reverse proxy server, and it is often used for load balancing in service development.

There are three main ways for Nginx to achieve load balancing:

Polling

Weighted polling

IP Hash polling

Let's do a special configuration and test for weighted polling of Nginx.

Demonstration of Nginx weighted polling

Nginx implements load balancing through the Upstream module, in which the configuration of weighted polling can add a weight to the relevant services, and the corresponding load may be set according to the performance and load capacity of the server.

The following is a configuration of weighted polling load. I will listen on ports 3001-3004 locally, configuring the weights of 1meme, 2meme, 3jin4, respectively:

# configure load balancer upstream load_rule {server 127.0.0.1 weight=1; server 3001 weight=1; server 127.0.1 weight=2; server 127.0.1 weight=1; server 127.0.1 weight=3; server 127.0.0.1 weight=1; server 3004 weight=4;}. Server {listen 80; server_name load_balance.com www.load_balance.com; location / {proxy_pass http://load_rule;}}

I configured the virtual domain name address of www.load_balance.com in the local / etc/hosts directory.

Next, use Go language to open four HTTP port listening services. The following is the Go program listening on port 3001. The others only need to modify the port:

Package mainimport ("net/http"os"strings") func main () {http.HandleFunc ("/ buy/ticket", handleReq) http.ListenAndServe (": 3001", nil)} / / processes the request function and writes the response result information to the log func handleReq (w http.ResponseWriter, r * http.Request) {failedMsg: = "handle in port:" writeLog (failedMsg) according to the request. ". / stat.log")} / / write func writeLog (msg string, logPath string) {fd, _: = os.OpenFile (logPath, os.O_RDWR | os.O_CREATE | os.O_APPEND, 0644) defer fd.Close () content: = strings.Join ([] string {msg, "rn"}, "3001") buf: = [] byte (content) fd.Write (buf)}

I wrote the requested port log information into the. / stat.log file, and then used the AB stress test tool to do the pressure test:

Ab-n 1000-c 100 http://www.load_balance.com/buy/ticket

According to the results in the log, ports 3001-3004 get 100,200,300,400 requests respectively.

This is in good agreement with the weight ratio I configured in Nginx, and the traffic after load is very uniform and random.

Type selection of second kill rush purchase system

Let's go back to the question we first mentioned: how does the train ticket second kill system provide normal and stable services under high concurrency?

From the above introduction, we know that user second kill traffic is evenly distributed to different servers through layers of load balancing. even so, the QPS borne by the stand-alone in the cluster is also very high.

How to optimize the performance of a single machine to the extreme?

To solve this problem, we need to figure out one thing: usually the booking system has to deal with the three basic stages of order generation, inventory deduction and user payment.

What our system needs to do is to ensure that train ticket orders are not oversold, many sold, each ticket sold must be paid to be effective, but also to ensure that the system bears a very high concurrency.

How should the sequence of these three stages be allocated more reasonably? Let's analyze it:

Place an order to reduce inventory

If you wait for the user to pay the order and reduce the inventory, the first feeling is that you will not sell less.

But this is a big taboo of concurrency architecture, because in extreme concurrency cases, users may create a lot of orders.

When inventory is reduced to zero, many users find that the grabbed orders can no longer be paid, which is the so-called "oversold". Nor can you avoid concurrent operations on the database disk IO.

Withholding inventory

In order to ensure the atomicity of withholding inventory and generating orders, it is necessary to use transaction processing, then take inventory judgment, reduce inventory, and finally submit transactions, the whole process has a lot of IO, and the operation of the database is blocked.

This method is not suitable for high concurrency second kill system at all. Next, we optimize the scheme of deducting inventory on a single machine: local deduction of inventory.

We allocate a certain amount of inventory to the local machine, reduce the inventory directly in memory, and then create the order asynchronously according to the previous logic.

The improved stand-alone system looks like this:

Problems come one after another. in the case of high concurrency, we cannot guarantee the high availability of the system, if two or three machines on these 100 servers can't handle concurrent traffic or other reasons.

Then the orders on these servers cannot be sold, which results in less sales of orders.

To solve this problem, we need a unified management of the total order quantity, which is the next fault-tolerant scheme.

The server should not only reduce inventory locally, but also reduce inventory remotely.

With the remote unified inventory reduction operation, we can allocate some extra "Buffer inventory" to each machine according to the machine load to prevent machine downtime.

Let's analyze it in detail in conjunction with the following architecture diagram:

We use Redis to store unified inventory, because the performance of Redis is very high, which claims that stand-alone QPS can resist 10W concurrency.

After local inventory reduction, if there is an order locally, we will request Redis to reduce inventory remotely. After both local and remote inventory reduction are successful, we will return to the user a prompt of successful ticket grabbing, which can also effectively ensure that the order will not be oversold.

When there is a machine downtime, because there are reserved Buffer tickets on each machine, the remaining tickets on the downtime machine can still be made up on other machines and a lot of sales are guaranteed.

In theory, the more Buffer is set, the more machines the system tolerates downtime, but too large a Buffer setting will also have a certain impact on Redis.

Although the anti-concurrency ability of Redis in-memory database is very high, the request will still go through the network IO, in fact, the number of requests for Redis in the process of ticket grabbing is the total amount of local inventory and Buffer inventory.

Because when the local inventory is insufficient, the system will directly return the user's "sold out" message, and will no longer follow the logic of unified deduction of inventory.

To some extent, this also avoids the huge amount of network requests that overwhelm the Redis, so the setting of the Buffer value requires the architect to seriously consider the load capacity of the system.

Code demonstration:

Go language is originally designed for concurrency. I use Go language to show you the specific process of grabbing tickets on a stand-alone machine.

Initialization work

The Init function in the Go package is executed before the Main function, and some preparatory work is mainly done at this stage.

The preparatory work that our system needs to do is to initialize the local inventory, initialize the Hash key value of the remote Redis storage unified inventory, and initialize the Redis connection pool.

You also need to initialize an Int type Chan of size 1 in order to implement the function of distributed locks.

You can also use read-write locks directly or other ways such as Redis to avoid resource competition, but using Channel is more efficient, which is the philosophy of the Go language: don't communicate through shared memory, share memory through communication.

The Redis library uses Redigo, and the following is the code implementation:

/ / localSpike package structure definition package localSpiketype LocalSpike struct {LocalInStock int64 LocalSalesVolume int64}... / / remoteSpike definition of hash structure and redis connection pool package remoteSpike// remote order storage health value type RemoteSpikeKeys struct {SpikeOrderHashKey string / / redis in the second kill order hash structure key TotalInventoryKey string / / hash structure total order inventory key QuantityOfOrderKey string / / hash structure existing order quantity key} / / initialize redis Connection pool func NewPool () * redis.Pool {return & redis.Pool {MaxIdle: 10000 MaxActive: 12000, / / max number of connections Dial: func () (redis.Conn, error) {c, err: = redis.Dial ("tcp", ": 6379") if err! = nil {panic (err.Error ())} return c, err} }}. Func init () {localSpike = localSpike2.LocalSpike {LocalInStock: 150, LocalSalesVolume: 0,} remoteSpike = remoteSpike2.RemoteSpikeKeys {SpikeOrderHashKey: "ticket_hash_key", TotalInventoryKey: "ticket_total_nums", QuantityOfOrderKey: "ticket_sold_nums",} redisPool = remoteSpike2.NewPool () done = make (chan int 1) done = ticket_sold_nums) then return redis.call ('HINCRBY', ticket_key, ticket_sold_key, 1) end return 0` / / remote unified buckle inventory func (RemoteSpikeKeys * RemoteSpikeKeys) RemoteDeductionStock (conn redis.Conn) bool {lua: = redis.NewScript (1, LuaScript) result, err: = redis.Int (lua.Do (conn, RemoteSpikeKeys.SpikeOrderHashKey, RemoteSpikeKeys.TotalInventoryKey) RemoteSpikeKeys.QuantityOfOrderKey) if err! = nil {return false} return result! = 0}

We use the Hash structure to store the information of total inventory and total sales. When the user requests, we determine whether the total sales is greater than the inventory, and then return the relevant Bool value.

Before starting the service, we need to initialize the initial inventory information for Redis:

Hmset ticket_hash_key "ticket_total_nums" 10000 "ticket_sold_nums" 0

Respond to user information

We start a HTTP service and listen on a port:

Package main...func main () {http.HandleFunc ("/ buy/ticket", handleReq) http.ListenAndServe (": 3005", nil)}

We have finished all the initialization work above, and then the logic of handleReq is very clear. We can judge whether the ticket grabbing is successful or not and return the information to the user.

Package main// processes the request function and writes the response result information to the log func handleReq (w http.ResponseWriter, r * http.Request) {redisConn: = redisPool.Get () LogMsg: = "" according to the request.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.