Shulou(Shulou.com)06/01 Report--

How to add SESSION synchronization function to DPVS, I believe many inexperienced people are at a loss about this. Therefore, this paper summarizes the causes and solutions of the problem. Through this article, I hope you can solve this problem.

Add the preface of SESSION synchronization function to DPVS

DPVS is an excellent open source software based on DPDK by iqiyi (https://github.com/iqiyi/dpvs)). Making use of the characteristic that DPDK works in user space, compared with LVS in kernel space, we can use a series of tools / middleware in user space to accomplish many functions that are difficult to accomplish in kernel space.

Just for fun

Although the author develops Java middleware in his daily work, he always has a great passion and curiosity about the underlying technology, especially at the network level. When I came across DPDK, a user-mode data plane development kit, I couldn't help itching after reading its official documentation and source code, so I tried to add a Session synchronization function to DPVS. Although it has little to do with work, the fun of technology lies in constantly messing around, Just For Fun! Of course, due to energy reasons, only to write a prototype and test successfully, there is still a long way to go from the production environment, after all, do not rely on this for a living ^ _ ^.

DPVS

DPVS is actually a load balancer software derived from LVS. What we often call Virtual IP (VIP) can be supported by DPVS, as shown in the following figure:

This time, the author adds the Session synchronization function to the master-slave mode in DPVS in FullNAT mode. As shown in the following figure:

What happens without SESSION synchronization?

Because the data forwarding of DPVS distributes packets through the internal session table, if there is no Session synchronization function, the corresponding database will be discarded because the corresponding Session can not be found. If the Client side is connected through tcp:

Then an error will be reported after the configured tcp retransmission timeout.

TCP Client RealServer

What happens if SESSION synchronizes

If the Session is synchronized, because the newly promoted DPVS2 Master can still know which RealServer to send the Packet to, if the TCP connection is used, the stability of the connection can still be guaranteed after a retransmission.

SESSION synchronization method

This time, the author tries to synchronize the Session of FullNat in master-slave mode. In fact, it only needs to synchronize the two Session tables (Session_IN and Session_OUT) under FullNat from Master to Slave.

How to synchronize LVS in kernel state if it works

Because software such as LVS works in kernel state, it is necessary to communicate between master and slave using more complex and difficult problems to debug, as shown in the following figure:

The debugging of kernel state is relatively complex than that of user mode, and there is no useful middleware, so the author has not tried to do this.

In user mode, the author uses Redis Pub/Sub synchronization.

In user mode, there are too many tools available, so I chose to use Redis's subscription / publish (Pub/Sub) feature to synchronize Session table information from Master to Slave, as shown below:

Because FullNat uses five tuples, the author's Key for Pub in Redis is:

Session_key_ (af protocol suite) _ (proto protocol) _ (client source address) _ (client port number) _ (vip address) _ (vip port number) _ (localIP) _ (localPort) _ (RealServer destination address) _ (RS destination port number) _ (current session CPUID) SESSION synchronous worker thread

First of all, the main function started by DPVS uses pthread to create two new threads for reids's Send (Pub) and Receive (Sub) in addition to DPVS's thread.

Communication between threads to publish information to Redis

The DPDK thread communicates with the Send/Recv thread while the ring_buffer communicates. So at the beginning of the creation, a rte_ring (session_rings) is created for each DPDK thread. Every time there is a new connection action, the DPDK thread encapsulates the new connection action into a message and throws it into it, which is then consumed by the SendPub thread. As shown in the following figure:

Because ring_buffer is limited, message loss may occur. The DPVS runtime stack for the new connection is:

_ _ dp_vs_in |-> conn_sched |-> tcp_conn_sched (tcp protocol) / * only TCP-SYN without other flag can be scheduled * / / * that is, only TCP-SYN packets can create a new connection * / |-> dp_vs_schedule | |-> dp_vs_snat_schedule (FullNAT mode) |

In the final dp_vs_snat_schedule code, add a piece of code:

Static struct dp_vs_conn * dp_vs_snat_schedule (.) {conn = dp_vs_conn_new (mbuf,iph,¶m,dest,0);. / / add logical session_info_enqueue (conn) to put conn information into session_buffer; return conn;}

Putting in the logic is actually assembling the information of conn into a sesion_msg structure, and then extracting the nine pieces of information from the previous session_key from conn:

Void session_info_enqueue (struct dp_vs_conn* conn) {. Int cid = rte_lcore_id (); struct session_msg* msg; if (rte_mempool_get (message_pool, (void**) & msg)

< 0){ ...... return; } copy_conn_to_msg(conn,msg); if(rte_ring_enqueue(session_rings[cid],msg) != 0){ ... rete_mempool_put(message_pool,msg); return; }}从Redis订阅消息 同样的，有一个Recv(Sub)线程从Redis订阅信息，然后Recv(Sub)线程和DPDK间的线程也用ring_buffer来同步，不过另用了一个session_subscribe_buffer。 如图中所示，从Redis订阅到信息之后，将消息重新塞到session_subscribe_buffer(每个线程都有)里面。然后利用DPVS的job回调方法在每个线程中处理subscribe消息并通过此消息重建session表: lcore_job_recv_fwd |->

Lcore_process_session_subscribe_ringvoid lcore_process_session_subscribe_ring (...) {struct rte_ring* ring = session_subscribe_ rings [cid];. Struct session_msg* msg; if (rte_ring_dequeue (ring, (void**) & msg)

< 0){ return; } new_dpvs_conn(msg); rte_mempool_put(message_pool,msg);} 笔者在new_dpvs_conn里面做了FullNAT的两张session表同步操作。 void dp_vs_conn_new_from_session(struct session_msg* msg){ ...... /*init inbound conn tuple hash*/ // SESSION IN 表项构建 t->

Af = msg- > af; t-> proto = msg- > proto;. / * init outbound conn tuple hash*/ SESSION OUT table item to build new- > af = msg- > af; new- > proto = msg- > proto;. / / bind dest err = dp_vs_conn_bind_dest (new,dest);. / / bind hash table dp_vs_conn_hash (new);} MQ consumption replay

Using Redis to do Pub/Sub is just a choice made by the author to keep the coding simple. If it is officially used in the production line, the author thinks that it is still necessary to send this kind of Session to Kafka this kind of queue, then the change of Session can be dropped locally. In this way, when both master and slave are down, the Session table can be rebuilt by consuming the messages already in the Kafka.

A small pit encountered.

When the author was testing, one of the problems encountered was that after Session synchronization, although the Session entries were synchronized correctly, the tcp connection was always disconnected, after adding a variety of Print judgments and TCP dump. Only to find that DPVS itself will rewrite the sequence of the TCP to add the toa field, so that the TCP sequence does not match, and the connection is disconnected. For the sake of simplicity, the author commented out this code, and finally succeeded!

Static int tcp_fnat_in_handle (...) {struct tcphdr * th;. / / tcp_in_add_toa (conn,mbuf,th); / / tcp_in_adjust_seq (conn,th); th- > source = conn- > lport; th- > dest = conn- > dport; return tcp_send_csum (af,iphdrlen,th,conn,mbuf);} deficiencies

At present, the author has only done the synchronization of Session new actions, and other actions such as Session deletion still need to be considered slowly. In addition, due to the limitation of time and energy, the coding of DPVS is only equivalent to a simple prototype verification, which is far from meeting the requirements of high availability of the production line.

However, when the test is successful and the other Slave is connected immediately after the Master goes down, the long connection (the MySQL Client is used for testing) remains continuous, and the query data is still silky, as if nothing had happened (if there is no this function, you can only wait for the card owner for about 25s to time out, tcp_retries2=5), you will feel a great sense of achievement!

After reading the above, have you mastered how to add SESSION synchronization to DPVS? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.