Shulou(Shulou.com)06/03 Report--

This article introduces the knowledge of "how to customize the protocol". In the operation of the actual case, many people will encounter such a dilemma. Then let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

Why customize the protocol?

The benefits of using standard protocols directly are obvious, and there are several advantages that I personally understand:

Since the standard protocol specification has become a standard, many systems can be directly docked and seamlessly integrated.

The most important point of the protocol is coding and decoding. Standard protocols often have ready-made codec packets, which can be used directly to reduce development time.

There are many third-party testing tools around standard protocols, which can be tested easily.

Since there are so many advantages, why do we want to customize the protocol, roughly out of the following considerations:

Since it is a standard protocol, it often takes into account more things, resulting in relatively large protocol data, which may not be tolerated in some systems that pursue performance and traffic.

There are many standard protocols, and no one is suitable for any scenario, so if there is no established standard protocol in a scenario, there will be a variety of private protocols.

As long as the two parties agree on the data structure, the custom protocol is not universal and theoretically more secure. Of course, many standard protocols now have secure versions, such as https,sftp and so on.

The above is just a personal understanding, welcome to add; on how to customize protocols, you can actually refer to some mainstream standard protocols or private protocols, in fact, there are a lot of common points to learn from; let's take a brief look at those mainstream protocols

Mainstream protocol

Let's take a look at how some mainstream standard protocols or private protocols define their own data structures, which is a very good reference for us.

Http protocol

Everyone is most familiar with the http protocol, which is called Hypertext transfer Protocol. The whole request message can be divided into three parts: the request line, the request header and the request body.

Request line

GET / test.html HTTP/1.1 (CRLF line wrap)

Request header

Accept-Encoding: gzip, deflateContent-Length: 38 Content-Encoding: gzip...

There are many request headers, each of which represents its own meaning, so it is not listed here one by one. We pay more attention to the structure of the whole message here.

Request text

This has a body only when a POST request is made, which stores business data, such as common json text strings. The length of the specific body can be determined according to the Content-Length in the header.

Dubbo protocol

The dubbo protocol format can be directly referenced to the following images provided on the official website:

Looking at the figure above, in fact, the whole protocol packet is roughly divided into two parts: the fixed part and the variable part, or the message header and message body.

The fixed part has a total of 4 "8" 4 "16 bytes, as shown below:

Header {Magic High = 8bit; / magic number high bit Magic Low = 8bit; / / magic number low bit Req/Res = 1bit; / / identify whether the request or response is 2 Way = 1bit; / / whether the tag expects to return a value of Event = 1bit from the server; / / identify whether it is an event message Serialization ID = 5bit; / / identify the serialization type Status = 8bit / / identify the status of the response Request ID = 64bit; / / identify the unique request Data Length = 32bit; / / the serialized content length}

The variable part determines the length according to the Data Length in the fixed part.

Redis protocol

The client and server of Redis use a network communication protocol called RESP (Redis Serialization Protocol) to exchange data, which is relatively simple. Here is the general form of this protocol:

*

CR LF$

CR LF

CR LF...$

CR LF

CR LF

The above outline introduces three representative protocols. Although each protocol has its own usage scenario, if we define the protocol ourselves, we still have something in common.

How to customize a protocol

Next, let's focus on what we need to pay attention to to customize the protocol. I have sorted out the following concerns according to my own understanding:

Complete packet

Agreement number

Header identification

Business data

Reserved field

The following are described in detail one by one:

Complete packet

We often talk about data packets, but TCP only has the concept of stream, but there is no concept of data packets. A very important point is how our program knows that all the business data has been received and can be processed as a complete data packet. If we do not deal with it, there will be the problems of half-packet and sticky packet. There are two mainstream processing methods:

Add a description of the packet length to the header, such as the dataLength field that appears in http and dubbo protocols

Use a special string as the end of the packet, so that when we accept the data, the predetermined special string indicates that the packet is complete.

Agreement number

Different protocols may have different names. I call it the agreement number here. According to this protocol number, the server knows what logic to execute; for example, the service name + version number in the / test.html,dubbo protocol in the http protocol request line, and what key is to be executed in redis.

Header identification

Whether this is necessary or not depends on the respective scenarios. For example, the redis protocol is simple enough and does not need any identification, and everything is agreed upon by both sides. But there are still some requirements for many other protocols. In addition to the above-mentioned dataLength can be specified in the header, there are many other things that can be specified, such as:

Business data format: text format, json format, html format, etc.

Compression format: data packets may be compressed in order to pursue the traffic packet size, such as gzip, deflater, snappy, etc.

Encryption algorithm: may need to encrypt my business data to ensure the security of business data AES, DES, etc.

Business data

Business data is often the largest and variable part of the whole data packet. What we have done above is actually serving business data, which needs to be transmitted over the network. The most important thing is serialization. Generally speaking, there are two ways:

Text method: serialize text document text, or json string, xml format, etc.

Binary mode: common ones such as protobuf,thrift,kyro, etc.

Reserved field

Whether you need to reserve fields depends on the situation. For example, the entire header of the http protocol is variable, and there is an identification for each line. Until a blank line is read, the text below indicates the end of the header. It can be understood that http uses two ways to ensure the complete packet, the header ends with special characters, and the body uses the way of specifying dataLength; in the header. In fact, its whole expansibility is very good.

The other is like dubbo. In fact, its head is equivalent to 16 bytes fixed. In this case, can you reserve a few bytes to prevent subsequent changes?

This is the end of "how to customize the Protocol". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.