Shulou(Shulou.com)06/03 Report--

How to use Protobuf, I believe that many inexperienced people do not know what to do, so this article summarizes the causes of the problem and solutions, through this article I hope you can solve this problem.

In previous articles, we talked about when to use Yaml, when to use JSON, and when to use Protobuf:

Human writing machine reading, using Yaml

Machine writing, human reading, using JSON

Machine write, machine read, use JSON or Protobuf

As a serialization format supported by almost every language, JSON has been widely used in many places. However, there is a drawback. JSON is full of curly braces, square braces and double quotes, resulting in too many redundant characters and a very large amount of data. In scenarios with high requirements for transmission speed, the larger the amount of data, the greater the transmission bandwidth, and the less data is transmitted per unit time.

Protobuf is a binary serialization format developed by Google. Compared with JSON, Protobuf's data is so compact that it doesn't even have a field name for the data. For example, there is a piece of data like this:

A = {'name':' kingname', 'salary': 99999,' address': 'Shanghai', 'skill': [' Python', 'reptile', 'Golang']}

If expressed in Protobuf, the binary form of the data is as follows:

This binary data has only values, but no field names, so to parse the data, you must include an additional field name in the code. So you need to define a xxx.proto file and mark the information of each field in it. At any time in any language, where serialization and deserialization are needed, use the protoc command in advance to generate a xxx_pb2 file based on the .proto file and process the data by importing data objects from the xxx_pb2 file.

Therefore, we say that the proto format, although it does simplify the amount of data transferred in the network, adds a considerable amount of work to developers.

Recently, there is a new binary serialization format: CBOR, which has less data than JSON, but is much simpler to develop than Protobuf.

Let's take a look at serializing the above data using CBOR. First install CBOR in Python:

Python3-m pip install cbor2

After the installation is complete, let's serialize the data:

Import cbor2 a = {'name':' kingname', 'salary': 99999,' address': 'Shanghai', 'skill': [' Python', 'reptile', 'Golang']} result = cbor2.dumps (a) print (result)

The running effect is shown in the following figure:

Note that what is printed is binary data, not a string. As you can see, the data has its own field name, and there are special characters between the field name and the value. CBOR can automatically recognize these special symbols to distinguish between the field name and the field value.

After my test, a large JSON file of 150MB, read into memory, and then re-serialized through CBOR to write the file, the file size can be reduced to about 60MB. Although the compression ratio is not as good as Protobuf, readability is not as good as JSON;, but the compression ratio is higher than JSON, readability is better than Protobuf, and almost no extra workload.

When writing microservices or communication between front and back end of the website, you can consider giving CBOR-Concise Binary Object Representation a try | Overview [1].

After reading the above, have you mastered how to use Protobuf? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.