Shulou(Shulou.com)06/01 Report--

What is Google Protocol Buffer?

Google Protocol Buffer (Protobuf for short) is a mixed language data standard within Google, which has more than 48162 message format definitions and more than 12183 .proto files. They are used in RPC systems and continuous data storage systems.

Protocol Buffers is a lightweight and efficient structured data storage format that can be used for structured data serialization, or serialization. It is very suitable for data storage or RPC data exchange format. Can be used in communication protocols, data storage and other fields of language-independent, platform-independent, extensible serialized structural data format. API is available in many languages, including C++, C#, GO, JAVA, PYTHON

If you don't know what Protobuf can do, it is recommended to combine google search keywords, take a look at the entry-level article, or take a look at the Developer Guide in the official documentation, or the development guide in Chinese. There are various language-related examples in the official documentation, so you can take a look at the actual usage with the code.

Many people say why not use json (or xml). The answer is simple: Protobuf is smaller, more concise, and serialization and deserialization are faster!

Google's latest open source gRpc framework defaults to using Protobuf as the data transfer format and service description file. GRpc will not be introduced in detail. If you are interested, you can take a look at the official website.

Back to the point, in the actual use of Protobuf, I found that Protobuf can not only write the content of the description message (Message), but also express other methods (similar to the methods in Rpc), mainly what you see in gRpc. At the same time, in the package of the Protobuf code generation tool, there is a directory like this, which is not clear about what it is for, as shown in the following figure:

There are a large number of defined proto files in the directory, but these files are actually Protobuf description files, similar to metadata. Describe yourself with your own syntax, and generate code such as metadata classes of the corresponding language through these files. For example, you can see the classes generated by the above description file in the C # version of Google.Protobuf, as shown in the following figure.

The most important of these description files is the descriptor.proto file, which is the description class of the entire proto syntax and describes the structure of the actual Protobuf syntax at all levels. Let's take a look at some code of this file. The above code describes the syntax definition of the proto file definition, such as the first two fields mean optional name, the optional package field, with multiple message_type (Message) in the middle. Definitions such as service (Rpc Service), enum_type (enumeration), etc., and then decomposed layer by layer. Basically, you can understand the full picture and extension points of Protobuf grammar.

Message FileDescriptorProto {optional string name = 1; / / file name, relative to root of source tree optional string package = 2; / / e.g. "foo", "foo.bar", etc. / / Names of files imported by this file. Repeated string dependency = 3; / / Indexes of the public imported files in the dependency list above. Repeated int32 public_dependency = 10; / / Indexes of the weak imported files in the dependency list. / / For Google-internal migration only. Do not use. Repeated int32 weak_dependency = 11; / / All top-level definitions in this file. Repeated DescriptorProto message_type = 4; repeated EnumDescriptorProto enum_type = 5; repeated ServiceDescriptorProto service = 6; repeated FieldDescriptorProto extension = 7; optional FileOptions options = 8; / / This field contains optional information about the original source code. / / You may safely remove this entire field without harming runtime / / functionality of the descriptors-the information is needed only by / / development tools. Optional SourceCodeInfo source_code_info = 9; / / The syntax of the proto file. / / The supported values are "proto2" and "proto3". Optional string syntax = 12;}

At the same time, there is also a plugin directory under the compiler directory, in which the plugin.proto file is very intriguing. Let's take a look at the contents of this file first.

Syntax = "proto3"; package google.protobuf.compiler;option java_package = "com.google.protobuf.compiler"; option java_outer_classname = "PluginProtos"; option csharp_namespace = "Google.Protobuf.Compiler"; option go_package = "plugin_go"; import "google/protobuf/descriptor.proto"; message CodeGeneratorRequest {repeated string file_to_generate = 1; string parameter = 2; repeated FileDescriptorProto proto_file = 15;} message CodeGeneratorResponse {string error = 1; message File {string name = 1 String insertion_point = 2; string content = 15;} repeated File file = 15;}

After removing the unnecessary comments, we can see that there are only two types defined in this file, one is the code generation request, the other is the code generation response, and in CodeGeneratorRequest there is the information of the FileDescriptorProto class that we saw in descriptor.proto before, you can think with the thigh that this should be the entry point for the code generation plug-in to obtain metadata, so what do you do?

From the code generation example of gRpc, we can see that Protobuf actually supports custom generated code plug-ins, as shown below:

% PROTOC%-I../../protos-- csharp_out Greeter. /.. / protos/helloworld.proto-- grpc_out Greeter-- plugin=protoc-gen-grpc=%PLUGIN%

In theory, we can implement our own plug-ins to generate any format we need, including all kinds of code, even documentation. But this information is very little, there are few related articles, and finally found an article about plugin http://www.expobrain.net/2015/09/13/create-a-plugin-for-google-protocol-buffer/, you are interested to see, but the focus of the article is this sentence:

The core part is the interface code to read a request from the stdin, traverse the AST and write the response on the stdout.

It turns out that the interface code of the plug-in actually reads the stream from standard input, and then outputs what you want to generate to standard output. I finally know how to use these.

Lift up your sleeves and start working and generate plugin.proto code from the protoc command line

Protoc-I../../protos-- csharp_out test.. /.. / protos/plugin.proto

Create a new console project, copy the code into the project, and add the test code to the Program.cs code

Using Google.Protobuf;using Google.Protobuf.Compiler;using System;namespace DotBPE.ProtobufPlugin {class Program {static void Main (string [] args) {Console.OutputEncoding = System.Text.Encoding.UTF8; var response = new CodeGeneratorResponse (); try {CodeGeneratorRequest request Using (var inStream = Console.OpenStandardInput ()) {request = CodeGeneratorRequest.Parser.ParseFrom (inStream);} ParseCode (request, response);} catch (Exception e) {response.Error + = e.ToString () } using (var output = Console.OpenStandardOutput ()) {response.WriteTo (output); output.Flush ();}} private static void ParseCode (CodeGeneratorRequest request, CodeGeneratorResponse response) {DotbpeGen.Generate (request,response);}

Haha started to compile, but the compilation failed! Oh, cheat on your father! The classes that have been generated by Google.Protobuf in the original C# version are all internal access rights and cannot be referenced from the outside. But Google.Protobuf is open source. And I can also generate the classes I need to use in the same project through the protoc command, or set it to public access. For convenience, I directly copy the Google.Protobuf source code into our project, this time compile again, the code runs perfectly, the next work is just to fill in the DotbpeGen.Generate code, this is just manual work.

As for what methods CodeGeneratorRequest and CodeGeneratorResponse have, in fact, you can see the proto file to know. The following are the generated code classes I use in the project for your reference

Using Google.Protobuf.Compiler;using Google.Protobuf.Reflection;using System;using System.Collections.Generic;using System.IO;using System.Linq;using System.Text;namespace DotBPE.ProtobufPlugin {public class DotbpeGen {public static void Generate (CodeGeneratorRequest request, CodeGeneratorResponse response) {foreach (var protofile in request.ProtoFile) {try {GenerateByProtoFile (protofile, response) } catch (Exception ex) {using (Stream stream = File.Create (". / error.txt")) {byte [] err = Encoding.UTF8.GetBytes (ex.Message+ex.StackTrace); stream.Write (err,0,err.Length) } response.Error + = ex.Message;} private static void GenerateSourceInfo (FileDescriptorProto protofile, CodeGeneratorResponse response) {bool genericDoc; protofile.Options.CustomOptions.TryGetBool (DotBPEOptions.GENERIC_MARKDOWN_DOC,out genericDoc) If (! genericDoc) {return;} StringBuilder sb = new StringBuilder (); foreach (var location in protofile.SourceCodeInfo.Location) {string path = String.Join (",", location.Path); string span = String.Join (",", location.Span) String leadingDetachedComments = String.Join ("\ r", location.LeadingDetachedComments); string trailingComments = String.Join ("\ r", location.TrailingComments) Sb.AppendLine ("{\" Path\ ",\" + path+ "\",\ "Span\",\ "+ span+"\ ",\" LeadingComments\ ",\"+ location.LeadingComments +"\ ",\" LeadingDetachedComments\ ",\"+ leadingDetachedComments +"\ ",\" TrailingComments\ ",\" + trailingComments + "\"} ") } var nfile = new CodeGeneratorResponse.Types.File {Name = GetFileName (protofile.Name) + "SI.txt", Content = sb.ToString ()}; response.File.Add (nfile) } private static void GenerateByProtoFile (FileDescriptorProto protofile, CodeGeneratorResponse response) {GenerateSourceInfo (protofile, response); GenerateServer (protofile, response); GenerateClient (protofile, response);} private static void GenerateServer (FileDescriptorProto protofile, CodeGeneratorResponse response) {bool genericServer; protofile.Options.CustomOptions.TryGetBool (DotBPEOptions.DISABLE_GENERIC_SERVICES_SERVER, out genericServer) If (genericServer) {return;} if (protofile.Service = = null | | protofile.Service.Count _ .Substring (0,1). ToUpper () + _ .Substring (1));} private static string GetTypeName (string typeFullName) {return ConvertCamelCase (typeFullName.Split ('.'). Last ());}

Then we write a proto file to test the following

/ / benchmark.protosyntax = "proto3"; package dotbpe;option csharp_namespace = "DotBPE.IntegrationTesting"; import public "dotbpe_option.proto"; option optimize_for = SPEED;//Benchmark Test Service service BenchmarkTest {option (service_id) = 50000; / / setting Service ID / / Test sending Echo message rpc Echo (BenchmarkMessage) returns (BenchmarkMessage) {option (message_id) = 1; / / setting message ID} / / comment at the end of Echo / / Test sends exit message rpc Quit (Void) returns (Void) {option (message_id) = 10000; / / setting message ID}; / / comment at the end of Quit} / / I am void message message Void {} / / I am the comment string field1 before the message BenchmarkMessage {/ / field of BenchmarkMessage message / / comment after the field / / the number of words in front of the field int32 field2 = 2; / / the comment after the field / * the special format in front of the field * the special format in front of the field * / int32 field3 = 3; string field4 = 4; repeated fixed64 field5 = 5; string field9 = 9; string field18 = 18; bool field80 = 80; bool field81 = 81; int32 field280 = 280 Int32 field6 = 6; int64 field22 = 22; bool field59 = 59; string field7 = 7; int32 field16 = 16; int32 field130 = 13; bool field12 = 12; bool field17 = 17; bool field13 = 13; bool field14 = 14; int32 field104 = 104; int32 field100 = 100; int32 field101 = 101; string field102 = 102; string field103 = 103; int32 field29 = 29; bool field30 = 30; int32 field60 = 60; int32 field271 = 271; int32 field272 = 272; int32 field150 = 150; int32 field23 = 23 Bool field24 = 24; int32 field25 = 25; bool field78 = 78; int32 field67 = 67; int32 field68 = 68; int32 field128 = 128; string field129 = 129; int32 field131 = 131;} / / dotbpe_option.proto// [START declaration] syntax = "proto3"; package dotbpe;// [END declaration] / / [START csharp_declaration] option csharp_namespace = "DotBPE.ProtoBuf"; / / [END csharp_declaration] import "google/protobuf/descriptor.proto" / / extended service extend google.protobuf.ServiceOptions {int32 service_id = 51001; bool disable_generic_service_client = 51003; / / whether to generate client code bool disable_generic_service_server = 51004; / / whether to generate server code} extend google.protobuf.MethodOptions {int32 message_id = 51002;} extend google.protobuf.FileOptions {bool disable_generic_services_client = 51003; / / whether to generate client code bool disable_generic_services_server = 51004 / / whether to generate server code bool generic_markdown_doc = 51005; / / whether to generate documentation}

The above dotbpe_option.proto our proto file has a custom extension to add some additional information we need. In fact, all the extensions are extensions to the messages in descriptor.proto.

Then we use the command to generate it. Here is a special convention. We must pay attention to when we set the

When the name of the protoc-gen-dotbpe=../../tool/ampplugin/dotbpe_amp.exe plug-in is protoc-gen-dotbpe, the output directory must be written as-- dotbpe_out, and the two names must match.

Set-excd $(dirname $0) /.. /.. / test/IntegrationTesting/PROTOC=protocPLUGIN=protoc-gen-dotbpe=../../tool/ampplugin/dotbpe_amp.exeIntegrationTesting_DIR=./DotBPE.IntegrationTesting/$PROTOC-I=./protos-- csharp_out=$IntegrationTesting_DIR-- dotbpe_out=$IntegrationTesting_DIR\. / protos/benchmark.proto-- plugin=$PLUGIN

Almost over, the relevant code can be found in https://github.com/xuanye/dotbpe/tree/develop/src/tool, this is I recently wrote a C# rpc framework, now completed the basic functions, need to be further improved, have the opportunity to introduce again.

Descriptor.proto information mining

We notice that the descriptor.proto file contains a message: SourceCodeInfo with the following fields in the message body

Optional string leading_comments = 3; optional string trailing_comments = 4; repeated string leading_detached_comments = 6

This is a very interesting definition, meaning that comments in the proto file can be obtained at run time. This can help us generate documentation or code comments, but the reading logic is complex, and there is a logic inside it that locates elements through Path and Span. Because in practice, you usually want to get comments on Service and Message, let's focus on how to get these two types of comments.

Here is an example of how we need to use Path in SourceCodeInfo.Location

* comments on [4, m]-Message * [4, m, 2, f]-comments on field (field) in Message * [6, s]-comments on Service * [6, s, 2, r]-comments on Rpc methods in Service

Where:

The index of the Message in the m-proto file (that is, the defined Message), starting at 0

The index of the Field field in f-Message (that is, the number of fields), starting with 0

The index of Service in the s-proto file, starting at 0

Index of the Rpc method in r-Service, starting at 0

Like this:

/ / [4,0] is the comment message MyMessage {/ / [4,0,2,0] here int32 field1 = 1; / / [4,0,2,0] is also here} / / [4,0] is the comment / / [6,0] here! service MyService {/ / [6,0,2,0] is here! Rpc (MyMessage) returns (MyMessage);}

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.