Shulou(Shulou.com)06/03 Report--

Look at hadoop PRC again.

Previously wrote an article called "hadoop RPC from entry to temporary abandonment", now roughly read the "Hadoop 2.xHDFS source code analysis" for the first time, feel that the overall understanding of HDFS has risen to a level, and re-read the dynamic agent, protocol buffer, so this time re-write the understanding of hadoop RPC.

The following is mainly through the ClientProtocol protocol, that is, the communication protocol between hdfs client and NameNode, to introduce the understanding of hadoopRPC.

Client

First of all, when you write a program to operate hadoop hdfs, you usually use DFSClient. The DFSClient class here holds an object instance of ClientNameNodeProtocolTranslatorPB (hereinafter referred to as translator) that implements the ClientProtocol interface. The main function of this instance is to transform ClientProtocol into a protocol buffer serialized request. In fact, it is mainly in which the dynamic proxy is used to obtain the proxy object of ClientNamenodeProtocolPB to achieve the transformation. When getting translator, you also need to consider whether NameNode is in HA mode.

Secondly, the most important method in the Client part is the Client.call method, which needs to obtain the connection object (Connection) of the corresponding RPCServer when calling the call method, but because establishing a connection is a very time-consuming operation in any system, connection will cache the connection to the corresponding server. In addition to establishing a connection, the Connection object also evokes the requesting thread after obtaining the corresponding reply to the corresponding request. The PRC request includes not only the parameters serialized by protocol buffer, but also the identity field of the corresponding request protocol. When the corresponding reply is received, the waiting thread will be called according to the callId. Here, we need to consider whether to call the normal accept reply method or the abnormal reply method according to whether the RPC call is successful or not.

two。 Server

Take NameNode's RPC server as an example, it has a class called NameNodeRpcServer that specializes in PRC requests sent from clients, DataNode, and other NameNode. Because you need to deal with a variety of PRC protocols (such as ClientNameNodeProtocolPB sent from the client), you need a class called BlockingService in this class. According to the information about the calling protocol in the request, find out the corresponding serverSideTranslator to process the corresponding * PB protocol into a real protocol. For example, the ClientNamenodeProtocolPB protocol sent from the client is transformed into a ClientProtocol protocol that can be recognized by jvm through the BlockingService of clientNNPbService.

In addition, the RPC server also refers to the reactor design model, using the idea of thread pool and nio to improve the concurrency ability. For details, you can refer to the previous "hadoop RPC from entry to temporary abandonment".

The above is almost a complete understanding of hadoopRPC, and new knowledge will be updated in a timely manner.

Kencao is in Shenzhen

2017-03-12

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.