Shulou(Shulou.com)06/01 Report--

Editor to share with you the example analysis of hbase callQueue, I believe that most people do not know much about it, so share this article for your reference, I hope you can learn a lot after reading this article, let's go to know it!

1. Hbase RPC Server

First, briefly describe the main process of Rpc request on the server side (Master/RegionServer):

When Server starts, it will configure the corresponding number of Rpc request processing threads handler to listen to CallQueues, and when a task comes, it will be added to CallQueue,Handler to consume processing requests.

/ / when the server receives the Client request, it will call RpcScheduler to distribute the request. The default RpcScheduler is SimpleRpcSchedulerorg.apache.hadoop.hbase.ipc.ServerRpcConnection#processRequest this.rpcServer.scheduler.dispatch (new CallRunner (this.rpcServer, call)) # 703 line / / SimpleRpcScheduler will add the request to CallQueueorg.apache.hadoop.hbase.ipc.SimpleRpcScheduler#dispatch / / each handler will fetch task processing from its own CallQueue, org.apache.hadoop.hbase.ipc.RpcExecutor.Handler#run ()

So when did these listening threads start?

/ / when master or regionserver starts Create rpcServicesorg.apache.hadoop.hbase.master.HMaster#createRpcServices 719org.apache.hadoop.hbase.regionserver.HRegionServer#createRpcServices 781org.apache.hadoop.hbase.master.MasterRpcServices#MasterRpcServicesorg.apache.hadoop.hbase.regionserver.HRegionServer#createRpcServices org.apache.hadoop.hbase.regionserver.RSRpcServices#RSRpcServices (org.apache.hadoop.hbase.regionserver.HRegionServer) / / create a server-side org.apache.hadoop.hbase.regionserver.RSRpcServices#createRpcServer 1294L// that handles RPC requests and create the default RpcServer (SimpleRpcScheduler) org. Apache.hadoop.hbase.regionserver.SimpleRpcSchedulerFactory#create (org.apache.hadoop.conf.Configuration Org.apache.hadoop.hbase.ipc.PriorityFunction, org.apache.hadoop.hbase.Abortable) / / launch handlers org.apache.hadoop.hbase.regionserver.RSRpcServices#start org.apache.hadoop.hbase.ipc.SimpleRpcServer#start org.apache.hadoop.hbase.ipc.SimpleRpcScheduler#start2 for different RpcExecutor. RpcScheduler classification

There are mainly two kinds of RpScheduler in hbase: SimpleRpcScheduler and FifoRpcScheduler.

Its usage type is determined by hbase.region.server.rpc.scheduler.factory.class, and the default is SimpleRpcSchedulerFactory.class.

We can also use org.apache.hadoop.hbase.ipc.PhoenixRpcSchedulerFactory after we install phoenix

3. SimpleRpcScheduler

There are four kinds of CallQueue in SimpleRpcScheduler, and the corresponding rpcExecutor of each CallQueue may be different under different conditions.

In fact, we are more concerned about the following four values of each callExecuotr: the number of callQ, the number of numCallQueues,handler, the implementation class of rpcExecutor, maxQueueLength

We can see the description of handler and queue in the log or hbase ui as shown above.

1. Call Queue normal queue

HandlerCount=hbase.regionserver.handler.count default 30numCallQueues=hbase.regionserver.handler.count*0.1callqReadShare=hbase.ipc.server.callqueue.read.ratio default 0callQueueType=hbase.ipc.server.callqueue.type default fifomaxQueueLength=hbase.ipc.server.max.callqueue.length default 10*handlerCount

If callqReadShare > 0

Indicates the separation of read and write. Assuming 0.7, it means that the read queue accounts for 70% and the write queue accounts for 30%.

Suppose handlerCount=100, then numCallQueues=handlerCount*0.1=10, there are seven read queues and three write queues

At this time, the implementation class of rpcExecutor is RWQueueRpcExecutor,handlers and the corresponding relationship between callQueue and RWQueueRpcExecutor,handlers is as follows

Name=default.RWQ

If callqReadShare 0 has this queue

At this time, the implementation class of rpcExecutor is FastPathBalancedQueueRpcExecutor

Name=replication.FPBQ

4. Meta Transition Queue meta Migration queue

MetaTransitionHandler=hbase.master.meta.transition.handler.count default 1maxQueueLength=hbase.ipc.server.max.callqueue.length default 10*handlerCountmaxPriorityQueueLength=hbase.ipc.server.priority.max.callqueue.length default = maxQueueLength

This queue is only available when metaTransitionHandler > 0.

At this time, the implementation class of rpcExecutor is FastPathBalancedQueueRpcExecutor

Name=metaPriority.FPBQ

The following figure details the options and conditions for four rpcExecutor.

4. The relationship between hanlder and queues

When Master or RegionServer starts, it starts a corresponding number of handlers. What do these handlers do? to put it simply, they handle various rpc requests from clients to the regionserver. A handler is a thread. How does handler handle requests? Instead of processing the request directly, it goes to the specified queue, that is, all requests come to advance a certain queue, and then there is handler to consume it. The existence of a queue can be thought of as a buffer when there are so many requests that handler cannot consume in time.

Well, we are more concerned about the following issues:

What is the correspondence between handler and queue in the program?

How many handler are there?

How many queues are there?

What is the length of each queue?

* * answer these questions through a picture below: * * there are four kinds of callQueue for hbase. Here, taking ordinary callQueue as an example, the principle is the same.

First introduce two parameters, and then answer the above four questions:

Hbase.regionserver.handler.count defaults to 30

The number of RPC Listener instances started on RegionServers, that is, the number of threads processing rpc requests. The number of Master handlers is also configured with this parameter. Too much handler can backfire. Can be set to a multiple of the number of CPU. If it is mainly read-only, the same number of handler and cpu will be fine. You can start with twice the number of CPU and then make adjustments.

Hbase.ipc.server.callqueue.handler.factor defaults to 0.1

Factors that determine the number of callQueue. A value of 0 means that a single queue is shared across all handlers. A value of 1 means that each handler has its own queue.

Hbase.ipc.server.max.callqueue.length defaults to 10 times the size of hander

The queue length of each callQueue, that is, how many requests can be accommodated

Answer the question:

So the number of handler is 30 by default. We can modify it through hbase.regionserver.handler.count to answer question 2.

The number of callQueue = hbase.regionserver.handler.count * hbase.ipc.server.callqueue.handler.factor, that is, if the default 30 handler,factor is 0.1, there will be 3 callQueue, corresponding to question 3

Each callQueue is a maxQueueSizeInBytesmaxQueueSizeInBytes parameter: hbase.ipc.server.max.callqueue.size defaults to 1024*1024*1024=1G*/// Enforcing the callqueue size, this triggers a retry in the client / / This is a bit late to be doing this check-we have already read in the / / total request. If ((totalRequestSize + this.rpcServer.callQueueSizeInBytes.sum ()) > this.rpcServer.maxQueueSizeInBytes) {final ServerCall callTooBig = createCall (id, this.service, null, totalRequestSize, null, 0, this.callCleanup); this.rpcServer.metrics.exception (RpcServer.CALL_QUEUE_TOO_BIG_EXCEPTION) CallTooBig.setResponse (null, null, RpcServer.CALL_QUEUE_TOO_BIG_EXCEPTION, "Call queue is full on" + this.rpcServer.server.getServerName () + ", is hbase.ipc.server.max.callqueue.size too small?"); callTooBig.sendResponseIfReady (); return;} / / or / 2. In this case, the rpcScheduler dispatch will determine whether the current queue size exceeds maxQueueLength. If it exceeds the queue size, it will directly return false if (! this.rpcServer.scheduler.dispatch (new CallRunner (this.rpcServer, call)) {this.rpcServer.callQueueSizeInBytes.add (- 1 * call.getSize ()); this.rpcServer.metrics.exception (RpcServer.CALL_QUEUE_TOO_BIG_EXCEPTION)). Call.setResponse (null, null, RpcServer.CALL_QUEUE_TOO_BIG_EXCEPTION, "Call queue is full on" + this.rpcServer.server.getServerName () + ", too many items queued?"); call.sendResponseIfReady ();}

We know from the code that there are two situations in Call queue is full: CALL_QUEUE_TOO_BIG_EXCEPTION

A kind of hbase.ipc.server.max.callqueue.size too small?

A kind of too many items queued?

Try the solution:

When you apply for hbase.ipc.server.max.callqueue.size too small, you can try to increase the hbase.ipc.server.max.callqueue.size. The default is 1024102410241G.

When you report too many items queued, you can try to increase the hbase.ipc.server.max.callqueue.length default = `handlerCount*10

The above is all the content of this article "sample Analysis of hbase callQueue". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.