Shulou(Shulou.com)06/03 Report--

Warm Tip: to see the high-definition no-code picture, please open it with your mobile phone and click the picture to enlarge.

1. Problem description

Through the CDH management platform and entering the Zookeeper management interface, the average request delay, minimum request delay and maximum request delay indicator trend chart of Zookeeper remains unchanged, and the indicator data is abnormal.

two。 Problem recurrence

Log in to the CDH platform, go to the Zookeeper management page, and check the status of each Zookeeper server. The page is as follows:

View the following metrics

The average request delay, the minimum request delay and the maximum request delay remain unchanged, and the index is abnormal.

3. Analysis of problems

3.1 Source of indicators

The indicator is obtained by CDH through ZookeeperJMX. For more information, please see the official website of Zookeeper.

Http://zookeeper.apache.org/doc/r3.5.3-beta/zookeeperJMX.html#ch\_jmx

Get Zookeeper metrics information by command

Through the above analysis, we can judge that the index data is not calculated by CDH, but is calculated by Zookeeper, and the problem of CM is eliminated.

3.2Analyzing Zookeeper source code

Command line calls Zookeeper monitoring metrics code snippet (MonitorCommand.java)

By looking at the source code, the metric data is to obtain the "request delay" indicator through ServerStats.

Analyze ServerStats code, code snippet (ServerStats.java)

ServerStats only provides updateLatency methods to update maxLatency and minLatency metrics.

The code snippet for calling the update ServerStats method is as follows: (FinalRequestProcessor.java)

3.3 abnormal index analysis

By analyzing the code, maxLatency and minLatency metrics data record each Request metrics data on the Zookeeper server startup.

Maximum request delay and minimum request delay among all requests of the maxLatency and minLatency metrics recording server to obtain the Zookeeper server

Maximum request delay indicator

Zookeeper server 1, the index value of the maxLatency that updates the ServerStats through A request is 100ms, and then all requests do not exceed the value that maxLatency is 100ms. Every time the CDH platform acquires the maxLatency index of Zookeeper server 1, it is 100ms, which causes the maxLatency index of CDH platform Zookeeper to remain "abnormal" at the beginning of the article.

Minimum request delay indicator

Zookeeper server 1, the index value of the minLatency that updates the ServerStats after passing the A request is 0ms, and after that, all requests are not less than the value that minLatency is 0ms. Every time the CDH platform acquires the minLatency index of Zookeeper server 1, it is 0ms, which leads to the "exception" that the minLatency index of CDH platform Zookeeper remains the same as 0ms at the beginning of the article.

Average request delay indicator

Zookeeper server 1 accumulates the delay time (totalLatency) of all requests, accumulates the total number of requests (count), and obtains the avgLatency index through totalLatency/count. As for the persistence of the avgLatency target to 0ms, due to totalLatency

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.