Shulou(Shulou.com)06/01 Report--

What this article shares with you is about the principle and implementation method of kafka traffic monitoring. The editor thinks it is very practical, so I share it with you to learn. I hope you can get something after reading this article.

Engineering capacity

As an excellent developer, the reliability of the monitoring alarm system in the process of project development can reflect a person's project management ability. Excellent monitoring and alarm system can save a lot of energy consumption, such as maintenance, fault prediction, timely and accurate fault notification, fault location and troubleshooting and so on.

You can imagine how terrible such a black box would be if there was no monitoring alarm system after the project was put online.

For big data project, the data generally needs to be entered into the message queue, such as kafka, and then separated from the line and decoupled and diverted in real time for real-time processing and offline processing. Benefits of message queuing:

The subscribers of the message queue can be expanded at any time as needed, and the users of the data can be well extended.

The horizontal expansion of message queues to increase throughput is still very simple. This uses the traditional database, sub-library sub-table is still very troublesome.

Due to the existence of message queues, it can also help us to resist the peak and avoid the downtime of the whole business processing caused by the excessive pressure on the back-end processing during the peak period.

Kafka plays a very important role in the big data project, so its monitoring and alarm is very important. Here, we are mainly talking about the monitoring and alarm of kafka traffic, and its purpose is obviously to make it easier for us to understand the overall situation and fluctuations of the data, so as to adjust the processing back-end, such as spark streaming,flume.

There are many kafka monitoring tools, such as kafka manager,KafkaOffsetMonitor,kafka eagle,kafka tools, and so on. Kafka manager is the most frequently used in Langjian. It is also recommended that we use this tool, which not only has monitoring functions but also management functions. For specific usage, please refer to:

Kafka Management artifact-kafkamanager

Monitoring index

Kafka has both the indicator server and the client. For more information on metrics, please see the official website of kafka:

Http://kafka.apache.org/0102/documentation.html#monitoring

The easiest way to view available metrics is to start jconsole and point it to a running kafka client or server; this will allow you to browse all metrics using JMX.

Friends who are familiar with kafka manager should have seen broker-related information, such as the number of incoming messages per second, the size of inflows per second, the size of outflows, and so on.

You can easily view it with kafka manager. However, this does not allow us to find fluctuations in data traffic in time, or we want to draw a curve to compare historical traffic in detail, which is impossible. So, we have to find a way to get these indicators, and then do our own presentation. There is also a warning of traffic fluctuations.

The interfaces of only a few metrics in the figure are made here at Langjian:

Def getBytesInPerSec (kafkaVersion: KafkaVersion, mbsc: MBeanServerConnection, topicOption: Option [String] = None) = {getBrokerTopicMeterMetrics (kafkaVersion, mbsc, "BytesInPerSec", topicOption)}

Def getBytesOutPerSec (kafkaVersion: KafkaVersion, mbsc: MBeanServerConnection, topicOption: Option [String] = None) = {getBrokerTopicMeterMetrics (kafkaVersion, mbsc, "BytesOutPerSec", topicOption)}

Def getBytesRejectedPerSec (kafkaVersion: KafkaVersion, mbsc: MBeanServerConnection, topicOption: Option [String] = None) = {getBrokerTopicMeterMetrics (kafkaVersion, mbsc, "BytesRejectedPerSec", topicOption)}

Def getFailedFetchRequestsPerSec (kafkaVersion: KafkaVersion, mbsc: MBeanServerConnection, topicOption: Option [String] = None) = {getBrokerTopicMeterMetrics (kafkaVersion, mbsc, "FailedFetchRequestsPerSec", topicOption)}

Def getFailedProduceRequestsPerSec (kafkaVersion: KafkaVersion, mbsc: MBeanServerConnection, topicOption: Option [String] = None) = {getBrokerTopicMeterMetrics (kafkaVersion, mbsc, "FailedProduceRequestsPerSec", topicOption)}

Def getMessagesInPerSec (kafkaVersion: KafkaVersion, mbsc: MBeanServerConnection, topicOption: Option [String] = None) = {getBrokerTopicMeterMetrics (kafkaVersion, mbsc, "MessagesInPerSec", topicOption)}

Jmx client

Server connected to jmx can use jconsole, but it doesn't meet our needs. So, we use JMXConnectorFactory to connect to jmx. When using JMXConnectorFactory to link jmx, the parameter url of JMXServiceURL must be connected in service:jmx mode. The specific link creation method is very simple, just a few lines of code, as follows:

Val jmxHost = "hostname" val jmxPort = 9999val urlString = s "service:jmx:rmi:///jndi/rmi://$jmxHost:$jmxPort/jmxrmi" val url = new JMXServiceURL (urlString) val jmxc = JMXConnectorFactory.connect (url)

Val mbsc = jmxc.getMBeanServerConnection

Println (KafkaMetrics.getMessagesInPerSec (Kafka_0_10_2_1,mbsc,Some ("test")) .fifteenth MinuteRate) jmxc.close ()

Open the jmx port of kafka

The jmx service of kafka is disabled by default. If it is enabled, you only need to add a line of code to the startup script kafka-server-start.sh of kafka server, with the content export JMX_PORT= "9999". The added location is as follows:

If ["x$KAFKA_HEAP_OPTS" = "x"]; then

Export KAFKA_HEAP_OPTS= "- Xmx1G-Xms1G"

Export JMX_PORT= "9999"

Fi

test

The test here is relatively simple, mainly to type out the number of messages, and you can adjust them according to your needs, such as sending SMS alarms when the average value is greater than the threshold.

A complete set of kafka monitoring, including:

Consumer monitoring, mainly survival alarm, consumption lag alarm.

Producer monitoring, mainly survival alarm, producer consumption upstream data ability alarm.

Broker monitoring, mainly survival alarm, traffic alarm, isr list, topic exception alarm, control transformation alarm.

These are the principles and implementation methods of kafka traffic monitoring. The editor believes that there are some knowledge points that we may see or use in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.