Shulou(Shulou.com)06/01 Report--

This article mainly shows you "how to use prometheus to monitor zookeeper", the content is easy to understand, clear, hope to help you solve doubts, the following let the editor lead you to study and learn "how to use prometheus to monitor zookeeper" this article.

Prometheus Monitoring zookeeper

Official introduction: there are two main ways, four-word command and jmx

1. Four-word command:

The three most commonly used stat, srvr and cons in https://zookeeper.apache.org/doc/r3.4.14/zookeeperAdmin.html#sc_zkCommands

Three of the more interesting commands: "stat" gives some general information about the server and connected clients, while "srvr" and "cons" give extended details on server and connections respectively.

$echo srvr | nc svc-zk.o.svc.mhc.local 2181 Zookeeper version: 3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03 ax 23 10:13 2017 GMTLatency min/avg/max: 0/0/802Received: 5833450Sent: 6074801Connections: 84Outstanding: 0Zxid: 0x2d9407Mode: standaloneNode count: 12977

It is convenient to use four-word commands to monitor zookeeper that is already in production, that is, to send commands such as mntr to the zookeeper service.

For more information, please see https://github.com/jiankunking/zookeeper_exporter.

Follow the above reference documentation to build a mirror or use the mirrored carlpett/zookeeper_exporter, and start exporter

Docker run-it-p 9141-- name=zookeeper_exporter_m-d xxx.xxx.com/library/zookeeper_exporter:1.0.2-zookeeper svc-zk.m.svc.uuu.local:2181curl http://127.0.0.1:9141/metrics. # HELP zk_avg_latency Average latency of requests# TYPE zk_avg_latency gaugezk_avg_latency "HELP zk_ephemerals_count Number of ephemeral nodes# TYPE zk_ephemerals_count gaugezk_ephemerals_count 262" HELP zk_max_file_descriptor_count Maximum number of open file descriptors# TYPE zk_max_file_descriptor_count counterzk_max_file_descriptor_count 1.048576e+06# HELP zk_max_latency Maximum seen latency of requests# TYPE zk_max_latency gaugezk_max_latency 1# HELP zk_min_latency Minimum seen latency of requests# TYPE zk_min_latency gaugezk_min_latency 0# HELP zk_num_alive_connections Number of active connections# TYPE zk_num_alive_connections gaugezk_num_alive_connections 82# HELP zk_open_file_descriptor_count Number of open file descriptors# TYPE zk_open_file_descriptor_count gaugezk_open_file_descriptor_count 10 "HELP zk_outstanding_requests Number of outstanding requests# TYPE zk_outstanding_requests gaugezk_outstanding_requests" HELP zk_packets_received Number of packets received# TYPE zk_packets_received counterzk_packets_received 4 "HELP zk_packets_sent Number of packets sent# TYPE zk_packets_sent counterzk_packets_sent 4" HELP zk_server_state Server state (leader/ Follower) # TYPE zk_server_state untypedzk_server_state {state= "standalone"} "HELP zk_up Exporter successful# TYPE zk_up gaugezk_up" HELP zk_watch_count Number of watches# TYPE zk_watch_count gaugezk_watch_count 2210 "HELP zk_znode_count Number of znodes# TYPE zk_znode_count gaugezk_znode_count 1310" HELP zookeeper_exporter_build_info A metric with a constant'1' value labeled by version Revision, branch, and goversion from which zookeeper_exporter was built.# TYPE zookeeper_exporter_build_info gaugezookeeper_exporter_build_info {branch= "", goversion= "go1.10.8", revision= "", version= ""} 1

The prometheus configuration content is added as follows, and the custom tag env is also added here:

-job_name: zk-u static_configs:-targets:-172.21.11.101 labels: env: U-job_name: zk-m static_configs:-targets:-172.21.11.101 labels: env: M-job_name: zk-h static_configs:-targets: -172.21.11.101 labels: env: h-job_name: zk-c static_configs:-targets:-172.21.11.101 zk-c static_configs 9144 labels: env: C

Reload effective configuration

Curl-XPOST http://prometheus:9090/-/reload

View target on the prometheus page:

Grafana presentation:

2. Jmx monitoring

Is the use of jvm_exporter exposure indicators, refer to prometheus official jmx_exporter https://github.com/prometheus/jmx_exporter

Jvm-exporter reads jmx data from jvm-based applications and exposes it to prometheus crawling as http.

Jmx is a commonly used technology that outputs the status information of running applications and can control it (for example, using jmx to trigger gc)

Jvm-exporter is a Java application that collects app and jvm metrics using jmx apis and runs in the same jvm as Java agent.

Jvm-exporter is written in Java, published as jar package, download address

There is an ansible role (https://github.com/alexdzyoba/ansible-jmx-exporter)), and the configuration file contains the rules for rewriting the JMX MBean to the Prometheus presentation format metric. It is basically an regexp collection that converts MBeans strings into Prometheus strings.

Zookeeper is a key component of many systems, such as Hadoop and Kafka and Clickhouse, so monitoring is essential.

2.1. run in java-agent mode (officially recommended):

Input parameters to enable when zookeeper is started, and there is no need to enable jmx.

$cat java.env export SERVER_JVMFLAGS= "- javaagent:/root/jmx_prometheus_javaagent-0.12.0.jar=7070:/root/config.yaml $SERVER_JVMFLAGS"

View the startup log and check the results:

$netstat-tlnp | grep 7070tcp 0 0 0.0.0.0 grep 7070tcp 7070 0.0.0.0 grep 7070tcp * LISTEN 892Unijava $curl-s localhost:7070/metrics | head# HELP jvm_threads_current Current thread count of a JVM# TYPE jvm_threads_current gaugejvm_threads_current 16.00 HELP jvm_threads_daemon Daemon thread count of a JVM# TYPE jvm_threads_daemon gaugejvm_threads_daemon 12.00 HELP jvm_ Threads_peak Peak thread count of a JVM# TYPE jvm_threads_peak gaugejvm_threads_peak 16.0# HELP jvm_threads_started_total Started thread count of a JVM

With jmx-exporter you can scrape the metrics of running JVM applications. Jmx-exporter runs as a Java agent (inside the target JVM) scrapes JMX metrics, rewrite it according to config rules and exposes it in Prometheus exposition format.

Enable jvm_exporter in the container

Based on the github repository of the official zookeeper, modify the official dockerfile to add startup parameters

The link below is modified by me and is available for personal testing.

Https://github.com/weifan01/zookeeper-docker/tree/master/3.4.14

Built zk-3.4.14 version image: docker pull 3070656869/zookeeper:3.4.14-jvm-exporter

After starting OK, visit port 8080 to view the result

2.2. run independently in http mode

Refer to the official document: https://github.com/prometheus/jmx_exporter

Git clone https://github.com/prometheus/jmx_exporter.gitcd jmx_exportermvn package# modify configuration file example_configs/httpserver_sample_config.yml# modify the listening address and port sh run_sample_httpserver.sh in the startup script

I tested the configuration as follows:

$cat http_server_config.yaml-- # jmx address and port hostPort: 172.21.10.248:6666username: password: rules: # replicated Zookeeper-pattern: "org.apache.ZooKeeperService (\\ w+)" name: "zookeeper_$2"-pattern: "org.apache.ZooKeeperService (\\ w+)" name: "zookeeper_$3" labels: replicaId: "$2"-pattern: "org.apache.ZooKeeperService (\\ W+) "name:" zookeeper_$4 "labels: replicaId:" $2 "memberType:" $3 "- pattern:" org.apache.ZooKeeperService (\\ w+) "name:" zookeeper_$4_$5 "labels: replicaId:" $2 "memberType:" $3 "# standalone Zookeeper-pattern:" org.apache.ZooKeeperService (\\ w+) "name:" zookeeper_$2 "- pattern : "org.apache.ZooKeeperService (\\ w +)" name: "zookeeper_$2"

Start:

Java-Dcom.sun.management.jmxremote.ssl=false-Dcom.sun.management.jmxremote.authenticate=false-Dcom.sun.management.jmxremote.port=6667-jar jmx_prometheus_httpserver-0.12.1-SNAPSHOT-jar-with-dependencies.jar 172.21.10.248 Dcom.sun.management.jmxremote.authenticate=false 6668 http_server_config.yaml

Visit:

The above is all the contents of the article "how to use prometheus to monitor zookeeper". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.