Shulou(Shulou.com)06/02 Report--

How to use prometheus to achieve remote storage, in view of this question, this article introduces the corresponding analysis and answer in detail, hoping to help more partners who want to solve this problem to find a more simple and feasible method.

Experimental system configuration

Mac 10.14

Docker

Engine: Version: 18.09.2 API version: 1.39 (minimum version 1.12) Go version: go1.10.6 Git commit: 6247962 Built: Sun Feb 10 04:13:06 2019 OS/Arch: linux/amd64 Experimental: false

Deploy pseudo-distributed kafka clusters in docker

Create a new topic named metrics

[Mac uses docker to build kafka cluster + Zookeeper + kafka-manager] (https://learnku.com/articles/19221)docker run-d-- name kafka--publish 9092-- link zookeeper-- env KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181-- env KAFKA_ADVERTISED_HOST_NAME=localhost-- env KAFKA_ADVERTISED_PORT=9092-- volume / etc/localtime:/etc/localtime wurstmeister/kafka:latest docker run-d-- name kafka- p 9092-e KAFKA_BROKER_ID=0-e KAFKA_ZOOKEEPER_CONNECT=10 .3.0.170: 2181-e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://1localhost:9092-e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092-t wurstmeister/kafkadocker run-d-- name kafka1-p 9093 virtual 9093-e KAFKA_BROKER_ID=1-e KAFKA_ZOOKEEPER_CONNECT=10.3.0.170:2181-e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9093-e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9093-t wurstmeister/kafkadocker run-d-- name kafka2- P 9094 KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9094 9094-e KAFKA_BROKER_ID=1-e KAFKA_ZOOKEEPER_CONNECT=10.3.0.170:2181-e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9094-t wurstmeister/kafkabin/kafka-topics.sh-- create-- zookeeper localhost:2181-- replication-factor 2-- partitions 2-- topic metricsbin/kafka-topics.sh-- describe-- zookeeper localhost:2181-- topic partopicbin/kafka-console-producer.sh-- broker -list localhost:9092-topic metricsbin/kafka-console-consumer.sh-bootstrap-server localhost:9092-topic metrics-from-beginning remote storage

The local storage design of Prometheus can reduce the complexity of its own operation and management, and at the same time meet the needs of most users' monitoring scale. But local storage also means that Prometheus cannot persist data, store large amounts of historical data, and flexibly scale and migrate.

In order to keep Prometheus simple, Prometheus does not try to solve the above problems in itself, but defines two standard interfaces (remote_write/remote_read) to allow users to save data to any third-party storage service based on these two interfaces, which is called Remote Storage in Promthues.

Remote storage of official documents

This article mainly explains the process of remote storage to kafka.

The github address of the adapter

Operation step 1

It is troublesome and important to pull down the github code and use goland to open it for compilation.

Install go and goland on mac

Compile the source code on goland, many of which cannot be downloaded, especially / golang.org/x/... Type of packages, because they are walled, so you need to add proxies. Direct execution

Local port of $export http_proxy=socks5://127.0.0.1:1080 / / ladder $export https_proxy=$http_proxy$ export GOPROXY= https://goproxy.io

Problem 1: solve the problem that the golang.org/x package cannot be downloaded

Problem 2: solving the problem of golang grpc installation

Question three

$go test# pkg-config-cflags rdkafka Package rdkafka was not found in the pkg-config search path. Perhaps you should add the directory containing `rdkafka.pc' to the PKG_CONFIG_PATH environment variable No package 'rdkafka' found pkg-config: exit status 1

Result

1. Git clone https://github.com/edenhill/librdkafka.gitcd librdkafka./configuremakemake install step 2

First modify the kafkaBrokerList in the source code config.go to the address of the local kafka.

Compile the prometheus-kafka-adapter source code:

Go testgo build

After there is no problem, an executable file named prometheus-kafka-adapter appears in the source code and executes. 3. Modify the configuration file prometheus.yml of prometheus by adding:

Remote_write:-url: "http://:8080/receive"

If you spend money on the kafka side, you can see the monitoring data.

Prometheus writes influxdb/opentsdb system configuration

Mac 10.14

Docker

Engine: Version: 18.09.2 API version: 1.39 (minimum version 1.12) Go version: go1.10.6 Git commit: 6247962 Built: Sun Feb 10 04:13:06 2019 OS/Arch: linux/amd64 Experimental: false

Deploy influxdb in docker

Execution

1. Prometheus uses influxdb for permanent storage

2. Prometheus high availability (2): understanding remote storage

3. Prometheus remote read and write Influxdb

Compilation (difficulty)

Execute under the parent directory:

Go build. / documentation/examples/remote_storage/remote_storage_adapter/ on how to use prometheus to achieve remote storage questions is shared here, I hope the above content can be of some help to you, if you still have a lot of doubts to be solved, you can follow the industry information channel for more related knowledge.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.