Shulou(Shulou.com)06/02 Report--

This article mainly explains "how Flume collects and sends messages to kafka". The content in the article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "how Flume collects and sends messages to kafka".

The Flume agent script and configuration are as follows

Mac

Installation:

Brew install flume

Start the script:

Nohup / usr/local/Cellar/flume/1.9.0/bin/flume-ng agent-- conf/ data/flume/run/flume/conf/conf.d/-- conf-file / data/flume/run/flume/conf/conf.d/mg-res_test.conf-- name mg-res_test-Dflume.codeadmin.logger=INFO,console &

Linux

Installation:

Http://flume.apache.org/download.html

Start the script:

Nohup / data/flume/run/flume/bin/flume-ng agent-- conf/ data/flume/run/flume/conf/conf.d/-- conf-file / data/flume/run/flume/conf/conf.d/mg-res_test.conf-- name mg-res_test-Dflume.codeadmin.logger=INFO,console &

Configuration description

/ data/flume/run/flume/conf/conf.d/mg-res_test.conf

Flume main configuration information. Just pay attention to the parts described in Chinese.

# Name the components on this agentmg-res_test.sources = mg-res_testmg-res_test.sinks = mg-res_testmg-res_test.channels = mg-res_test# Describe/configure the sourcemg-res_test.sources.mg-res_test.type = TAILDIR# flume record offset file mg-res_test.sources.mg-res_test.positionFile=/data/logs/test/res-test/mg-res_test.jsonmg-res_test.sources.mg-res_test. Filegroups=f1# log file to be collected mg-res_test.sources.mg-res_test.filegroups.f1=/data/logs/test/res-test/app.logmg-res_test.sources.mg-res_test.fileHeader=true# Describe the sinkmg-res_test.sinks.mg-res_test.channel = mg-res_testmg-res_test.sinks.mg-res_test.type = org.apache.flume.sink.kafka.KafkaSink# kafka topic configuration is based on the actual mg-res _ test.sinks.mg-res_test.kafka.topic = res-test# kafka configuration information is based on the actual mg-res_test.sinks.mg-res_test.kafka.bootstrap.servers = localhost:9092mg-res_test.sinks.mg-res_test.kafka.flumeBatchSize = 2mg-res_test.sinks.mg-res_test.kafka.producer.acks = 1mg-res_test.sinks.mg-res_test.kafka.producer.linger.ms = 1mg-res_test.sinks. Mg-res_test.kafka.producer.compression.type = snappy# Use a channel which buffers events in memorymg-res_test.channels.mg-res_test.type = memorymg-res_test.channels.mg-res_test.capacity = 100000mg-res_test.channels.mg-res_test.transactionCapacity = 1000 million Bind the source and sink to the channelmg-res_test.sources.mg-res_test.channels = mg-res_testmg-res_test.sinks.mg-res_test.channel = mg-res_test

/ data/flume/run/flume/conf/conf.d/flume-env.sh

Just configure java home.

Export JAVA_OPTS= "- Xms128m-Xmx128m-Dcom.sun.management.jmxremote" JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home

Verification

The local environment can directly start kafka consumption to see if there are any messages coming in. In the test environment, please query the increment of kafka messages or the actual consumption of the business system.

Kafka-console-consumer-- bootstrap-server localhost:9092-- topic res-test-- from-beginning Thank you for your reading. The above is the content of "how Flume collects and sends messages to kafka". After the study of this article, I believe you have a deeper understanding of how Flume collects and sends messages to kafka, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.