Shulou(Shulou.com)05/31 Report--

This article shows you how to integrate flume1.6.0 and kafka. The content is concise and easy to understand. It will definitely brighten your eyes. I hope you can get something through the detailed introduction of this article.

Start kafka and use the default configuration

① starts zookeeper,bin/-server-start.sh-daemon config/server.properties

② starts kafka:bin/kafka-server-start.sh-daemon config/server.properties

③ creates theme: bin/kafka-topics.sh-create-zookeeper localhost:2181-replication-factor 1-partitions 1-topic spider

④ starts consumer: bin/kafka-console-consumer.sh-zookeeper localhost:2181-topic spider-from-beginning

Configure flume and start it

For the detailed configuration of flume, please see my previous detailed configuration of flume.

Change the sink of flume (1.6.0, which will be added later on the blog by custom sink) to:

Agent1.sinks.log-sink1.channel = ch2

Agent1.sinks.log-sink1.type = org.apache.flume.sink.kafka.KafkaSink

Agent1.sinks.log-sink1.topic = spider

Agent1.sinks.log-sink1.brokerList = 192.168.57.133 9092

Agent1.sinks.log-sink1.requiredAcks = 1

Agent1.sinks.log-sink1.batchSize = 1

Start flume

III. Testing

The test class here is still the example of the last time log4j integrates flume. Run the test class. If you see the following output, the integration is successful.

The above content is how to integrate flume1.6.0 and kafka. Have you learned any knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.