Network Security Internet Technology Development Database Servers Mobile Phone Android Software Apple Software Computer Software News IT Information

In addition to Weibo, there is also WeChat

Please pay attention

WeChat public account

Shulou

In non-kerberos environment, flume collects logs to kafka

2025-04-08 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > Internet Technology >

Share

Shulou(Shulou.com)06/02 Report--

Kafkaflume.conf

Agent.sources = s1agent.channels = c1agent.sinks = k1agent.sources.s1.type=execagent.sources.s1.command=tail-F / usr/local/src/flume/testflume2.logagent.sources.s1.channels=c1agent.channels.c1.type=memoryagent.channels.c1.capacity=10000agent.channels.c1.transactionCapacity=100# set Kafka receiver # agent.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink## set broker address and port number of Kafka agent.sinks.k1.kafka.bootstrap.servers = master:9092,node01:9092 Node02:9092## sets Topicagent.sinks.k1.kafka.topic of Kafka = mydemo5## sets serialization mode # # agent.sinks.k1.kafka.flumeBatchSize = 20agent.sinks.k1.kafka.producer.acks = 1agent.sinks.k1.kafka.producer.linger.ms = 1agent.sinks.ki.kafka.producer.compression.type = snappyagent.sinks.k1.channel=c1

Create a topic

Kafka-topics-create-zookeeper master:2181/kafka-replication-factor 2-partitions 3-topic mydemo5

Start flume as a message producer to write on kafka

Flume-ng agent-c conf-f kafkaflume.conf-n agent-Dflume.root.logger=INFO,console

Consume kafka data

Kafka-console-consumer-bootstrap-server master:9092-topic mydemo5-from-beginning

Read the contents of a file and write to append to a new file

#! / bin/bashwhile read linedo echo $line sleep 0.01echo-e $line > > / usr/local/src/flume/testflume2.logdone < / etc/sudo-ldap.conf

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.

Views: 0

*The comments in the above article only represent the author's personal views and do not represent the views and positions of this website. If you have more insights, please feel free to contribute and share.

Share To

Internet Technology

Wechat

© 2024 shulou.com SLNews company. All rights reserved.

12
Report