Shulou(Shulou.com)06/01 Report--

This article mainly introduces the example analysis of flume online configuration, which has a certain reference value. Interested friends can refer to it. I hope you will learn a lot after reading this article.

Hdfs_agent.sources = R1

Hdfs_agent.sinks = K1

Hdfs_agent.channels = C1

# Describe/configure the source

Hdfs_agent.sources.r1.type = org.apache.flume.source.kafka.KafkaSource

# hdfs_agent.sources.r1.zookeeperConnect = 10.69.40.17 2181 Magazine 10.69.40.12 Fringe 2181 Magazine 10.69.40.7 Fraser 2181

# hdfs_agent.sources.r1.groupId = online_kafka_to_hdfs_gk_groupid_000001

# hdfs_agent.sources.r1.topic = IRT_Dev_Status_STD

Hdfs_agent.sources.r1.kafka.bootstrap.servers = 10.69.40.10 9092Jing 10.69.40.13 Fringe 9092Jing 10.69.40.14 Fringe 9092Jing 10.69.40.15Rd 9092

Hdfs_agent.sources.r1.kafka.consumer.group.id = online_kafka_to_hdfs_gk_groupid_000001

Hdfs_agent.sources.r1.kafka.topics = workcondition_info_flume

Hdfs_agent.sources.r1.batchSize = 2000

Hdfs_agent.sources.r1.interceptors = i1

Hdfs_agent.sources.r1.interceptors.i1.type = host

# Describe the sink

Hdfs_agent.sinks.k1.type = hdfs

Hdfs_agent.sinks.k1.hdfs.path = hdfs://rootcloud/ods/%Y%m%d/%H

Hdfs_agent.sinks.k1.hdfs.filePrefix = log-% {host}

Hdfs_agent.sinks.k1.hdfs.fileType = DataStream

Hdfs_agent.sinks.k1.hdfs.rollInterval = 3600

Hdfs_agent.sinks.k1.hdfs.rollSize = 104857600

Hdfs_agent.sinks.k1.hdfs.rollCount = 0

Hdfs_agent.sinks.k1.hdfs.batchSize = 2000

Hdfs_agent.sinks.k1.hdfs.callTimeout = 300000

Hdfs_agent.sinks.k1.hdfs.idleTimeout = 600,

# Use a channel which buffers events in memory

Hdfs_agent.channels.c1.type = memory

Hdfs_agent.channels.c1.capacity = 2000000

Hdfs_agent.channels.c1.transactionCapacity = 3000

# Bind the source and sink to the channel

Hdfs_agent.sources.r1.channels = C1

Hdfs_agent.sinks.k1.channel = C1

Thank you for reading this article carefully. I hope the article "sample Analysis of flume online configuration" shared by the editor will be helpful to you. At the same time, I also hope that you will support and pay attention to the industry information channel. More related knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.