Shulou(Shulou.com)05/31 Report--

Today, I will talk to you about the log analysis of Spark Streaming combined with Flume and Kafka, which may not be well understood by many people. in order to make you understand better, the editor has summarized the following content for you. I hope you can get something according to this article.

Spark Streaming combined with Flume, Kafka log analysis 1. Modify the corresponding configuration file

According to http://my.oschina.net/sunmin/blog/692994

Integrated installation of Flume+Kafka+SparkStreaming

Modify the log output file that needs to be monitored by flume/conf/producer.conf to the local log path: / var/log/nginx/www.eric.aysaas.com-access.log2. Import related jar packages

(shortcut key Ctrl + Alt + Shift + s), click "Modules" on the left side of the Project Structure interface to display the following interface

The jar package compiles itself, or loads the http://search.maven.org/#search|ga|1|g%3A%22org.apache.spark%22%20AND%20v%3A%221.6.1%22

3. The new log/KafkaTest.scala code is as follows: import kafka.serializer.StringDecoderimport org.apache.spark.SparkConfimport org.apache.spark.streaming.dstream.DStreamimport org.apache.spark.streaming. {Seconds, StreamingContext} import org.apache.spark.streaming.kafka.KafkaUtils/** * flume+kafka+SparkStreaming Real-time nginx Log acquisition * Created by eric on 16-6-29. * / object KafkaLog {def main (agrs: Array [String]): Unit = {val sparkConf = new SparkConf (). SetMaster ("local [1]"). SetAppName ("StreamingTest") val ssc = new StreamingContext (sparkConf, Seconds (20)) / / represents an instance val topic = "HappyBirthDayToAnYuan" val topicSet = topic.split (""). ToSet / / create direct kafka stream val kafkaParams = Map [String] with brokers and topics String] ("metadata.broker.list"-> "localhost:9092") / / pull information directly from kafka brokers without using any receiver. Val messages = KafkaUtils.createDirectStream [String, String, StringDecoder, StringDecoder] (ssc, kafkaParams, topicSet) val lines = messages.map (_. _ 2) lines.print () val words: DStream [String] = lines.flatMap (_ .split ("\ n") words.count (). Print () / / launch ssc.start () ssc.awaitTermination ()} 4.Test

Visit the local page to generate log http://www.eric.aysaas.com/app/admin

The total number of log lines generated in these 20 seconds is:

After reading the above, do you have any further understanding of Spark Streaming's log analysis combined with Flume and Kafka? If you want to know more knowledge or related content, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.