Shulou(Shulou.com)06/03 Report--

Demand:

/ solution is a hot ranking within each sub-module-- "ranking must be done with sortBy--" (it may be a simple sort, or a secondary ranking)-- "

/ / there is a wordCount--- to "calculate the number of times--" what to consider as a key

/ / under a website, the top 2 hottest pages under each sub-module:

/ / under a website, the top 2 hottest pages under each sub-module-"the top 2 of the number of pages under each sub-module

/ / in actual development, the real code time may only account for 20-30%. The rest of the time is to understand the requirements and think about ideas.

Import java.net.URLimport org.apache.spark. {SparkConf, SparkContext} object Data_anlysis {def main (args: Array [String]) {val conf = new SparkConf (). SetAppName ("data-anysis"). SetMaster ("local [2]") val sc = new SparkContext (conf) val linesRDD = sc.textFile ("d://web_data.log") val rdd01 = linesRDD.map (line = > {val lineSplited = line.split ("\ t") (lineSplited (1)) 1)}) val rdd02 = rdd01.reduceByKey (_ + _) val rdd03 = rdd02.map (tup = > {val url = tup._1 Val host = new URL (url). GetHost (host,url,tup._2)}) val rdd04 = rdd03.groupBy (_. _ 1) val rdd05 = rdd04.mapValues (iter = > {iter.toList.sortBy (_. _ 3) .explorse.take (2)}) println (rdd05.collect (). ToBuffer) sc.stop ()}}

The running results will display the log data of the site in accordance with the requirements of a certain module to list the top two results.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.