Shulou(Shulou.com)06/03 Report--

Word count:

MapReduce process:

Write a class that inherits mapper and declares input (basically fixed) output (depending on requirements) types

If you rewrite map, the map method will be called many times. Each time you call the map method to read a row of data passed by split, you need to cut this line of data (StringTokeizer class, see space cutting by default).

While traversal, output through context

To write a program main entry class, package the program to JobTracker (mobile computing instead of mobile data)

Note that because it is a local program, the program is packed into a xxx.jar package and executed on the server of the namenode node.

Hadoop/bin has a hadoop command

#. / hadoop (enter) expand all commands combined with hadoop

#. / hadoop jar path/xxx.jar ww.cola.JobRun

-

Friend recommendation system: Cartesian product operation

In the relationship of a pair of friends, each individual is both master and follower, that is, one person does both K and V

Therefore, the data processed by map will increase, each time the map method is called, the data is output twice, and K and V

Switch.

Map process:

After the default shuff processing of the system, the data with the same K worth are merged.

It appears as follows: (Key and Value are separated by colons):

Avatar B C D E / / to deal with the possible relationship between them, simply apply the value as a Cartesian product

B: G

C:F

. . .

. .

Reduce process:

Use Set to store because it can be duplicated.

Accurate advertising push: and sort from top to bottom according to attention

Case: advertise Xiaomi's mobile phone on Sina Weibo and find people who follow the phone, which are used to pop up ads as soon as they log in and rank them by level of attention.

Attention weight formula: W=TF*Log (N/DF) (scoring)

TF: the number of times the current keyword appears in the Weibo content of the movie

DF: the number of current keywords that appear in all Weibo content, for example, "Xiaomi", appearing 4 times in a Weibo, only recorded as 1 piece of data

N: total Weibo

The results show that: (in order to reduce the overhead of disk content, different keywords are identified with unique numbers)

Weibo id keyword 1: score keyword 2: score keyword 3: score. .

After that, I will write a special blog.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.