Shulou(Shulou.com)06/03 Report--

I have been engaged in software development for nearly four years and have been keen on various technical languages, but I feel that I always do not know which aspect of development I like to engage in. I have been doing software outsourcing to Japan in Dalian in the first two years of graduation. After two years, I feel that it really doesn't make any sense. The software design is not done by myself, but I just need to write the program according to the requirements, so I left my job with several colleagues and later went to Beijing. In fact, what I want to do when I come to Beijing is not the same as the reality. In the end, I still worked on a similar outsourcing project. Writing this diary, I have always been very uncertain about what kind of technology I like and what kind of language I like, so this time I am going to start writing blog to record my study. My priority is mobile phone development. Fortunately, my current job gives me a lot of free time. It gives me a lot of room to study, so here I give myself a blog that encourages me to strive for at least one article about my study every week. My direction of learning is now divided into two directions, and finally to determine which one I like, and then work hard for it. One is the direction of the mobile phone, and the other is the direction of Hadoop. I think the software market in China will be very large in the future. needless to say, the mobile Internet is still very hot now, and it may gradually stabilize in the future, but in short, the past 10 years or so should be very stable. My view on the future is that data is king in the future. Whoever has more data will be able to take the initiative, but it is useless to master more data, so it is necessary to analyze all the data reasonably. In order to turn the data into the corresponding value. So the other direction is to study the treatment of big data and big data. Today's blog article will talk about Hadoop's notes on big data's handling experience. The book I am reading now is the second edition of the authoritative Guide to Hadoop in Chinese. Through learning to grow step by step, it can be regarded as your own learning record.

Start taking notes:

Hadoop is an open source project in apache. The core is Google's framework for dealing with big data, based on the MapReduce software design model. It is an open source software project of big data distributed system, which is now the mainstream software framework for dealing with big data analysis.

Hadoop processing: data input-> MapReduce program and configuration-> output.

Map task: key,value generation of data. Output to the local disk, because it is an intermediate result, it would be a mountain out of a molehill if you put it on HDFS (distributed file system).

Reduce task: parse the data and output the expected results. Output to HDFS to achieve reliable data.

I understand that Reduce is quite a leader. If it needs some kind of regular data, it needs map to provide it. In map, it can be processed by providing partitions to different Reduce. When there is no Reduce, map is sent directly to HDFS.

There can be an excessive merge function in map and Reduce, and the use of merge function in map can reduce the amount of data output to map. Effectively reduce the amount of data transmission between map and Reduce. However, the merge function is still used on a case-by-case basis. For example, you can't use it when you need full data, such as averaging.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.