Shulou(Shulou.com)06/03 Report--

PS: The follow-up articles will decompose the content of my practice into small modules, which will facilitate everyone's learning and communication. I will also attach the relevant code. Come on! Three years of learning big data principles, has not been practiced. Recently, I am preparing to leave my job. It is just a good time to put all the big data I have learned into practice, and it will not only be pure theory. In the face of practice, we must first have an empty cup mentality. After emptying ourselves, we can learn more. Come on! I also hope that everyone will pay more attention to it and pay more attention to the combination of practice and principle in the future.

environment construction

For big data, the focus is on the underlying architecture of Hadoop. Although Spark architecture is still used more. But hadoop is still the foundation. There is also a reason why it should be based on Linux, mainly because most of the current website servers are in Linux systems. At this point, personal feeling or in the file read and data call this aspect has a lot of advantages. Linux's open source makes it easier for most programmers to understand the system. Linux is also very helpful for Python programming. As for the rumor: life is short, you have to use Python. Personal understanding, programming in Python will be very simple. As long as you have Linux installed.

For this part of the environment, during the operation. The biggest difficulty is the NAT settings, for this step, there is no step-by-step operation according to the video content. I am using wireless to operate, and most personal PCs in the wireless link to obtain the IP is 192.168 network segment, when the virtual machine to communicate with the host in NAT mode, we must set in different network segments, and I personally change the virtual machine to 172.20 this network segment. It will not affect the subsequent operation.

Java Guide

For this part, since I learned about setting environmental variables in this block in the R Principle class earlier. Then in the virtual machine linux related settings can be in-depth understanding of its principles. and direct the original guidance to the relevant path.

The main difficulty in this part is the understanding of vim editor and the interpretation of related commands. At that time, the biggest confusion was how to edit and save a file after opening it. Later, through the search Baidu, see some technical posts after gradually understand the use of vim editor. There is also an understanding of the principle, there is also a good English, after encountering errors, to know the solution from ×××, and practice implementation. Stuck in a place, must be solved within a day, otherwise the enthusiasm for learning a great blow.

Building Hadoop

In this section, vim command is the most. That is, how to set the relevant parameters under hadoop-1.2.1, and finally whether the key processes such as datanode,jobtrack,tasktrack,namenode are up. And this part, is oneself after formatting namenode, open the service many times before getting up. This block may involve the interconnection of three virtual machines and the mutual transmission of parameters. So this one is the slowest. Another difficulty is that if you don't have a public key file on your virtual machine (mine doesn't). This requires touch to establish, and when detecting user permissions, use the chmod command. In short, this part of the content is a bit difficult, you need to be able to write vim commands comprehensively, and at the same time understand the relevant processes of hadoop.

summary

Now the Python command is missing, and I feel that theory is really different from practice. In the process of continuous learning, not only do I have to overcome the inherent defects in the code, but I also have a deeper understanding of the kernel principle. Fortunately, the good habits that have been developed will record the operations in the work one by one. Easy to follow up and understand. Everyone is welcome to discuss it together.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.