Shulou(Shulou.com)06/02 Report--

Editor to share with you how to use Hadoop for distributed parallel programming, I believe that most people do not know much about it, so share this article for your reference, I hope you can learn a lot after reading this article, let's go to know it!

Distributed parallel programming with Hadoop

Introduction to Hadoop

Hadoop is an open source distributed parallel programming framework that can run on large-scale clusters. Because distributed storage is essential for distributed programming, this framework also includes a distributed file system HDFS (HadoopDistributedFileSystem).

Hadoop may not be so well known so far, and its version number is only 0.16, which seems like a long way to go, but it's absolutely famous to mention Hadoop's other two open source projects, Nutch and Lucene (both of which were founded by DougCutting). Lucene is an open source high-performance full-text search toolkit developed with Java. It is not a complete application, but a set of easy-to-use API. In the world, there have been numerous software systems, Web website based on Lucene to achieve full-text search function, and then DougCutting created * open source Web search engine (http://www.nutch.org)Nutch, which adds web crawlers and some Web-related functions on the basis of Lucene, some plug-ins for parsing various document formats, etc., in addition, Nutch also contains a distributed file system for storing data. Since the Nutch0.8.0 version, DougCutting has separated the distributed file system in Nutch and the code that implements the MapReduce algorithm to form a new open source project, Hadoop. Nutch has also evolved into an open source search engine based on Lucene full-text search and Hadoop distributed computing platform.

Based on Hadoop, you can easily write distributed parallel programs that can handle large amounts of data and run them on a large-scale computer cluster composed of hundreds of nodes. Judging from the current situation, Hadoop is destined to have a brilliant future: "cloud computing" is a popular technical term at present. Major IT companies around the world are investing in and promoting this new generation of computing model, and Hadoop is used by several major companies as important basic software in their "cloud computing" environment. For example, Yahoo is using the power of Hadoop open source platform to fight Google, in addition to funding the Hadoop development team. Also under development is the Hadoop-based open source project Pig, a distributed computing program that focuses on massive dataset analysis. Based on Hadoop, Amazon launched AmazonS3 (AmazonSimpleStorageService), which provides reliable, fast and scalable network storage services, as well as a commercial cloud computing platform AmazonEC2 (AmazonElasticComputeCloud). Hadoop is also an important basic software in IBM's cloud computing project "Blue Cloud Project". Google is working with IBM to promote Hadoop-based cloud computing.

Welcome the change of programming method

Under the action of Moore's Law, programmers did not have to consider that the performance of the computer would not keep up with the development of the software, because about every 18 months, the main frequency of the CPU would double, and the performance would be doubled, and the software could enjoy a free performance improvement without any changes at all. However, as the transistor circuit is gradually approaching its physical performance limit, Moore's Law began to fail around 2005, and people can no longer expect the speed of a single CPU to double every 18 months, providing us with faster and faster computing performance. Chip manufacturers such as Intel,AMD,IBM begin to tap the performance potential of CPU from the perspective of multi-core. With the arrival of multi-core era and Internet era, great changes will take place in the way of software programming. Multi-thread concurrent programming based on multi-core and distributed parallel programming based on large-scale computer clusters are the main ways to improve software performance in the future.

Many people think that this major change in programming will bring a software concurrency crisis, because our traditional software approach is basically the sequential execution of a single instruction and single data stream, which is very much in line with human thinking habits. But it doesn't fit in with concurrent parallel programming. Cluster-based distributed parallel programming allows software and data to run on many computers connected to a network at the same time, and each computer here can be an ordinary PC. The advantage of such a distributed parallel environment is that it is easy to expand new computing nodes by adding computers, thus obtaining incredible massive computing power and strong fault tolerance at the same time. The failure of a number of computing nodes will not affect the normal operation of the calculation and the correctness of the results. Google does this by using a parallel programming model called MapReduce for distributed parallel programming, running on a distributed file system called GFS (GoogleFileSystem), and providing search services for hundreds of millions of users around the world.

Hadoop implements Google's MapReduce programming model, provides an easy-to-use programming interface, and provides its own distributed file system HDFS. Unlike Google, Hadoop is open source, and anyone can use this framework for parallel programming. If distributed parallel programming is difficult enough to intimidate ordinary programmers, the emergence of open source Hadoop has greatly lowered its threshold. After reading this article, you will find that programming based on Hadoop is very simple. Without any parallel development experience, you can easily develop distributed parallel programs and make them run on hundreds of machines at the same time. Then complete the calculation of massive data in a short time. You may think that you won't have hundreds of machines to run your parallel programs, but in fact, with the popularity of "cloud computing", anyone can easily gain such massive computing power.

These are all the contents of the article "how to use Hadoop for distributed parallel programming". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.