Shulou(Shulou.com)06/01 Report--

This article introduces the relevant knowledge of "what are the four module files commonly used in hadoop". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

1.core-site.xml (tool module). Includes the utility classes commonly used in Hadoop, renamed from the original Hadoopcore section. It mainly includes system configuration tool Configuration, remote procedure call RPC, serialization mechanism and Hadoop abstract file system FileSystem and so on. They provide basic services for building a cloud computing environment on general hardware and provide the required API for software development running on the platform.

2.hdfs-site.xml (data Enclosure). Distributed file system, which provides high throughput, high scalability and high fault tolerance access to application data. It is the foundation of data storage management in Hadoop system. It is a highly fault-tolerant system that can detect and respond to hardware failures and is used to run on low-cost general-purpose hardware. HDFS simplifies the file consistency model, provides high-throughput application data access through streaming data access, and is suitable for applications with large datasets.

Namenode+ datanode + secondarynode

3.mapred-site.xml (data processing module), a parallel processing system for large data sets based on YARN. It is a kind of calculation model, which is used to calculate a large amount of data. The MapReduce implementation of Hadoop, together with Common and HDFS, constitutes the three components in the early stage of Hadoop development. MapReduce divides the application into two steps: Map and Reduce, in which Map performs specified operations on the independent elements on the dataset to generate intermediate results in the form of key-value pairs. Reduce specifies all values of the same "key" in the intermediate result to get the final result. The function partition such as MapReduce is very suitable for data processing in a distributed parallel environment composed of a large number of computers.

4.yarn-site.xml (Job scheduling + Resource Management platform), task scheduling and cluster resource management. Resourcemanager + nodemanager

This is the end of the content of "what are the four module files commonly used in hadoop". Thank you for your reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.