Shulou(Shulou.com)06/01 Report--

This article mainly shows you "what is the core of the hadoop framework structure", the content is easy to understand, clear, hope to help you solve your doubts, the following let the editor lead you to study and learn "what is the core of the hadoop framework structure" this article.

Figure: DKhadoop technology architecture diagram

The core of hadoop framework structure:

The core design of hadoop frame structure is: HDFS and MapReduce. HDFS provides storage for massive data, and MapReduce provides computing for massive data.

Big data integrated development framework:

Big data's application development is too inclined to the bottom, and the technical aspects of the design are very extensive, so it is naturally much more difficult to learn. It is even more difficult for beginners. DKhadoop is a fast search that reencapsulates a series of technical frameworks at the bottom. Some common and reused basic codes and algorithms in the development of big data are encapsulated into class libraries, which lowers the learning threshold of Dazhong Bureau and reduces the difficulty of development.

DKhadoop frame structure module:

Let's take the DKhadoop distribution as an example:

1. The framework consists of six parts: data source and SQL engine, data acquisition (custom crawler) module, data processing module, machine learning algorithm module, natural language processing module and search engine module.

2. Fast big data General Computing platform (DKH) has integrated all the components of the development framework with the same version number. If a fast development framework is deployed on the open source big data framework, the components of the platform need to be supported as follows:

(1) data sources and SQL engines: DK.Hadoop, spark, hive, sqoop, flume, kafka

(2) data acquisition: DK.hadoop

(3) data processing module: DK.Hadoop, spark, storm, hive

(4) Machine learning and AI:DK.Hadoop, spark

(5) NLP module: upload server-side JAR package and directly support it

(6) search engine module: do not publish independently

Dkhadoop is a deeply integrated, recompiled HADOOP distribution that can be released separately. Independent deployment of FreeRCH (Express big data integrated development framework), the necessary components. DK.HADOOP integrates NOSQL database and simplifies the programming between file system and non-relational database; DK.HADOOP improves the cluster synchronization system and makes the data processing of HADOOP more efficient.

The above is all the content of the article "what is the Core of hadoop Framework structure". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.