Shulou(Shulou.com)06/01 Report--

Editor to share with you what the structure of the DKhadoop framework is, I believe most people do not know much about it, so share this article for your reference, I hope you can learn a lot after reading this article, let's go to understand it!

In recent years, with the development of the Internet, especially the development of the mobile Internet, the growth of data shows an explosive growth momentum. Google's crawler alone downloads more than 100 million web pages a day (2000 data,) the explosive growth of data directly promotes the development of massive data processing technology. The three major technical frameworks of large table, distributed file system and distributed computing put forward by Google have solved the problem of massive data processing. Google immediately opened up its design ideas and published three epoch-making papers, and soon the open source framework based on Google's design ideas emerged, namely, hadoop, Maperduce and many Nosql systems that are very popular today. These three technologies are also the core foundation of big data's technology.

At present, there are many commercial distributions of hadoop in China. Most of these commercial editions of hadoop are issued by foreign countries. There are not many domestic distributions, such as DKhadoop. It can be said that it is better to do commercial versions of hadoop independently at home. Let's take the big fast search DKhadoop as an example to introduce the hadoop framework structure!

Figure: DKhadoop technology architecture diagram

The core of hadoop framework structure:

The core design of hadoop frame structure is: HDFS and MapReduce. HDFS provides storage for massive data, and MapReduce provides computing for massive data.

Big data integrated development framework:

Big data's application development is too inclined to the bottom, and the technical aspects of the design are very extensive, so it is naturally much more difficult to learn. It is even more difficult for beginners. DKhadoop is a fast search that reencapsulates a series of technical frameworks at the bottom. Some common and reused basic codes and algorithms in the development of big data are encapsulated into class libraries, which lowers the learning threshold of Dazhong Bureau and reduces the difficulty of development.

DKhadoop frame structure module:

Let's take the DKhadoop distribution as an example:

1. The framework consists of six parts: data source and SQL engine, data acquisition (custom crawler) module, data processing module, machine learning algorithm module, natural language processing module and search engine module.

2. Fast big data General Computing platform (DKH) has integrated all the components of the development framework with the same version number. If a fast development framework is deployed on the open source big data framework, the components of the platform need to be supported as follows:

(1) data sources and SQL engines: DK.Hadoop, spark, hive, sqoop, flume, kafka

(2) data acquisition: DK.hadoop

(3) data processing module: DK.Hadoop, spark, storm, hive

(4) Machine learning and AI:DK.Hadoop, spark

(5) NLP module: upload server-side JAR package and directly support it

(6) search engine module: do not publish independently

Dkhadoop is a deeply integrated, recompiled HADOOP distribution that can be released separately. Independent deployment of FreeRCH (Express big data integrated development framework), the necessary components. DK.HADOOP integrates NOSQL database and simplifies the programming between file system and non-relational database; DK.HADOOP improves the cluster synchronization system and makes the data processing of HADOOP more efficient.

The above is all the content of the article "what is the structure of the DKhadoop framework?" Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.