Shulou(Shulou.com)06/03 Report--

What is big data? Since the beginning of this century, especially after 2010, with the development of the Internet, especially the mobile Internet, the growth of data has shown a trend of explosion. it has been difficult to estimate how much data is stored in electronic devices all over the world. The unit of measurement describing the amount of data in the data system has been rising from MB (1MB equals about 1 million bytes), GB (1024MB), TB (1024GB). PB (equal to 1024TB) level data system has been very common. With the continuous increase of mobile personal data, social networking sites, scientific computing, securities transactions, website logs and sensor network data, the total amount of data owned in China has long exceeded the ZB level.

The traditional data processing method is: with the increase of the amount of data, constantly update the hardware indicators, and adopt measures such as more powerful CPU and larger capacity disks, but the reality is that the speed of data increase is far beyond the speed of stand-alone computing and storage capacity.

The processing method of "big data" is to adopt the method of multi-machine and multi-node to deal with a large amount of data, and to adopt this new processing method, a new big data system is needed to guarantee it. The system needs to deal with a series of problems such as communication coordination and data separation between multiple nodes.

In a word, the way of using multi-machine and multi-node to solve the problems of communication coordination, data coordination and calculation coordination of each node and to deal with massive data is the thinking of "big data". Its characteristic is that with the continuous increase of the amount of data, the number of machines can be increased and expanded horizontally. A big data system can be as many as tens of thousands of machines or more.

At first, Hadoop mainly consists of two parts: distributed file system HDFS and computing framework MapReduce, which is an independent project from Nutch. In version 2.0, resource management and task scheduling functions are split from MapReduce to form YARN, so that other frameworks can run on Hadoop like MapReduce. Compared with the previous distributed computing framework, Hadoop hides many tedious details, such as fault tolerance, load balancing and so on.

Hadoop also has a strong ability to scale out, and it is easy to connect new computers to the cluster to participate in computing. With the support of the open source community, Hadoop continues to develop and improve, and integrates many excellent products, such as non-relational database HBase, data warehouse Hive, data processing tool Sqoop, machine learning algorithm library Mahout, consistency service software ZooKeeper, management tool Ambari, etc., forming a relatively complete ecological circle and the de facto standard of distributed computing.

The fast big data General Computing platform (DKH) has integrated all the components of the development framework with the same version number. If a fast development framework is deployed on the open source big data framework, the components of the platform need to be supported as follows:

Data sources and SQL engines: DK.Hadoop, spark, hive, sqoop, flume, kafka

Data acquisition: DK.hadoop

Data processing module: DK.Hadoop, spark, storm, hive

Machine Learning and AI:DK.Hadoop, spark

NLP module: upload server-side JAR package for direct support

Search engine module: do not publish independently

Da Kuai big data platform (DKH) is an one-stop search engine level designed by Daxuai Company to open the channel between big data ecosystem and traditional non-big data company, big data general computing platform. Through the use of DKH, traditional companies can easily bridge big data's technology gap and achieve search engine-level big data platform performance.

L DKH, effectively integrates all the components of the whole HADOOP ecosystem, and deeply optimizes and recompiles into a complete and higher performance big data general computing platform, realizing the organic coordination of various components. Therefore, compared with the open source big data platform, DKH has a performance improvement of up to 5 times (maximum) in computing performance.

L DKH, through the unique middleware technology of Fast, simplifies the complex configuration of big data cluster to three nodes (master node, management node, computing node), greatly simplifies the management, operation and maintenance of the cluster, and enhances the high availability, high maintainability and high stability of the cluster.

L DKH, although highly integrated, still retains all the advantages of the open source system and is 100% compatible with the open source system. Big data applications developed on the open source platform can run efficiently on DKH without any changes, and the performance will be improved by up to 5 times.

L DKH is integrated with the fast big data integrated development framework (FreeRCH). The FreeRCH development framework provides more than 20 classes commonly used in big data, search, natural language processing and artificial intelligence development. Through a total of more than 100 methods, the development efficiency has been improved more than 10 times.

The SQL version of DKH also provides the integration of distributed MySQL, the traditional information system, which can be seamlessly implemented for big data and distributed leapfrogging.

Technical architecture diagram of DKH standard platform

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.