Shulou(Shulou.com)05/31 Report--

In this issue, the editor will bring you about how to analyze the core components of the Spark big data analysis framework. The article is rich in content and analyzes and narrates it from a professional point of view. I hope you can get something after reading this article.

The core components of Spark big data analysis framework include RDD memory data structure, Streaming stream computing framework, GraphX graph computing and mesh data mining, MLlib machine learning support framework, Spark SQL data retrieval language, Tachyon file system, SparkR computing engine and other main components. Here is a brief introduction.

1. RDD memory data structure

Big data analysis system generally includes data acquisition, data cleaning, data processing, data analysis, report output and other subsystems. In order to facilitate data processing and improve performance, Spark specially introduces RDD data memory structure, which is very similar to the mechanism of R. The user program only needs to access the structure of RDD, and the data scheduling and exchange with the storage system are driven by the provider. RDD can interact with Haoop's HBase, HDFS, etc., and can be used as a data storage system. Of course, many other data storage systems can be supported by extension.

Because of RDD, it is important that the application model is separated from physical storage and that it is easier to handle traversing searches for a large number of data records. Because the structure of Hadoop is mainly suitable for sequential processing, it is very inefficient to go back and retrieve data repeatedly, and there is a lack of a unified implementation framework, so it is up to the algorithm developers to figure out how to implement it. There is no doubt that this is quite difficult. With the emergence of RDD, this problem has been solved to a certain extent. However, because RDD is the core component and difficult to implement, the performance, capacity and stability of this block directly determine the implementation degree of other algorithms. From the current point of view, there are still frequent problems with the memory overload occupied by RDD.

II. Streaming flow Computing Framework

Stream is now an important data form of Twitter, Weibo, Wechat, picture services, Internet of things, location services and so on, so stream computing is more important than ever before. Stream computing framework is the core infrastructure of all Internet service providers. Amazon and Microsoft have launched Event message bus cloud service platform, while facebook\ twitter and others have made their own stream computing framework open source.

Spark Streaming is specifically designed to handle streaming data. Through Spark Streaming, the data can be quickly pushed into the processing link, processed as quickly as an assembly line, and fed back to use in the shortest possible time.

Third, GraphX diagram calculation and mesh data mining.

The topological structure of physical network, the connection relationship of social network and the Emurr relationship of traditional database are all typical Graph data models. Hadoop is mainly suitable for situations where there is a large amount of data, there is almost no support for relationship processing, and Hbase is also a very weak relationship processing ability. Graph data structures often need to scan the data quickly and many times. The introduction of RDD enables Spark to deal with graph-based data structures more efficiently, which makes it possible to store and process large-scale graph networks. Similar systems dedicated to diagrams include neo4j and so on.

Compared with the relational connection of traditional database, GraphX can handle larger-scale and deeper topological relations, and can operate on multiple cluster nodes, which is indeed a sharp tool for modern data relationship research.

IV. MLlib machine learning support framework

By transplanting the machine learning algorithm to the Spark architecture, on the one hand, we can make use of the low-level large-scale storage and the fast data access ability of RDD, but also make use of the graph data structure and the processing power of cluster computing, so that the machine learning operation can be carried out on a large-scale cluster system, that is, it greatly expands the application ability of machine learning algorithm.

Spark SQL data Retrieval language

This is similar to an Hive-based implementation, but based on RDD theoretically provides better performance and makes it easier to handle operations such as join and relational retrieval. This is designed as a standardized entrance to interact with users.

VI. Tachyon file system

Tachyon is an implementation similar to HDFS, but feels closer to the user, while HDFS is primarily block-oriented.

7. SparkR computing engine

Apply the capabilities of R language to the Spark basic computing architecture and provide it with an algorithm engine.

The above is the editor for you to share how to analyze the core components of the Spark big data analysis framework, if you happen to have similar doubts, you might as well refer to the above analysis to understand. If you want to know more about it, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.