Shulou(Shulou.com)06/02 Report--

Editor to share with you what the four major components of spark are, I believe most people do not know much about it, so share this article for your reference, I hope you can learn a lot after reading this article, let's go to know it!

The four components of spark are: 1, SparkStreaming, the component of streaming computing for real-time data; 2, SparkSQL, the component used to operate structured data; 3, the framework and algorithm library provided by GraphX,Spark for graph computing; 4, MLlib, a machine learning algorithm library.

Four components of spark

1 、 SparkStreaming:

There is a strong demand for streaming computing of real-time data in many application fields, such as web server logs in the network environment or message queues composed of status updates submitted by users, which are real-time data streams. Spark Streaming is a component of streaming computing for real-time data on the Spark platform, which provides a rich API for dealing with data streams. Because these API correspond to the basic operations in Spark Core, developers will be more comfortable writing Spark Streaming applications after they are familiar with Spark core concepts and programming methods. From the perspective of the underlying design, Spark Streaming supports the same level of fault tolerance, throughput, and scalability as Spark Core.

2 、 SparkSQL:

Spark SQL is a component that Spark uses to manipulate structured data. With Spark SQL, users can query data using the SQL or Apache Hive version of SQL dialect (HQL). Spark SQL supports a variety of data source types, such as Hive tables, Parquet, and JSON. Spark SQL not only provides a SQL interface for Spark, but also enables developers to integrate SQL statements into the development process of Spark applications. Whether using Python, Java or Scala, users can simultaneously perform SQL queries and complex data analysis in a single application. Because of its close integration with the rich computing environment provided by Spark, Spark SQL stands out from other open source data warehouse tools. Spark SQL was first introduced in Spark l.0. Before Spark SQL, the University of California, Berkeley tried to modify Apache Hive to run on Spark, and then proposed the component Shark. However, with the proposal and development of Spark SQL, it is more closely integrated with Spark engine and API, so that Shark has been replaced by Spark SQL.

3 、 GraphX:

GraphX is the framework and algorithm library provided by Spark for graph computing. The concept of flexible distributed attribute graph is put forward in GraphX, and on this basis, the organic combination and unity of graph view and table view are realized, and rich operations are provided for graph data processing, such as subgraph operation subgraph, vertex attribute operation mapVertices, edge attribute operation mapEdges and so on. GraphX also combines with Pregel, and can directly use some common graph algorithms, such as PageRank, triangle counting and so on.

4 、 MLlib:

MLlib is a machine learning algorithm library provided by Spark, which contains a variety of classical and common machine learning algorithms, such as classification, regression, clustering, collaborative filtering and so on. MLlib not only provides additional functions such as model evaluation and data import, but also provides some lower-level machine learning primitives, including a general gradient descent optimization algorithm. All of these methods are designed to scale easily on the cluster.

The above is all the content of the article "what are the four components of spark?" Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.