Shulou(Shulou.com)06/01 Report--

This article is about how to carry out Spark data analysis, the editor thinks it is very practical, so I share it with you to learn. I hope you can get something after reading this article.

Introduction to Spark data analysis

1.Spark is a platform for fast and general-purpose cluster computing, extending the MapReduce computing model to support more computing models, including interactive queries and stream processing

two。 Including Spark Core, Spark SQL, Spark Streaming (memory streaming computing), MLlib (machine learning), GraphX (graph computing)

3. Suitable for data science applications and data processing applications

II. Download and get started with Spark

1.Spark applications use a driver program (driver program) to initiate various parallel operations on the cluster. The driver accesses the Spark through a SparkContext object, which represents a connection to the computing cluster.

III. RDD programming

1.RDD (Resilient Distributed Dataset, flexible distributed dataset) is a collection of distributed elements. In Spark, all operations on data are to create RDD, convert RDD, and call RDD operations to evaluate.

two。 Mode of work:

Create an input RDD from external data

Use a conversion operation such as filter () to convert RDD to define a new RDD

Tell Spark to persist () on the intermediate result RDD that needs to be requisitioned

Use action actions (such as count (), first (), etc.) to trigger a parallel computation, which is optimized by Spark before execution

The conversion operations of 3.RDD are all evaluated lazily, and Spark will not start to calculate before the action operation is called.

4. Common conversion operations: map () and filter ()

4. Key-value pair operation

1.pair RDD (key-value pair RDD), Spark provides some proprietary operations

2.Spark programs can reduce communication overhead by controlling RDD partitioning, which is helpful only if the dataset is used multiple times in key-based operations such as connections

3. Use the partitioner () method to get the partition mode of RDD in Java

Many operations of 4.Spark introduce the process of shuffling data across nodes according to keys, which benefit from partitioning.

V. data reading and saving

1. When a text file is read as RDD, each line entered becomes an element of RDD, or multiple complete files can be read into a pair RDD at one time.

2.JSON data is read as a text file and then mapped to values in RDD using a JSON parser. You can also use a custom Hadoop format to manipulate JSON data in Java and Scala

3.SequenceFile is a commonly used Hadoop format consisting of key-value pairs without relative relationship structure, with synchronization marks that Spark can use to locate a point in the file and then align it with the boundary of the record.

VI. Advanced Spark programming

1. Accumulator: provides a simple syntax for aggregating values from a worker node into a driver program, often used to count events during job execution when debugging

two。 Broadcast variable: allows the program to efficiently send a large read-only value to all working nodes for use by one or more Spark operations

The pipe () method of 3.Spark allows us to implement part of the logic in Spark jobs in any language, as long as we can read and write Unix standard streams.

The numerical operation of 4.Spark is realized by streaming algorithm, which allows the model to be built one element at a time.

Run Spark on the cluster

1. In the distributed environment, the Spark cluster adopts the master / slave structure, the central coordination node is called the Driver node, and the working node is called the executor node, which can launch the Spark application on the machine in the cluster through the external service of the cluster manager (Cluster Manager).

two。 Driver program: turn the user program into a task; schedule tasks for the actuator node

3. Deploy using bin/spark-submit

4. You can use other cluster managers: Hadoop YARN, Apache Mesos, etc.

VIII. Spark tuning and debugging

1. Modify the run-time configuration options for Spark applications, using the SparkConf class

two。 Key performance considerations: parallelism, serialization format, memory management, hardware provisioning

IX. Spark SQL

1. Three major functions:

It is possible to read data from various structured data sources

Supports not only data queries using SQL statements in Spark programs, but also external tools that connect to Spark SQL through a standard database connector (JDBC/ODBC).

Supports high integration with regular Python/Java/Scala code, including RDD and SQL tables, exposed custom SQL function interfaces, etc.

two。 SchemaRDD is provided, which is the RDD for storing Row objects. Each Row object represents a row of records, so structural information can be used to store data more efficiently.

10. Spark Streaming

1.Spark Streaming: allows users to write streaming computing applications using a set of API that is very similar to batch processing, so that a large number of batch application technologies and even code can be reused

2.Spark Streaming uses a discretization stream (discretized stream) as an abstract representation, called DStream, which is a sequence of data received over time

Machine Learning based on MLlib

1.MLlib:Spark provides a library of machine learning functions designed for parallel operations on clusters, including many machine learning algorithms that represent data in the form of RDD and then call various algorithms on distributed datasets

two。 According to the training data (training data), the machine learning algorithm maximizes the mathematical goal of representing the behavior of the algorithm, and uses it to predict or make decisions, solving problems including classification, regression, clustering, etc.

The above is how to carry out Spark data analysis, the editor believes that there are some knowledge points that we may see or use in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.