Shulou(Shulou.com)06/01 Report--

This article is to share with you about the nature of Spark and how to use Spark for data analysis, the editor thinks it is very practical, so I share it with you to learn. I hope you can get something after reading this article.

1. What is Apache Spark?

Apache Spark is a cluster computing platform designed for speed and general goals.

From a speed perspective, Spark inherits from the popular MapReduce model and can more effectively support many types of computation, such as interactive queries and stream processing. Speed is very important in the processing of large data sets, which can determine whether users can process the data interactively or wait a few minutes or even hours. An important feature of Spark for speed is that it can run calculations in memory, and even for complex disk-based applications, Spark is still more efficient than MapReduce.

In terms of generality, Spark can handle tasks that previously require multiple independent distributed systems, including batch applications, interactive algorithms, interactive queries, and data streams. By supporting these tasks with the same engine, Spark makes it easy to merge different processing types, which are frequently used in production data analysis. Moreover, Spark reduces the administrative burden of maintaining different tools.

Spark is designed to be highly accessible, providing simple API in Python, Java, Scala, and SQL, as well as rich built-in libraries. Spark is also integrated with other big data tools. In particular, Spark can run on Hadoop clusters and can access any Hadoop data source, including Cassandra.

two。 A unified stack

The Spark project contains several tightly integrated components. As its core, Spark is a "computing engine" responsible for scheduling, distributing, and monitoring applications composed of computing tasks between multiple working machines or on a computing cluster. The Spark core engine is fast and versatile and can drive different components for a wide variety of loads, such as SQL or machine learning. These components can interact closely so that you can combine them in a software project like a library program.

The tightly coupled approach has many benefits. All libraries and high-level components in the stack can benefit from improvements in lower-level components. For example, when the core engine of Spark is optimized, SQL and machine learning libraries accelerate automatically. Second, the overhead of running the stack is minimized, because there is no need to run 5-10 independent software systems, only one is sufficient. These operating costs include deployment, maintenance, testing, support, and other operations. This means that whenever a new component is added to the Spark stack, the team using Spark can immediately try the new component. In the past, it took downloading, deployment, and learning to try a new data analysis software, but now you only need to upgrade Spark.

Finally, one of the benefits of a tightly coupled approach is the ability to build applications that can constantly merge different processing models. For example, through Spark, you can write an application that uses machine learning to classify data in real time according to the input data stream; at the same time, analysts can query the result data in real time through SQL. Moreover, more data engineers and data scientists can use Pyhton Shell to access data for advertising analysis. Other people may access data in separate batch applications. From beginning to end, the IT team only needs to maintain one system.

Here we will briefly introduce each component of Spark, as shown in figure 1-1

Figure 1-1 Spark stack

3. Spark core components

The core components of Spark include the basic functions of Spark, including task scheduling components, memory management components, fault-tolerant recovery components, components that interact with the storage system, and so on. The core components of Spark provide API for defining resilient distributed datasets (resilient distributed datasets,RDDs), a set of API that is the main programming abstraction of Spark. RDDs represents a collection of data distributed on many different machine nodes and can be processed in parallel. The Spark core component provides a number of API to create and manipulate these collections.

Spark SQL

Spark SQL is the package that Spark uses to process structured data. It makes it possible to query data through SQL statements like the Hive query language (Hive Query Language, HQL), supporting a variety of data sources, including Hive tables, Parquet, and JSON. In addition to providing a SQL interface for Spark, Spark SQL allows developers to combine SQL queries and data programming operations supported by RDDs through Python, Java, and Scala into a single application, combining SQL with complex analysis. Tight integration with computing-intensive environments makes Spark SQL different from any other open source data warehouse tool. Spark SQL introduced Spark in Spark version 1. 0.

Shark is an older SQL project on Spark developed by the University of California and the University of Berkeley that runs on Spark by modifying Hive. It has now been replaced by Spark SQL to provide better integration with the Spark engine and API.

Spark stream (Spark Streaming)

As a component of Spark, Spark stream can handle real-time stream data. An example of streaming data is a log file generated by a Web server in a production environment, and a user requests a Web service to include a status update message. Spark streams provide an API for manipulating data streams that closely match the Spark core RDD API, making it easier for programmers to understand the project and quickly switch between applications that operate memory data, disk data, and real-time data. Spark streams are designed to provide the same level of fault tolerance, throughput, and scalability as Spark core components.

MLlib

Spark contains a library about machine learning called MLlib. MLlib provides many types of machine learning algorithms, including classification, regression, clustering and collaborative filtering, and supports model evaluation and data import functions. MLlib also provides a low-level machine learning primitive, including a general gradient descent optimization algorithm. All of these methods can be applied to a cluster.

GraphX

GraphX is a library for operating diagrams (such as social network friend diagrams) and performing graph-based parallel computing. Similar to spark streams and Spark SQL, GraphX extends Spark RDD API, allowing us to create a directed graph with any attributes bound to each node and edge. GraphX also provides a variety of operators for operating diagrams, as well as a library of general graph algorithms.

Cluster Manager Cluster Managers

At the bottom, Spark can effectively scale from one computing node to hundreds of nodes. To achieve this goal with flexibility, Spark can run on multiple cluster managers, including Hadoop YARN,Apache Mesos and a simple cluster manager called a stand-alone scheduler included in Spark. If you install Spark on an empty cluster, the stand-alone scheduler provides an easy way; if you already have a Hadoop YARN or Mesos cluster, Spark supports your application to allow on these cluster managers. Chapter 7 gives different choices and how to choose the right cluster manager.

4. Who uses Spark? What do you do with Spark?

Because Spark is a general framework for cluster computing, it can be used in many different applications. In the preface, we point out two kinds of readers of the book: data scientists and data engineers. Let's take a closer look at these two types of people and the way they use Spark. Obviously, the typical use cases are different, but we can roughly divide them into two categories, data science and data applications.

Of course, this is an imprecise classification and usage pattern, and many people have two skills at the same time, sometimes playing the role of a data mining scientist, and then writing a data processing application. Nevertheless, it makes sense to be divided into two groups and their use cases.

The task of data science

Data science, a discipline that has emerged in recent years, focuses on analyzing data. Although there is no standard definition, we believe that the main job of a data scientist is to analyze and model data. Data scientists may SQL, statistics, predictive models (machine learning), and program in Python, MATLAB, or R. Data scientists can format data for further analysis.

Data scientists use relevant techniques to analyze data in order to answer a question or conduct in-depth research. Usually, their work involves special analysis, so they use interactive shell so that they can see query results and code snippets in the shortest possible time. Spark's speed and simple API interface are well suited to this goal, and its built-in library means that many algorithms can be used at any time.

Spark supports different data science tasks through several components. Spark shell makes it easy to analyze interactive data with Python or Scala. Spark SQL also has a separate SQL shell that can be used for data analysis with SQL, or Spark SQL can be used in Spark programs or in Spark shell. The MLlib library supports machine learning and data analysis. Moreover, it supports calling programs written in external MATLAB or R languages. Spark allows data scientists to use tools such as R or Pandas to deal with problems that contain large amounts of data.

Sometimes, after the initial data processing phase, the work of data scientists will be productised, expanded, reinforced (fault-tolerant), and then become a production data processing application as a component of commercial applications. For example, the research of a data scientist may produce a product recommendation system that is integrated into a web application to generate product recommendations to users. The work of a data scientist is usually productised by another person, such as an engineer.

Data processing application

Another major use of Spark can be described from an engineer's point of view. In this case, engineers refer to a large number of software developers who use Spark to build production data processing applications. These developers understand the concepts and principles of software engineering, such as packaging, interface design, and object-oriented programming. They usually have degrees in computer science. They use their software engineering skills to design and build software systems that implement a business usage scenario.

For engineers, Spark provides a simple way to parallelize these applications between clusters, hiding the complexity of distributed systems, network communication, and fault-tolerant processing. The system makes engineers have sufficient authority to monitor, check and adjust the application while realizing the task. API's modular features make it easy to reuse existing work and local tests.

Spark users use Spark as their data processing application because it provides rich features, easy to learn and use, and mature and reliable.

5. Introduction to the history of Spark

Spark is an open source project maintained by several different developer communities. If you or your team use Spark for many times, you may be interested in its history. Spark was created by UC Berkeley RAD Lab (now AMP Lab) as a research project in 2009. Lab researchers, who previously worked on Hadoop MapReduce, found that MapReduce was inefficient for iterative and interactive computing tasks. Therefore, in the initial stage, Spark is mainly designed for interactive queries and iterative algorithms, supporting memory storage and efficient fault-tolerant recovery.

Shortly after the creation of Spark in 2009, academic articles about Spark were published, and in some specific tasks, Spark can be 10-20 times faster than MapReduce.

Some of the Spark users are other groups in UC Berkeley, including machine learning researchers, such as the Mobile Millennium project team, which uses Spark to monitor and predict traffic congestion in the San Francisco Bay area. In a very short period of time, many external organizations began to use Spark. Now, more than 50 organizations are using Spark, and some organizations have announced their use in Spark communities such as Spark Meetups and Spark Summit. The main contributors to Spark are Databricks, Yahoo and Intel.

In 2011, AMP Labs began to develop upper-level components on Spark, such as Shark and Spark streams. All of these components are sometimes called the Berkeley data Analysis Stack (Berkeley Data Analytics Stack,BDAS).

Spark opened source in March 2010, moved to the Apache Software Foundation in June 2014, and is now its project.

6. Spark version and release

Since its inception, Spark has been a very active project and community, and the number of contributors to each release has grown. Spark 1. 0 has more than 100 contributors. Despite the rapid growth in the level of activity, the community still releases an updated version of Spark with a fixed plan. Spark 1.0 was released in May 2014. This book is mainly based on Spark 1.1.0, but its concepts and examples also apply to earlier versions.

7. Spark storage tier

Spark can create distributed datasets from any file stored in the Hadoop distributed File system (HDFS), or from other Hadoop API-supported storage systems, such as local file systems, Amazon S3, Cassandra, Hive,HBase, and so on. It is important to keep in mind that Hadoop is not necessary for Spark, and Spark can support any storage system that implements Hadoop API. Spark supports text files, sequence files, Avro, Parquet, and any other Hadoop input format.

The above is the nature of Spark and how to use Spark for data analysis. The editor believes that there are some knowledge points that we may see or use in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.