Shulou(Shulou.com)06/01 Report--

This article mainly explains "how to speed up Spark by Redis". Interested friends may wish to have a look at it. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn "Redis how to speed up Spark"!

Apache Spark has gradually become a model for the next generation of big data processing tools. By borrowing from open source algorithms and distributing processing tasks to clusters of computing nodes, the generation of Spark and Hadoop easily outperforms traditional frameworks in terms of the type of data analysis they can perform on a single platform and the speed at which they perform these tasks. Spark uses memory to process data, so it is much faster (100x faster) than disk-based Hadoop.

But with a little help, Spark can run even faster. If you combine Spark with Redis (the popular in-memory data structure storage technology), you can once again greatly improve the performance of processing analysis tasks. This is due to Redis's optimized data structure and its ability to minimize complexity and overhead when performing operations. Speed can be further accelerated by accessing Redis data structures and API,Spark with connectors.

How much is the speed increase? If Redis is used in conjunction with Spark, it turns out that processing data (to analyze the time series data described below) is 45 times faster than Spark using process memory or out-of-heap cache alone to store data-not 45% faster, but a full 45 times faster!

Why is this important? Many companies increasingly need to analyze transactions as fast as business transactions themselves. More and more decisions are becoming automated, and the analysis needed to drive these decisions should be carried out in real time. Apache Spark is an excellent general-purpose data processing framework; although it is not real-time, it is a big step towards making data useful in a more timely manner.

Spark uses resilient distributed datasets (RDD), which can be stored in volatile memory or in persistent storage systems such as HDFS. The RDD does not change and is distributed on all nodes of the Spark cluster, which can be transformed to create other RDD.

Spark RDD

RDD is an important abstract object in Spark. They represent a fault-tolerant approach to efficiently presenting data to iterative processes. Because the processing is done in memory, this means that the processing time is several orders of magnitude shorter than using HDFS and MapReduce.

Redis is specifically designed for high performance. The sub-millisecond delay benefits from the optimized data structure, which improves efficiency by allowing operations to be performed adjacent to the data storage. This data structure can not only make efficient use of memory and reduce the complexity of applications, but also reduce network overhead, bandwidth consumption and processing time. Redis data structures include strings, collections, ordered collections, hashes, bitmaps, hyperloglog, and geospatial indexes. Developers can use Redis data structures like Lego bricks-they are simple conduits that provide complex functionality.

To visually demonstrate how this data structure simplifies the processing time and complexity of an application, we might as well take the ordered set (Sorted Set) data structure as an example. An ordered set is basically a group of members sorted by score.

Redis ordered set

You can store multiple types of data here, which are automatically sorted by scores. Common data types stored in ordered collections include: goods (by price), commodity names (by quantity), time series data such as stock prices, and sensor readings such as timestamps.

The charm of ordered collections lies in the built-in operation of Redis, which allows scope queries, multiple ordered collections to intersect, search by member levels and scores, and more transactions to be executed easily, with * * speed, and can be executed on a large scale. The built-in operation not only saves the code to be written, but also shortens the network delay and saves bandwidth when performing the operation in memory, so it can achieve high throughput with sub-millisecond delay. If ordered sets are used to analyze time series data, performance can usually be improved by several orders of magnitude compared to other memory key / value storage systems or disk-based databases.

The goal of the Redis team is to improve the analysis capabilities of Spark, for which Spark-Redis connectors are developed. This package allows Spark to use Redis as one of its data sources. This connector exposes the data structure of Redis to Spark, which can greatly improve performance for all types of analysis.

Spark Redis connector

To demonstrate the benefits to Spark, the Redis team decided to perform time slice (range) queries in several different scenarios to horizontally compare time series analysis in Spark. These scenarios include: Spark stores all data in in-heap memory, Spark uses Tachyon as the out-of-heap cache, Spark uses HDFS, and a combination of Spark and Redis.

The Redis team used Cloudera's Spark time series package to build a Spark-Redis time series package that uses Redis ordered sets to speed up time series analysis. In addition to giving Spark access to all of Redis's data structures, the package does two other things:

Automatically ensures that the Redis nodes are consistent with the Spark cluster, thus ensuring that each Spark node uses local Redis data, thus optimizing latency.

Integrate with Spark data frames and data source API to automatically convert Spark SQL queries into the kind of retrieval mechanism that is efficient for data in Redis.

In a nutshell, this means that users do not have to worry about operational consistency between Spark and Redis, and can continue to use Spark SQL for analysis, while greatly improving query performance.

The time series data used for this horizontal comparison include randomly generated financial data of 1024 stocks per day over a time range of 32 years. Each stock is represented by its own ordered collection, the score is the date, and the data members include the opening price, the * price, the closing price, the trading volume, and the adjusted closing price. The following figure describes the data representation in the Redis ordered collection for Spark analysis:

Spark Redis time series

In the above example, in the case of an ordered collection AAPL, there are scores representing each day (1989-01-01) and multiple values represented as a related row throughout the day. As long as you use a simple ZRANGEBYSCORE command in Redis, you can do this: get all the values of a time slice, and thus all the stock prices within the specified date range. Redis executes this type of query 100 times faster than other key / value storage systems.

This horizontal comparison confirms the improvement in performance. It is found that Spark uses Redis to execute timeslice queries 135 times faster than Spark using HDFS, and 45 times faster than Spark using in-heap memory or Spark using Tachyon as out-of-heap cache. The following figure shows the average execution time compared for different scenarios:

Spark Redis horizontal comparison

This guide will guide you step by step through the installation of typical Spark clusters and Spark-Redis packages. It also uses a simple word counting example to show how Spark and Redis can be used together. After you have tried the Spark and Spark-Redis packages, you can further explore more scenarios that take advantage of other Redis data structures.

Although ordered collections are suitable for time series data, other data structures of Redis, such as collections, lists, and geospatial indexes, can further enrich Spark analysis. Imagine this: a Spark process attempts to obtain information about the effectiveness of new product launches based on crowd preferences and proximity to the city center. Now imagine that built-in analysis of built-in data structures, such as geospatial indexes and collections, can greatly speed up this process. The combination of Spark-Redis has a bright future.

Spark supports a wide range of analyses, including SQL, machine learning, graphical computing, and Spark Streaming. Using the memory processing capabilities of Spark can only allow you to reach a certain scale. With Redis, however, you can go a step further: not only can you improve performance by taking advantage of Redis's data structures, but you can also more easily extend Spark, that is, by taking full advantage of the shared distributed data storage mechanism provided by Redis to handle millions or even billions of records.

The example of time series is just the beginning. The use of Redis data structures for machine learning and graphic analysis is also expected to significantly reduce execution time for these workloads.

At this point, I believe you have a deeper understanding of "Redis how to speed up Spark". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.