How to use spark-redis components to access cloud database Redis

This article focuses on "how to use spark-redis components to access cloud database Redis", interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn how to use spark-redis components to access cloud database Redis.

Create a service

Let's take the EMR-3.21.0 version and Redis 4.0 as examples. The version of Spark installed in the EMR cluster is 2.4.3, and we need to use the corresponding version 2.4 of Spark-Redis, which can support version 2.9.0 of Redis.

EMR and Redis need to be created in the same VPC network. At the same time, after the cloud database Redis instance is launched, the EMR cluster IP address needs to be added to the "whitelist setting" (see Redis Quick start document, https://help.aliyun.com/document_detail/107043.html).

Start Spark Shell

Next, we log in to the EMR Master node to start Spark Shell. If the Master node can connect to the public network, you can use package to load spark-redis-related jar packages:

Spark-shell-packages com.redislabs:spark-redis:2.4.0\-conf spark.redis.host=hostname\-conf spark.redis.port=6379\-conf spark.redis.auth=password

Parameters such as spark.redis.host can be specified on the command line, configured in spark-defaults.conf, or specified in code. Where:

Spark.redis.host:Redis private network connection address

Spark.redis.port:Redis service port number

Spark.redis.auth: the password specified when creating the Redis instance

You can also specify the dependent jar package by-- jars:

Spark-shell-- jars spark-redis-2.4.0.jar,jedis-3.1.0-m1.jar,commons-pool2-2.0.jar\-- conf spark.redis.host=hostname\-- conf spark.redis.port=6379\-- conf spark.redis.auth=password writes data to Redis (RDD) scala > import com.redislabs.provider.redis._ import com.redislabs.provider.redis._ through Spark

Scala > val data = Array (("key1", "v1"), ("key2", "world"), ("key3", "hello"), ("key4", "Hong"), ("key5", "Kong") data: Array [(String, String)] = Array ((key1,v1), (key2,world), (key3,hello), (key4,Hong), (key5,Kong)

Scala > val distData = sc.parallelize (data) distData: org.apache.spark.rdd.RDD [(String, String)] = ParallelCollectionRDD [0] at parallelize at: 29

Scala > sc.toRedisKV (distData) read Redis (RDD) scala > val stringRDD = sc.fromRedisKV ("key*"). Map {kv = > kv._2} stringRDD: org.apache.spark.rdd.RDD [String] = MapPartitionsRDD [3] at map at: 27

Scala > val values = stringRDD.collect () values: Array [String] = Array (world, hello, v1, Kong, Hong)

Scala > println (values.mkString (",") world,hello,v1,Kong,HongSpark DataFrame writes Redisscala > case class Person (name: String, age: Int) defined class Person

Scala > val personSeq = Seq (Person ("John", 30), Person ("Peter", 45) personSeq: Seq [Person] = List (Person (John,30), Person (Peter,45))

Scala > val df = spark.createDataFrame (personSeq) df: org.apache.spark.sql.DataFrame = [name: string, age: int]

Scala > df.write.format ("org.apache.spark.sql.redis"). Option ("table", "person"). Save () so far, I believe you have a deeper understanding of "how to use spark-redis components to access cloud database Redis". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!