Shulou(Shulou.com)06/02 Report--

This article mainly introduces "how to use Spark SQL". In daily operation, I believe many people have doubts about how to use Spark SQL. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful to answer the doubts about "how to use Spark SQL". Next, please follow the editor to study!

Pom.xml

Org.apache.spark

Spark-core_2.10

2.1.0

Org.apache.spark

Spark-sql_2.10

2.1.0

Java:

Import java.io.Serializable

Import java.util.Arrays

Import org.apache.spark.SparkConf

Import org.apache.spark.api.java.JavaRDD

Import org.apache.spark.api.java.JavaSparkContext

Import org.apache.spark.sql.Dataset

Import org.apache.spark.sql.Row

Import org.apache.spark.sql.SQLContext

Import org.apache.spark.sql.SparkSession

Public class SparkSqlTest {

Public static class Person implements Serializable {

Private static final long serialVersionUID =-6259413972682177507L

Private String name

Private int age

Public Person (String name, int age) {

This.name = name

This.age = age

}

Public String toString () {

Return name + ":" + age

}

Public String getName () {

Return name

}

Public void setName (String name) {

This.name = name

}

Public int getAge () {

Return age

}

Public void setAge (int age) {

This.age = age

}

}

Public static void main (String [] args) {

SparkConf conf = new SparkConf () .setAppName ("Test") .setMaster ("local")

JavaSparkContext sc = new JavaSparkContext (conf)

SparkSession spark = SparkSession.builder () .appName ("Test") .getOrCreate ()

JavaRDD input = sc.parallelize (Arrays.asList ("abc,1", "test,2"))

JavaRDD persons = input.map (s-> s.split (",") .map (s-> new Person (s [0], Integer.parseInt (s [1])

/ / [abc: 1, test: 2]

System.out.println (persons.collect ())

Dataset df = spark.createDataFrame (persons, Person.class)

/ *

+-+

| | age | name |

+-+

| | 1 | abc |

| | 2 | test |

+-+

, /

Df.show ()

/ *

Root

|-- age: integer (nullable = false)

|-- name: string (nullable = true)

, /

Df.printSchema ()

SQLContext sql = new SQLContext (spark)

Sql.registerDataFrameAsTable (df, "person")

/ *

+-+

| | age | name |

+-+

| | 2 | test |

+-+

, /

Sql.sql ("SELECT * FROM person WHERE age > 1") .show ()

Sc.close ()

}

}

At this point, the study on "how to use Spark SQL" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.