How to use Spark SQL
This article mainly introduces "how to use Spark SQL". In daily operation, I believe many people have doubts about how to use Spark SQL. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful to answer the doubts about "how to use Spark SQL". Next, please follow the editor to study!
Pom.xml
Org.apache.spark
Spark-core_2.10
2.1.0
Org.apache.spark
Spark-sql_2.10
2.1.0
Java:
Import java.io.Serializable
Import java.util.Arrays
Import org.apache.spark.SparkConf
Import org.apache.spark.api.java.JavaRDD
Import org.apache.spark.api.java.JavaSparkContext
Import org.apache.spark.sql.Dataset
Import org.apache.spark.sql.Row
Import org.apache.spark.sql.SQLContext
Import org.apache.spark.sql.SparkSession
Public class SparkSqlTest {
Public static class Person implements Serializable {
Private static final long serialVersionUID =-6259413972682177507L
Private String name
Private int age
Public Person (String name, int age) {
This.name = name
This.age = age
}
Public String toString () {
Return name + ":" + age
}
Public String getName () {
Return name
}
Public void setName (String name) {
This.name = name
}
Public int getAge () {
Return age
}
Public void setAge (int age) {
This.age = age
}
}
Public static void main (String [] args) {
SparkConf conf = new SparkConf () .setAppName ("Test") .setMaster ("local")
JavaSparkContext sc = new JavaSparkContext (conf)
SparkSession spark = SparkSession.builder () .appName ("Test") .getOrCreate ()
JavaRDD input = sc.parallelize (Arrays.asList ("abc,1", "test,2"))
JavaRDD persons = input.map (s-> s.split (",") .map (s-> new Person (s [0], Integer.parseInt (s [1])
/ / [abc: 1, test: 2]
System.out.println (persons.collect ())
Dataset df = spark.createDataFrame (persons, Person.class)
/ *
+-+
| | age | name |
+-+
| | 1 | abc |
| | 2 | test |
+-+
, /
Df.show ()
/ *
Root
|-- age: integer (nullable = false)
|-- name: string (nullable = true)
, /
Df.printSchema ()
SQLContext sql = new SQLContext (spark)
Sql.registerDataFrameAsTable (df, "person")
/ *
+-+
| | age | name |
+-+
| | 2 | test |
+-+
, /
Sql.sql ("SELECT * FROM person WHERE age > 1") .show ()
Sc.close ()
}
}
At this point, the study on "how to use Spark SQL" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!
