Shulou(Shulou.com)05/31 Report--

This article will explain in detail the example analysis of Spark SQL programming for you. The editor thinks it is very practical, so I share it with you for reference. I hope you can get something after reading this article.

# Spark SQL programming Guide #

# # introduction # # Spark SQL supports the execution of relational query expressions of SQL or HiveQL in Spark. Its core component is a new RDD type JavaSchemaRDD. The JavaSchemaRDD consists of a Row object and a schema that represents the data type of each column of the row. An JavaSchemaRDD is similar to a table in a traditional relational database. JavaSchemaRDD can be created from an existing RDD,Parquet file, JSON dataset, or by running HiveSQL to get the data stored on the Apache Hive.

Spark SQL is currently an alpha component. Although we will try to minimize API changes, some API can be changed later in the release.

# # getting started # # in Spark, the entry point for all relational functions is the JavaSQLContext class. Or his subclass. To create a basic JavaSQLContext, all you need is a JavaSparkContext.

JavaSparkContext sc =...; / / An existing JavaSparkContext.JavaSQLContext sqlContext = new org.apache.spark.sql.api.java.JavaSQLContext (sc)

# # data sources # # Spark SQL supports the operation of various data sources through the JavaSchemaRDD interface. A single dataset is loaded, and it can be registered as a table or even connected to data from other sources.

# one of the types of tables supported by RDDs### Spark SQL is RDD of JavaBeans. BeanInfo defines the schema of this table. Currently, Spark SQL does not support JavaBeans that includes nested or complex types such as Lists or Arrays. You can create a JavaBeans by creating a class that implements Serializable and all its fields have getters and setters methods.

Public static class Person implements Serializable {private String name; private int age; public String getName () {return name;} public void setName (String name) {this.name = name;} public int getAge () {return age;} public void setAge (int age) {this.age = age;}}

A schema can be applied to an existing RDD by calling applySchema and providing the class object of the JavaBean.

/ / sc is an existing JavaSparkContext.JavaSQLContext sqlContext = new org.apache.spark.sql.api.java.JavaSQLContext (sc) / / Load a text file and convert each line to a JavaBean.JavaRDD people = sc.textFile ("examples/src/main/resources/people.txt"). Map (new Function () {public Person call (String line) throws Exception {String [] parts = line.split (","); Person person = new Person (); person.setName (parts [0]) Person.setAge (Integer.parseInt (parts [1]. Trim ()); return person;}}); / / Apply a schema to an RDD of JavaBeans and register it as a table.JavaSchemaRDD schemaPeople = sqlContext.applySchema (people, Person.class); schemaPeople.registerAsTable ("people"); / / SQL can be run over RDDs that have been registered as tables.JavaSchemaRDD teenagers = sqlContext.sql ("SELECT name FROM people WHERE age > = 13 AND age = 13 AND age = 13 AND age)

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.