Shulou(Shulou.com)06/03 Report--

Create a table format in hive and store it in orc format

Create table user (id int,name string) stored as orc

Spark writes files

Val jsons = "hdfs://localhost:9000/test/artist_orc.json" val people = sc.textFile (jsons) val schemaString = "id name" val schema = StructType (schemaString.split (") .map (fieldName = > {if (fieldName = =" name ") StructField (fieldName, StringType, true) else StructField (fieldName, IntegerType True)) val rowRDD = people.map (line= > {JSONObject.fromObject (line)}) .map (p = > Row (new Integer (p.get ("id") .toString), p.get ("name")) val hiveContext = new org.apache.spark.sql.hive.HiveContext (sc) val peopleSchemaRDD = hiveContext.createDataFrame (rowRDD, schema) peopleSchemaRDD.write.format ("orc") .save ("hdfs://localhost:9000/user/xb/warehouse/artist_orc/adf")

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.