What is it like to develop spark2.0 dataFrame based on eclipse
This article introduces how to develop spark2.0 dataFrame based on eclipse. The content is very detailed. Interested friends can use it for reference. I hope it will be helpful to you.
I was going to study the spark-sql of spark2.0 a few days ago.
My main tool is eclipse
The version is hadoop2.7+spark2.0
Cut the crap and go straight to the code wordcount
Import org.apache.spark.sql.SparkSession
Import org.apache.spark._
Import org.apache.spark.SparkContext._
Import scala.io.Source
/ * *
* Created by Administrator on 2016-7-29 0029.
, /
Object wordcount2 {
Def main (args: Array [String]) {
Val sparksession=SparkSession.builder ()
.master ("local")
.getOrCreate ()
Import sparksession.implicits._
/ / val ds=sparksession.read.textFile ("hdfs://192.168.153.11:9000/user/spark/a.txt"). As [String]
/ / val ds=sparksession.read.text ("file:/D:/a.txt"). As [String]
/ / var ds = sparksession.read.format ("text") .load ("file:/D:/a.txt") .as [String]
/ / val ds=sparksession.sparkContext.textFile (args (0)) .toDS ()
/ / val result=ds.flatMap (_ .split (")) .filter (_ .length > 0) .groupBy () .count ()
Val warehouseLocation = "file:/" / / "hdfs://192.168.21.8:9000/" / / "file:/"
Val sparkSession1 = SparkSession.builder (). Master ("local"). AppName ("tst"). Config ("spark.sql.warehouse.dir", warehouseLocation). GetOrCreate ()
Var ds = sparksession.read.format ("text"). Load ("file:/D:/a.txt"). As [String]
Val counts = ds.flatMap (line = > line.split (""))
Val words = counts.groupByKey (_ .toLowerCase ())
Val datas = words.count ()
Datas.show ()
/ / result.show ()
/ / the main point of attention is var ds = sparksession.read.format ("text"). Load ("file:/D:/a.txt"). As [String]
/ / pay attention to the path of load here
}
}
On the development of spark2.0 based on eclipse dataFrame is how to share here, I hope that the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.
