Shulou(Shulou.com)05/31 Report--

This article introduces how to develop spark2.0 dataFrame based on eclipse. The content is very detailed. Interested friends can use it for reference. I hope it will be helpful to you.

I was going to study the spark-sql of spark2.0 a few days ago.

My main tool is eclipse

The version is hadoop2.7+spark2.0

Cut the crap and go straight to the code wordcount

Import org.apache.spark.sql.SparkSession

Import org.apache.spark._

Import org.apache.spark.SparkContext._

Import scala.io.Source

/ * *

* Created by Administrator on 2016-7-29 0029.

, /

Object wordcount2 {

Def main (args: Array [String]) {

Val sparksession=SparkSession.builder ()

.master ("local")

.getOrCreate ()

Import sparksession.implicits._

/ / val ds=sparksession.read.textFile ("hdfs://192.168.153.11:9000/user/spark/a.txt"). As [String]

/ / val ds=sparksession.read.text ("file:/D:/a.txt"). As [String]

/ / var ds = sparksession.read.format ("text") .load ("file:/D:/a.txt") .as [String]

/ / val ds=sparksession.sparkContext.textFile (args (0)) .toDS ()

/ / val result=ds.flatMap (_ .split (")) .filter (_ .length > 0) .groupBy () .count ()

Val warehouseLocation = "file:/" / / "hdfs://192.168.21.8:9000/" / / "file:/"

Val sparkSession1 = SparkSession.builder (). Master ("local"). AppName ("tst"). Config ("spark.sql.warehouse.dir", warehouseLocation). GetOrCreate ()

Var ds = sparksession.read.format ("text"). Load ("file:/D:/a.txt"). As [String]

Val counts = ds.flatMap (line = > line.split (""))

Val words = counts.groupByKey (_ .toLowerCase ())

Val datas = words.count ()

Datas.show ()

/ / result.show ()

/ / the main point of attention is var ds = sparksession.read.format ("text"). Load ("file:/D:/a.txt"). As [String]

/ / pay attention to the path of load here

}

}

On the development of spark2.0 based on eclipse dataFrame is how to share here, I hope that the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.