Shulou(Shulou.com)05/31 Report--

This article mainly introduces "what are the operating methods of window and groupBy". In the daily operation, I believe that many people have doubts about what the operating methods of window and groupBy are. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful to answer the doubts about "what are the operating methods of window and groupBy?" Next, please follow the editor to study!

Window operation import spark.implicits._val words =. / / streaming DataFrame of schema {timestamp: Timestamp, word: String} / / Group the data by window and word and compute the count of each groupval windowedCounts = words.groupBy (window ($"timestamp", "10 minutes", "5 minutes"), $"word") .count ()

Window operation is a continuous flow-specific operation, setting the size of the time window, performing groupBy operations according to the window size, and so on.

Look at the groupBy operation on dataset.

GroupBy operation

Definition:

Def groupBy (cols: Column*): RelationalGroupedDataset = {

RelationalGroupedDataset (toDF (), cols.map (_ .expr), RelationalGroupedDataset.GroupByType)

}

Generate a new RelationalGroupedDataset object. The most important method for this object:

Private [this] def toDF (aggExprs: Seq [Expression]): DataFrame = {val aggregates = if (df.sparkSession.sessionState.conf.dataFrameRetainGroupColumns) {groupingExprs + + aggExprs} else {aggExprs} val aliasedAgg = aggregates.map (alias) groupType match {case RelationalGroupedDataset.GroupByType = > Dataset.ofRows (df.sparkSession, Aggregate (groupingExprs, aliasedAgg, df.logicalPlan)) case RelationalGroupedDataset.RollupType = > Dataset.ofRows (df.sparkSession Aggregate (Seq (Rollup (groupingExprs)), aliasedAgg, df.logicalPlan) case RelationalGroupedDataset.CubeType = > Dataset.ofRows (df.sparkSession, Aggregate (Seq (Cube (groupingExprs)), aliasedAgg, df.logicalPlan) case RelationalGroupedDataset.PivotType (pivotCol, values) = > val aliasedGrps = groupingExprs.map (alias) Dataset.ofRows (df.sparkSession, Pivot (Some (aliasedGrps), pivotCol, values, aggExprs, df.logicalPlan)}}

Let's take a look at one:

Dataset.ofRows (df.sparkSession, Aggregate (groupingExprs, aliasedAgg, df.logicalPlan))

See what its implementation mechanism is?

Aggregate here is a kind of LogicPlan, so we only need to look at the implementation mechanism of Aggregate.

The implementation mechanism of Aggregate is about to involve the related classes in the catalyst package.

At this point, the study on "what are the operating methods of window and groupBy" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.