Shulou(Shulou.com)06/02 Report--

This article is from the Apache Spark Meetup held in Adobe Systems Inc on September 19, 2018.

The upcoming version 2.4 of Apache Spark is the fifth version of the 2.x series. This article gives an overview of the main features and enhancements of Apache Spark 2.4.

The new scheduling model (Barrier Scheduling) enables users to properly embed distributed deep learning training into the stage of Spark to simplify the distributed training workflow. 35 higher-order functions have been added to manipulate the array / map in Spark SQL. Add a new native AVRO data source for Databricks-based spark-avro modules. PySpark also introduces an eager evaluation model (eager evaluation mode) for all operations of teaching and debugability. Spark on K8S supports PySpark and R, and supports client mode (client-mode). Various enhancements to Structured Streaming. For example, stateful operators in continuous processing. Various performance improvements to built-in data sources. For example, Parquet nested mode pruning (schema pruning). Scala 2.12 is supported.

Click on the demo website to download this PPT.

Summary

The above are the new features of Apache Spark 2018 introduced by the editor to you. I hope it will be helpful to you. If you have any questions, please leave me a message and the editor will reply to you in time. Thank you very much for your support to the website!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.