Shulou(Shulou.com)06/01 Report--

This post is about what Spark2.4.0 does. Xiaobian thinks it is quite practical, so share it with everyone for reference. Let's follow Xiaobian and have a look.

SparkCore and SQL

Added Barrier ExecutionMode for better integration with deep learning frameworks

30+ built-in functions and higher-order functions are also introduced to handle more complex data types

Improved integration with k8s

Support for Scala 2.12

Built-in support of the Avro format data source, this feeling is good, after the wave gives test cases, after the pb support that is better.

performance and stability

This optimization is more and more worthy of attention.

Connector optimization

This optimization is mainly Parquet, orc, csv and avro optimization upgrade

MLlib

MLlib supports image format data sources

StructuredStreaming

Use foreachBatch (Python, Scala, and Java supported) to expose the output rows of each microbatch as DataFrames.

Foreach and ForeachWriter added to Python API

Support for reading committed messages produced to kafka topic by producers using transactions using "kafka.isolation.level".

Thank you for reading! About "Spark2.4.0 has what function" this article is shared here, I hope the above content can have some help to everyone, so that everyone can learn more knowledge, if you think the article is good, you can share it to let more people see it!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.