Shulou(Shulou.com)05/31 Report--

This article mainly explains "what are the new features of Spark3.0". Interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Let's let Xiaobian take you to learn "What are the new features of Spark3.0"!

The main new features of Spark 3.0 are as follows:

Compared to Spark2.4, the performance is improved by 2 times, mainly in adaptive query execution, dynamic partition pruning and so on.

Pandas API changes, including Python type hints and UDF functions.

PySpark exception handling has been enhanced.

New Structured Streaming UI page.

A lot of Jira problems have been solved.

Structured Streaming was originally introduced in Spark 2.0 and stopped updating SparkStreaming. It is obvious that Structured Streaming appeared in order to compete with Flink, which supports better watermarking, windows, etc. in the field of real-time computing.

3.0 Version adds a special UI for Structured Streaming, which makes it easy to view the execution information of stream jobs.

Although Structured Streaming still has a long way to go compared to Flink, you can expect continued enhancements to Structured Streaming in Spark 3.0.

At this point, I believe that everyone has a deeper understanding of "what are the new features of Spark3.0". Let's actually operate it! Here is the website, more related content can enter the relevant channels for inquiry, pay attention to us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.