Shulou(Shulou.com)06/01 Report--

This article mainly introduces "Apache Beam2.23.0 updated what content", in daily operation, I believe many people in Apache Beam2.23.0 updated what content questions there are doubts, Xiaobian consulted all kinds of information, sorted out simple and easy to use operation methods, I hope to answer "Apache Beam2.23.0 updated what content" doubts helpful! Next, please follow the small series to learn together!

profile

Apache Beam 2.23.0 is now available. Apache Beam is a project contributed by Google to the Apache Foundation in February 2016. Its main goal is to unify the programming paradigm of batch processing and stream processing, and provide a simple, flexible, feature-rich and powerful SDK for infinite, out-of-order, web-scale dataset processing. Apache Beam focuses on the programming paradigm and interface definition of data processing, and does not involve the implementation of specific execution engines. Apache Beam hopes that data processing programs developed based on Beam can be executed on arbitrary distributed computing engines.

Main update: Highlights

Twister2 Runner（BEAM-7304）。

Python 3.8 support (BEAM-8494).

I/Os

Added support for Snowflake reading (Java)(BEAM-9722).

Added support for writing Splunk (Java)(BEAM-8596).

Added support for assume role (Java)(BEAM-10335).

A new transform readable from BigQuery has been added: apache_beam.io.gcp.bigquery.ReadFromBigQuery. This transformation is experimental. It reads data from BigQuery by exporting it to Avro files and reading those files. It also supports reading data by exporting to JSON files. Fields related to time and date differ slightly in behavior.

Add dispositions for SnowflakeIO.write (BEAM-10343)

New Features/Improvements

Update Snowflake JDBC dependency and add application=beam to connection URL (BEAM-10383).

Breaking Changes

RowJson.RowJsonDeserializer, JsonToRow, and PubsubJsonTableProvider now accept implicit nulls by default when deserializing JSON (Java). Previously nulls could only be represented by explicit null values, such as {"foo": "bar", "baz": null}, whereas implicit null values like {"foo": "bar"} raised exceptions. Both JSON strings now yield the same result by default. You can override this behavior with RowJson.RowJsonDeserializer#withNullBehavior.

Fixed a bug in Python GroupIntoBatches experimental conversion that was actually a keypress to group batches. This changes the output type of this conversion (BEAM-6696).

Deprecations

Remove the Gearpump runner.（BEAM-9999）

Remove the Apex Runner.（BEAM-9999）

RedisIO.readAll() is deprecated and will be removed in 2 versions, users must use RedisIO.readKeyPatterns() instead (BEAM-9747).

At this point, the study of "What is updated in Apache Beam 2.23.0" is over, hoping to solve everyone's doubts. Theory and practice can better match to help everyone learn, go and try it! If you want to continue learning more relevant knowledge, please continue to pay attention to the website, Xiaobian will continue to strive to bring more practical articles for everyone!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.