Shulou(Shulou.com)05/31 Report--

In this issue, the editor will bring you an example analysis of the performance improvement of Kafka2.6.0. The article is rich in content and analyzed and described from a professional point of view. I hope you can get something after reading this article.

The Kafka2.6 version was released recently, less than four months after the 2.5.0 release.

Kafka 2.6.0 contains many important new features. The following is a summary of some important changes:

TLSv1.3 is enabled by default for Java 11 or later

Performance is significantly improved, especially when agents have a large number of partitions

It is easier to extend the application of Kafka Streams

Issued when Kafka Streams supports changes

New metrics provides better operational insight

Kafka Connect can automatically create a Topic when configured to connect

Improved error reporting options for receiver connectors in Kafka Connect

New filters in Kafka Connect and conditional application of SMT

The default value for "client.dns.lookup" configuration is now "use_all_dns_ips".

Upgrade Zookeeper to 3.5.8

New function

Add KStream#repartition Action

Make SSL context / engine configuration extensible

TLSv1.3 is enabled by default and some older protocols are disabled

Conditional application of SMT

Add task-level activity process ratio to flow metrics

Restructure the main loop to process multiple records of one task at a time

Improve

Enhanced TransformerSupplier / ProcessorSupplier

Clean up task management

Merge "onAssignment" stream with "partitionsAssigned" task creation

Open disk read and write metrics

Allow consumers to explicitly trigger rebalancing

Update gradle to 6.0 +

Support for Java 14

Switch the default version to Scala 2.13

-improve the performance of "matchingAcls"

Console producer supports setting of client.id

Upgrade Guide:

If you are upgrading from a version prior to 2.1.x, refer to the following notes for changes to the schema used to store consumer offsets. After you change inter.broker.protocol.version to the latest version, you cannot downgrade to a version prior to 2.1.

For rolling upgrades:

Update server.properties on all agents and add the following properties. CURRENT_KAFKA_VERSION refers to the version you are upgrading. CURRENT_MESSAGE_FORMAT_VERSION refers to the version of the message format currently in use. If the message format version was previously overwritten, its current value should be retained. Or, if you are upgrading from a version prior to 0.11.0.x, you should set CURRENT_MESSAGE_FORMAT_VERSION to match CURRENT_KAFKA_VERSION.

Inter.broker.protocol.version = CURRENT_KAFKA_VERSION (e. G. 2.5, 2.4, etc.)

Log.message.format.version = CURRENT_MESSAGE_FORMAT_VERSION

If you are upgrading from 0.11.0.x or later and the message format has not been overwritten, you only need to overwrite the inter-agent protocol version.

Inter.broker.protocol.version = CURRENT_KAFKA_VERSION (e. G. 2.5, 2.4, etc.)

Upgrade one agent at a time: shut down the agent, update the code, and then restart. When this is done, the agent will run the latest version, and you can verify that the cluster behaves and performs as expected. If you have any problems, you can still downgrade at this time.

After verifying the behavior and performance of the cluster, change the protocol version 2.6 by editing the protocol version inter.broker.protocol.version and setting it to.

Restart the agents one by one for the new protocol version to take effect. After the agent starts using the latest version of the protocol, it will no longer be able to downgrade the cluster to an older version.

If you have overwritten the message format version as described above, you need to scroll and restart again to upgrade it to the latest version. Once all (or most) users have upgraded to version 0.11.0 or later, change log.message.format.version to 2.6 on each agent, and then restart them one by one. Note that older Scala clients that are no longer maintained do not support the message format introduced in 0.11, and newer Java clients must be used to avoid conversion costs.

2.6.0 pay attention

Kafka Streams adds a new processing mode (requires Broker 2.5 or later) that uses a full one-time guarantee to improve the scalability of the application.

TLSv1.3 is enabled by default for Java 11 or later. If both the client and server support TLSv1.3, the protocol will be negotiated, otherwise it will fall back to TLSv1.2.

TLSv1.3 is enabled by default for Java 11 or later. If both the client and server support TLSv1.3, the protocol will be negotiated, otherwise it will fall back to TLSv1.2.

NotLeaderForPartitionException has been deprecated and replaced with NotLeaderOrFollowerException. If the agent is not a copy, get requests and other requests for leaders or followers only will return NOT_LEADER_OR_FOLLOWER (6) instead of REPLICA_NOT_AVAILABLE (9) to ensure that this temporary error during reassignment is handled by all clients as retryable exceptions.

The above is the example of Kafka2.6.0 performance improvement shared by Xiaobian. If you happen to have similar doubts, you might as well refer to the above analysis to understand. If you want to know more about it, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.