Shulou(Shulou.com)05/31 Report--

This article introduces the relevant knowledge of "what are the characteristics of storm Transactional spouts". In the operation of practical cases, many people will encounter such a dilemma. Then let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

Trident processes tuple in small batches (batch), and each batch is assigned a unique transaction id. Different spout have different characteristics, and a transactionalspout will have the following features:

1. Batch with the same txid must be the same. When a batch corresponding to a txid is replayed, the same tuples in the previous batch of the corresponding txid must be replayed.

2. There is no intersection between each batch. Each tuple can only belong to one batch

3. Every tuple belongs to a batch, without exception

This is a very easy-to-understand type of spout where tuple streams are divided into fixed batch and never change. Trident-kafka has an implementation of transactional spout. )

You might ask: why don't we always use transactional spout? It's easy to understand. One reason is that not all places need fault tolerance. For example, the way TransactionalTridentKafkaSpout works is that a batch contains tuple from all the partition in a kafka topic. Once the batch is issued, any time the batch is reissued, it must contain all the original tuple to satisfy the semantics of the transactional spout. Now let's assume that a batch is issued by TransactionalTridentKafkaSpout, the batch is not processed successfully, and at the same time a node of the kafka is down. You will not be able to replay an exact batch as before (because the node down of the kakfa is lost, part of the partition of the topic may not be available), and the whole processing will be interrupted.

This is why "opaque transactional" spouts (opaque transaction spout) exist-they are fault-tolerant for the loss of source nodes and can still help you achieve semantics that are processed only once. This kind of spout will be introduced later.

Before we talk about "opaque transactional" spout, let's take a look at how to design a State implementation with exactly-once semantics for transactional spout. The type of this State is "transactionalstate" and it takes advantage of the semantics that any txid always corresponds to the same tuple sequence.

Suppose you have a topology that calculates the number of occurrences of words, and you want to store the number of occurrences of words in the database as key/value pairs. Key is the word, and value is the number of times the word appears. You have seen that just storing a quantity is not enough to know if you have processed a batch. You can store value and txid together in the database. That way, before updating the count, you can compare the txid stored in the database with the txid you want to store now. If so, skip doing nothing, because the value has been dealt with before. If it is different, the storage is performed. The premise that this logic works is that txid never changes, and Trident ensures that state updates are done in a strict sequence between batch.

Considering the running logic of the following example, suppose you are dealing with a batch with a txid of 3 that contains the following tuple:

["man"]

["man"]

["dog"]

Assume that the following key/value pairs are currently saved in the database:

Man = > [count=3, txid=1]

Dog = > [count=4, txid=3]

Apple = > [count=10, txid=2]

The txid corresponding to the word "man" is 1. Because the current txid is 3, you can be sure that you haven't updated the number of words for the tuple in this batch. So you can safely add 2 to count and update txid to 3. At the same time, the txid of the word "dog" is the same as the current txid, so you can skip this update. At this point, the data in the database is as follows:

Man = > [count=5, txid=3]

Dog = > [count=4, txid=3]

Apple = > [count=10, txid=2]

This is the end of the content of "what are the features of storm Transactional spouts"? thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.