Shulou(Shulou.com)06/01 Report--

Today, I will talk to you about the basic knowledge of message queuing Kafka and what the .NET Core client is like. Many people may not know much about it. In order to make you understand better, the editor summarized the following. I hope you can get something from this article.

Preface

Message queuing is used for message transmission in the latest project. Kafka was chosen to cooperate with other java projects, so I learned about Kafka and took notes.

I won't talk about the differences between Kafka and some other message queues, including performance and how they are used.

Brief introduction

Kafka is a service that implements distributed, partitioned, and replicated logs. It provides the function of message system middleware through a set of unique design. It is a messaging system with publish and subscribe functions.

Some nouns

If you are using Kafka, there are some nouns you need to know in Kafka, and the text does not discuss whether these nouns have the same meaning in other message queues. All nouns refer to Kafka.

Message

A message, which is the content to be sent, is generally packaged as a message object.

Topic

In popular terms, it is the place where the "message" is placed, that is, a container for message delivery. If you think of a message as an envelope, then Topic is a mailbox.

Partition & & Log

Partition partition can be understood as a logical partition, like the disk of our computer.

Kafka maintains a log Log file for each partition.

Each partition is an ordered, immutable queue of messages. When the message comes, it is appended to the log file, which is performed according to the commit command.

Each message in the partition has a number called offset id, and this id is unique and incremental in the current partition.

Log is used to record the messages received in the partition, because each Topic can deliver messages to one or more partitions at the same time, so in fact, when storing logs, each partition will correspond to a log directory, and its naming rule is generally -, which is a commit log log file of a partition.

The Kafka cluster stores all published information over a period of time, which can be configured regardless of whether the message has been consumed or not. For example, if the log storage period is set to 2 days, all messages published within 2 days can be consumed, while the previous messages will be discarded in order to free up space. The performance of Kafka has nothing to do with the amount of data, so saving a large amount of message data does not cause performance problems.

The log is partitioned for the following purposes: first, this allows log to scale more than a single server to go online, and the size of each independent partition is limited by the volume of a single server, but a topic can have a lot of partition so that it can handle data of any size. Second, it can be used as a separate unit in parallel processing.

Producer Producers

Like other message queues, the producer is usually the producer of the message.

In Kafka, it determines the partition on which the message is sent to the specified Topic.

Consumer Consumers

The consumer is the user of the message, and there are several nouns that need to be distinguished on the consumer side.

In general, message queues have two modes of consumption, namely, queue mode and subscription mode.

Queue mode: one-to-one, that is, a message can only be consumed by one consumer and cannot be consumed repeatedly. In general, queues support multiple consumers, but for a message, only one consumer can consume it.

Subscription model: one to many, a message may be consumed multiple times, and the message producer publishes the message to Topic, as long as consumers who subscribe to Topic can consume it.

Consumer & & Subscriber

Group: a group is a collection of consumers. Each group has one or more consumers. In Kafka, messages can only be consumed once within a group.

In the publish and subscribe model, consumers subscribe in groups, namely Consumer Group, and their relationship is shown below:

Every message published on Topic is delivered to a consumer in each consumer group that subscribes to the Topic, that is, each group is delivered, but only one consumer in each group consumes the message.

At the beginning, it is introduced that Kafka is a message queue with publish-subscribe function, so in Kafka, the queue pattern is implemented through a single consumer group, that is, there is only one consumer group in the whole structure, and the load balance between consumers.

Kafka cluster

Borker: the Kafka cluster consists of multiple servers, each of which is called a Broker. Messages of the same Topic are stored on different Broker according to certain key and algorithm.

The above image is quoted from: http://blog.csdn.net/lizhitao

Because Kafka's cluster is realized by distributing partitions to each Server, that is to say, each server in the cluster shares partition data and requests with each other, and the log files of each partition are copied into specified scores and scattered across cluster machines to achieve failover.

For each partition, there will be one server as its "leader" and zero or more servers as the "followers". The leader server handles all read and write requests about the partition, while the followers server passively replicates the leader server. If a leader server fails, then one of the followers servers will be automatically elected as the new leader. Each server acts as the leader of some partition as well as the follower of other servers, thus realizing the load balancing of the cluster.

.net Core Kafka client

In the .NET Core, there is a corresponding open source kafka sdk project, Rdkafka. It supports both .NET 4.5, cross-platform support, and can run on both Linux,macOS and Windows.

RdKafka Github: https://github.com/ah-/rdkafka-dotnet

RdKafka Nuget: Install-Package RdKafka

Producer API// Producer accepts one or more BrokerList

Using (Producer producer = new Producer ("127.0.0.1 new Producer 9092") / / is sent to a Topic named testtopic, and if not, a u is created.

Sing (Topic topic = producer.Topic ("testtopic")) {

/ / convert message to a byte [] byte [] data = Encoding.UTF8.GetBytes ("Hello RdKafka"); DeliveryReport deliveryReport = await topic.Produce (data); Console.WriteLine ($"send to partition: {deliveryReport.Partition}, Offset is: {deliveryReport.Offset}");} Consumer API

Because Kafka is consumed as a consumer group, you need to specify a GroupId.

In the internal implementation, consumers monitor Topic messages through a polling mechanism, which is also recommended by Kafka. The polling interval in Rdkafka is 1 second.

/ / configure consumer groups

Var config = new Config ()

{

GroupId = "example-csharp-consumer"}

Using (var consumer = new EventConsumer (config, "127.0.0.1))

{/ / Register an event consumer.OnMessage + = (obj, msg) = > {

String text = Encoding.UTF8.GetString (msg.Payload, 0, msg.Payload.Length); Console.WriteLine ($"Topic: {msg.Topic} Partition: {msg.Partition} Offset: {msg.Offset} {text}");}; / subscribe to one or more Topic consumer.Subscribe (new [] {"testtopic"}); / / launch consumer.Start (); Console.WriteLine ("Started consumer, press enter to stop consuming") Console.ReadLine ();} after reading the above, do you have any further understanding of the basics of message queuing Kafka and what the .NET Core client is? If you want to know more knowledge or related content, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.