Shulou(Shulou.com)06/02 Report--

This article mainly explains "how to understand the use and features of Kafka". The content of the explanation is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "how to understand the use and features of Kafka".

1. Interviewer: can you briefly describe the more important keywords in the Kafka architecture?

For example, Partition,Broker, how do you understand it?

Problem analysis: Kafka basic knowledge investigation, because of the excellent performance of Kafka, in the cluster structure is also different, some new conceptual design names may not be understood by beginners, to build a Kafka cluster first need to understand the concept of topics, producers, consumers, and brokers.

A: I have done some in-depth understanding of Kafka, and its design ideas are worth using for reference, among which there are six key name concepts. Understanding these concepts can better understand the working mechanism of Kafka.

Producer

The producer of the message, such as the payment system confirms that the user has paid, the payment system notifies the order system and the logistics system, and the payment system is the producer.

Consumer

The recipient of consumption, in the case of Producer, the logistics system is the consumer. The first two are relatively simple, so I won't say much about them.

Topic

Each message posted to the MQ cluster has a category called topic, which can be understood as the name of a class of messages. All messages have been categorized as units by topic.

Partition

Kafka is physically the concept of partitioning in which each Topic is dispersed over one or more Partition. When the data of a Topic is too large, it is divided into small pieces. Kafka introduces a multi-copy model for the partition, and the design of "one leader multi-follower" is adopted between replicas. Automatic failover is realized through multiple copies to ensure availability.

Broker:

Can be understood as a node of a server, and the cluster contains one or more servers, which are called broker. For applications, the producer sends out the consumption and doesn't care. Consumers spend slowly at their own rate. There may be a lot of news during this period, and consumer pressure is still within a certain range. The decoupling between producers and consumers is a caching service broker.

Kafka Cluster

A cluster is a collection of Broker, and multiple Broker form a highly available cluster.

The relationship between Producer and Consumer

The relationship between topic and Partition

A topic can be stored in multiple Partition, each Partition ordered.

The interviewer didn't interrupt me here. I went on.

So why did we choose Kafka?

1. Here we no longer list the functions of similar products, but directly summarize the practical information and the unique features of Kafka:

two。 Compared with the similar middleware RabbitMQ or ActiveMQ,Kafka, which supports batch pulling messages, it greatly increases the message throughput of Kafka.

Multiple delivery scenarios are supported:

1. Send and forget.

two。 Send synchronously.

3. Asynchronously send + callback function.

Although the three methods are different in time, it does not mean that the faster the time, the better. Which method to use depends on the specific business scenario. For example, the business requires that messages must be sent sequentially, can be sent synchronously using the second method, and can only be sent on one partation. If the business is only concerned with the throughput of messages, allows a small number of messages to fail, and does not care about the order in which messages are sent, then send and forget can be used. If the business needs to know whether the message was sent successfully and does not care about the order of the message, it can send the message in the way of async + callback

3. Distributed and highly scalable. The Kafka cluster can be expanded transparently, adding new servers to the cluster.

Only talking about the advantages of Kafka, then other similar products are not good? Of course not, existence is the truth, each product can survive, must have its own advantages, such as RabbitMQ, slightly inferior to Kafka in terms of throughput, but their starting point is different, RabbitMQ supports the reliable delivery of messages, supports transactions, does not support batch operations, technical selection, choose the most suitable for you, you know the most familiar.

Distributed high-performance persistence and scalability support multi-partition high-throughput data persistence support multi-copy low latency fault tolerance high support for multi-subscribers high level of concurrency online expansion based on ZooKeeper scheduling time complexity O (1) message distribution automatic balancing

After talking about how fast Kafka is compared to other products, I finally succeeded in digging a hole for myself. (?), followed my train of thought and asked.

Interviewer: then why is the throughput of Kafka much higher than that of other similar middleware?

Problem analysis: many years of experience summary, the biggest loss in the interview is that you are not familiar with things written on your resume, and that you know the results, do not know the principle, the source code has not been read, at least know the use of clever design.

A: Kafka is a high-throughput distributed messaging system and provides persistence. Its high performance has two important features:

1. Taking advantage of the fact that the performance of continuous disk reading and writing is much higher than that of random reading and writing, the internal batch processing of messages, zero-copy mechanism, data storage and acquisition are local disk sequential batch operations, with O (1) complexity, and the message processing efficiency is very high.

two。 Concurrently, split a topic into multiple partition, and the unit of kafka read and write is partition, so splitting a topic into multiple partition can improve throughput. However, there is a premise that different partition needs to be on different disks (which can be on the same machine). If multiple partition are located on the same disk, it means that multiple processes read and write multiple files on one disk at the same time, which makes the operating system schedule disk reads and writes frequently, that is, it destroys the continuity of disk reads and writes.

In the linkedlin test, each machine loaded 6 disks, and did not do ra, in order to make full use of multi-disk concurrent read and write, but also to ensure that each disk continuous read and write characteristics.

The same topic will be distributed to multiple shards and processed in parallel.

In-depth analysis of Demo, the production and consumption model of Kafka messages

Pseudo code: send a message using the KafKa client

Public class MqProducer {private final Logger LOG = LoggerFactory.getLogger (MqProducer.class); @ Resource private Producer payProducer; public void sendPayMsg (String msg) {try {LOG.debug ("send msg: {}", msg); payProducer.send (msg); / / send a message. } catch (MQException e) {LOG.error ("mq message exception message: {}", msg, e);}

What does it look like?

That is, the value of msg in payProducer.send (msg):

{"businessType": 1, "cityId": 10, "ctime": 1567426767077, "dataKey": 20190902, "logType": 1, "phone": "13212341234", "uid": 12345678, "userType": 1, "uuid": "32EA02C86D78863"}

Regardless of the length of the message, it can be regarded as a json string, and the message is transmitted in the form of key-value.

Pseudo code: receives a message

Public class DemoConsumer {/ * Note: the server has a limit on the number of consumer instances with the same topic and queue created by a single ip, and more than 100 refuse to create. * * / private static IConsumerProcessor consumer; public static void main (String [] args) throws Exception {Properties properties = new Properties (); properties.setProperty (ConsumerConstants.SubscribeGroup, "dache.risk.log.queue.v2"); / / create a consumer object corresponding to topic (note that a new instance is generated for each build call) consumer = KafkaClient.buildConsumerFactory (properties, "topic.xxx.xxx") / / call recvMessageWithParallel to set listener consumer.recvMessageWithParallel (String.class, new IMessageListener () {@ Override public ConsumeStatus recvMessage (Message message, MessagetContext context) {/ / TODO: consumption logic code try {System.out.println ("message= [" + message.getBody () + "] partition=" + message.getParttion ()) on the business side } catch (Exception e) {e.printStackTrace ();} return ConsumeStatus.CONSUME_SUCCESS;}});}} Appendix: message management tools

If you have just set up a Kafka cluster and do not have a complete page management system, you might as well learn about these open source tools and show your leaders their ability to solve problems.

In order to simplify the work of developers and service engineers in maintaining Kafka clusters, page-based management tools are essential.

Common Kafka open source management tools:

Kafka Manager: developed by the yahoo team.

Thank you for your reading, the above is the content of "how to understand the use and characteristics of Kafka". After the study of this article, I believe you have a deeper understanding of how to understand the use and characteristics of Kafka. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.