Shulou(Shulou.com)06/01 Report--

What are the 20 knowledge points of Kafka? in view of this question, this article introduces the corresponding analysis and solution in detail, hoping to help more partners who want to solve this problem to find a more simple and feasible way.

3 key functions of 1.kafka?

Publish and subscribe record streams, similar to message queues or enterprise messaging systems.

The record stream is stored in a fault-tolerant and persistent manner.

Process the record stream.

2.kafka is usually used for two major types of applications.

Establish a real-time streaming data pipeline to reliably obtain data between systems or applications

Build a real-time streaming application to transform or respond to data streams

3.kafka features?

Message persistence

High throughput

Expansibility

Multi-client support

Kafka Streams

Security mechanism

Data backup

Lightweight

Message compression

5 core Api of 4.kafka?

Producer API

Consumer API

Streams API

Connector API

Admin API

5. What is Broker (Agent)?

In a Kafka cluster, an kafka instance is called an Broker node.

6. What is Producer (producer)?

The producer of the message is called Producer. Producer sends messages to the topic specified by the cluster for storage, and a custom algorithm determines which partition to send the message record to?

7. What is Consumer (Consumer)?

The consumer of the message that reads the message from the topic specified in the kafka cluster.

8. What is Topic (theme)?

Topic, kafka records messages of different business types through different topics.

9. What is Partition (partition)?

Each Topic can have one or more Partition.

10. What is a Replication?

Each topic is required to have its number of copies when it is created (default 1).

11. What is a Record?

Data that is actually written to the kafka cluster and can be read by consumers. Each record contains a key, value, and timestamp.

What scenarios is 12.kafka suitable for?

Log collection, message system, activity tracking, operational indicators, streaming processing, time source, etc.

13.kafka disk selection?

The performance of SSD is better than that of ordinary disk, as we all know, in practice, we can use ordinary disk. It mostly uses sequential read and write operations, which to some extent avoids the biggest disadvantage of mechanical disks, that is, random read and write operations are slow, so SSD does not have much advantage.

14. What are the advantages of using RAID?

Provide redundant disk storage space

Provide load balancing

15. How many factors need to be taken into account in disk capacity planning?

Number of new messages

Message retention time

Average message size

Number of backups

Whether compression is enabled

16.Broker uses a single? Multiple file directory path parameters?

Log.dirs multiple log.dir single

17. Generally speaking, which parameter is selected to configure the path? Benefits?

Log.dirs benefits: improve read and write performance, high throughput for multiple physical disks at the same time. Failover. One disk hung up and transferred to another.

18. What are the relevant parameters for automatically creating a theme?

Auto.create.topics.enable

19. Resolve kafka message loss?

Instead of using producer.send (msg), use producer.send (msg, callback).

Set acks = all.

Set retries to a larger value.

Set unclean.leader.election.enable = false.

Set replication.factor > = 3.

Set min.insync.replicas > 1.

Make sure replication.factor > min.insync.replicas.

Ensure that the message consumption is completed before submitting.

20.kafka message repetition problem?

Be idempotent. Database aspects (unique keys and primary keys) can avoid duplication. Exercise control over the business.

The answers to the 20 questions about Kafka are shared here. I hope the above content can be of some help to everyone. If you still have a lot of doubts to be solved, you can follow the industry information channel to learn more about it.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.