Shulou(Shulou.com)06/01 Report--

This article mainly shows you "what is Kafka in CDP", the content is easy to understand, clear, hope to help you solve your doubts, the following let the editor lead you to study and learn "what is Kafka in CDP" this article.

Apache Kafka is a high performance, high availability, redundant streaming message platform.

Introduction to Kafka

Kafka functions much like a publish / subscribe messaging system, but with higher throughput, built-in partitioning, replication, and fault tolerance. Kafka is a good solution for large-scale message processing applications. It is usually used with Apache Hadoop and Spark Streaming.

You might think of logs as files or data tables sorted by time. Over time, newer entries are appended to the log from left to right. The log entry number can easily replace the timestamp.

Kafka integrates this unique abstraction with traditional publish / subscribe messaging concepts such as producers, consumers, and brokers, parallelism, and enterprise capabilities to improve performance and fault tolerance.

The initial use case of Kafka is to track user behavior on a Web site. Site activities (page browsing, search, or other actions that users may perform) are posted to the central theme, one for each type of activity.

Kafka can be used to monitor operational data and aggregate statistics from distributed applications to generate centralized data feeds. It is also suitable for log aggregation, with low latency and convenient support for multiple data sources.

Kafka provides the following:

Persistent message delivery with O (1) disk structure, which means that the execution time of the Kafka algorithm is independent of the size of the input. The execution time is constant, even if messages with several TB are stored.

High throughput, even with moderate hardware, can support hundreds of thousands of messages per second.

Message partitioning through the Kafka server is explicitly supported. It allocates consumption on the consumer cluster while maintaining the order of message flows.

Support for loading parallel data into Hadoop.

Kafka architecture

Learn about the architecture of Kafka and its comparison with the ideal publish-subscribe system.

The ideal publish-subscribe system is simple: the message from publisher A must reach subscriber A, the message from publisher B must reach subscriber B, and so on.

Figure 1. Ideal publish-subscribe system

The ideal system has the following advantages:

Infinite backtracking. New subscriber A1 can read the stream of publisher An at any point in time.

The message is reserved. No messages are missing.

Unlimited storage space. The publish-subscribe system has unlimited message storage.

No downtime. The publish-subscribe system never crashes.

Unlimited expansion. Publish-subscribe systems can handle any number of publishers and / or subscribers with constant message delivery delays.

However, the architecture of Kafka deviates from this ideal system. Some of the main differences are:

Messaging is implemented on top of replicated distributed commit logs.

The client has more functionality and, therefore, more responsibility.

Messaging is optimized for batch processing rather than for individual messages.

Messages are retained even if they are consumed; they can be used again.

The results of these design decisions are:

Extremely high level of scalability

Extremely high throughput

High availability

Different semantics and message passing guarantees

Kafka terminology

Kafka uses its own terminology when it comes to basic building blocks and key concepts. The usage of these terms may be different from other technologies. The following provides a list and definition of the most important concepts of Kafka:

Broker: an agent is a server that stores messages sent to the topic and serves consumer requests.

Topic: a topic is a message queue written by one or more producers and read by one or more consumers.

Producer: the producer is the external process of sending records to Kafka topics.

Consumer (consumer): the consumer is an external process that receives the topic stream from the Kafka cluster.

Client (client): a client is a term for producers and consumers.

Record: a record is a publish-subscribe message. The record consists of key / value pairs and metadata containing timestamps.

Partition: Kafka divides the record into multiple partitions. You can think of partitions as a subset of all records for a topic.

The above is all the content of the article "what is Kafka in CDP?" Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.