Shulou(Shulou.com)06/03 Report--

Blog outline:

I. Overview of Kafka

1) message queuing

2) Why use message queuing?

3) what is Kafka?

4) characteristics of Kafka

5) Kafka architecture

6) the difference between Topic and Partition

7) kafka flow chart

8) File storage mechanism of Kafka

9) guarantee of reliability and persistence of data

10) leader election

2. Deploy stand-alone Kafka

1) deploy Kafka

2) Test Kafka

Deploy Kafka clusters

1) Environmental preparation

2) deploy zookeeper cluster

3) deploy Kafka cluster

I. Overview of Kafka 1) message queuing

1) peer-to-peer mode (one-to-one, the consumer actively pulls the data, and the message is cleared after the message is received)

The peer-to-peer model is usually a messaging model based on pull or polling, which requests information from the queue rather than pushing the message to the client. This model is characterized by the fact that messages sent to the queue are received and processed by one and only one receiver, even if there are multiple message listeners

2) publish / subscribe model (one-to-many, push to all subscribers after data production)

The publish-subscribe model is a push-based messaging model. The publish-subscribe model can have many different subscribers, temporary subscribers receive messages only when actively listening to topics, while persistent subscribers listen for all messages on topics, even if the current subscriber is unavailable and offline.

2) Why use message queuing? 1) decoupling

Allows you to extend or modify processes on both sides independently, as long as you make sure they follow the same interface constraints.

2) redundancy

Message queues persist data until they have been fully processed, avoiding the risk of data loss. In the insert-get-delete paradigm used by many message queues, before deleting a message from the queue, your processing system needs to clearly indicate that the message has been processed to ensure that your data is safely saved until you have finished using it.

3) scalability

Because message queues decouple your processing, it is easy to increase the frequency of queuing and processing of messages, as long as you add additional processing.

4) flexibility & & cutting peak and filling valley

Applications still need to continue to play a role in the face of a surge in traffic, but such bursts of traffic are not common. It is undoubtedly a huge waste to invest resources at any time in order to be able to handle this kind of peak access. The use of message queuing enables key components to withstand sudden access pressure without completely collapsing due to sudden overloaded requests.

5) recoverability

When some components of the system fail, the whole system will not be affected. Message queuing reduces the coupling between processes, so even if a process that processes messages dies, messages added to the queue can still be processed after the system is restored.

6) sequence guarantee

In most usage scenarios, the order of data processing is important. Most message queues are sorted by nature, and it is guaranteed that the data will be processed in a specific order. (Kafka guarantees the ordering of messages in a Partition)

7) buffer

It is helpful to control and optimize the speed of data flow through the system and solve the inconsistency between the processing speed of production messages and consumer messages.

8) Asynchronous communication

In many cases, users do not want or need to process messages immediately. Message queuing provides an asynchronous processing mechanism that allows users to put a message on the queue, but does not process it immediately. Put as many messages as you want in the queue, and then process them when needed.

3) what is Kafka?

Kafka is an open source stream processing platform released by the Apache Software Foundation and written by Scala and Java. It is a high-throughput distributed publishing subscription messaging system that can handle all action flow data in consumer-scale websites. This action (web browsing, search and other user actions) is a key factor in many social functions on the modern web. This data is usually resolved by processing logs and log aggregations due to throughput requirements. For log data and offline analysis systems like Hadoop, but requiring real-time processing limitations, this is a feasible solution. The purpose of Kafka is to unify online and offline message processing through Hadoop's parallel loading mechanism, as well as to provide real-time messages through clusters.

4) characteristics of Kafka

Kafka is a high-throughput distributed publish and subscribe messaging system with the following characteristics:

1) provide message persistence through disk data structure, which can maintain stable performance for a long time even for message storage with TB.

2) persistence: use file storage, log files to store messages, need to write to the hard disk, and use a certain threshold to write to the hard disk, so as to reduce disk I and O, if the kafka suddenly goes down, part of the data will be lost.

3) High throughput: even very ordinary hardware kafka can support millions of messages per second

4) support partitioning messages through kafka servers and consumer machine clusters

5) support Hadoop parallel data loading

5) Kafka architecture

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.