Shulou(Shulou.com)06/02 Report--

Today, I will talk to you about the application scenarios of kafka and message queuing, which may not be well understood by many people. in order to make you understand better, the editor has summarized the following content for you. I hope you can get something from this article.

Interprocess communication

The purpose of process communication

Data transmission

One process needs to send its data to another process, and the amount of data sent is between one byte and several megabytes.

Share data

Multiple processes want to manipulate shared data, and one process wants to share data

Notice

One process needs to send a message to another or group of processes informing it that something has happened (such as notifying the parent process when the process terminates).

Resource sharing

The same resources are shared among multiple processes. To do this, the kernel is required to provide locking and synchronization mechanisms.

Process control

Some processes want to have complete control over the execution of another process (such as the Debug process). At this time, the control process wants to be able to intercept all traps and exceptions of another process and know its state change in time.

Piping (pipe)

Pipeline is one of the most basic IPC mechanism, which acts between consanguineous processes to complete data transmission. One process needs to send its data to another process, and the amount of data sent is between one byte and a few megabytes.

Semaphore (semophore)

A semaphore is a counter that can be used to control the access of multiple processes to shared resources. It is often used as a locking mechanism to prevent other processes from accessing a shared resource when it is being accessed. Therefore, it is mainly used as a means of synchronization between processes and between different threads within the same process.

Message queuing (message queue)

Message queuing is a linked list of messages, including Posix message queuing system V message queuing. Processes with sufficient permissions can add messages to the queue, and processes that are given read permission can read away messages from the queue. Message queue overcomes the shortcomings that the signal carries less information, the pipeline can only carry unformatted byte flow and the buffer size is limited.

Signal (singal)

A signal is a more complex form of communication that notifies the receiving process that an event has occurred.

It is mainly used as a means of synchronization between processes and between different threads of the same process.

Shared memory (shared memory)

Shared memory is to map a piece of memory that can be accessed by other processes, which is created by one process, but can be accessed by multiple processes. Shared memory is the fastest IPC mode, and it is specially designed for other inter-process communication modes to run inefficiently. It is often used in conjunction with other communication mechanisms, such as semaphores, to achieve synchronization and communication between processes.

Allows multiple processes to access the same memory space, which is the fastest available IPC form. It is designed for the low efficiency of other communication mechanisms. It is often used in conjunction with other communication mechanisms, such as semaphores, to achieve synchronization and mutual exclusion between processes.

Socket (socket)

Socket is also an inter-process communication mechanism. Unlike other communication mechanisms, it can be used for process communication between different machines.

A more general inter-process communication mechanism can be used for inter-process communication between different machines.

Originally developed by the BSD branch of the Unix system, it is now generally portable to other types of Unix systems: both Linux and System V variants support sockets.

Usage scenarios of message queuing

Asynchronous processing

Scenario description: after registration, users need to send registration email and registration SMS. Common treatment methods: 1. The way of synchronization; 2. Parallel mode; 3. Asynchronous mode

(1) Serial mode: after successfully writing the registration information to the database, send the registration email, and then send the registration text message. When all the above three tasks are completed, return to the client

(2) parallel mode: after the registration information is successfully written into the database, the registration email is sent and the registration SMS is sent at the same time. After the above three tasks are completed, return to the client. The difference with serial is that parallel approach can improve processing time.

Scenario description: after registration, users need to send registration email and registration SMS. Common treatment methods: 1. Serial mode; 2. Parallel mode; 3. The introduction of message queuing in an asynchronous way will not be a necessary business logic for asynchronous processing. The modified structure is as follows:

Application decoupling

Scenario description: after the user places an order, the order system needs to notify the inventory system. Traditionally, the order system calls the interface of the inventory system. The figure below is as follows

Disadvantages of the traditional model:

If the inventory system is not accessible, the order reduction will fail, resulting in order failure

Coupling of order system and inventory system

Scenario description: after the user places an order, the order system needs to notify the inventory system. Message queue is introduced to decouple the system. The figure below is as follows

Order system: after the user places an order, the order system completes the persistence processing, writes the message to the message queue, and returns the user's order to be issued successfully.

Inventory system: subscribe to the order message and use the pull / push method to obtain the order information. The inventory system carries out inventory operation according to the order information.

If: the inventory system does not work properly when placing an order. It also does not affect the normal issuance of the order, because after the order is issued, the order system writes to the message queue and no longer cares about other subsequent operations. Realize the application decoupling of order system and inventory system

Flow peaking

Application scenario: flash sale activity, usually because the traffic is too large, the traffic will surge and the application will fail. In order to solve this problem, it is generally necessary to join the message queue at the front end of the application.

The number of people who can control the activity

It can relieve the application of high flow collapse in a short time.

After receiving the user's request, the server first writes to the message queue. If the message queue length exceeds the maximum number, abandon the user request directly or jump to the error page

The second kill service does subsequent processing according to the request information in the message queue.

Log processing

Application scenario: log processing refers to the use of message queues in log processing, such as the application of Kafka, to solve the problem of mass log transmission. The architecture is simplified as follows

The log collection client is responsible for log data collection and regularly writes to the Kafka queue

Kafka message queue, which is responsible for receiving, storing and forwarding log data

Log processing applications: subscribe to and consume log data in kafka queues

Message communication

Message communication means that message queues generally have built-in efficient communication mechanisms, so they can also be used in pure message communication. Such as implementing peer-to-peer message queues or chat rooms

1, peer-to-peer communication: client An and client B use the same queue for message communication.

2, chat room communication: client A, client B, client N subscribe to the same topic, publish and receive messages. Achieve a chat room-like effect.

After reading the above, do you have any further understanding of the application scenario of kafka and message queuing? If you want to know more knowledge or related content, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.