Shulou(Shulou.com)05/31 Report--

Today, I would like to talk to you about which of Redis, Kafka or RabbitMQ is better suited to micro-services. Many people may not know much about it. In order to make you understand better, the editor has summarized the following for you. I hope you can get something from this article.

When asynchronous communication is used for micro services, message brokers are usually used. The agent ensures reliable and stable communication between different microservices, that messages are managed and monitored within the system, and that messages are not lost. You can choose some message agents that vary in size and data capabilities. This blog post will compare the three most popular brokers: RabbitMQ,Kafka and Redis.

But first, let's take a look at microservice communications.

Microservice communication: synchronous and asynchronous

There are two common ways of communication between microservices: synchronous and asynchronous. In synchronous communication, the caller waits for a response before sending the next message, and it runs as the REST protocol over HTTP. In contrast, in asynchronous communication, messages can be sent without waiting for a response. This applies to distributed systems, where message agents are usually required to manage messages.

The type of communication you choose should take into account different parameters, such as the structure of the microservice, appropriate infrastructure, latency, size, dependencies, and communication purpose. The establishment of asynchronous communication may be more complex and more components need to be added to stack, but the benefits of using asynchronous communication for micro services far outweigh the disadvantages.

Advantages of asynchronous communication

First, by definition, asynchronous communication is non-blocking. It also supports better scaling than synchronization. Third, in the case of a microservice crash, asynchronous communication mechanisms provide a variety of recovery techniques that are generally better at dealing with crash-related errors. In addition, when using a proxy instead of the REST protocol, the services that receive the communication do not actually need to know each other. After the old service has been running for a long time, you can even introduce a new service, that is, a better decoupled service.

Finally, when choosing asynchronous operations, you will enhance your ability to create centralized discovery, monitoring, load balancing, and even policy executors in the future. This will provide you with flexibility, scalability, and more functionality in code and system building.

Select the appropriate message agent

Asynchronous communication is usually managed through a message broker. There are other methods, such as aysncio, but they are rarer and more limited.

When choosing an agent to perform asynchronous operations, you should consider the following:

Agent size-the number of messages sent per second in the system. Data persistence-the ability to recover messages. Consumer ability-whether brokers have the ability to manage one-to-one and / or one-to-many consumers.

one-for-one

One to many

We checked the latest and best services there to find the strongest providers in the three categories.

RabbitMQ (AMQP)

Scale: depending on the configuration and resources, the operating speed here is about 50K msg per second.

Persistence: support for persistent messages and instant messages.

One-to-one and one-to-many consumers: both.

RabbitMQ was released in 2007 and is one of the first common message agents to be created. It is an open source code that delivers messages through point-to-point and pub-sub methods by implementing the Advanced message queuing Protocol (AMQP). It is designed to support complex routing logic.

There are some managed services that allow you to use it as a SaaS, but it is not part of the stack of major native cloud providers. RabbitMQ supports all major languages, including Python,Java,.NET,PHP,Ruby,JavaScript,Go,Swift, etc.

In persistent mode, you may encounter some performance problems.

Kafka

Size: up to 1 million messages can be sent per second.

Persistence: yes.

One-to-one vs one-to-many consumers: only one-to-many (strange at first glance, right?! ).

Kafka was created by Linkedin in 2011 to handle high-throughput, low-latency processing. As a distributed streaming platform, Kafka replicates a publish-subscribe service. It provides data persistence and stores record streams, enabling it to exchange high-quality messages.

Kafka used to manage SaaS on Azure,AWS and Confluent. They are both creators and major contributors to the Kafka project. Kafka supports all major languages, including Python,Java,C / C + +, Clojure,.NET,PHP,Ruby,JavaScript,Go,Swift, etc.

Redis

Size: up to 1 million messages can be sent per second.

Persistence: basically not, it's a data store in memory.

One-to-one and one-to-many consumers: both.

Redis is a little different from other message agents. The core of Redis is an in-memory data store that can be used as a high-performance key-value store or a message broker. Another difference is that Redis does not have persistence, but dumps its memory into Disk / DB. It is also very suitable for real-time data processing.

Initially, Redis is not one-to-one and one-to-many. However, with the introduction of pub-sub in Redis 5.0, the functionality has been enhanced and one-to-many has become a real choice.

Message broker for each use case

We introduced some features of RabbitMQ,Kafka and Redis. All three animals fall into their category, but as mentioned above, they operate in very different ways. This is our suggestion that the correct message broker should be used according to different use cases.

Short-lived message: Redis

Redis's in-memory database is almost suitable for short-lived use cases of messages that do not require persistence. Because Redis provides very fast service and memory capabilities, it is ideal for short retention messages, where persistence is not very important, and you can tolerate some loss. With the release of the pub-sub stream in 5.0, it has also become a candidate for one-to-many use cases, which is absolutely needed because of its limitations and old Redis capabilities.

Massive data: Kafka

Kafka is a high-throughput distributed queue for storing large amounts of data for long periods of time. Kafka is ideal for one-to-many use cases that require persistence.

Complex routing: RabbitMQ

RabbitMQ is an older but mature agent with many features that support complex routing. When the required rate is not high (more than tens of thousands of msg / sec), it will even support complex routed traffic.

Consider your software stack

Of course, the last thing to consider is your current software stack. If you are looking for a relatively simple integration process and do not want to maintain other agents in the stack, you may prefer to use agents that are already supported by the stack.

For example, if you use Celery for Task Queue in a system on top of RabbitMQ, you will get the power to use it with RabbitMQ or Redis instead of Kafka that does not support Kafka and requires some rewriting.

We use all of the above through the development and growth of the platform, and then use some of it! It is important to remember that each tool has its own advantages and disadvantages, which is related to understanding them and choosing the right tools for work and specific times, situations, and requirements.

After reading the above, do you know more about which of Redis, Kafka or RabbitMQ is more compatible with microservices? If you want to know more knowledge or related content, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.