Shulou(Shulou.com)06/01 Report--

In this issue, Xiaobian will bring you about the advantages and disadvantages of Kafka and Redis. The article is rich in content and analyzed and described from a professional perspective. After reading this article, I hope you can gain something.

preface

Big data technologies complement each other, and no technology is without shortcomings and develops in isolation. Today, let's analyze the comparison between Kafka and Redis and see what the advantages and disadvantages of distributed publishing and subscription are.

What is Redis?

Redis is an open source, free, BSD-compliant, high-performance key-value, non-relational database. Some friends may ask, Redis as a storage database, how to compare with distributed publish and subscribe message system Kafka? The two are not on the same level.

However, Redis has a queue data type, which is used to make a publish/subscribe system. This can be compared with kafka.

What is Kafka?

Kafka is a high-throughput, distributed, publish-subscribe-based messaging system. Using Kafka technology, large-scale messaging systems can be built on cheap PC servers. Kafka features message persistence, high throughput, distributed, multi-client support, real-time, etc.

Now dry goods come, the difference between kafka and Redis

Different storage media

Redis queue data is stored in memory, although there are AOF and RDB persistence methods, but still mainly memory.

Kafka is stored on a hard drive.

different performance

Because the storage medium is different, in theory, the performance of redis queue is better than kafka, but in the actual use process, this experience is not very obvious, usually only some high concurrency scenarios need to use redis queue, such as red packets, you can first disassemble the red packets and then push to redis queue, in the moment of grab can be very good support concurrency.

different costs

Here we need to draw a line, draw a line, draw a line.

Kafka is stored on the hard disk, the cost will be much smaller than the memory, the specific difference is 1 or 2 orders of magnitude, in the case of a very large amount of data, using kafka can save a lot of server costs. The most common are logs generated by applications. These logs are generally of large magnitude. If there is a need to process them, you can use the kafka queue.

This is just a brief introduction to the original gap, so let's look at the core comparison--the pros and cons as message queues.

Redis as a Message Queue

redis publish (pub), subscribe (sub) mode

Publish-subscribe in redis consists of three parts. Publisher (producer), channel (similar to topic), subscriber (consumer), the specific structure is as follows:

Redis pub-sub mode is very much like Western fast food, fast food fast disappear, all because Redis is to use memory to do access, all the messages you produce will be immediately processed by consumers at one time, and there will be no trace, at the same time because memory is always precious, so there will be restrictions on memory, when producers and consumers come up, it will also affect the efficiency of Redis. And Redis is intolerably slow at publishing and consuming big size(10K+ files) data.

Consider using Redis as a message queue if you have the following scenarios

If your demand is a fast-growing, fast-disappearing, instant consumption scenario, and the message produced is consumed immediately by the consumer,

If speed is something that you value very much, for example, if you are ten million seconds slower,

If the message loss scenario is allowed

If you don't need the system to save the messages you send, you can leave without a trace.

If the amount of data that needs to be processed isn't that huge

KafKa as a message queue

KafKa's design is exquisite, supporting distributed, highly available deployment, and a large queue is divided into multiple partitions (partitions) to improve the throughput of message enqueue, divide and conquer the idea. And when consuming support group concept, can support multiple clients consuming the same queue, and a group can increase the number of consumers to expand the processing capacity of consumption.

KafKa is not affected by the number of producers, because the throughput is sufficient support, even on a cheap stand-alone server can also have 100,000 message transmission per second, and consumers can want to consume whenever they want, the message is there, very flexible, do not worry about panic. The ability to persist messages with a certain policy (e.g. delete within a certain period of time, or empty at a certain capacity)

When there is a scenario you might consider using KafKa as a message queue

If you want stable message queues

If you want the messages you send to be kept for a certain period of time, not untraceable

If you can't stand the loss of data

If the speed doesn't have to be that fast

If you need to deal with huge amounts of data

Redis stores data in columns by hash of keys, and when Redis is used as a cluster, the corresponding application corresponds to a Redis, which will cause data skew to some extent, resulting in data loss.

From the previous deployment of Kafka cluster, a topic of kafka can have multiple partitions (copies), and it is evenly distributed on the Kafka cluster, which will not have the data skew like redis. Kafka also has the redundancy mechanism of Redis, such as Redis cluster if there is a machine down is likely to cause data loss, and Kafka because it is evenly distributed on the cluster host, even if a machine down, it will not affect the use. At the same time, Kafka, as a subscription message system, also has high throughput of millions per second, persistent and distributed characteristics.

The above is what are the advantages and disadvantages of Kafka and Redis shared by Xiaobian. If there are similar doubts, please refer to the above analysis for understanding. If you want to know more about it, please pay attention to the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.