Shulou(Shulou.com)06/01 Report--

Big data in how to achieve Kafka and activemq comparison, I believe that many inexperienced people do not know what to do, so this paper summarizes the causes of the problem and solutions, through this article I hope you can solve this problem.

Compared with Kafka and activemq, background analysis message queuing has always been a very important component, and I firmly believe this conclusion when I have been through two enterprises. The most extensive function of queue is decoupling. Broadly speaking, it can effectively integrate the work of different departments. Based on an agreed format, the two sides can develop without interfering with each other. It can be said that this idea of production and consumption applies not only to programs but also to many places. At present, for what I see, kafka is more of a data source, a data bridge, and a communication between different businesses. For example, if you need to access the business data of Department An in real time, there will be such a means:

The data landed on the HDFS will be used for some algorithmic offline processing, while the kafka side is for consumers who need real-time. In fact, the consumption of data is nothing more than real-time and offline.

Comparison between Kafka and activemq

It is really very different from the activemq,kafka that is often used in the past. Make a comparison to deepen the impression.

Contrast

Activemq

Kafka

Interface protocol

Comply with JMS specification and support all languages well.

It does not follow the standard MQ interface protocol, so it is more complex to use.

Throughput

Lower, disk random read and write

Higher, disk sequential read and write

Vernier position

AMQ to manage, unable to read historical data

The client manages it by itself, and you can even read it again if you don't like it.

HA mechanism

Master-slave replication, competitive lock to elect a new master node

Like the hadoop series, all nodes are managed by zk

In the final analysis, as a consumer of kafka, the biggest difference you can feel lies in a few:

1. The throughput is indeed very high. Historical data can be reread, but there are also some disadvantages: 1. Conceptually, it is complicated. Compared with AMQ, you only need to know ip and queue name to get data. Kafka is very complicated to use.

Basic concepts of Kafka (excerpt)

1.Broker: message middleware processing node. A Kafka node is a broker, and multiple broker can form a Kafka cluster. 2.Topic: a class of messages, such as page view logs, click logs, etc., can exist in the form of topic, and the Kafka cluster can be responsible for the distribution of multiple topic at the same time. 3.Partition:topic physical packets, a topic can be divided into multiple partition, each partition is an ordered queue. 4.Segment:partition is physically composed of multiple segment. 5.offset: each partition consists of a series of ordered, immutable messages that are continuously appended to the partition. Each message in partition has a consecutive sequence number called offset, which is used by partition to uniquely identify a message.

Common parameters on the consumer side of Kafka

Properties props = new Properties ()

/ / the address of the zk server xxxx:2181

Props.put ("zookeeper.connect", zookeeper)

/ / the name of the group, which is different from other group, otherwise the data will not be received

Props.put ("group.id", groupId)

Props.put ("zookeeper.session.timeout.ms", "8000")

Props.put ("zookeeper.connection.timeout.ms", "20000")

Props.put ("zookeeper.sync.time.ms", "2000")

Props.put ("auto.commit.interval.ms", "5000")

Props.put ("rebalance.max.retries", "5")

Props.put ("rebalance.backoff.ms", "60000")

Props.put ("auto.commit.enable", "true")

/ / key parameters, whether to start reading from the front of offset every time

Props.put ("auto.offset.reset", "smallest")

After reading the above, have you mastered how to achieve the comparison between Kafka and activemq in big data? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.