Shulou(Shulou.com)05/31 Report--

Editor to share with you how to integrate spring kakfa, I believe that most people do not know much about it, so share this article for your reference, I hope you can learn a lot after reading this article, let's learn about it!

I. production end

1.1 kafka-producer.xml configuration description

Classpath:/properties/kafka-producer.properties / / kafka service cluster / / grouping / / number of retries / / batch quantity / / topic name

1.2 kafka-producer.properties properties file

Bootstrap.servers=192.168.0.75:9092192.168.0.75:9093192.168.0.75:9094group.id=testGroupretries=1batch.size=16384linger.ms=1buffer.memory=33554432defaultTopic=topic-test

1.3 production side interface encapsulation description:

1) Class name: com.rkhd.ienterprise.kafka.producer.KafkaProducerServer

2) methods:

/ * send messages (not partitioned) * @ param data the data to be sent * @ return returns a map. If the successful code is 0, others fail * / public Map sendDefault (Object data); / * * send message (non-partitioned) * @ param key key to be sent * @ param data data to be sent * @ return returns a map. If the successful code is 0, others are failed * / public Map sendDefault (Object key, Object data); / * * send information (partition) * @ param partitionNum partition number (greater than 1). Note that the number of partitions is specified when the topic is created. You cannot change the key to be sent by * @ param key * @ param data to send data * @ return returns a map. If the successful code is 0, others are failed * / public Map sendDefault (int partitionNum, Object key, Object data); / * * send message (non-partition) * @ param topic send the destination topic name. If topic is null or "", a map will be returned using the data to be sent by defaultTopic * @ param data configured in xml * @ return. If the successful code is 0, others are failed * / public Map sendMessage (String topic, Object data); / * * send message (non-partitioned) * @ param topic send the destination topic name. If topic is null or "", a map will be returned using the key to send * @ param data * @ return configured in xml * @ param data. If successful code is 0, others are failed * * / public Map sendMessage (String topic, Object key, Object data) / * * send information (partition) * @ param topic send the destination topic name. If topic is null or "", the number of defaultTopic * @ param partitionNum partitions configured in xml (greater than 1) will be used. Note that the number of partitions is specified when topic is created. You cannot change the data to be sent by * @ param data * @ return and return a map. If the success code is 0, the rest is a failure * / public Map sendMessage (String topic, Integer partitionNum, Object data) / * * send information (partitions) * @ param topic sends the destination topic name. If topic is null or "", the number of partitions to be sent (greater than 1) will be sent using the key * @ param value to be sent by defaultTopic * @ param value configured in xml. Please note that the number of partitions is specified when topic is created, and the number of partitions returned by * @ return cannot be changed. If the success code is 0, the rest is a failure * * / public Map sendMessage (String topic, int partitionNum, Object key, Object value); second, the consumer side

2.1 kafka-consumer.xml configuration description

/ / kafka service cluster / / grouping / / whether auto-commit / / auto-commit interval / / session expiration time / / configure the number of consumers

2.2 kafka-consumer.properties properties file

Bootstrap.servers=192.168.0.75:9092192.168.0.75:9093192.168.0.75:9094group.id=testGroupenable.auto.commit=falseauto.commit.interval.ms=1000session.timeout.ms=15000topicName=ahao-test

2.3 description of consumer interface encapsulation

1) Class name: com.rkhd.ienterprise.mq.client.consumer.client.KafkaConsumerClient

2) provide abstract methods (according to different business implementations):

Public abstract void onConsumer (ConsumerRecord record)

3) implementation description: each business line implements the abstract method by inheriting this class

III. Overview of Kafka technology

3.1Features of Kafka

High throughput and low latency: kafka can process hundreds of thousands of messages per second, with a delay of at least a few milliseconds

Scalability: kafka clusters support hot expansion

Persistence, reliability: messages are persisted to the local disk, and data backup is supported to prevent data loss

Fault tolerance: nodes in the cluster are allowed to fail (if the number of replicas is n, 1 node is allowed to fail)

High concurrency: supports thousands of clients reading and writing at the same time

3.2 Kafka architecture components

The object that publishes subscriptions in Kafka is topic. We can create a topic for each type of data, and the client that publishes messages to topic is called producer, and the client that subscribes messages from topic is called consumer. Producers and consumers can read and write data from multiple topic simultaneously. A kafka cluster consists of one or more broker servers that are responsible for persisting and backing up specific kafka messages.

Topic: the directory where the message is stored is the topic

Producer: the side of the production message to the topic

Consumer: the party that subscribes to topic consumption messages

The service instance of Broker:Kafka is a broker.

3.3 kafka application scenarios

Log collection: a company can use Kafka to collect the log of various services, and open it to various consumer through kafka as a unified interface service, such as hadoop, Hbase, Solr, etc.

Message system: decoupling and producers and consumers, caching messages, etc.

User activity tracking: Kafka is often used to record various activities of web users or app users, such as browsing, searching, clicking and other activities. These activity information is published to the topic of kafka by each server, and then subscribers subscribe to these topic to do real-time monitoring and analysis, or load to hadoop or data warehouse for offline analysis and mining.

Operational indicators: Kafka is also often used to record operational monitoring data. This includes collecting data from various distributed applications and producing centralized feedback on various operations, such as alarms and reports.

Streaming: such as spark streaming and storm

Event source

The above is all the content of this article "how to integrate spring kakfa". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.