Shulou(Shulou.com)06/01 Report--

This article will explain in detail how to analyze the application of Kafka in the environment of big data. The content of the article is of high quality, so the editor will share it with you for reference. I hope you will have some understanding of the relevant knowledge after reading this article.

We live in an era of data explosion, the huge growth of data has brought pressure on our business processing, at the same time, the huge amount of data has also brought us considerable wealth. As big data integrates the data of users, operators and service providers of various industries into big data's environment, or users access huge amounts of data in big data's environment, the message processing between business platforms will become particularly complex. How to collect and use data efficiently and how to reduce the pressure on various business systems are becoming more and more prominent. In the early system implementation, the business is relatively simple. Even if the amount of data and business is relatively large, big data environment can also deal with it. However, with the increase of access systems, the amount of data and business increases, big data environment, business systems can appear certain bottlenecks. Let's look at a few scenes.

Scene 1: we have developed a device information mining platform. This platform needs to store the status information of the routing nodes collected by the Internet gateway into the data center in real time. Usually, a gateway needs to report dozens or even hundreds of changing routing information at a time. There are tens of thousands of such Internet gates in the region. When the information collection platform writes or updates these changed data information to the database, it will put great pressure on the database agent and even hang up the database directly. This puts forward high requirements for our data acquisition system. The requirement of how to update messages to the database stably and efficiently is laid out.

Scenario 2: the data processed by the data center needs to be shared with several different organizations in real time. The method we often use is to store the data in batches in the data acquisition machine, and the branch office collects it regularly, or the branch office obtains the data from the data center in real time through JDBC, RPC, http or other mechanisms. There are some problems in these two methods, the former lies in the lack of real-time performance and involves the problem of data integrity; the latter is that when the amount of data is large, multiple branches read the data at the same time, which will cause great pressure on the data center and cause a great waste of resources.

In order to solve the problems raised in the above scenario, we need such a messaging system:

Buffering capacity, the system can provide a buffer, when there is a large amount of data coming, the system can reliably buffer the data for subsequent modules to deal with.

Subscription and distribution capabilities, the system can receive messages reliably cached, can also publish reliable cached data to consumers.

This requires us to find a high-throughput system that can meet the needs of subscription and publishing.

Kafka is a distributed, high-throughput, publish / subscribe-based messaging system. Using kafka technology, a large-scale message system can be built on cheap PC Server. Kafka has many characteristics, such as message persistence, high throughput, distributed, real-time, low coupling, multi-client support, reliable data and so on. It is suitable for online and offline message processing.

Use kafka to solve the problems we mentioned above.

The Internet gateway collects the changing routing information, and passes the collected information into kafka through the producer of kafka. Kafka caches the collected information according to the receiving order and joins the queue to be consumed. The consumer of Kafka reads the queue information and updates the obtained information to the database according to a certain processing strategy. Complete the storage of data to the data center.

When the data in the data center needs to be shared, the producer of kafka first reads the data from the data center, then passes it into the kafka cache and joins the queue to be consumed. As data consumers, each branch structure initiates consumption actions, reads data from kafka queues, and processes the acquired data.

The code produced by Kafka is as follows:

Public void produce () {/ / production message preprocessing produceInfoProcess () Pro.send (ProducerRecord New Callback () {@ Override onCompletion () {if (metadata = = null) {/ / failed failedSend () } else {/ / sent successfully! " SuccessedSend ();}}

According to the demand, the message producer flexibly defines the produceInfoProcess () method to deal with the relevant data. And handle the callback mechanism according to the situation when the data is published to kafka. The failedSend () method is defined when the data transmission fails, and the successedSend () method is defined when the data is successfully sent.

The code for Kafka consumption is as follows:

Public void consumer () {/ / configuration file properties (); / / iterator iterator = stream.iterator () to get the current data; while (iterator.hasNext ()) {/ / fetch message MessageAndMetadata next = iterator.next () MessageProcess ();}}

The Kafka consumer will establish a connection with the kafka cluster. Read the data from kafka, call the messageProcess () method, and deal with the acquired data flexibly.

The high throughput capacity and cache mechanism of Kafka can effectively solve the problem of peak traffic impact. Practice shows that before kafka is introduced into the system, when the amount of data sent by the Internet gateway is large, the relational database is often suspended and the data is often lost. After the introduction of kafka, the update program can process messages independently, without data loss, the pressure fluctuation of the relational database will not change too significantly, and the database lock will not occur.

Relying on the subscription distribution mechanism of kafka, it realizes the function of publishing once, and each branch subscribes independently according to the demand. It avoids the situation that each branch office requests data directly from the data center, or the data center transmits data to the branch office in batches in turn, so that the real-time performance is insufficient. Kafka improves the real-time performance, reduces the pressure on the data center, and improves the efficiency.

On how to analyze the application of Kafka in big data environment to share here, I hope that the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.