Shulou(Shulou.com)06/02 Report--

Scene introduction

Speaking of Kafka, many users love and hate it. Kafka is a distributed, publish / subscribe-based messaging system, and its ultimate experience is unstoppable, but users still need to pay a lot of work behind worrying about operation and maintenance, complex security policies, lack of reliability and ease of use and so on. Even if you are a veteran of Kafka, you will inevitably have the same troubles mentioned above.

Instead of worrying about Kafka deployment all day, why don't you try the surprise of Kafka on the cloud? At present, the mainstream domestic cloud service providers provide Kafka services on the cloud to provide asynchronous message queuing services for application systems. Through the highly available message buffer queue, it can realize application decoupling, burst traffic processing and interworking and integration with third parties, which has the characteristics of large-scale, high reliability, high concurrent access, scalability and complete hosting. For example, Huawei Cloud distributed messaging service DMS, which helps decouple application components in the cloud, is very cost-effective. DMS embraces open source and provides support for compatible open source Kafka interfaces. Users can migrate seamlessly and use them on demand.

The Kafka proprietary version of distributed message Service (Data Message Service, referred to as DMS) supports collecting incremental transaction message data and uploading it to Huawei Cloud object Storage Service (Object Storage Service, referred to as OBS). By analyzing the uploaded transaction data through a real-time streaming computing framework (such as Flick) or Huawei Cloud Real-time Computing Service (Cloud Stream Service, referred to as CS), it can analyze recent transaction indicators and other information to support users to make trading decisions.

Apply for environment and resources

1. Turn on dump

▲ can enable message dump by clicking enable dump in more columns.

two。 Create a dump task

After waiting for the dump task to be enabled, you can add a dump task to dump. Click the instance link to enter the instance management tab with the dump management tab. Click create dump task to create a message dump task to dump the message, as shown below:

After clicking, the details of creating a dump task will pop up:

You need to enter the required parameters with an asterisk. Here, we mainly focus on the parameter Topics to be entered, that is, the Topic name to be dumped. Multiple Topic inputs or regular representations are supported to subscribe to Topic messages, and the message contents are uploaded to OBS buckets. Click the select button in the dump address line, and the select data dump address pops up, which lists which OBS buckets can be dumped.

It's worth mentioning that if you don't have an OBS bucket, you need to create an OBS bucket to hold the data that DIS dumps.

3. View dump task

After creating a dump task, you can see the task information on the dump management page. Click View dump information to view the progress of the message dump of the dump Topic in real time.

View OBS information and analysis data

Log in to Huawei Cloud OBS to view the files with partitions in the dump address bucket, which have been uploaded to the dms-obs bucket according to the time format set at the time of the dump.

You can use real-time data such as Flink to analyze the file contents of the uploaded directory and display the summary results after the analysis.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.