Shulou(Shulou.com)06/01 Report--

Big data platform log storage analysis system solution is what, I believe that many inexperienced people do not know what to do, so this paper summarizes the causes of the problem and solutions, through this article I hope you can solve this problem.

The log storage and analysis system of big data platform is based on big data platform to store and analyze the log records generated by the business system. The log data comes from the log history data stored by ElasticSearch, and the log data that needs to be stored and analyzed is read from ElasticSearch to Kafka. After hearing the data message coming in, the consumer of Kafka obtains the data by stream and writes it to the HBase. The data stored in HBase provides services for the retrieval and analysis of log data. The Web front end retrieves the data in the HBase by calling the back-end API and visually displays the PivotTable. The displayed data supports the CSV export function.

The system functions included in this solution are:

Read Kafka messages and store them in HBase

Retrieve HBase data and provide API

CSV export of display content

Technical scheme

The prerequisites for determining the technical solution include:

The message pushed to Kafka is ready.

The data structure of HBase has been determined

Log retrieval conditions are fixed

The backend API has determined that the Web front end will call the API

Web front-end development is ready

Technical architecture

The entire technical architecture is shown in the following figure:

The system is divided into three modules:

LogData Collector: subscribe to the Topic of Kafka through Spark Streaming, collect the log data needed by the business, and then write it to HBase. HBase is built in the cluster of HDFS.

LogData Searching: this is a Spring Boot service that exposes the API interface to the front end through @ RestController. Inside, Spark is integrated and Spark SQL is used to query HBase database.

Web UI front end: responsible for invoking the LogData Searching service, presenting the data to the UI, and realizing the export of the display data at the front end.

Note: there may be a change in the export function. If the front end is displayed in pages, and the exported data is full data that meets the retrieval conditions, you need to call the LogData Searching service during export to get the data needed for export, rather than the data already displayed in the front end.

After reading the above, have you mastered the solution of big data platform log storage analysis system? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.