Shulou(Shulou.com)06/01 Report--

In this issue, the editor will bring you about how to build big data self-help analysis platform. The article is rich in content and analyzes and narrates it from a professional point of view. I hope you can get something after reading this article.

What is a self-help analysis platform

The self-help analysis platform is built on big data platform and relies on the data research and development capability of big data platform. Through unified data service, it realizes the unified management of data query and analysis, and provides efficient data decision support for enterprise business analysis. At the same time, it also prevents data engineers from falling into complicated data collection requirements. Self-help analysis platform is a front-end product that computer-based business personnel can quickly get started with. It not only needs the processing performance of big data, but also needs visual analysis capabilities that are simple and easy to use. Only by enabling business personnel to quickly master the method of use, combined with the company's business, the self-help analysis platform is valuable. In fact, all along, the major companies' data analysis platform has only one goal-to kill Excel. What modules should the self-help analysis platform have?

As mentioned above, self-help analysis platform is used to query data, explore data, need to have the existing functions of Excel, but also better than Excel. Support multi-data source access self-service analysis platform should be able to support a variety of data sources, different data types of file access, so that data engineers and business personnel can quickly import data into the self-service analysis platform. Need to support traditional relational database, Hive, file import (Excel, CSV, TXT, etc.). Multi-dimensional analysis can quickly query, filter, aggregate, sort, associate and other dynamic operations on the imported data. For example, the business staff already have some basic user information, it can import the user name, through the user name associated with the corresponding user analysis data. And can carry on the grouping aggregation operation to different types of users. All of the above operations need to be drag-and-drop, and there is no need for the business staff to write a single line of code. Rich visualization needs to support commonly used visualization graphics, such as pie chart, ring chart, coaxial curve, bar chart, scatter chart, etc., and users need to bind the data imported by themselves or cleaned through the platform. you can quickly produce corresponding analysis charts and make visual reports. Authority control self-service analysis platform is used for all business personnel of the company, and needs to have corresponding authority control. For example, the data chart made by A user can not be viewed by B user, but can be viewed only after An is authorized to B. The data in the self-help analysis platform should also be subject to permission control, such as sensitive data can not be open to all users, downloading data requires process approval, and so on. High-performance data analysis query should be fast, self-help analysis should be fast, and visualization should be fast. Many self-help analysis platforms eventually become data download platforms, a large part of which is not fast enough. Although big data is much faster than Excel, in actual business exploration, most of the time the amount of data is less than one million. If it is not as fast as Excel, why do people use your platform? Therefore, whether it is a large amount of data, or a small amount of data, it should be fast! Technically, should we consider the use of query computing engines that cannot be used for large and medium and small amounts of data? III. Architecture of self-help analysis platform

Self-help analysis engine for complex query analysis with a large amount of data, we can use Spark to submit tasks to achieve self-help analysis. For small and medium-sized data, we use MPP database to achieve fast query. Visualization we can use echarts to support various types of chart presentation, or use open source self-help analysis projects such as superset for presentation.

Authority

In order to achieve mutual isolation and data security, the background management and control system controls the authorization of data through conditional restrictions, and uses encryption algorithms for sensitive information such as mobile phone numbers, identity numbers, mailboxes and other sensitive information to prevent data leakage.

In fact, both the business staff and the IT team have their own ideas about the construction of the self-help analysis platform, and they also want to do something for the company through the data, so when establishing a self-help analysis platform, they can constantly communicate with the business staff, set some subject data, show the results, share them with the business staff and leaders, and let them participate in the evaluation and suggestions, optimize and improve constantly, when the relevant personnel have a sense of participation. The self-help analysis platform will develop for a long time. Finally, I would like to remind you that the purpose of the self-help analysis platform is to "kill Excel" so that all the analysis results can be stored online and must not be reduced to a data download platform.

The above is the editor for you to share how to build big data self-help analysis platform, if you happen to have similar doubts, you might as well refer to the above analysis to understand. If you want to know more about it, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.