Shulou(Shulou.com)06/01 Report--

Overall data center solution

1 preface

With the development of the global economy and the rapid transformation from the Internet era to the DT era, enterprises develop to a certain stage, integration, industrialization and interconnection are bound to become new opportunities for the second take-off. But within the enterprise, the long-standing decentralized development of business, leading to information islands, business chain breakpoints and other problems, these problems are essentially the problem of data, data does not achieve deep value. Enterprises to achieve rapid development, to solve the problem of data, mining the value of data is an imperative road. Among them, the three main ideas of data value discovery are:

To solve the enterprise information isolated island, to achieve business integration; this idea is the main solution for enterprises to solve the information isolated island.

Data integration provides deep value mining for the development of enterprises; the essence of this idea is to discover the intrinsic nature of enterprises and solve enterprise problems based on data.

Data operation, the external service and operation of its own data, control the entrance; at present, this idea is carried out more in the Internet industry and the industry that has mastered the data, through mining the value of the data to achieve data value-added.

Combined with several current ideas of data value-added, as well as the problems encountered by large group enterprises at present, after several years of research and practice, Beijing Jinghang Computing and Communication Research Institute has successfully released the enterprise data resource management and analysis platform V2.0. The goal is to solve data problems for enterprises and achieve data value-added.

2 overall solution of aircraft enterprise data center

The overall solution version 1.0 of the airline data center has been launched since 2011, starting from the basic data management needs of users, and after several years of development and practice, the concept of integrated management of business subject data, full-cycle data collection, management, application and analysis is creatively put forward. It has been successfully applied in Aerospace Science and Industry Group and its subordinate units at all levels.

In 2015, after nearly 5 years of experience, the flight enterprise data resource management and analysis platform was successfully released, realizing one-click solutions for enterprise data centers, and providing best business practices to different industries.

2.1 data resource modeling method with metadata as the core

Figure 1: the idea of metadata as the core

Combined with many years of information construction experience, Jinghang Institute puts forward a set of flight data resource modeling method which is in line with group enterprises, which can adapt to the different definitions of metadata model description methods in different projects. and the adjustment of the metadata model description method in different application stages of the system. At the same time, the modeling method provides users with a custom extension interface.

Data modeling is the description of data objects, including data identification and elements of data composition. Metadata model is a kind of data specification, which covers data naming, interpretation, data structure and other information. It can be used to create database structure, understand data model, and perform cross-department data exchange and sharing. Create a metadata model under the resource directory to manage the master data model. The definition of metadata model is used to describe the main data elements to form complete data information, data model information, data reference relation information and so on, which supports the management and application of data. At the same time, the version management of the metadata model is carried out to meet the needs of data management changes, and at the same time have an impact on the data entity model and data content.

Based on the data resource modeling method, the data modeling of the data resource object is realized through the platform, and the storage structure, data format and the relationship with other models of the data resource are established through the data model. To adapt to the adjustment of the data model that needs to be carried out with the change of time and management methods, the model can be changed and upgraded through the system configuration, and the security of the data resources and the smooth change of the data model can be ensured.

2.2 Management and application of data resources based on data model

Based on the data model, it realizes data maintenance, management, query and retrieval, supports data process audit, and realizes multi-role data collaborative management. The data resource management interface and data items are customized through configuration, and the data contents of different interfaces are applied to different management scenarios.

At the same time, combined with many years of experience, Jinghang Institute puts forward a set of data model set with enterprise management as the core, and provides a standard data model. At the same time, the platform provides flexible model relationship definition to achieve a unified view of enterprise data.

2.3 flexible coding engine

For enterprises, especially group enterprises, there are numerous basic data that need to be managed by coding, such as unique coding for personnel, coding for contracts, coding for materials, coding for products, and so on. coded data is the key node of the enterprise business, such as the joint of the human body, is the key lifeline of all business.

The flight data resource management and analysis platform combines the coding data requirements of enterprises, and combines the problems such as coding changes caused by business development, the co-existence of new and old codes, and multiple coding rules, and provides a set of flexible and configurable coding engine. The engine supports various forms of code segment rule management, such as fixed code segment, feature code segment, date code segment, pipelined code segment and so on. Among them, the feature code segment manages the eigenvalue and feature code comparison table of the feature attribute. Through the coding engine, the coding data requirements of the enterprise can be met.

2.4 data quality improvement based on Rule engine

In order to ensure the effectiveness and accuracy of the data center, the platform provides a data quality management module based on rule engine, that is, through the definition of quality rules and the association of rules with the data model, the evaluation report and analysis results are generated through evaluation and scheduling.

Quality rules are created in the definition of quality rules for the needs of data evaluation, and the elements and verification rules that need to be verified in each type of data model in the system are configured in the data model rule settings. The frequency and verification content of generating evaluation results are set in the quality evaluation scheduling. Data quality assessment provides manual generation of evaluation reports for elements of the data model.

2.5 MPP-based data Mart

The self-developed MPP data Mart scheme is adopted, which adopts a similar MapReduce computing framework, draws lessons from the concept of incremental stream computing, and uses new technologies such as memory computing, in-library computing, distributed communication, column storage distributed file system to develop and implement a big data computing processing database system with distributed parallel real-time processing, complex query optimization and complex analysis.

2.6 Fast analysis and continuous iteration based on agility

The business intelligence system in the era of data warehouse and OLAP requires users to put forward the requirements of analysis and statistics in advance. On this basis, expand the data modeling work, then import the data, and then create the Cube. After these tasks are completed, business intelligence applications can be developed, which is a typical data-driven model. Business-driven business intelligence system directly imports detailed data, no longer requires users to put forward specific analysis and statistical requirements in advance, and no longer has the process of creating Cube, which greatly simplifies the work of the data layer, shortens the response cycle of the data layer, and the whole business intelligence system is transformed from data-driven to business-driven.

In the era of data warehouse + OLAP, a new analysis requirement may take a month to implement, and now it only takes a week or even a day. It used to take a year to build a business intelligence system, but now we can develop the first data analysis application in less than a week.

2.7 user self-service business intelligence based on exploration

Exploratory BI system believes that data applications such as Reporting and Dashboard are the portals, entrances rather than endpoints of business intelligence systems. In such a system, users can further interact with data (Interactive) based on analysis techniques such as Filter, Drill, Brush, Associate, Transform, Dynamic Calculation, and so on.

When the data layer is thin, the business layer is conditionally thick. Industry-leading enterprises in good hierarchical planning and hierarchical management, from general managers to front-line employees, departments at all levels can put forward and develop their own data analysis applications, and finally create a self-service business intelligence system on demand. Because most of the data analysis applications are developed by users or people close to users, the number of people involved in the development of applications is reduced, the management of the whole organization is more flattened, and the response time is greatly shortened. the management ability and strategic decision-making level of enterprises are improved. Compared with the traditional business intelligence system, the self-service business intelligence system is more optimized and efficient.

This data modeling technology, which directly imports detailed data, transforms the relationship between data and applications from tight coupling to loose coupling, so that most analytical applications do not cause any change in the data layer, while business intelligence systems based on MPP architecture can directly carry out high-performance analysis of detailed data. In this way, users can quickly develop data applications and then conduct real-time analysis.

3 concluding remarks

With the arrival of the DT era, the value of data will be more and more discovered, especially for large group enterprises, data from the generation, transmission, use, replication, reports, graphics and other life cycle process, each stage will play a different role, only a deeper role in discovering data, giving more role to data, data will play a more role for enterprises.

That's why the flight data center was born!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.