Shulou(Shulou.com)06/01 Report--

This article mainly explains "what is the design idea of big data's data warehouse platform". The content of the explanation in the article is simple and clear, and it is easy to learn and understand. let's study and learn "what is big data's design idea of data warehouse platform"!

The following figure shows the overall technical architecture of the warehouse:

An excellent and reliable data warehouse must have a clear structure and layering, rather than the more layers and themes the better, to ensure that in a clear state, you can quickly find the location of the data. The division of the subject area can be divided according to the internal business line and structure of the company.

The BDL layer in the figure, as the buffer data layer, is actually the same table as the source data table. Extract data directly from the source database (traditional relational business database), and store only the latest daily data snapshots (mainly incremental data)

The BAK layer is the decimation of the BDL layer, which stores the whole history of business data and plays the role of backup and error checking.

The FDL layer is the basic data layer, which is the basic data table developed based on the topic and warehouse model. Normal form modeling and dimensional modeling (that is, star model) are mainly used here, which contain the finest-grained data, so it is called the basic data layer. Popular understanding, this is the core basic data layer of the entire data warehouse, because the data table structure of the BAK layer and the BDL layer is completely consistent with the data of the business database, and the data of the FDL layer begins to be cleaned for the first time in the real sense, unwanted field removal. A process of removing non-standard field names, unifying data, and standardizing standardization.

The GDL layer is the general data layer, that is, the wide surface layer. It is also a wide table developed based on the theme and warehouse model. The third paradigm may be sacrificed to integrate the relevant dimensions or attributes into one table. This kind of table is characterized by more fields and a large amount of data, but it can help to eliminate duplicate queries. Because this kind of table can be used by various requirements, it is called the general data layer.

The ADL layer is the summary data layer, which is a summary data table developed based on the topic and warehouse model. Only dimensional modeling (that is, star model) is used here. The index library can be placed on this level.

The APP layer is open to users, where users can develop their own data and retrieve data. Desensitization data for users can also be put on this layer. The data needed for the data products we develop can also be done at this layer.

There are also two layers that are special:

One is TMP (temporary layer). To put it bluntly, it is to store intermediate result tables. Some business scenarios are more complex, requiring table disassembly, multiple calculations, and some intermediate result set tables, so this layer is also an auxiliary layer.

DIM layer is the dimensional surface, a variety of time dimensions, provincial, municipal and regional dimensions can be stored, according to the company's business form to divide, ADL layer can be associated with the DIM layer of multi-dimensional, do a variety of reports, easy to capture.

The following is the data flow framework composition of the data warehouse:

Thank you for your reading. the above is the content of "what is big data's design idea of data warehouse platform". After the study of this article, I believe you have a deeper understanding of what is the design idea of big data's data warehouse platform. The specific use of the situation also needs to be verified by practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.