Shulou(Shulou.com)06/01 Report--

DB, DW, DM, ODS, OLAP, OLTP and BI concept is what, I believe many inexperienced people are helpless, for this reason this article summarizes the causes of the problem and solutions, through this article I hope you can solve this problem.

Today, I specially checked some official explanations and many excellent blog articles, and wrote down some nouns about the number of warehouses. First, I will explain these abbreviations:

DB (DataBase): Database, generally referred to as OLTP database, online transaction database, used to support production. DB keeps the latest status of data information, only one status!

DW (Data Warehouse): The data warehouse stores the state of the data at different points in time. For the same data information, the state of different points in time is kept, which is convenient for us to do statistical analysis.

3. About DM, there are two kinds of statements on the Internet at present, one is Data Mart; the other is Data Mining. Baidu Encyclopedia gives the concept of data mining. I explain these two statements here:

DM (Data Mart): Data mart, a local DW established with a business application as the starting point. DW only cares about the data it needs, and does not consider the overall data architecture and application of the enterprise. Each application has its own DM.

DM (Data Mining): Data Mining, also known as Knowledge Discovery in Database (KDD), is a non-trivial process of obtaining effective, novel, potentially useful, and ultimately understandable patterns from a large amount of data. Simply put, data mining is to extract or "mine" knowledge from a large amount of data.

ODS (Operating Data Store): Operational data warehouse, the earliest data warehouse model. The characteristic is that the data model adopts the design of sticking source, and the structure of ODS database is what the data structure of business system is. The difference is that ODS database can provide the history of data changes, so each table in ODS database will add a date type to represent the time point of data, and the daily data changes will be stored, which is conducive to data analysis.

OLTP (on-line transaction processing): OLTP is the main application of traditional relational databases, mainly basic, daily transaction processing, such as banking transactions.

OLAP (On-Line Analytical Processing): OLAP is the main application of data warehouse system, supporting complex analytical operations, focusing on decision support, and providing intuitive and understandable query results.

7. BI (Business Intelligence): Business intelligence, leaders, decision makers, after obtaining the statistical information of OLAP and the scientific rules obtained by DM, make appropriate adjustments to production, such as ordering supermarket personnel to put beer and diapers together for sale, which will react to DB modifying inventory data-this is the role of the whole BI!

Overall Data Center Architecture

The overall architecture of data warehouse, metadata of each system is synchronized to operational data warehouse ODS through ETL, ODS data is subject-oriented or modeled to form DW (data warehouse), DM is to establish a model for a certain business field, and specific users (decision layer) view reports generated by DM.

Let's talk about some of the relationships between them:

Relationship between Data Warehouse and Data Mining

If Data Warehouse is compared to a mine, Data Mining is the work of mining deep into the mine. After all, Data Mining was not magic or alchemy. Without rich and complete data, it was hard to expect Data Mining to discover meaningful information.

To convert large amounts of data into useful information, information must first be collected efficiently. With the advancement of technology, a fully functional database system has become the best tool for collecting data. A data warehouse is simply a collection of useful data from other systems and stored in an integrated repository. Therefore, it is actually a relational database that has been processed and integrated and has a large capacity to store the data required by the decision support system for decision support or data analysis. From an information technology perspective, the goal of a data warehouse is to get the right data to the right people at the right time in an organization.

Many people confuse Data Warehouse and Data Mining and don't know how to tell them apart. In fact, data warehouse is a new theme of database technology, using computer systems to help us operate, calculate and think, so that the way of operation changes, and the way of decision-making changes.

A data warehouse itself is a very large database that stores data consolidated from organizational job databases, especially OLTP (On-Line Transactional Processing) data. Putting this consolidated data into a database and using it to make decisions for corporate decision makers is the biggest challenge in building a data warehouse, however, the process of transforming and integrating data. Because transforming the data in the job into useful strategic information is the focus of the entire data warehouse. In summary, a data warehouse should have these data: integrated data, detailed and summarized data, historical data, and interpretive data. Mining useful information and knowledge for decision-making from data warehouses is the biggest purpose of building data warehouses and using Data Mining. The essence and process of the two are different. In other words, the data warehouse should be established first, so that Data mining can be carried out efficiently, because the data contained in the data warehouse itself is clean (there will be no wrong data mixed in it), complete, and integrated. Data Mining is a process and technique for extracting useful information from huge data warehouses.

2. ODS to DW integration example

integral examples

Will OLAP replace Data Mining?

OLAP (Online Analytical Process) means an online analytical process linked from a database. Some people say,"I already have OLAP tools, so I don't need Data Mining." In fact, the two are quite different, the main difference being that Data Mining is used to generate hypotheses, while OLAP is used to verify hypotheses. Simply put, OLAP is user-driven, where users make assumptions and then verify them using OLAP, while Data Mining is used to help users generate assumptions. So when using OLAP or other Query tools, the user is doing exploration by himself, but Data Mining is using tools to help do exploration.

For example, a market analyst planning shelf space for a supermarket may assume that baby diapers and baby formula are products that are often purchased together. Then, OLAP tools can be used to test whether this assumption is true and see how obvious the evidence is. But Data Mining is different, the implementation of Data Mining will be a large amount of billing data after sorting, do not need to assume or expect possible results, through Mining technology can find the potential rules in the data, so we may get, for example, diapers and beer are often bought at the same time unexpected discovery, which OLAP can not do.

Data Mining can often mine relationships beyond the scope of induction, but OLAP can only use manual queries and visual reports to confirm certain relationships. Data Mining's ability to automatically find data models and relationships that are not even suspected has actually exceeded the limits of our experience, education, and imagination. OLAP can complement Data Mining, but this feature cannot be replaced by OLAP.

Summary: DM is intelligent OLAP

Relationship between Data Warehouse and Data Mart

A data warehouse is an enterprise-level, which provides decision support for the operation of various departments of the entire enterprise; a data mart is a miniature data warehouse, which usually has less data, less subject areas, and less historical data, so it is department-level, and generally can only serve managers within a certain local scope, so it is also called a department-level data warehouse.

After reading the above content, do you know what the concepts of DB, DW, DM, ODS, OLAP, OLTP and BI are? If you still want to learn more skills or want to know more related content, welcome to pay attention to the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.