Shulou(Shulou.com)06/03 Report--

This article introduces the relevant knowledge of "the basic process of Hive data warehouse development". Many people will encounter such a dilemma in the operation of actual cases, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

1. Get data from business system

Sqoop imports data from database: spoop can transfer data between Hive and traditional database, import data from a relational database into HDFS of Hadoop, and import data from HDFS into relational database.

Flume collects text data: Flume can collect various types of files and store them in hdfs.

Ftp file server: download the source data (incremental data, full data) needed for analysis from the file server.

two。 Data storage

Data Warehouse layered ods layer, dw layer, da layer:

Source data layer ODS: direct reference to peripheral data that is not uniformly formatted will not be directly applied and is not conducive to analysis.

Data warehouse layer DW: from ODS, you have to go through the process of ETL to unify and clean the data.

Data application layer DA: to use the DW layer data to the real data consumer.

Data Mart: also known as data Mart, data Mart is to meet the needs of specific departments or users and store it in a multi-dimensional way, including defining dimensions, indicators to be calculated, levels of dimensions, and so on. Generate a data cube for decision analysis needs.

3. Configure scheduling system

4. Export data & display

This is the end of the content of "the basic process of Hive warehouse development". Thank you for your reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.