Shulou(Shulou.com)06/01 Report--

How does big data load the data? What problems should be paid attention to in data loading? What are the differences between relational databases and Impala and Hive loading data?

Before understanding loading data, we must make clear a concept of "data validation". Unlike other relational databases, Impala and Hive are verified when we write. For example, when we create a table, when we load data to it, it will verify whether the data and data type meet the requirements. If not, the data cannot be loaded.

In Hive and Impala, it is checked as it is read. Why is it designed in this way? Mainly to improve the speed of writing and the efficiency of loading. So, will the test data go wrong in this way? Of course not. If you can't query this data, it will be expressed as "null". If you want to guarantee the reliability of the query and ensure that every data can be found and matched, you can make a guarantee at the front end through other engineering languages.

Load data from HDFS

(1) in order to load data, you can simply add files to the table directory of HDFS, which is done directly using the hdfs dfs command.

Example: load data from HDFS to sales table

(2) use the LOAD DATA INPATH command to do it in Hive or Impala. This operation will move the data within the HDFS, and as with the previous command, the data source can be a file or directory.

2. Overlay data

(1) add the OVERWRITE keyword to delete all records before importing. Remove all files from the table directory, and then move the new files to that directory.

3. Append selected records to the table

(1) insert data through query

1. Use INSERT INTO to add results to the existing Hive table

2. Specify WHERE conditions to control which records will be appended

Use the Metabase Manager to load data

(1) the metadata manager provides two ways to load data

Loading data from a relational database

The Sqoop inline supports importing data into Hive and Impala. You can add the-- hive-import option to the Sqoop command, create tables in the hive Metabase, and import data from RDBMS into the HDFS table directory.

Note-tables created by hive-import are accessible in both Hive and Impala

If you have any questions, come at any time, and colleagues will get together and chat together!

Welcome to the Wechat official account "big data cn".

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.