Shulou(Shulou.com)06/03 Report--

Both Impala and Hive are data query tools built on top of Hadoop, but each has different emphasis, so why should we use both tools at the same time? Is it not possible to use Hive or Impala alone?

First, introduce Impala and Hive

(1) both Impala and Hive provide SQL query tools for HDFS/Hbase data, Hive will be converted into MapReduce, and access to HDFS data can be realized by scheduling with the help of YARN, while Impala directly queries HDFS data. But they all provide the following standard SQL statements that run in the fuselage.

(2) Apache Hive is a high-level abstraction of MapReduce, and HiveQL,Hive can be used to generate MapReduce or Spark jobs running on Hadoop clusters. Hive was originally developed by Facebook around 2007 and is now an open source project for Apache.

Apache Impala is a high-performance dedicated SQL engine, using Impala SQL, because Impala does not need any framework to directly query data blocks, so the query latency is millisecond. Impala, inspired by Google's Dremel project, was developed by Cloudera in 2012 and is now an open source Apache project.

2. What's the difference between Impala and Hive?

(1) Hive has many features:

1. Broader support for complex data types (such as arrays and maps) and window analysis

2. High scalability

3. Usually used for batch processing

(2) Impala is faster

1. Professional SQL engine, providing better performance from 5x to 50x

2. Ideal interactive query and data analysis tools.

3. More features are being added

III. High-level overview:

4. Why use Hive and Impala?

The main contents are as follows: 1. It brings massive data analysis ability for data analysts, does not need software development experience, and uses SQL knowledge to analyze data.

2. It is more productive than writing MapReduce or Spark directly. 5 lines of HiveQL/Impala SQL is equivalent to 200 or more lines of Java code.

3. Provide good interoperability with other systems, such as through Java and external scripting extensions, and many business intelligence tools support Hive and Impala.

Use cases of Hive and Impala

(1) Log file analysis

Log is a common data type and an important data source in big data era. Its structure is not fixed. Log collection can be put into HDFS through Flume and kafka, then the structure of log is analyzed, a table is established according to the separator of log, and then Hive and Impala are used to analyze the data. For example:

(2) emotional analysis

Many organizations use Hive or Impala to analyze social media coverage. For example:

(3) Business intelligence

Many leading BI tools support Hive and Impala

After reading the function and effect of Hive and Impala, it plays an important role in mastering Hadoop data processing. In ordinary times, we should accumulate and sum up experience and constantly improve our skills. I usually not only sum up my own experiences and lessons, but also like to look at the knowledge shared by others, learning from each other to make up for my weaknesses, which plays an important role in improving my knowledge structure. Wechat service numbers like "CSDN" forum, "big data cn" and "big data Times Learning Center" are all quite good. In short, if we strive to learn from many aspects of knowledge, we will make greater progress!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.