Shulou(Shulou.com)06/03 Report--

This article mainly shares the three major analysis tools of Hadoop: Hive, Pig and Impala.

Hive and Pig are high-level data languages, based on Mapreduce, and the underlying processing will be converted to Mapreduce to submit. Hive and Pig are both open source, Hive was originally developed by Facebook, and Pig was originally developed by Yahoo! Development, which is described below:

What is Hive?

Hive can be seen as a mapper from SQL to Mapreduce, that is, you don't have to develop Mapreduce, you just need to understand SQL. HiveQL is a subset of standard SQL92, which is not exactly the same as standard SQL. HiveQL itself has an extension of 20%, and about 80% of the syntax is consistent with standard SQL, like

This standard SQL is supported, so for data analysts, it is very convenient to cut into the Hadoop platform to do data analysis.

What is Pig?

Pig is a data flow language for dealing with large data sets. What is data flow? That is, the process of processing data can be defined step by step, such as the first step of loading, the second step of conversion, the third step of conversion, and the fourth step of storage, which can define the direction of the data step by step, which is very similar to the series of processing processes that we carry out in data mining. Because pig is the language of data flow, it is very suitable for material data exploration and non-processing of ETL phase data. His idea is very similar to that of Spark, so it can also be said that Spark is the correct Pig. Why do you say that? Because both Pig and Spark are handled like data streams, pig has transformations and action operations, as well as in spark.

Pig data flow language

Pig is widely used in the ETL phase, and it is very suitable for some data miners, especially to find out some unknown data. Because you don't need to specify any name or type, you can load it first, and then match all the data, and then you can observe what the data looks like and analyze how to do the conversion. Pig is a very precise semantic language, so it will be very convenient to learn.

Comparison between hive and pig

What is Impala?

Although we have hive, hive is based on mapreduce, and its analysis efficiency is not high. Everyone strives to find a high-performance SQL engine, and the emergence of impala solves this problem. Impala is a high-performance SQL engine dealing with massive data, its query can reach seconds and, even some data can reach millisecond level, low latency, 10 to 50 times faster than Hive, Pig or MapReduce, its SQL is also similar to HiveQL query language, it and the standard SQL also has 80% syntax repetition, but also has its own expansion part. Impala uses the same data as Hive, just like creating a table in Hive, Impala is also accessible, and vice versa. Impala runs on the Hadoop cluster, the data is stored in HDFS, can not use MapReduce, it has its own architecture, but also the main memory structure, each service can directly access the data block. Impala is developed by Cloudera, 100% open source and released under the Apache software license.

So there are three kinds of data analysis solutions, in practice, how do we use them? Generally speaking, Pig is not used as much as Hive and Impala, but they each have their own advantages. Let's describe their respective conditions of use: we know that Impala is a near-real-time query, using the same data as Hive, so we ask, why use Hive? There are some complex text analysis can only use Hive, such as some CSV files, some high-frequency word analysis, statistical analysis can only be supported by Hive,Impala. There are also some complex types, such as arrays and complex structures, which can only be used in Hive. Impala is mainly used for timely and interactive analysis. Hive is used for jobs with high stability and low real-time performance. Pig can also support some complex types, but pig does not have a fixed model, which can be used if you do some temporary data exploration.

Compare Hive, Pig and Impala

So can they replace RDBMS? Of course not, relational data supports transactions with low latency and can be modified at any time, while Hive and Impala can not replace relational databases. Pig, Hive and Impala are mainly suitable for large amounts of data reading and low-cost extensive expansion.

Analyze workflow schematic

The above is the data analysis content shared by the author according to his own knowledge system, mainly aiming at the respective characteristics, application and distinction of Hive, Pig and Impala, as well as the difference with the traditional database, which plays an important role in in-depth understanding of the application of data analysis tools in practice. In my actual work and study, I like to pay attention to some real-time information about big data, such as "big data cn", which plays an important role in understanding and grasping the development of big data. I also like to see some knowledge structures shared by others, such as the "big data era Learning Center", to constantly enrich and improve my knowledge system, which have greatly promoted my development and recommended to you.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.