Shulou(Shulou.com)06/01 Report--

This article introduces the relevant knowledge of "what is the difference between Pig and Hive". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

Pig is a programming language that simplifies the common tasks of Hadoop. Pig can load data, express transformed data, and store the final result. Pig's built-in operations make semi-structured data meaningful (such as log files). At the same time, Pig can extend the use of custom data types added in Java and support data conversion. Hive acts as a data warehouse in Hadoop. Hive adds the structure of the data in HDFS (hive superimposes structure on data in HDFS) and allows data queries using syntax similar to SQL. Like Pig, the core functionality of Hive is extensible. Hive is more suitable for data warehouse tasks, while Hive is mainly used for static structures and tasks that require frequent analysis. The similarity between Hive and SQL makes it an ideal intersection of Hadoop and other BI tools. Pig gives developers more flexibility in the big data set domain and allows the development of concise scripts to transform data streams for embedding into larger applications. Pig is relatively lightweight compared to Hive, and its main advantage is that it can significantly reduce the amount of code compared to using Hadoop Java APIs directly. Essentially talk about Pig and Hive. After the conversion of Pig Latin into a MapReduce job, through the MapReduce multiple threads, processes or independent systems to execute the result set of parallel processing for classification and induction. The Map () and Reduce () functions run in parallel, running a set of tasks at the same time, even if not on the same system, and when all processing is complete, the results are sorted, formatted, and saved to a file. Pig uses MapReduce to divide the computing into two phases. The first stage is divided into small blocks and distributed to each node where the data is stored for execution, dispersing the computing pressure. The second stage aggregates the results of the first stage, which can achieve very high throughput. With less code and workload, thousands of machines can be driven to parallel computing, making full use of computer resources. Remove the bottleneck in operation. In other words, the biggest function of Pig is to implement a set of shell scripts for the mapreduce algorithm (framework), which is similar to the familiar SQL statement, which is called Pig Latin in Pig. In this script, we can sort, filter, sum, group (group by) and Joining the loaded data, and Pig can also operate on the dataset by user-defined functions. That is the legendary UDF (user-defined functions). The concluding reading feeling is: Pig is used to write some real-time scripts, such as the leader asks you for a piece of data and needs to be out in half an hour; Hive, it is a product manager who comes over and asks what's going on? So you Hive, a concise SQL-like statement... Done! "what's the difference between Pig and Hive" is introduced here, thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.