Shulou(Shulou.com)05/31 Report--

This article introduces to you the principle and characteristics of pig, the content is very detailed, interested friends can refer to, hope to be helpful to you.

Apache Pig is an abstraction of MapReduce. It is a tool / platform for analyzing large data sets and representing them as data streams. Pig is usually used with Hadoop; we can use Apache Pig to perform all the data processing operations in Hadoop.

To analyze data using Apache Pig, programmers need to write scripts in the Pig Latin language. All of these scripts are internally converted to Map and Reduce tasks. Apache Pig has a component called Pig Engine that takes Pig Latin scripts as input and converts them into MapReduce jobs.

Why use Apache Pig

With Pig Latin, programmers can easily execute MapReduce jobs without having to type complex code into Java.

Apache Pig uses a multi-query approach, which reduces the length of the code. For example, an operation that requires typing 200 lines of code (LoC) into Java can be easily done by entering as little as 10 LoC in Apache Pig. In the end, Apache Pig reduced development time by nearly 16 times.

Pig Latin is a language similar to SQL, when you are familiar with SQL, it is easy to learn Apache Pig.

Apache Pig provides many built-in operators to support data manipulation, such as join,filter,ordering. In addition, it provides nested data types, such as tuple (tuple), bag (package), and map (mapping) that MapReduce lacks.

Apache Pig has the following characteristics:

Rich set of operators-it provides many operators to perform operations such as join,sort,filer.

Easy to program-Pig Latin is similar to SQL, if you are good at using SQL, it is easy to write Pig scripts.

Optimization opportunities-tasks in Apache Pig automatically optimize their execution, so programmers only need to focus on the semantics of the language.

Extensibility-using existing operators, users can develop their own capabilities to read, process, and write data.

User-defined functions-Pig provides the ability to create user-defined functions in other programming languages, such as Java, and can be called or embedded in Pig scripts.

Processing all kinds of data-Apache Pig analyzes a variety of data, whether structured or unstructured, and stores the results in HDFS.

Apache Pig and MapReduce

The main differences between Apache Pig and MapReduce are listed below.

Apache PigMapReduceApache Pig is a data flow language. MapReduce is a data processing mode.

It is a high level language.

MapReduce is low-level and rigid. Performing Join operations in Apache Pig is very simple. It is very difficult to perform Join operations between datasets in MapReduce. Any novice programmer with basic knowledge of SQL can easily work with Apache Pig. It is necessary to use MapReduce to expose to Java. Apache Pig uses a multi-query approach, which greatly reduces the length of the code. MapReduce will need almost 20 times the number of rows to perform the same task. There is no need to compile. When executed, each Apache Pig operator is internally converted to a MapReduce job. The MapReduce job has a long compilation process. Apache Pig Vs SQL

The main differences between Apache Pig and SQL are listed below.

PigSQLPig Latin is a programming language. SQL is a declarative language. In Apache Pig, the mode is optional. We can store data without the need for design patterns (values stored as 01001 01,02, etc.) schemas are required in SQL. The data model in Apache Pig is a nested relationship. The data model used in SQL is a flat relationship. Apache Pig provides limited opportunities for query optimization. There are more opportunities for query optimization in SQL.

In addition to the above differences, Apache Pig Latin:

Allows splitting in pipeline (pipeline).

Allows developers to store data anywhere in pipeline.

Declare the execution plan.

Provides operators to perform ETL (Extract extraction, Transform transformation, and Load loading) functions.

Apache Pig VS Hive

Both Apache Pig and Hive are used to create MapReduce jobs. In some cases, Hive runs on HDFS in a manner similar to Apache Pig. In the following table, we list several important points that distinguish Apache Pig from Hive.

Apache PigHiveApache Pig uses a language called Pig Latin (originally created in Yahoo). Hive uses a language called HiveQL (originally created in Facebook). Pig Latin is a data flow language. HiveQL is a query processing language. Pig Latin is a process language, which is suitable for pipelined paradigm. HiveQL is a declarative language. Apache Pig can handle structured, unstructured and semi-structured data. Hive is mainly used for structured data. Application of Apache Pig

Apache Pig is often used by data scientists to perform tasks that involve specific processing and rapid prototyping. Use Apache Pig:

Deal with large data sources, such as Web logs.

Perform data processing for the search platform.

Processing loading of time-sensitive data

The language used to analyze data in Hadoop using Pig is called Pig Latin, a high-level data processing language that provides a rich set of data types and operators to perform various operations on the data.

To perform specific tasks, programmers using Pig need to write Pig scripts in the Pig Latin language and execute them using any execution mechanism (Grunt Shell,UDFs,Embedded). Once executed, these scripts generate the required output by applying a series of transformations to the Pig framework.

Architecture of Apache Pig

Apache Pig component

As shown in the figure, there are various components in the Apache Pig framework. Let's look at the main components.

Parser (parser)

Initially, the Pig script is processed by the parser, which checks the syntax, type checking, and other miscellaneous checks of the script. The output of the parser will be DAG (directed acyclic graph), which represents Pig Latin statements and logical operators. In DAG, the logical operator of a script is represented as a node and the data flow is represented as an edge.

Optimizer (optimizer)

The logical plan (DAG) is passed to the logical optimizer, which performs logical optimizations, such as projection and pushdown.

Compiler (compiler)

The compiler compiles the optimized logical plan into a series of MapReduce jobs.

Execution engine (execution engine)

Finally, the MapReduce job is submitted to Hadoop in sort order. These MapReduce jobs are executed on Hadoop and produce the desired results.

Pig Latin data model

Pig Latin's data model is completely nested, allowing complex non-atomic data types, such as map and tuple.

Atom (atom)

Any single value in Pig Latin, regardless of its data type, is called Atom. It is stored as a string and can be used as a string and a number. Int,long,float,double,chararray and bytearray are the atomic values of Pig. A piece of data or a simple atomic value is called a field. Example: "raja" or "30"

Tuple (tuple)

Records formed by a collection of ordered fields are called tuples, and fields can be of any type. Tuples are similar to rows in the RDBMS table. Example: (Raja,30)

Bag (package)

A package is an unordered set of tuples. In other words, a collection of tuples (not unique) is called a package. Each tuple can have any number of fields (flexible mode). The package is represented by "{}". It is similar to a table in RDBMS, but unlike a table in RDBMS, each tuple does not need to contain the same number of fields, or fields in the same location (column) have the same type.

Example: {(Raja,30), (Mohammad,45)}

A package can be a field in a relationship; in this case, it is called an inner bag.

Example: {Raja,30, {9848022338, rajaqigmail.com,}}

Map (Mapping)

A map (or data map) is a set of key-value pairs. Key needs to be of type chararray and should be unique. Value can be of any type, represented by "[]"

Example: [name#Raja,age#30]

Relation (relationship)

A relationship is a tuple package. Relationships in Pig Latin are unordered (there is no guarantee that tuples will be processed in any particular order).

On the principles and characteristics of pig what is shared here, I hope that the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.