Installation and use of pig
1. Download the software:
Wget http://apache.fayea.com/pig/pig-0.15.0/pig-0.15.0.tar.gz
two。 Decompression
Tar-zxvf pig-0.15.0.tar.gz
Mv pig-0.15.0 / usr/local/
Ln-s pig-0.15.0 pig
3. Configure environment variables:
Export PATH=PATH=$HOME/bin:/usr/local/hadoop/bin:/usr/local/hadoop/sbin:/usr/local/pig/bin:$PATH
Export PIG_CLASSPATH=/usr/local/hadoop/etc/hadoop
4. Enter grunt shell:
Log in to pig in local mode: all files and execution processes in this mode are local and are generally used for testing programs
[hadoop@host61 ~] $pig-x local
15-10-03 01:14:09 INFO pig.ExecTypeProvider: Trying ExecType: LOCAL
15-10-03 01:14:09 INFO pig.ExecTypeProvider: Picked LOCAL as the ExecType
2015-10-03 01 Apache Pig version 1414 09756 [main] INFO org.apache.pig.Main-Apache Pig version 0.15.0 (r1682971) compiled Jun 01 2015, 11:44:35
2015-10-03 01 Logging error messages to 1414 09758 [main] INFO org.apache.pig.Main-Logging error messages to: / home/hadoop/pig_1443860049744.log
2015-10-03 01 Default bootup file 14013 [main] INFO org.apache.pig.impl.util.Utils-Default bootup file / home/hadoop/.pigbootup not found
2015-10-03 01 fs.default.name is deprecated 1414 main INFO org.apache.hadoop.conf.Configuration.deprecation-fs.default.name is deprecated. Instead, use fs.defaultFS
2015-10-03 01 mapred.job.tracker is deprecated 14 main 12656 [mapred.job.tracker is deprecated] Instead, use mapreduce.jobtracker.address
2015-10-03 01 INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine 1412 885 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine-Connecting to hadoop file system at: file:///
2015-10-03 01 io.bytes.per.checksum is deprecated 1415 [main] 13573. Instead, use dfs.bytes-per-checksum
Grunt >
Log in in Mapreduce mode: actual working mode:
[hadoop@host63 ~] $pig
15-10-03 02:11:54 INFO pig.ExecTypeProvider: Trying ExecType: LOCAL
15-10-03 02:11:54 INFO pig.ExecTypeProvider: Trying ExecType: MAPREDUCE
15-10-03 02:11:54 INFO pig.ExecTypeProvider: Picked MAPREDUCE as the ExecType
2015-10-03 02 Apache Pig version 1115 086 [main] INFO org.apache.pig.Main-Apache Pig version 0.15.0 (r1682971) compiled Jun 01 2015, 11:44:35
2015-10-03 02 Logging error messages to 1115 087 [main] INFO org.apache.pig.Main-Logging error messages to: / home/hadoop/pig_1443863515062.log
2015-10-03 02 Default bootup file 11 main 55271 [main] INFO org.apache.pig.impl.util.Utils-Default bootup file / home/hadoop/.pigbootup not found
2015-10-03 02 mapred.job.tracker is deprecated 11 main 59735 [mapred.job.tracker is deprecated]. Instead, use mapreduce.jobtracker.address
2015-10-03 02 fs.default.name is deprecated 11 main 59740 [fs.default.name is deprecated]. Instead, use fs.defaultFS
2015-10-03 02 Connecting to hadoop file system at 11 Connecting to hadoop file system at 59742 [main] Connecting to hadoop file system at: hdfs://host61:9000/
2015-10-03 02 mapred.job.tracker is deprecated 12 main [06256] mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-10-03 02 Connecting to map-reduce job tracker at 12 Connecting to map-reduce job tracker at 06257 [main] Connecting to map-reduce job tracker at: host61:9001
2015-10-03 02 fs.default.name is deprecated 12 fs.default.name is deprecated 06265 [main]. Instead, use fs.defaultFS
Grunt >
5.pig operates in the following three ways:
1. Script
2.grunt
3. Embedded system
6. Log in to pig and use the common commands:
[hadoop@host63 ~] $pig
15-10-03 06:01:01 INFO pig.ExecTypeProvider: Trying ExecType: LOCAL
15-10-03 06:01:01 INFO pig.ExecTypeProvider: Trying ExecType: MAPREDUCE
15-10-03 06:01:01 INFO pig.ExecTypeProvider: Picked MAPREDUCE as the ExecType
2015-10-03 06 Apache Pig version 01lo 01412 [main] INFO org.apache.pig.Main-Apache Pig version 0.15.0 (r1682971) compiled Jun 01 2015, 11:44:35
2015-10-03 006 Logging error messages to 01lo 01413 [main] INFO org.apache.pig.Main-Logging error messages to: / home/hadoop/pig_1443877261408.log
2015-10-03 06 Default bootup file 01lo 01502 [main] INFO org.apache.pig.impl.util.Utils-Default bootup file / home/hadoop/.pigbootup not found
2015-10-03 006 mapred.job.tracker is deprecated 01V 03657 [main] mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-10-03 006 fs.default.name is deprecated 01V 03657 [main] fs.default.name is deprecated. Instead, use fs.defaultFS
2015-10-03 006 Connecting to hadoop file system at 01lo 03662 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine-Connecting to hadoop file system at: hdfs://host61:9000/
2015-10-03 006 INFO org.apache.hadoop.conf.Configuration.deprecation-mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-10-03 06 Connecting to map-reduce job tracker at 01lo 05968 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine-Connecting to map-reduce job tracker at: host61:9001
2015-10-03 006 INFO org.apache.hadoop.conf.Configuration.deprecation-fs.default.name is deprecated. Instead, use fs.defaultFS
Grunt > help
Commands:
;-See the PigLatin manual for details: http://hadoop.apache.org/pig
File system commands:
Fs-Equivalent to Hadoop dfs command: http://hadoop.apache.org/common/docs/current/hdfs_shell.html
Diagnostic commands:
Describe [:: cd /
Show the current directory:
Grunt > ls
Hdfs://host61:9000/in
Hdfs://host61:9000/out
Hdfs://host61:9000/user
Grunt > cd / in
Grunt > ls
Hdfs://host61:9000/in/jdk-8u60-linux-x64.tar.gz 181238643
Hdfs://host61:9000/in/mytest1.txt 23
Hdfs://host61:9000/in/mytest2.txt 24
Hdfs://host61:9000/in/mytest3.txt 4
View file information:
Grunt > cat mytest1.txt
This is the first file
Copy the files in hdfs to the operating system:
Grunt > copyToLocal / in/mytest5.txt / home/hadoop/mytest.txt
[hadoop@host63 ~] $ls-l mytest.txt
-rw-r--r--. 1 hadoop hadoop 102 Oct 3 06:23 mytest.txt
Use the sh+ operating system commands to execute commands in the operating system in grunt:
Grunt > sh ls-l / home/hadoop/mytest.txt
-rw-r--r--. 1 hadoop hadoop 102 Oct 3 06:23 / home/hadoop/mytest.txt
7.pig 's data model:
Bag: tabl
Tuple: OK, record
Field: attribute
Pig does not require that different tuple in the same bag have the same number or type of field.
Common statements of 8.pig latin:
LOAD: point out how to load data
FOREACH: scan line by line and do some processing
FILTER: filtering rows
DUMP: display the results to the screen
STORE: save the results to a file
9. Sample data processing:
Generate test files:
[hadoop@host63 tmp] $ls-l / | awk'{if (NR! = 1) print $NF "#" $5}'> / tmp/mytest.txt
[hadoop@host63 tmp] $cat / tmp/mytest.txt
Bin#4096
Boot#1024
Dev#3680
Etc#12288
Home#4096
Lib#4096
Lib64#12288
Lost+found#16384
Media#4096
Mnt#4096
Opt#4096
Proc#0
Root#4096
Sbin#12288
Selinux#0
Srv#4096
Sys#0
Tmp#4096
Usr#4096
Var#4096
Load the file:
Grunt > records = LOAD'/ tmp/mytest.txt' USING PigStorage ('#') AS (filename:chararray,size:int)
2015-10-03 07 fs.default.name is deprecated. 35 main 48 479. Instead, use fs.defaultFS
2015-10-03 07 mapred.job.tracker is deprecated. 35 mapred.job.tracker is deprecated 35 main. Instead, use mapreduce.jobtracker.address
2015-10-03 07 io.bytes.per.checksum is deprecated 35 io.bytes.per.checksum is deprecated 35 main. Instead, use dfs.bytes-per-checksum
2015-10-03 0715 INFO org.apache.hadoop.conf.Configuration.deprecation-mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-10-03 07 io.bytes.per.checksum is deprecated. 35 main 48 723. Instead, use dfs.bytes-per-checksum
2015-10-03 07 fs.default.name is deprecated. 35 main 48 723. Instead, use fs.defaultFS
Display the file:
Grunt > DUMP records
(bin,4096)
(boot,1024)
(dev,3680)
(etc,12288)
(home,4096)
(lib,4096)
(lib64,12288)
(lost+found,16384)
(media,4096)
(mnt,4096)
(opt,4096)
(proc,0)
(root,4096)
(sbin,12288)
(selinux,0)
(srv,4096)
(sys,0)
(tmp,4096)
(usr,4096)
(var,4096)
Displays the structure of the records:
Grunt > DESCRIBE records
Records: {filename: chararray,size: int}
Filter records:
Grunt > filter_records = FILTER records BY size > 4096
Grunt > DUMP fileter_records
(etc,12288)
(lib64,12288)
(lost+found,16384)
(sbin,12288)
Grunt > DESCRIBE filter_records
Filter_records: {filename: chararray,size: int}
Grouping:
Grunt > group_records = GROUP records BY size
Grunt > DUMP group_records
(0, {(sys,0), (proc,0), (selinux,0)})
(1024, {(boot,1024)})
(3680, {(dev,3680)})
(4096, (var,4096), (usr,4096), (tmp,4096), (srv,4096), (root,4096), (opt,4096), (mnt,4096), (media,4096), (lib,4096), (home,4096), (bin,4096)})
(12288, {(etc,12288), (lib64,12288), (sbin,12288)})
(16384, {(lost+found,16384)})
Grunt > DESCRIBE group_records
Group_records: {group: int,records: {(filename: chararray,size: int)}}
Format:
Grunt > format_records = FOREACH group_records GENERATE group, FLATTEN (records)
De-weight:
Grunt > dis_records = DISTINCT records
Sort:
Grunt > ord_records = ORDER dis_records BY size desc
Take the first three rows of data:
Grunt > top_records=LIMIT ord_records 3
Find the maximum value:
Grunt > max_records = FOREACH group_records GENERATE group,MAX (records.size)
Grunt > DUMP max_records
(0pl 0)
(1024pr 1024)
(3680, 3680)
(4096 and 4096)
(12288, 12288)
(16384pm 16384)
View the execution plan:
Grunt > EXPLAIN max_records
Save the recordset:
Grunt > STORE group_records INTO'/ tmp/mytest_group'
Grunt > STORE filter_records INTO'/ tmp/mytest_filter'
Grunt > STORE max_records INTO'/ tmp/mytest_max'
10.UDF
Pig supports writing UDF using java,python,javascript
