Shulou(Shulou.com)05/31 Report--

This article is about how Hive-f encapsulates support for passing parameters. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

Demand

} Hive-f

} hiveF encapsulates hive-f aa.sql

} support passing any number of parameters to separate shell script from sql file

} Java class name * .sql-date "2013-01-01"... .

Analysis.

Encapsulate the partition field ${date} in sql with hiveF to separate the value passed by the date field from the sql file

Create table tmp_test1 as select session_id, url, getcmsid (url) from page_views where dt=' {$date} 'and url like'% topicId%'

The java code is as follows

Main.java

Package hive.hiveF;import java.io.File;public class Main {/ * * @ param.. / * .sql-date "2013-01-01" * @ throws Exception * / public static void main (String [] args) throws Exception {/ / args = new String [5]; / / args [0] = "c:/test.sql"; / / args [1] = "- date" / / args [2] = "2013-01-01"; / / args [3] = "- date1"; vi hi// args [4] = "2013-11-11"; ParseArgs parse = new ParseArgs (args); String sql = Utils.getSql (args [0]); System.out.println (Utils.parse (sql, parse.getMap () / / System.out.println (parse.getMap (). Get ("date")); / / System.out.println ("select * from T1 limit 2");}}

ParesArgs.java (separate the date value in the-date "$date" in the shell script for execution in the sql file, see hive_test.sh,hive_test.hql)

Package hive.hiveF;import java.util.HashMap;import java.util.Map;public class ParseArgs {private Map map = null; public ParseArgs (String [] args) {map = new HashMap (); if (args.length = = 0) {return;} int iTuno; while (II) {value = args [I] .trim () If (value.startsWith ("\") | | value.startsWith ("\")) {value = value.substring (1meme value.length ()-1). Trim ();}} map.put (key, value); iTunes + } else {ionization;} public Map getMap () {return map;}}

Utils.java

Package hive.hiveF;import java.io.BufferedReader;import java.io.File;import java.io.FileNotFoundException;import java.io.FileReader;import java.util.HashMap;import java.util.Map;public class Utils {public static final String BEGIN = "{$"; public static final String END = "}"; public static String getSql (File file) throws Exception {BufferedReader bf = new BufferedReader (new FileReader (file)); StringBuffer sqlBuffer = new StringBuffer (); String temp = null While ((temp = bf.readLine ())! = null) {String tmp = temp.trim (); if (tmp.length () = = 0 | | tmp.startsWith ("#") | | tmp.startsWith ("-") {continue;} sqlBuffer.append (tmp+ ");} bf.close (); return sqlBuffer.toString () } / * replace the parameter reference in SQL with value * @ param sql * @ param map * / public static String parse (String sql, Map map) {int begin = sql.indexOf (BEGIN); while (begin! =-1) {String suffix = sql.substring (begin + BEGIN.length ()) in map / / data} int end = begin+BEGIN.length () + suffix.indexOf (END); String key = sql.substring (begin+BEGIN.length (), end). Trim (); if (mapping null & & map.get (key)! = null) {sql = sql.substring (0Magne begin) + map.get (key) + sql.substring (end+1,sql.length ()) } else {throw new RuntimeException ("Incalid Expression...");} begin = sql.indexOf (BEGIN);} return sql;}}

Prepare for

1. Create table page_views in hive

Create table page_views (sessionid string, url string) partitioed by (dt string) row format delimited fields terminated by

2. Create the file vi pageviews under the root directory

1 http://www.baidu.com/topicId=123123

2 http://www.baidu.com/topicId=14325245

3 http://www.baidu.com/topicId=432w32452

4 http://www.baidu.com/topicId=542351

4 http://www.baidu.com/topicId=5423e

5 http://www.baidu.com/topicId=54325342

6 http://www.baidu.com/topicId=2452

3 http://www.baidu.com/topicId=543222

3. Load data into page_views

Load data local inpath'/ root/pageviews' into table page_views partition (dt='2013-08-08')

4. Import a finished udf (getCmsId) reference UDF and UDAF development into / usr/hiveF/lib

Operation

1. Create a directory in the Linux file. HiveF/lib and HiveF/bin store jar packages and shell files, respectively.

2. Upload the above three java programs into HiveF/lib files as hiveF3.jar packages.

3. Go to the HiveF/bin directory and create two files, hive_test.sh,hive_test.hql,hiveF

Vi hiveF

. / etc/profile sql= `java-jar / usr/hiveF/lib/hiveF3.jar $* `echo "$sql" hive-e "$sql"

Vi hive_test.hql

Add jar / usr/hiveF/getId.jar; create temporary function getcmsid as' hive.GetCmsID';drop table tmp_test1;create table tmp_test1 as select sessionid, url, getcmsid (url) from page_views where dt=' {$date} 'and url like'% topicId%'

Vi hive_test.sh

#! / bin/bash#set-x. / etc/profilecd / opt/beifengbegin= `date-d-1days +% Ymure% m mure% d`begin = "2013-08-08" # hive-f. / hive_test.hqlhiveF3 / usr/hiveF/bin/hive_test.hql-date "$begin"

4. Add / usr/hiveF/bin to Path to save exit in / etc/profile

Source / etc/profile

5. Execute the shell script. / after the hive_test.sh is completed

Thank you for reading! This is the end of the article on "how to encapsulate support parameters for Hive-f". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it for more people to see!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.