Shulou(Shulou.com)06/02 Report--

This article focuses on "how to write UDF functions", interested friends may wish to have a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn how to write UDF functions.

1. Why do I need UDF?

1), because the internal function can not meet the demand.

2), hive itself is a flexible framework, allowing the use of custom module functions, such as UDF, serde, input and output, etc.

What is 2.UDF?

UDF:user difine function, user-defined function, one to one. Common udaf:user define aggregate function, user-defined aggregate function, many-to-one. Udtf:user define table_generate function, user-defined tables generate functions, one-to-many.

3. How to write UDF function?

1) this class needs to inherit UDF and override evaluate () to allow the method to be overloaded.

2), you can also inherit generic UDF, and you need to override initliaze (), getDisplay (), evaluate ()

The use of 4.UDF

The first one: (currently session is valid)

Package edu.qianfeng.UDF;import org.apache.hadoop.hive.ql.exec.UDF;/** * use Java custom UDF * @ author lyd * * 1603 1603_class * / public class FirstUDF extends UDF {/ / rewrite evaluate () public String evaluate (String str) {/ / judge if (str = = null) {return null;} return str+ "_ class";}}

1. Add the jar package of custom UDF

Hive > add jar / home/h3h.jar

2. Create a temporary function

Hive > create temporary function myfunc as "edu.qianfeng.UDF.FirstUDF"

3. Test whether the addition is good:

Show functions;select myfunc ("1603")

4. You can delete it when it is determined to be useless:

Drop temporary function myfunc

Second: (currently session is valid)

Package edu.qianfeng.UDF;import org.apache.hadoop.hive.ql.exec.UDF;import org.json.JSONException;import org.json.JSONObject;public class KeyToValue extends UDF {public String evaluate (String str,String key) {if (str = = null | | key = = null) {return null;} / / sex=1&hight=180&weight=130&sal=28000 / / {sex:} String str1 = str.replace ("=", ":"); String str2 = str1.replace ("&", ","); String str3 = "{" + str2+ "}" String value= "; try {JSONObject jo = new JSONObject (str3); value= jo.get (key). ToString ();} catch (JSONException e) {e.printStackTrace ();} return value;} public static void main (String [] args) {System.out.println (new KeyToValue (). Evaluate (" sex=1&hight=180&weight=130&sal=28000&facevalue=900 "," facevalue ");}}

1. Add the jar package of custom UDF (the jar of hive.aux.jars.path in this directory will be loaded automatically when hive starts)

Hive.aux.jars.path $HIVE_HOME/auxlib

Cp / home/h3h.jar $HIVE_HOME/auxlib/

2. Start hive and create a temporary function

Hive > create temporary function ktv as "edu.qianfeng.UDF.KeyToValue"

3. Test whether the addition is good:

Show functions;select myfunc ("1603")

4. You can delete it when it is determined to be useless:

Drop temporary function myfunc

Third: (currently session is valid)

1. Create an initialization file:

Vi. / init-hiveadd jar / home/h3h.jar;create temporary function ktv1 as "edu.qianfeng.UDF.KeyToValue"

2. Start the command:

Hive-I. / init-hive

3. Test whether the addition is good:

Show functions;select myfunc ("1603")

4. You can delete it when it is determined to be useless:

Drop temporary function myfunc

Fourth: (make it permanent)

Package edu.qianfeng.UDF;import java.text.ParseException;import java.text.SimpleDateFormat;import java.util.Calendar;import java.util.Date;import org.apache.hadoop.hive.ql.exec.UDF / * author lyd * * 2000-04-19 17 * 2000-04-20 16 * 2000-04-21 16 * * / public class BirthdayToAge extends UDF {public static void main (String [] args) {System.out.println (new BirthdayToAge (). Evaluate ("2000-04-20"));} public String evaluate (String birthday) {if (birthday = = null | | birthday.trim (). IsEmpty ()) {return null;} String age = "" Try {/ / get the time of birth SimpleDateFormat sdf = new SimpleDateFormat ("yyyy-MM-dd"); / / get the timestamp of birth long bithdayts = sdf.parse (birthday). GetTime (); / get today's timestamp long nowts = new Date (). GetTime (); / / get the number of seconds since birth long alldays = (nowts-bithdayts) / 1000lb 60lb 60lb 24; if (alldays < 0) {return null } / / judge leap year int birthdayyear = Integer.parseInt (birthday.split ("-") [0]); Calendar ca = Calendar.getInstance (); int nowyear = ca.get (Calendar.YEAR); / / Loop search int rnday = 0; for (int I = birthdayyear; I < nowyear; iyears +) {if ((I% 400 = = 0) | | (I% 4 = = 0 & i0! = 0) {rnday + +;}} / / subtract the number of frontal days in leap years by age = (alldays-rnday) / 365 + "" } catch (ParseException e) {return null;} return age;}}

The source code needs to be compiled. 1) copy the written Jave file to ~ / install/hive-0.8.1/src/ql/src/java/org/apache/hadoop/hive/ql/UDF/

Cd ~ / install/hive-0.8.1/src/ql/src/java/org/apache/hadoop/hive/ql/UDF/ls-lhgt | head

2) modify ~ / install/hive-0.8.1/src/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java to add import and RegisterUDF

Import com.meilishuo.hive.UDF.UDFIp2Long; / / add importregisterUDF ("ip2long", UDFIp2Long.class, false); / / add register

3) run ant-Dhadoop.version=1.0.1 package under ~ / install/hive-0.8.1/src

Cd ~ / install/hive-0.8.1/srcant-Dhadoop.version=1.0.1 package

4) replace the jar package of exec. The newly generated package is in the / hive-0.8.1/src/build/ql directory. Replace the link.

Cp hive-exec-0.8.1.jar / hadoop/hive/lib/hive-exec-0.8.1.jar.0628rm hive-exec-0.8.1.jarln-s hive-exec-0.8.1.jar.0628 hive-exec-0.8.1.jar

5) restart for testing

At this point, I believe you have a deeper understanding of "how to write UDF functions", might as well come to the actual operation of it! Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.