In addition to Weibo, there is also WeChat
Please pay attention
WeChat public account
Shulou
2025-03-18 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > Development >
Share
Shulou(Shulou.com)06/02 Report--
This article focuses on "how to write UDF functions", interested friends may wish to have a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn how to write UDF functions.
1. Why do I need UDF?
1), because the internal function can not meet the demand.
2), hive itself is a flexible framework, allowing the use of custom module functions, such as UDF, serde, input and output, etc.
What is 2.UDF?
UDF:user difine function, user-defined function, one to one. Common udaf:user define aggregate function, user-defined aggregate function, many-to-one. Udtf:user define table_generate function, user-defined tables generate functions, one-to-many.
3. How to write UDF function?
1) this class needs to inherit UDF and override evaluate () to allow the method to be overloaded.
2), you can also inherit generic UDF, and you need to override initliaze (), getDisplay (), evaluate ()
The use of 4.UDF
The first one: (currently session is valid)
Package edu.qianfeng.UDF;import org.apache.hadoop.hive.ql.exec.UDF;/** * use Java custom UDF * @ author lyd * * 1603 1603_class * / public class FirstUDF extends UDF {/ / rewrite evaluate () public String evaluate (String str) {/ / judge if (str = = null) {return null;} return str+ "_ class";}}
1. Add the jar package of custom UDF
Hive > add jar / home/h3h.jar
2. Create a temporary function
Hive > create temporary function myfunc as "edu.qianfeng.UDF.FirstUDF"
3. Test whether the addition is good:
Show functions;select myfunc ("1603")
4. You can delete it when it is determined to be useless:
Drop temporary function myfunc
Second: (currently session is valid)
Package edu.qianfeng.UDF;import org.apache.hadoop.hive.ql.exec.UDF;import org.json.JSONException;import org.json.JSONObject;public class KeyToValue extends UDF {public String evaluate (String str,String key) {if (str = = null | | key = = null) {return null;} / / sex=1&hight=180&weight=130&sal=28000 / / {sex:} String str1 = str.replace ("=", ":"); String str2 = str1.replace ("&", ","); String str3 = "{" + str2+ "}" String value= "; try {JSONObject jo = new JSONObject (str3); value= jo.get (key). ToString ();} catch (JSONException e) {e.printStackTrace ();} return value;} public static void main (String [] args) {System.out.println (new KeyToValue (). Evaluate (" sex=1&hight=180&weight=130&sal=28000&facevalue=900 "," facevalue ");}}
1. Add the jar package of custom UDF (the jar of hive.aux.jars.path in this directory will be loaded automatically when hive starts)
Hive.aux.jars.path $HIVE_HOME/auxlib
Cp / home/h3h.jar $HIVE_HOME/auxlib/
2. Start hive and create a temporary function
Hive > create temporary function ktv as "edu.qianfeng.UDF.KeyToValue"
3. Test whether the addition is good:
Show functions;select myfunc ("1603")
4. You can delete it when it is determined to be useless:
Drop temporary function myfunc
Third: (currently session is valid)
1. Create an initialization file:
Vi. / init-hiveadd jar / home/h3h.jar;create temporary function ktv1 as "edu.qianfeng.UDF.KeyToValue"
2. Start the command:
Hive-I. / init-hive
3. Test whether the addition is good:
Show functions;select myfunc ("1603")
4. You can delete it when it is determined to be useless:
Drop temporary function myfunc
Fourth: (make it permanent)
Package edu.qianfeng.UDF;import java.text.ParseException;import java.text.SimpleDateFormat;import java.util.Calendar;import java.util.Date;import org.apache.hadoop.hive.ql.exec.UDF / * author lyd * * 2000-04-19 17 * 2000-04-20 16 * 2000-04-21 16 * * / public class BirthdayToAge extends UDF {public static void main (String [] args) {System.out.println (new BirthdayToAge (). Evaluate ("2000-04-20"));} public String evaluate (String birthday) {if (birthday = = null | | birthday.trim (). IsEmpty ()) {return null;} String age = "" Try {/ / get the time of birth SimpleDateFormat sdf = new SimpleDateFormat ("yyyy-MM-dd"); / / get the timestamp of birth long bithdayts = sdf.parse (birthday). GetTime (); / get today's timestamp long nowts = new Date (). GetTime (); / / get the number of seconds since birth long alldays = (nowts-bithdayts) / 1000lb 60lb 60lb 24; if (alldays < 0) {return null } / / judge leap year int birthdayyear = Integer.parseInt (birthday.split ("-") [0]); Calendar ca = Calendar.getInstance (); int nowyear = ca.get (Calendar.YEAR); / / Loop search int rnday = 0; for (int I = birthdayyear; I < nowyear; iyears +) {if ((I% 400 = = 0) | | (I% 4 = = 0 & i0! = 0) {rnday + +;}} / / subtract the number of frontal days in leap years by age = (alldays-rnday) / 365 + "" } catch (ParseException e) {return null;} return age;}}
The source code needs to be compiled. 1) copy the written Jave file to ~ / install/hive-0.8.1/src/ql/src/java/org/apache/hadoop/hive/ql/UDF/
Cd ~ / install/hive-0.8.1/src/ql/src/java/org/apache/hadoop/hive/ql/UDF/ls-lhgt | head
2) modify ~ / install/hive-0.8.1/src/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java to add import and RegisterUDF
Import com.meilishuo.hive.UDF.UDFIp2Long; / / add importregisterUDF ("ip2long", UDFIp2Long.class, false); / / add register
3) run ant-Dhadoop.version=1.0.1 package under ~ / install/hive-0.8.1/src
Cd ~ / install/hive-0.8.1/srcant-Dhadoop.version=1.0.1 package
4) replace the jar package of exec. The newly generated package is in the / hive-0.8.1/src/build/ql directory. Replace the link.
Cp hive-exec-0.8.1.jar / hadoop/hive/lib/hive-exec-0.8.1.jar.0628rm hive-exec-0.8.1.jarln-s hive-exec-0.8.1.jar.0628 hive-exec-0.8.1.jar
5) restart for testing
At this point, I believe you have a deeper understanding of "how to write UDF functions", might as well come to the actual operation of it! Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!
Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.
Views: 0
*The comments in the above article only represent the author's personal views and do not represent the views and positions of this website. If you have more insights, please feel free to contribute and share.
Continue with the installation of the previous hadoop.First, install zookooper1. Decompress zookoope
"Every 5-10 years, there's a rare product, a really special, very unusual product that's the most un
© 2024 shulou.com SLNews company. All rights reserved.