Shulou(Shulou.com)06/03 Report--

Preface

Since hive 0.13, permanent function; has been added to allow user-defined function to increase the startup time of hive without adding create temporary function to the .hiverc file (without pre-executing the command to create temporary functions), and the udf jar package can be placed on hdfs for easy management without pushing udf to the hive client side. But there is a problem with permanent function, that is, you need to add the database name before the function name, that is, [database]. [function]; if you do not enter database, this will automatically complete the function with the current database.

With reference to traditional relational databases, there is generally a default schema. When searching for function, priority is given to searching the default schema;, such as iopostgresql's pg_catalog, etc. So I want to add the default database feature to Hive.

Design

Add two parameters

Hive.function.default.function.enabled defaults to: false, which means that this feature is disabled; set to true, which means that the feature is enabled; and when searching for functions, the default database is found first.

Hive.function.default.function defaults to: default; takes effect when hive.function.default.function.enabled=true; default function search path.

Realize

You need to add this feature, and you need to know when permanent function will use the current database to complete function.

FunctionRegistry.java

Private static FunctionInfo getFunctionInfoFromMetastore (String functionName) {FunctionInfo ret = null; try {String dbName; String fName; if (FunctionUtils.isQualifiedFunctionName (functionName)) {String [] parts = FunctionUtils.splitQualifiedFunctionName (functionName); dbName = parts [0]; fName = parts [1];} else {/ / otherwise, qualify using current db dbName = SessionState.get (). GetCurrentDatabase (). ToLowerCase (); fName = functionName } / / Try looking up function in the metastore HiveConf conf = SessionState.get (). GetConf (); Function func = Hive.get (conf) .getFunction (dbName, fName); if (func! = null) {/ / Found UDF in metastore-now add it to the function registry / / At this point we should add any relevant jars that would be needed for the UDf. Try {FunctionTask.addFunctionResources (func.getResourceUris ());} catch (Exception e) {LOG.error ("Unable to load resources for" + dbName + "." + fName + ":" + e.getMessage (), e); return null;} Class udfClass = Class.forName (func.getClassName (), true, Utilities.getSessionSpecifiedClassLoader ()) If (registerTemporaryFunction (functionName, udfClass)) {ret = mFunctions.get (functionName);} else {LOG.error (func.getClassName () + "is not a valid UDF class and was not registered.") } catch (HiveException e) {if (! ((e.getCause ()! = null) & & (e.getCause () instanceof MetaException)) & & (e.getCause (). GetCause ()! = null) & & (e.getCause (). GetCause () instanceof NoSuchObjectException) {LOG.info ("Unable to lookup UDF in metastore:" + e) } catch (ClassNotFoundException e) {/ / Lookup of UDf class failed LOG.error ("Unable to load UDF class:" + e);} return ret;} public static String getNormalizedFunctionName (String fn) {/ / Does the same thing as getFunctionInfo, except for getting the function info. Fn = fn.toLowerCase (); return (FunctionUtils.isQualifiedFunctionName (fn) | | mFunctions.get (fn)! = null)? Fn: FunctionUtils.qualifyFunctionName (fn, SessionState.get (). GetCurrentDatabase (). ToLowerCase ());} private static T getFunctionInfo (Map mFunctions, String functionName) {functionName = functionName.toLowerCase (); T functionInfo = null; if (FunctionUtils.isQualifiedFunctionName (functionName)) {functionInfo = getQualifiedFunctionInfo (mFunctions, functionName);} else {/ / First try without qualifiers-would resolve builtin/temp functions. / / Otherwise try qualifying with current db name. FunctionInfo = mFunctions.get (functionName); if (functionInfo = = null & &! FunctionUtils.isQualifiedFunctionName (functionName)) {String qualifiedName = FunctionUtils.qualifyFunctionName (functionName, SessionState.get (). GetCurrentDatabase (). ToLowerCase ()); functionInfo = getQualifiedFunctionInfo (mFunctions, qualifiedName);}} return functionInfo;}

FunctionUtils.java

Public static String [] getQualifiedFunctionNameParts (String name) throws HiveException {if (isQualifiedFunctionName (name)) {return splitQualifiedFunctionName (name);} String dbName = SessionState.get (). GetCurrentDatabase (); return new String [] {dbName, name};}

Add one to the code to determine whether hive.function.default.function.enabled is true, and if so, adjust the default dbName to hive.function.default.function.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.