Shulou(Shulou.com)06/01 Report--

Hive how to set variable transfer, I believe that many inexperienced people do not know what to do, so this paper summarizes the causes of the problem and solutions, through this article I hope you can solve this problem.

# org.apache.hadoop.hive.ql.parse.VariableSubstitution:

Public String substitute (HiveConf conf, String expr) {if (expr = = null) {return expr;} if (HiveConf.getBoolVar (conf, ConfVars.HIVEVARIABLESUBSTITUTE)) {l4j.debug ("Substitution is on:" + expr);} else {return expr;} int depth = HiveConf.getIntVar (conf, ConfVars.HIVEVARIABLESUBSTITUTEDEPTH); return substitute (conf, expr, depth);} 123456789101112

If hive.variable.substitute is enabled (default is enabled), further judgment is made using SystemVariables's substitute method and hive.variable.substitute.depth (default is 40):

Protected final String substitute (Configuration conf, String expr, int depth) {Matcher match = varPat.matcher (""); String eval = expr; StringBuilder builder = new StringBuilder (); int s = 0; for (; s depth) {throw new IllegalStateException ("Variable substitution depth is deeper than" + depth + "for expression" + expr);} return eval;} 1234567891011121315161718192021223242627293031323334

If the ${} parameter used exceeds the number of hive.variable.substitute.depth, an exception is thrown directly, so we directly add set hive.variable.substitute.depth=100; to the front of the statement to solve the problem!

The execution of the set command is specifically executed in the CommandProcessor implementation class SetProcessor, but the substitute statement is also called in CompileProcessor, that is, when the hive statement is compiled, so when oozie calls beeline to execute the statement when it is in use, the compile phase reports an exception.

But why does Hue have no problem executing this statement directly? Because hue uses beeswax developed by python when executing hive, and beeswax directly processes these variables, replaces the variables with the actual values of the variables, and then submits them to hive for execution:

Def substitute_variables (input_data, substitutions): "Replaces variables with values from substitutions." Def f (value): if not isinstance (value, basestring): return value new_value = Template (value). Safe_substitute (substitutions) if new_value! = value: LOG.debug ("Substituted% s->% s"% (repr (value), repr (new_value)) return new_value return recursive_walk (f, input_data) after reading the above, have you mastered the method of how to set variable transfer in hive? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.