Shulou(Shulou.com)06/01 Report--

This article mainly shows you "datasketches how to help Hive calculate quantiles", the content is easy to understand, clear, hope to help you solve doubts, the following let Xiaobian lead you to study and learn "datasketches how to help Hive calculate quantiles" this article.

1. Add the following three jar packages to hive's hive.aux.jars.path directory, org.apache.datasketches datasketches-hive 1.1.0-incubating org.apache.datasketches datasketches-java 1.3.0-incubating org.apache.datasketches datasketches-memory 1.2.0-incubating2. Open the hive client, execute the command to add jar packages, and add 3 jar packages to add jar xxx.jar3. Create a function to find the quartile of the double type as an example

Create the function Q1, and the corresponding class is org.apache.datasketches.hive.quantiles.DataToDoublesSketchUDAF,q1. The function is to aggregate the data of a group.

Create a function Q2 with the class org.apache.datasketches.hive.quantiles.GetQuantileFromDoublesSketchUDF, and take the result of Q1 as the input parameter of Q2, and pass in a position parameter, such as 0.25, which means the lower quartile.

Hive creates a function statement:

Create [temporary] function [dbname.] function_name AS 'class_name'

Using two custom functions in sql, you can calculate the quantiles in a set of data.

These are all the contents of the article "how datasketches assists Hive in calculating quantiles". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.