Shulou(Shulou.com)06/02 Report--

This article mainly explains "what is the difference and relationship between order by and distribute by in hive". Interested friends may wish to have a look. The method introduced in this paper is simple, fast and practical. Now let the editor take you to learn "what is the difference and relationship between order by and distribute by in hive?"

Order by

Order by sorts the data globally, just like order by in databases such as oracle and mysql, which is only done in one reduce, so it is very inefficient when the amount of data is very large.

And if limit is not specified when setting: set hive.mapred.mode=strict, an error will be reported when executing select, as shown below:

LIMIT must also be specified .

Sort by

Sort by is sorted separately in its own reduce, so global order is not guaranteed, it is generally executed with distribute by, and distribute by is written before sort by.

If mapred.reduce.tasks=1 has the same effect as order by, if it is greater than 1, it will be divided into several files and each file will be sorted according to the specified field, without ensuring global order.

Sort by is not affected by whether hive.mapred.mode is strict or nostrict.

Distribute by

DISTRIBUTE BY controls how the output in map is divided in reducer. Using DISTRIBUTE BY ensures that records of the same KEY are divided into a Reduce.

Cluster by

The combination of distribute by and sort by is equivalent to cluster by, but cluster by cannot specify a rule that sorts as asc or desc, only in ascending order.

At this point, I believe you have a deeper understanding of "what is the difference and relationship between order by and distribute by in hive". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.