Shulou(Shulou.com)05/31 Report--

This article mainly introduces "how to deal with Illegal partition for problems under hadoop". In daily operation, I believe many people have doubts about how to deal with Illegal partition for problems under hadoop. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful for you to answer the doubts of "how to deal with Illegal partition for problems under hadoop"! Next, please follow the editor to study!

1. Background:

Filter the key in one file, and the key that needs to be filtered is stored in another file. There are a large number of key to be filtered, hundreds of millions of them, so it is not appropriate to use methods such as bloomfilter. At the same time, there may be a lot of filtered key in the file or very little. Therefore, the secondary sorting method of value is used to filter, and the key that needs to be filtered is made into key0, and the real key is made into key1. Using the partition function and the key comparison function to send the same key to the same machine, you can determine whether the key needs to be filtered by determining whether the first key value, key0, exists.

two。 Question:

To use secondary sorting, it is necessary to implement the getPartition function. Considering that the hash function may return a negative number, the return uses such a function, Math.abs (k)% numReduceTasks. No problem was found at the beginning of the test, but after increasing the number of input files, it was found that the program would report an Illegal partition for error.

3. Process:

Directly turn to google, from the http://blog.csdn.net/hezuoxiang/article/details/6878026, you can see that the error is because the partition function returned a negative number, this value is unreasonable. I was a little confused for a moment. Didn't Math.abs (x) return a positive number? The return will be negative. Well, programmers using java to write hadoop are just a bunch of holes to step on.

4. Solution:

(1) change the hash function to make the hash value within the range that int can represent.

(2) take the absolute value Math.abs (k% numReduceTasks); for the value of the hash function.

At this point, the study on "how to deal with Illegal partition for problems under hadoop" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.