Shulou(Shulou.com)06/01 Report--

This article focuses on "how to optimize the parameters of Hadoop", interested friends may wish to have a look. The method introduced in this paper is simple, fast and practical. Now let the editor take you to learn "how to tune the parameters of Hadoop"!

1. Before building a cluster, configure many directories in hdfs-site.xml in advance

2. Dfs.nammenode.handler.count=20*log2 (total number of clusters), which is the Namenode thread pool, which handles datanode heartbeats, client requests, etc.

3. Editslog storage paths dfs.nammenode.edits.dir and dfs.nammenode.name.dir are configured separately

4. Yarn defaults to 8g of physical memory, which can be adjusted low if the task is stuck, and will not be dynamically allocated. Yarn.nodemanager.resource.memory-mb

5. Optimize the parameters of mr. The ring buffer size io.sort.mb defaults to 100m, which can be adjusted according to cluster performance. The percentage of overwritten files io.sort.spill.percent defaults to 80%, which can be adjusted to 90% and 95% according to cluster performance, as well as the number of map tasks and the number of reduce pulls.

At this point, I believe you have a deeper understanding of "how to tune the parameters of Hadoop". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.