Shulou(Shulou.com)06/02 Report--

This article shows you how to use HDFS tiered storage in CDH. The content is concise and easy to understand, which will definitely brighten your eyes. I hope you can get something through the detailed introduction of this article.

Configure and use HDFS tiered Stora

Modify the DataNode data directory on CM to set six SSD disks to SSD, another sixteen disks, six to ARCHIVE and ten to DISK

1. Test using SSD storage, execute wordcount

Capacity of disk space before jobs are not submitted

Set the policy of the HDFS data directory for submitting wordcount tasks to ALL_SSD

Execute the script that generates the data to generate 1TB test data

Looking at the disk after generating the data, only the SSD capacity has increased.

Submit wordcount task

After the completion of the wordcount task, check the disk. Because the directory where the intermediate data generated during the execution of wordcount falls on the disk does not specify a storage policy, the hot policy is used by default, resulting in an increase in the amount of directory data stored by DISK.

two。 Test using ARCHIVE storage, execute sort

Capacity of disk space before jobs are not submitted

Set the policy of the HDFS data directory for submitting sort tasks to cold

Execute the script that generates the data to generate 1TB test data

Looking at the disk after generating the data, only the disk capacity of the ARCHIVE type has increased.

Submit sort task

After the completion of the sort task, check the disk. Because the directory of the intermediate data generated by sort during execution does not specify a storage policy, the hot policy is used by default. As a result, in addition to the growth of the directory stored by ARCHIVE, the amount of directory data stored by DISK also increases.

3. Test using DISK storage, execute terasort

Capacity of disk space before jobs are not submitted

Set the policy of the HDFS data directory for submitting terasort tasks to hot

Execute the script that generates the data to generate 1TB test data

Looking at the disk after generating the data, only the directory data stored by DISK has grown.

Submit terasort task

After the completion of the terasort task, I looked at the disk and found that only the directory data stored by DISK had grown.

Summary

1. You can configure the data directory of HDFS on CM, configure the storage type of each disk, and then specify the storage policy for the corresponding HDFS when using HDFS, so that the specified data can be stored to the disk of the corresponding storage type, realizing the hierarchical storage of HDFS.

two。 When using HDFS hierarchical storage, you need to pay attention to the allocation of data. For frequently used data, you can store it on SSD, for archived data can be stored on ARCHIVE-type disks, and for some commonly used basic data can be stored on DISK-type disks. Reasonable distribution of data can make the best performance of all disks and achieve the highest cost performance.

Github of Fayson:

Https://github.com/fayson/cdhproject

The above is how CDH uses HDFS tiered storage. Have you learned any knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.