Shulou(Shulou.com)06/03 Report--

We mainly introduce two aspects of hardware planning: one is how to plan Master, the other is how to plan Slave, the choice is different. Other plans such as capacity and operating system selection will also be briefly introduced, which is important to understand the operating environment of Hadoop.

First of all, let's take a look at Slave, which is used to store data and then calculate, and the processor is usually given priority when choosing a configuration. We know that the core of Hadoop is not how complex operations are performed on a single machine, it is distributed, so the requirement for processors is not high, so you can choose a medium data set (for example, 2 * 6 cores 2.9 main frequency).

As for memory, it should be as high as possible, with the middle end to 256GB RAM and the high end to 512GB RAM. The middle end of the network is given to 1GB Ethernet, and the high end is given to 10GB Ethernet. The focus here is on disk drives, 16*3TB SATA drivers (mid-range) and 24*1TB SAS drivers (high-end). We find that the more high-end disks, the higher the access efficiency. In the disk drive here will also involve a concept of Non-RAID, we know about it, you can mine your own.

Switches use dedicated network facilities, Hadoop will make resources saturated, nodes are connected to rack switches, and the racks communicate with each other through core switches.

Next, we focus on Master nodes. Master has no business data and does not need to be calculated, but Master stores active data, so Master nodes are very important. When using machines, if conditions permit, it is best to choose high-end machines, operator-level hardware, dual power supplies, Ethernet cards, and all modules are redundant. It is configured with Raid because the Master is the source data and there is no copy of the data. The cluster with less than 20 nodes is configured with 64GB RAM,300. The cluster with less than 20 nodes is configured with 96GB memory, and the larger cluster is configured with 128GB memory.

As for capacity planning and the choice of operating system, we don't have to explain it too much, just understand it. Capacity planning We focus on the replica mechanism and temporary space, as well as the space needed by the server itself. Here we must be aware that Hadoop automatically uses new nodes, and that many clusters start to be small (less than 10 nodes) and grow as data and processing grow, and Hadoop clusters can grow to thousands of nodes. While the operating system generally chooses distributions that are good at management, you can also learn about several: CentOS: for servers, not workstations; RedHat Enterprise Edition, the very popular release of linux;Ubuntu; the version that uses LTS (long-term support); and the very popular distribution of SuSE in Europe.

The above is an introduction to the Hadoop hardware environment based on my own experience. If there is anything unclear, such as Non-raid, you can find your own resources to recharge. I usually also like to watch some learning knowledge shared by others, so as to make up for the lack of my own knowledge system, such as the big data era Learning Center. In addition, I like to look at some actual big data cases, try to analyze the problems in the case, and constantly improve their ability to transfer knowledge, such as "big data cn". We encourage and make progress together.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.