Shulou(Shulou.com)06/03 Report--

Deployment: divided into test environment and production environment.

Hadoop distributions: Apache, CDH (tar package, rpm package, parcel package, which must be managed by cloudera manager), HDP

Note: big data disables IPV6; in the framework while modifying the maximum number of processes for users and the maximum number of files opened by a single process.

Three major components of Hadoop 2: 1. HDFS 2.MapReduce 3 YARN

HDFS: distributed file system, which is functionally divided into namenode and datanode. Namenode is used to store cluster metadata, such as the file name layer, which datanode the file is stored on, whether the file is sliced, etc.; the data is stored in block; the data in namenode is stored in memory, and the local disk includes mirror files and editing logs; it is stored in blocks, and the default size in V2 is 128MB. Heartbeats are sent between datanode and namenode (proof of survival status, default is every 3 seconds, the heartbeat return result contains instructions sent by namenode to datanode, such as copying or deleting data blocks), block status report (reported every other hour by default), etc., are communicated through underlying protocols such as RPC The default replica is 3, and the default policy is that two replicas are on different nodes of the same rack, and the third replica is on a different rack.

YARN architecture: divided into node manager (each node) and resource manager (managing the whole cluster)

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.