Shulou(Shulou.com)06/01 Report--

This article is about what HDFS is suitable for in Hadoop. Xiaobian thinks it is quite practical, so share it with everyone for reference. Let's follow Xiaobian and have a look.

The Hadoop ecosystem has always been a hot spot in the big data space, including HDFS to talk about today, yarn to talk about in the future, mareduce, spark, hive, hbase, zookeeper to talk about, and so on.

HDFS, hadoop distributed file system, originated from Google GFS, but GFS is written in C++, Hadoop is Doug Cutting in Yahoo written in Java. Hadoop became an Apache top-level project in 2008.

application

What scenarios does HDFS apply to? Very large file storage, such as G or T units, because the basic unit of block inside HDFS is already 128MB. Note that there is a small file problem here. The mistake is that even a small file of 1K can occupy 128MB of hard disk. In fact, it still occupies 1K hard disk. However, the bottle neck of the small file problem is in the name node, because the name node needs to store the relevant information of files and blocks in memory. As the number of files increases, the memory of the name node is insufficient.(For example, millions of small files take up 300MB of memory). Of course, hdfs federation can solve the problem of insufficient memory of name node by sharding. HDFS is also suitable for "write once, read-many" scenarios, and its writing is append only, so it can't be changed if you want to. If you write multiple times, you should consider cassandra (see my previous article). HDFS files are usually written only by a single writer (a lease is used to ensure that only one writer can currently write to a file). HDFS is popular with businesses because it requires only common commodity hardware and does not require expensive high-availability hardware. HDFS is not suitable for data access methods that require low latency, because HDFS exchanges high throughput with latency.

Conceptual Blocks

HDFS, files are divided into block-sized chunks, each block is 128MB, some people will ask, why do you have to make it so large, mainly to shorten the proportion of seek time in the total hard disk read and write time, such as seek time needs 5 ms, and find time can only account for 0.5% of the total time, then the hard disk read and write time is about 1s, how many files can be worn in 1s, if the hard disk read and write is 128MB/s, then 128MB can be transferred, Therefore, the block size is defined as 128MB, which ensures that the time spent on hard disk operations is effectively spent on reads and writes rather than seeking. Of course, too big is not good, mapreduce map is usually in blocks, if the block is too small, mapreduce efficiency will be relatively low.

hdfs fsck $path -files -blocks -locations

The above command can be used to provide block information of the file, such as which machine the block is on and what the name is, so that you can further query the specific information of the block.

Namenodes and datanodes

Namesode manages namespaces, file system tree structures and file/directory metadata, which are persisted on hard disk in the following ways: namespace image and edit log. At the same time, the metadata of the block is also stored in namd node, which is stored in memory. As mentioned earlier, there are millions of small files that will take up 300MB of memory. Why isn't block information persistent? Because it changes and is rebuilt from datanodes when the system restarts.

There are several ways to backup a name node. One way is to write persistent hard disk information to both the local hard disk and the remote NFS mount. Another way is to run secondary namenode, which doesn't really play the role of namenode, but periodically merge namesapce image and edit log to prevent edit log from getting too large. It saves a merged namespace image, and once the primary fails, copies the metadata from the NFS to the secondary namespace, so that the secondary becomes the new primary.

The specific process is shown in the following figure, edit log and fsimage are both in the hard disk, edit log is WAL (Cassandra write operation also uses WAL means, WAL is very popular, you can pull it out once), fsimage is check point of the filesystem metadata. Write edit log first, then update in-memory representation of filesystem metadata (used to serve read requests), which is not shown in the figure.

Is there a better way? The above method fails to provide HA, and namenode is still a single point of failure. The new primary needs (1) to load namespace image into memory (2) to replay edit log (3) to receive enough block reports from datanodes (previously mentioned block information is in memory). This process can take 30 minutes or more. Client can't wait ~~

Hadoop 2 provides HA support. namenode uses active-standby configuration:

namenodes uses highly available shared storage to store edit logs. Each active write is read by standby and synchronized into its own memory.

Datanodes send block reports to all name nodes at the same time, remember that block mapping is in memory.

The client needs to be configured to handle namenode failover, which is actually the leader election of watch zookeeper (see zookeeper I talked about earlier)

This eliminates the need for a secondary namenode, standby replaces it and periodically generates check points.

The shared storage mentioned above mainly refers to QJM (quorum journal manager), usually configured with 3 (of course, I have also seen 50 nodes with 5 journal nodes), and quorum needs to be satisfied when writing.

This way, standby can withstand the active namenode failure immediately because the latest edit log and up-to-date block mapping are in memory.

HDFS write

HDFS read

CLI Exampletouch test.txthdfs dfs -mkdir /user/qingge/testdirhdfs dfs -copyFromLocal ./ test.txt /user/qingge/testdir/ hdfs dfs -ls /user/qingge/testdir/test.txthdfs dfs -chmod o-r /user/qingge/testdir/test.txt hdfs dfs -cat /user/qingge/testdir/test.txt |head -10 hdfs dfs -mv/user/qingge/testdir/test. txt/user/qingge/testdir/test2.txthdfs fsck /data/lalala -files -blocks -locationshdfs fsck -blockId block_101010HTTP access

(1) direct access: HDFS daemons server HTTP requests, embedded web servers in the name node and datanodes act as WebHDFS endpoionts.

(2)proxy access: Multiple HDFS proxies in the middle, for strictr firewall and bandwidth-limiting policies, RPC request and block request used between proxy and node.

HDFS Federation

Equivalent to namenode sharding, if you don't want to use HA, then namenode memory and how to do, answer partition ah, each namenode from the root directory to draw a few subdirectories, wireless partition wireless expansion, between each namenode well water does not interfere with river water, one explosion or waste does not affect the other.

Thinking Questions:

If HDFS has a capacity of 1 petabyte, the block size is 64MB, the average metadata size is 300 bytes per block, and the replication factor is 3, what is the minimum memory for a namenode?

A: Almost 1.56G, 1024*1024 * 1024MB/(64MB *3)* 300B/(1024 * 1024 * 1024) = 1.56GB

Thank you for reading! About "Hadoop HDFS applies to what scenarios" This article is shared here, I hope the above content can be of some help to everyone, so that everyone can learn more knowledge, if you think the article is good, you can share it to let more people see it!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.