Shulou(Shulou.com)05/31 Report--

This article is about how Hbase writes to hdfs. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

The following figure is a top-level structure diagram that describes the underlying storage structure of Hbase. You can see that Hbase will handle both HFile files (memstore generation) and HLog files (WAL generation). When it is really stored in HDFS, it will use DFS Client as the client of hdfs to write large quantities of these two kinds of data streams to multiple DataNode nodes.

In the previous analysis of the wal thread model source code, in order to highlight the wal thread model, it did not specify what the writer instance is in writer.append () and writer.sync (). It was modified by the volatile keyword in FSHLog and declared as an interface of WALProvider.Writer type:

Here we only discuss using hdfs as the file system for hbase, that is, fs (System) in the init parameter is an instance of DistributedFileSystem. In addition to the path parameter indicating the path of the file to be created in hdfs is important, there is also a replication parameter in its createNonRecursive implementation, which indicates that the number of backups is the number of datanode writes. Dfs in DistributedFileSystem is an instance reference to DFSClient, which is the DFSClient referred to in the original architecture diagram. Hbase uses DFSClient's create method to create a file to hdfs's namenode through a RPC call and constructs an output stream DFSOutputStream instance. Another key point of this method starts a pipeline, which is called streamer.start (), which is an implementation written by hbase to multiple datanode pipes of hdfs. Although what is analyzed here is the writing process of wal, in fact, keyvalue is written to memstore, and then written to HFile, which is also implemented by pipeline in this way.

The create function of NameNode is called through rpc, the namesystem.startFile function is called, and the startFileInternal function is called, which creates a new file with a state of under construction without any data block corresponding to it. When successfully created at the same time, an instance of DFSOutputStream type is returned, which is called wrappedStream in FSDataOutputStream, and this object is responsible for handling the communication between datanode and namenode.

Hdfs file structure, HDFS a file consists of multiple block (default 64MB). Through the comments here, you can see that HDFS reads and writes block in units of packet (64K by default for each packet). Each packet consists of several chunk (the default 512Byte). Chunk is the basic unit for data verification, generating a checksum (the default 4Byte) for each chunk and storing it.

After analyzing this, we can see that the process of writing hbase files to hdfs is not special. Hdfs regards hbase as the client of hdfs and then encapsulates it into chunk and assembles it into packet, and then writes data to datanode in batches. In order to ensure the orderly transmission of the data, the data sending queue dataqueue and the waiting confirmation queue ackqueue are used, and two threads DFSOutputStream$DataStreamer and DFSOutputStream$DataStreamer$ResponseProcessor (in run ()) are used to send the data to the corresponding block and confirm whether the data has arrived.

Another key point is how hbase gets the data to datanode's disk.

Here, we want to go back to the ProtobufLogWriter class because writer.sync () finally calls the writer method of ProtobufLogWriter, and its source code is as follows:

As explained in the comments, hflush is synchronized only to ensure that the new reader can see it, but there is no guarantee that it is really persisted in every datanode, that is, the fsync () system call in posix is not actually called. It simply brushes the data written by the client to the OS cache (store) of each DataNode, which will result in data loss if the DataNode of each copy is crash at the same time (for example, when the computer room is powered off).

Hdfs also provides the client with another semantic hsync:client that all the data is sent to each datanode of the copy, and each copy on the datanode completes the call to fsync in the posix, that is, the operating system has brushed the data to the disk (of course, the disk may also buffer the data). It is important to note that when calling fsync, only the current block will be flushed to the disk. If you want each block to be flushed to the disk, you must pass in the Sync tag when creating the stream.

Hbase currently chooses hflush semantics. The flushOrsync method called by both semantics, where the isSync called by hflush passes in false and the hsync passes in true.

WaitForAckedSeqno () is used to wait for the ack in the ackqueue to come back and be awakened.

Thank you for reading! This is the end of the article on "how to write Hbase into hdfs". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it out for more people to see!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.