Shulou(Shulou.com)05/31 Report--

What this article shares to you is about how to solve the problem of RegionServer crash caused by HBase Replication when a large number of data are written. The editor thinks it is very practical, so I share it with you. I hope you can get something after reading this article.

HBase provides Replication functionality in versions later than 0.90. these days, when I was testing this feature, I found that RegionServer crashed when a large amount of data (> 100W) was written. The exception log is as follows:

2014-10-29 10 DataInputStream.java:180 40 at java.io.DataInputStream.readLong 44 225 WARN org.apache.hadoop.hdfs.DFSClient: DFSOutputStream ResponseProcessor exception for block blk_-2223802775658985697_1410java.io.EOFException at java.io.DataInputStream.readFully (DataInputStream.java:180) at java.io.DataInputStream.readLong (DataInputStream.java:399) at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields (DataTransferProtocol.java:124) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run (DFSClient.java:2967) ) 2014-10-29 10-40 INFO org.apache.hadoop.hdfs.DFSClient: Could not obtain block blk_-2223802775658985697_1410 from any node: java.io.IOException: No live nodes contain current block. Will get new block locations from namenode and retry...2014-10-29 10 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_-2223802775658985697_1410 bad datanode [0] 192.168.11.55 WARN org.apache.hadoop.hdfs.DFSClient 40010 2014-10-29 10 WARN org.apache.hadoop.hdfs.DFSClient: Error while syncingjava.io.IOException: All datanodes 192.168.11.55 WARN org.apache.hadoop.hdfs.DFSClient 40010 are bad. Aborting... At org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError (DFSClient.java:3096) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2100 (DFSClient.java:2589) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run (DFSClient.java:2793) 2014-10-29 10 at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2100 40 44235 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not sync. Requesting close of hlogjava.io.IOException: Reflection at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync (SequenceFileLogWriter.java:310) at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer (HLog.java:1366) at org.apache.hadoop.hbase.regionserver.wal.HLog.sync (HLog.java:1476) at org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer (HRegion.java:5970) at Org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation (HRegion.java:2490) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate (HRegion.java:2190) at org.apache.hadoop.hbase.regionserver.HRegionServer.multi (HRegionServer.java:3888) at sun.reflect.GeneratedMethodAccessor33.invoke (Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke ( Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call (WritableRpcEngine.java:323) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run (HBaseServer.java:1434) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.GeneratedMethodAccessor35.invoke (Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke (Method.java:597 ) at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync (SequenceFileLogWriter.java:308)... 11 moreCaused by: java.io.IOException: DFSOutputStream is closed at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.sync (DFSClient.java:3669) at org.apache.hadoop.fs.FSDataOutputStream.sync (FSDataOutputStream.java:97) at org.apache.hadoop.io.SequenceFile$Writer.syncFs (SequenceFile.java:995)... 15 more

In fact, this problem is not caused by the Replication function, but by the client timeout during data-intensive writes

= the following comes from the network = =

Normally, the process for DFSClient to write block data is:

1. DFSClient side

A) DFSOutputStream is responsible for receiving and writing data, that is, obtaining data through the write method (synchronized) in DFSOutputSummer, while sync (main code synchronized (this)) writes data to dataQueue through enqueuePacket after establishing packet through FlushBuffer.

B) DataStreamer (Daemon thread) in DFSOutputStream, which is responsible for sending data to DataNode. It checks whether there is any data in dataQueue before sending it, and waits if it is not.

C) when DataStreamer sets up pipeline to transmit data, it will set up a ResponseProcessor (Thread) to the pipeline to get the feedback ack of DataNode, and judge whether there is any error, recoverBlock, etc.

2. DataNode side

A) in each packet transmission process, according to the pipleLine that establishes the data transmission, the upstream sends data to the downstream in turn, and the downstream sends ack to the upstream in turn.

B) the last node of pipeline (numTarget=0), PacketResponder will always run lastDatanodeRun? Method, which will result in about 1 dfs.socket.timeout 2 dfs.socket.timeout after the ack is sent (ackQueue.size () = 0). Send heartbeats to client along the pipeline all the time.

3. HBase side

The HBase side writes data to hdfs through the writer in hlog, and sync every time the data is written. At the same time, there is a logSyncer in HLog, and the default configuration is to call sync once per second, regardless of whether data is written or not.

This problem is caused by timeouts in the first place. Let's first analyze what happened on DFSClient and DataNode before and after timeouts.

1. Problem recurrence

A) the client ResponseProcessor reported a 69-second socket timeout, and the error point was PipelineAck.readFields (). After the error, catch directly and mark hasError=true,closed=true. This thread will not stop.

B) DataStreamer calls processDatanodeError in polling to process hasError=true. At this point, errorIndex=0 (the default) will first throw an exception for Recovery for Block. Then close the blockstream and re-recoverBlock based on the pipeline of the two nodes.

C) on DataNode, processDatanodeError () closes blockstream. This will cause the packetResponder in pipeline to be interrupted and terminated.

D) on DataNode, processDatanodeError () closes blockstream, causing the readToBuf in BlockReceiver's readNextPacket () to read no data and an exception to throw EOFException. This exception is thrown all the way up to the run of DataXceiver, which will cause DataXceiver to stop running, prompting DataNode.dnRegistration Error.

E) recoverBlock will proceed normally and will be completed first on two nodes (the second and third). Namenode then finds that there is not enough replicas and initiates a transfer block command to DataNode, which is an asynchronous process. However, during the hlog check, the transfer is likely to be unfinished and will be reported to pipeline error detected. Found 2 replicas but expecting 3 replicas . And close hlog.

The above is the error process that can be seen according to the log.

two。 Analysis of problems

A) Why did it time out and why didn't the heartbeat be sent?

According to the above analysis, the 69-second timeout of ResponseProcessor socket is the cause of the subsequent series of exceptions and hlog shutdown. So why did the socket timeout occur? ResponseProcessor should receive the HeartBeat packet within the time of 1 dfs.socket.timeout 2.

After printing the log, we found that the dfs.socket.timeout configured on DataNode is 180 seconds, while HBase uses the default configuration of 60 seconds when calling DFSClient. Therefore, DFSClient believes that the timeout time is 3 × nodes.length+60=69 seconds, while the data Node side sends the heartbeat timeout=1/2 × 180 seconds 90 seconds! Therefore, if there is no data writing, the DataNode will send the heartbeat packet 90 seconds later, when the DFSClient has already socketTimeout and leads to a series of subsequent phenomena.

B) Why didn't a new packet be sent in 69 seconds?

Let's first analyze the synchronization relationship between DFSOutputStream write data and sync. DFSOutputStream inherits from FSOutputSummer,DFSOutputStream to receive data through FSOutputSummer's write method, which is synchronized. The flushBuffer () and enqueuePacket () of the sync method are also in the synchronized (this) code block. That is, for a DFSOutputStream thread, if sync and write are called at the same time, a synchronous wait will occur. In the HBase scenario, the frequency of sync is very high, and sync has a good chance of grabbing the lock. In this way, it is very likely to be constantly sync, constantly flushBuffer, but failed to write data through write (blocked). This is the reason why no packet has been sent during the timeout.

To sum up, due to the characteristics of HBase business calls and DFSOutputStream's synchronized code block, it is likely that there will be no packet writes in 69 seconds. At this point, however, socket timeouts should not be allowed. Socket timeouts are the root cause of this problem, while sockets timeouts are due to inconsistent configuration.

3. Problem solving

On the HDFS side and the HBase side, set a larger value for dfs.socket.timeout, such as 300000 (300s)

The above is how to solve the problem of RegionServer crash caused by HBase Replication when a large number of data are written. The editor believes that there are some knowledge points that we may see or use in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.