Shulou(Shulou.com)06/01 Report--

The problem we encountered again today:

Not able to place enough replicas

2015-02-08 18 File 35 ERROR org.apache.hadoop.security.UserGroupInformation: 43978 ERROR org.apache.hadoop.security.UserGroupInformation: File / hbase/aaaa/fbade52c03733ec2aa6d5355052d9c89/recovered.edits/0000000000004181150.temp could only be replicated to 0 nodes, instead of 1

2015-02-08 18 INFO org.apache.hadoop.ipc.Server 35 call addBlock 43978 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 8020, call addBlock (/ hbase/aaaa/fbade52c03733ec2aa6d5355052d9c89/recovered.edits/0000000000004181150.temp, DFSClient_hb_m_m66,60000,1423391732102, null) from 192.168.0.66 on 42030: error: java.io.IOException: File / hbase/aaaa/fbade52c03733ec2aa6d5355052d9c89/recovered.edits/0000000000004181150.temp could only be replicated to 0 nodes, instead of 1

Java.io.IOException: File / hbase/aaaa/fbade52c03733ec2aa6d5355052d9c89/recovered.edits/0000000000004181150.temp could only be replicated to 0 nodes, instead of 1

At org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock (FSNamesystem.java:1920)

At org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock (NameNode.java:783)

At sun.reflect.NativeMethodAccessorImpl.invoke0 (NativeMethod)

At sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:39)

At sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:25)

At java.lang.reflect.Method.invoke (Method.java:597)

At org.apache.hadoop.ipc.RPC$Server.call (RPC.java:587)

At org.apache.hadoop.ipc.Server$Handler$1.run (Server.java:1432)

At org.apache.hadoop.ipc.Server$Handler$1.run (Server.java:1428)

At java.security.AccessController.doPrivileged (Native Method)

At javax.security.auth.Subject.doAs (Subject.java:396)

At org.apache.hadoop.security.UserGroupInformation.doAs (UserGroupInformation.java:1190)

At org.apache.hadoop.ipc.Server$Handler.run (Server.java:1426)

2015-02-08 18 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem 3544014 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Not able to place enough replicas, still in need of 3 to reach 3

Not able to place enough replicas

2015-02-08 18 nodes 35 44014 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:web cause:java.io.IOException: File / hbase/.META./1028785192/recovered.edits/0000000000004125376.temp could only be replicated to 0 nodes, instead of 1

2015-02-08 18 INFO org.apache.hadoop.ipc.Server 35 hbase/.META./1028785192/recovered.edits/0000000000004125376.temp could only be replicated to 44014 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 8020, call addBlock (/ hbase/.META./1028785192/recovered.edits/0000000000004125376.temp, DFSClient_hb_m_m66,60000,1423391732102, null) from 192.168.0.66 on 42030: error: java.io.IOException: File / hbase/.META./1028785192/recovered.edits/0000000000004125376.temp could only be replicated to 0 nodes, instead of 1

Java.io.IOException: File / hbase/.META./1028785192/recovered.edits/0000000000004125376.temp could only be replicated to 0 nodes, instead of 1

At org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock (FSNamesystem.java:1920)

At org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock (NameNode.java:783)

At sun.reflect.NativeMethodAccessorImpl.invoke0 (NativeMethod)

At sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:39)

At sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:25)

At java.lang.reflect.Method.invoke (Method.java:597)

At org.apache.hadoop.ipc.RPC$Server.call (RPC.java:587)

At org.apache.hadoop.ipc.Server$Handler$1.run (Server.java:1432)

At org.apache.hadoop.ipc.Server$Handler$1.run (Server.java:1428)

At java.security.AccessController.doPrivileged (Native Method)

At javax.security.auth.Subject.doAs (Subject.java:396)

At org.apache.hadoop.security.UserGroupInformation.doAs (UserGroupInformation.java:1190)

At org.apache.hadoop.ipc.Server$Handler.run (Server.java:1426)

Rebooting n polygons has no effect.

Delete it, find that the file still exists after you start hmaster, delete it and then restore it.

Simply delete it with the directory, ok

. / hadoop fs-rmr / hbase/aaaa/8aab6d49ca2235530d7bf992dcb15e55

. / hadoop fs-rmr / hbase/.META./1028785192

Start hmaser ok again

. / hbase-daemon.sh start master

Jps check that ok hmaster is started.

Hey, hadoop, it's hard not to be violent!

Damn it, I thought it was all ok, but my colleague found a problem during the test:

. / hbase shell

List

You can see the table.

But when you scan the meter, you say it doesn't exist.

It immediately occurred to me that something was wrong with hbase itself.

Decisive repair

. / hbase hbck

It is found that there is a hollow discontinuity.

. / hbase hbck-fixMeta-fixAssignments

. / hbase hbck-repair

After the repair is complete, again. / hbase hbck finds that there is still a problem.

Again. / hbase hbck-repair

I can't believe it's all ok this time.

Restart shell and scan the table. Now ok. Was

Restart hbase found that the hbase process can not be closed, close the client process, ok.

When I started again, I suddenly found that one of the bags was full, dizzy, and repaired decisively.

Finally, the application was started, and the calls were all normal.

There are so many twists and turns and a lot of changes. Secretly scold one in my heart, shit.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.