Shulou(Shulou.com)06/01 Report--

I. Preface

When I woke up in the morning, I opened Wechat, and my colleagues reported that the kafka cluster had failed frequently since yesterday morning. I quickly turned on the computer to check the machine monitoring and log information of the kafka cluster, and found that the cluster load of one of the nodes suddenly fell off from yesterday morning, which was about the same time as the feedback from my colleagues, so I logged on to the server and started working.

Second, troubleshooting

1. Check the machine monitoring to see if you can roughly locate which node is abnormal.

2. Locate one of the abnormal nodes according to the machine monitoring, log in to the server to view the kafka log, and find that there is an error log, and the log stays at this point in time:

[2017-06-01 16 5915] ERROR Processor got uncaught exception. (kafka.network.Processor) java.lang.OutOfMemoryError: Direct buffer memory at java.nio.Bits.reserveMemory (Bits.java:658) at java.nio.DirectByteBuffer. (DirectByteBuffer.java:123) at java.nio.ByteBuffer.allocateDirect (ByteBuffer.java:306) at sun.nio.ch.Util.getTemporaryDirectBuffer (Util.java:174) at sun.nio.ch.IOUtil.read (IOUtil.java:195) at sun. Nio.ch.SocketChannelImpl.read (SocketChannelImpl.java:379) at org.apache.kafka.common.network.PlaintextTransportLayer.read (PlaintextTransportLayer.java:108) at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel (NetworkReceive.java:97) at org.apache.kafka.common.network.NetworkReceive.readFrom (NetworkReceive.java:71) at org.apache.kafka.common.network.KafkaChannel.receive (KafkaChannel.java:160) at org .apache.kafka.common.network.KafkaChannel.read (KafkaChannel.java:141) at org.apache.kafka.common.network.Selector.poll (Selector.java:286) at kafka.network.Processor.run (SocketServer.scala:413)

3. Check the kafka process and listening port, and find that they are all normal, and Nima faked her death.

Ps-ef | grep kafka # # View the process netstat-ntlp of kafka | the listening port of grep 9092 # # 9092kafka

4. Since he has faked his death, he has no choice but to restart

Ps-ef | grep kafka | grep-v grep | awk'{print $2}'| xargs kill-9 / usr/local/kafka/bin;nohup. / kafka-server-start.sh.. / config/server.properties &

5. After rebooting, after observing the kafka log of the node, after an index reconstruction, the error message above was frantically brushed, and finally Google solved the problem.

3. Solution:

In

/ usr/local/kafka/binkafka-run-class.sh

Remove

-XX:+DisableExplicitGC

Add

-XX:MaxDirectMemorySize=512m

Restart kafka at one time, the problem is resolved.

PS: reference link: http://ju.outofmemory.cn/entry/75905

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.