Shulou(Shulou.com)06/01 Report--

How to parse the custom kafka parcel library and CDH installation kafka service can not install the past demining process, many novices are not very clear about this, in order to help you solve this problem, the following editor will explain in detail for you, people with this need can come to learn, hope you can get something.

1. Version

CDH5.9.1 failed to install KAFKA-2.0.2-1.2.0.2.p0.5-el6 https://archive.cloudera.com/kafka/parcels/2.0/

CDH5.9.1 failed to install KAFKA-2.1.0-1.2.1.0.p0.115-el6 https://archive.cloudera.com/kafka/parcels/2.1.0/

CDH5.10.0 failed to install KAFKA-2.1.1-1.2.1.1.p0.18-el6 https://archive.cloudera.com/kafka/parcels/latest/

two。 Error description

Customize the parcel library of kafka, install the kafka service, cannot install the past.

2.1 Open three log outputs of stdout,stderr,Role Log

When we have a problem, we will only watch Role log, and then throw the error "Will not attempt to authenticate using SASL (unknown error)". Then we analyze that kafka cannot connect to zk.

So Google, Baidu and so on, all kinds of inspection a lot, and finally did not solve the problem.

2.2 specific investigation:

Firewalls

Port number

/ etc/hosts file

Zoo.cfg file

User permissions

Zk jar package (client) of kafka and jar package version of zookeeper

The memory of the kafka broker process is too small, direct oom

3.Role log:

Opening socket connection to server sht-sgmhadoopdn-03/172.16.101.60:2181. Will not attempt to authenticate using SASL (unknown error)

4.stdout:

Java.lang.OutOfMemoryError: Java heap space

5.stderr:

+ grep-Q OnOutOfMemoryError / proc/14495/cmdline

+ RET=0

+'['0-eq 0']'

+ TARGET=14495

+ + date

+ echo Thu Mar 30 18:10:20 CST 2017

+ kill-9 14495

6. After n times of troubleshooting operation, it is invalid. It was only then that I looked carefully at the stdout and stderr logs and found the sentence "OnOutOfMemoryError".

I noticed this sentence at first, but I didn't think about it carefully, so it took about 1.5 working days.

7. We need to delete the kafka service and reinstall it, so it's best to delete the value of the log.dirs parameter of kafka to "/ var/local/kafka/data", which is manually removed from the installed machine.

8. So we opened the home interface of cdh and found that kafka had been installed, but it failed to start, then went to the configuration interface of kafka to search for the word "memory", changed it to 1G, and saved it.

9. Just start kafka.

10. Thinking

During the first installation, the value of this parameter was not configured in the interface.

Why, why is kafka's broker_max_heap_size parameter set so small during installation?

Is it helpful for you to read the above content? If you want to know more about the relevant knowledge or read more related articles, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.