Shulou(Shulou.com)06/01 Report--

How to solve a CDH5.10 clients.NetworkClient: Bootstrap broker ip:9092 disconnected problem, I believe that many inexperienced people do not know what to do, so this paper summarizes the causes of the problem and solutions, through this article I hope you can solve this problem.

1. Stable version combination used in the current environment

a. This environment CDH has been upgraded four times. Of course, the version is CDH-5.10.0-1.cdh6.10.0.p0.41.

B.KAFKA version is KAFKA-2.1.0-1.2.1.0.p0.115

C.SPARK2 version is SPARK2-2.0.0.cloudera1-1.cdh6.7.0.p0.113931

Analysis of 2.Spark2 installation troubleshooting

You will find on the Hosts-- > Parcels page that Spark2 can be upgraded to the 2.0.0.cloudera2 version of this version of release, which is 2.0.0.cloudera2-1.cdh6.7.0.p0.118100.

However, during the installation, we found that the spark history startup error of this version was reported. By analyzing the shell script stdout,stderr log, the error was reported as

The CSD version (2.0.0.cloudera1) is not compatible with the current Spark 2 version (2.0.0.cloudera2)

After further analysis, the current CSD_VERSION is 2.0.0.cloudera1. If you upgrade to the latest version, the SPARK2_VERSION is 2.0.0.cloudera2, so it is impossible to start the service at all.

Try to change 2.0.0.cloudera2 to 2.0.0.cloudera1 in the Metabase table, but the spark2 of parcel in the web interface is immediately unavailable. I really feel the power of cloudera!

Finally, I chose the same version of SPARK2-2.0.0.cloudera1-1.cdh6.7.0.p0.113931 as CSD_VERSION.

3.spark2_submit submits the jar package to yarn, and real-time spak reads data from kafka, but checks job's log and finds the following error

4. Analyze the error, replace all the referenced versions of the program's pom file with the current CDH,Kafka,Spark2 version, and then compile the jar package

(in fact, if you compile the Apache maven package, there is no dependency package, and the pom file is fine.)

Then think about the suspicion that the kafka jar package of spark2 on the cluster is inconsistent with the kafka version of CDH.

So put the previous version of bak, and then cpoy the jar package of the current kafka to the jars folder of spark2 (focus on this)

4.1pom file

Pom.rar

4.2 each of the cluster needs to do the following

Click (here) to collapse or open

[root@sh-hadoop-01 ~] # / opt/cloudera/parcels/SPARK2/lib/spark2/jars/

[root@sh-hadoop-01 jars] # ll

.

-rw-rw-r-- 1 root root 5001608 Dec 7 02:54 kafka_2.11-0.9.0-kafka-2.0.0.jar

-rw-rw-r-- 1 root root 649382 Dec 7 02:54 kafka-clients-0.9.0-kafka-2.0.0.jar

.

[root@sh-hadoop-01 jars] # mv kafka_2.11-0.9.0-kafka-2.0.0.jar kafka_2.11-0.9.0-kafka-2.0.0.jar.bak

[root@sh-hadoop-01 jars] # mv kafka-clients-0.9.0-kafka-2.0.0.jar kafka-clients-0.9.0-kafka-2.0.0.jar.bak

[root@sh-hadoop-01 jars] # cd / opt/cloudera/parcels/KAFKA/lib/kafka/libs

[root@sh-hadoop-01 libs] # cp / opt/cloudera/parcels/KAFKA/lib/kafka/libs/kafka_2.11-0.10.0-kafka-2.1.0.jar / opt/cloudera/parcels/SPARK2/lib/spark2/jars/

[root@sh-hadoop-01 libs] # cp / opt/cloudera/parcels/KAFKA/lib/kafka/libs/kafka-clients-0.10.0-kafka-2.1.0.jar / opt/cloudera/parcels/SPARK2/lib/spark2/jars/

[root@sh-hadoop-01 libs] # ll / opt/cloudera/parcels/SPARK2/lib/spark2/jars/

.

-rwxr-xr-x 1 root root 5156768 Mar 9 23:48 kafka_2.11-0.10.0-kafka-2.1.0.jar

-rw-rw-r-- 1 root root 5001608 Dec 7 02:54 kafka_2.11-0.9.0-kafka-2.0.0.jar.bak

-rwxr-xr-x 1 root root 747732 Mar 9 23:48 kafka-clients-0.10.0-kafka-2.1.0.jar

-rw-rw-r-- 1 root root 649382 Dec 7 02:54 kafka-clients-0.9.0-kafka-2.0.0.jar.bak

.

5. Solve the problem in the early morning and resubmit jar until now it has been running steadily for 10 hours.

After reading the above, have you mastered how to solve a clients.NetworkClient: Bootstrap broker ip:9092 disconnected problem in CDH5.10? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.