Shulou(Shulou.com)05/31 Report--

This article will explain in detail how to report errors in sending messages in kafka clusters. The editor thinks it is very practical, so I share it for you as a reference. I hope you can get something after reading this article.

Because the logs collected by logstash have to be sent to kafka for a queue mechanism, there is a problem sending messages after setting up the kafka cluster.

ERROR fetching topic metadata for topics [Set (order)] from broker [ArrayBuffer (id:0,host:slave4,port:9092)] failed (kafka.utils.Utils$)

Kafka.common.KafkaException: fetching topic metadata for topics [Set (order)] from broker [ArrayBuffer (id:0,host:slave4,port:9092)] failed

At kafka.client.ClientUtils$.fetchTopicMetadata (ClientUtils.scala:67)

At kafka.producer.BrokerPartitionInfo.updateInfo (BrokerPartitionInfo.scala:82)

At kafka.producer.async.DefaultEventHandler$$anonfun$handle$2.apply$mcV$sp (DefaultEventHandler.scala:78)

At kafka.utils.Utils$.swallow (Utils.scala:167)

At kafka.utils.Logging$class.swallowError (Logging.scala:106)

At kafka.utils.Utils$.swallowError (Utils.scala:46)

At kafka.producer.async.DefaultEventHandler.handle (DefaultEventHandler.scala:78)

At kafka.producer.async.ProducerSendThread.tryToHandle (ProducerSendThread.scala:104)

At kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply (ProducerSendThread.scala:87)

At kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply (ProducerSendThread.scala:67)

At scala.collection.immutable.Stream.foreach (Stream.scala:547)

At kafka.producer.async.ProducerSendThread.processEvents (ProducerSendThread.scala:66)

At kafka.producer.async.ProducerSendThread.run (ProducerSendThread.scala:44)

Caused by: java.net.ConnectException: Connection refused

At sun.nio.ch.Net.connect0 (Native Method)

At sun.nio.ch.Net.connect (Net.java:484)

At sun.nio.ch.Net.connect (Net.java:476)

At sun.nio.ch.SocketChannelImpl.connect (SocketChannelImpl.java:675)

At kafka.network.BlockingChannel.connect (BlockingChannel.scala:57)

At kafka.producer.SyncProducer.connect (SyncProducer.scala:141)

At kafka.producer.SyncProducer.getOrMakeConnection (SyncProducer.scala:156)

At kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend (SyncProducer.scala:68)

At kafka.producer.SyncProducer.send (SyncProducer.scala:112)

At kafka.client.ClientUtils$.fetchTopicMetadata (ClientUtils.scala:53)

... 12 more

ERROR Error in handling batch of 1 events (kafka.producer.async.ProducerSendThread)

Kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries.

At kafka.producer.async.DefaultEventHandler.handle (DefaultEventHandler.scala:90)

At kafka.producer.async.ProducerSendThread.tryToHandle (ProducerSendThread.scala:104)

At kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply (ProducerSendThread.scala:87)

At kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply (ProducerSendThread.scala:67)

At scala.collection.immutable.Stream.foreach (Stream.scala:547)

At kafka.producer.async.ProducerSendThread.processEvents (ProducerSendThread.scala:66)

At kafka.producer.async.ProducerSendThread.run (ProducerSendThread.scala:44)

Because you see Caused by: java.net.ConnectException: Connection refused, check selinux

Vim / etc/selinux/config

# This file controls the state of SELinux on the system.

# SELINUX= can take one of these three values:

# enforcing-SELinux security policy is enforced.

# permissive-SELinux prints warnings instead of enforcing.

# disabled-No SELinux policy is loaded.

SELINUX=disabled

# SELINUXTYPE= can take one of these two values:

# targeted-Targeted processes are protected

# mls-Multi Level Security protection.

SELINUXTYPE=targeted

Force shutdown of selinux

Setenforce 0

It's the same mistake to send a message.

View zookeeper

[zk: master:2181 (CONNECTED) 18] get / brokers/ids/0

{"jmx_port":-1, "timestamp": "1465262832441", "host": "localhost", "version": 1, "port": 9092}

CZxid = 0x70000005e

Ctime = Tue Jun 07 09:27:12 CST 2016

MZxid = 0x70000005e

Mtime = Tue Jun 07 09:27:12 CST 2016

PZxid = 0x70000005e

Cversion = 0

DataVersion = 0

AclVersion = 0

EphemeralOwner = 0x255213e787e0002

DataLength = 86

NumChildren = 0

Host is localhost, while my kafka cluster is on slave4, 5, and zk is on master,slave2, 3, so modify the host.name=slave4 and host.name=slave5 of server.properties on slave4 and 5, respectively.

Restart kafka to view zk

[zk: master:2181 (CONNECTED) 27] get / brokers/ids/0

{"jmx_port":-1, "timestamp": "1465266328389", "host": "slave4", "version": 1, "port": 9092}

CZxid = 0x70000008a

Ctime = Tue Jun 07 10:25:27 CST 2016

MZxid = 0x70000008a

Mtime = Tue Jun 07 10:25:27 CST 2016

PZxid = 0x70000008a

Cversion = 0

DataVersion = 0

AclVersion = 0

EphemeralOwner = 0x355213e7be80003

DataLength = 83

NumChildren = 0

The host on 0 becomes slave4.

The message was sent successfully and the message was received successfully

This is the end of the article on "how to send messages and report errors in kafka clusters". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, please share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.