Shulou(Shulou.com)06/03 Report--

Overview

Kafka is an open source stream processing platform developed by the Apache Software Foundation and written by Scala and Java. Kafka is a high-throughput distributed publish and subscribe messaging system that can handle all action flow data in consumer-scale websites. This action (web browsing, search and other user actions) is a key factor in many social functions on the modern web. This data is usually resolved by processing logs and log aggregations due to throughput requirements. For log data and offline analysis systems like Hadoop's, but requiring real-time processing limitations, this is a feasible solution. The purpose of Kafka is to unify online and offline message processing through Hadoop's parallel loading mechanism, as well as to provide real-time messages through clusters.

1 、 Kafka installation JDKtar xvf jdk1.8.0_231.tar.gz-C / usr/local & & cd / usr/localln-sv jdk1.8.0_231 jdk vim / etc/profile.d/java.sh JAVA_HOME=/usr/local/jdk PATH=$JAVA_HOME/bin:$PATHzookeeper install (or use the vim / usr/local/kafka/zookeeper/conf/zoo.cfg tickTime=2000 initLimit=10 syncLimit=5 dataDir=/data/zookeeper clientPort=2181 maxClientCnxns=0 # cluster that comes with kafka Version zookeeper adds the following configuration: # server.1=ip1:2888:3888 # server.2=ip2:2888:3888 # server.3=ip3:28888:3888 download kafka and install kakfawget https://archive.apache.org/dist/kafka/0.10.2.1/kafka_2.11-0.10.2.1.tgztar xvf kafka_2.11-0.10.2.1.tgz-C / usr/local & & cd / usr/localln-sv kafka_2.11-0 .10.2.1.tgz kafka modifies kafka startup memory vim / usr/local/kafka/bin/kafka-server-start.sh export KAFKA_HEAP_OPTS= "- Xmx2G-Xms2G" kafka start and stop / usr/local/kafka/bin/zookeeper-server-start.sh-deamon / usr/local/kafka/conf/zookeeper.properties/usr/local/kafka/bin/kafka-server-start.sh-deamon / usr/local/kafka/conf/server.properties/usr/local/kafka/bin/ Kafka-server-stop.sh / usr/local/kafka/conf/server.properties/usr/local/kafka/bin/zookeeper-server-stop.sh / usr/local/kafka/conf/zookeeper.properties start and stop method for installing kafka separately / usr/local/zookeeper/bin/zkServer.sh stop | stop2, Kafka sets prerequisites for public network access (provide public network access if necessary)

You need a domain name that resolves to the ip address of the private network, and the private network environment can also set / etc/hosts

Parameter setting

Host.name=kafka.test.com (corresponding domain name resolution needs to be solved to private network ip)

The high version has been deprecated. The lower version 0.10.2.1 is available, only if the listeners attribute is not configured, and has been replaced with the listeners attribute. Represents the hostname of the broker

Advertised.listeners=PLAINTEXT://kafka.test.com:9092 (for higher version, instead of host.name, set advertised.listeners instead of host.name)

A listener registered to zookeeper and provided to the client, using listeners if not configured.

Advertised.host.name (no setting is required, just for reference)

Has been deprecated. Used only if the advertised.listeners or listeners property is not configured. Advertised.listeners is recommended on the official website.

Listeners (no setting is required, just for reference)

The URL and protocols that need to be monitored, such as PLAINTEXT://myhost:9092,SSL://:9091 CLIENT://0.0.0.0:9092,REPLICATION://localhost:9093. If the configuration is not specified, the return value of the java.net.InetAddress.getCanonicalHostName () function is used

Modify the / etc/hosts file on broker

[private network ip] kafka.test.com

Modify the / etc/hosts file on the public network access server

[public network ip] kafka.test.com

3. Kafka consumption debugging producer / usr/local/kafka/bin/kafka-console-producer.sh-- broker-list IP:9092-- topic TOPIC consumer / usr/local/kafka/bin/kafka-console-consumer.sh-- bootstrap-server IP:9092-- topic TOPIC--from-beginning-- max-messages 1/usr/local/kafka/bin/kafka-console-consumer.sh-- bootstrap-server extranet IP:9092-- topic TOPIC--from-beginning-- max-messages 14, Troubleshooting of logstash debugging output {stdout {codec = > rubydebug {metadata = > true}} 5 and logstash's inability to consume kakfa logs

A, topics_pattern wildcard problem ". *", "." It must be necessary.

Topics_pattern= > "prefix-.*"

B. Match rules in filter. Pay attention to matching topic in kafka. The topic metadata corresponding to different filebeat and different logstash versions may not be the same. This should be noted.

If [type] = ~ "prefix-*" {grok {match = > ["[type]", "^ prefix- (?)"]} if [kafka] [topic] = ~ "prefix-*" {grok {match = > ["[kafka] [topic]" "^ prefix- (?. * $)]} if [@ metadata] [topic] = ~" prefix-* "{grok {match = > [" [@ metadata] [topic] "," ^ prefix- (?) "]} if [@ metadata] [kafka] [topic] = ~" prefix-* "{grok {match = > [[@ metadata] [kafka] [topic]" "^ prefix- (?. * $)"]}}

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.