Shulou(Shulou.com)05/31 Report--

In this article, the editor introduces in detail "how to build a kafka cluster in docker". The content is detailed, the steps are clear, and the details are handled properly. I hope this article "how to build a kafka cluster in docker" can help you solve your doubts.

1. Native Docker command

1. Delete all dangling data volumes (i.e. useless Volume, zombie files)

Docker volume rm $(docker volume ls-qf dangling=true)

two。 Delete all dangling images (that is, mirrors without tag)

Docker rmi $(docker images | grep "^" | awk "{print $3}"

3. Delete all closed containers

Docker ps-a | grep Exit | cut-d'- f 1 | xargs docker rm II. Image selection

The environment is M1 version of mbp:

Zookeeper adopts zookeeper

Kafka adopts wurstmeister/kafka

Kafka-Manager adopts scjtqs/kafka-manager

Mysql adopts mysql/mysql-server

III. Cluster planning

1. Create a new docker network

Docker network create docker-net-subnet 172.20.10.0/16docker network ls

two。 Cluster planning

HostnameIp addrportlistenerzook1172.20.10.112184:2181zook2172.20.10.122185:2181zook3172.20.10.132186:2181kafka1172.20.10.14 internal 9092, external 9192:9192kafka1kafka2172.20.10.15 internal 9093, external 9193:9193kafka2Kafka3172.20.10.16 internal 9094, external 9194:9194Kafka3 native (host Mbp) 172.20.10.2kafka manager172.20.10.109000:9000 IV, Zookeeper cluster installation

1. Create a new file zk-docker-compose.yml

Version: '3.4'services: zook1: image: zookeeper:latest restart: always hostname: zook1 container_name: zook1 # Container name Convenient to display a meaningful name in rancher: ports:-2183 Users/konsy/Development/volume/zkcluster/zook1/data:/data 2181 # map the default port number of this container to volumes: # Mount data volume-"/ Users/konsy/Development/volume/zkcluster/zook1/data:/data"-"/ Users/konsy/Development/volume/zkcluster/zook1/datalog:/datalog"-"/ Users/konsy/Development/volume/zkcluster/zook1/logs:/logs" Environment: ZOO_MY_ID: 1 # is the node value of zookeeper Also the brokerid value of kafka ZOO_SERVERS: server.1=zook1:2888:3888 2181 server.2=zook2:2888:3888;2181 server.3=zook3:2888:3888 2181 networks: docker-net: ipv4_address: 172.20.10.11 zook2: image: zookeeper:latest restart: always hostname: zook2 container_name: zook2 # Container name It is convenient to display the meaningful name ports:-2184 Users/konsy/Development/volume/zkcluster/zook2/data:/data 2181 # to map the default port number of this container to volumes:-"/ Users/konsy/Development/volume/zkcluster/zook2/data:/data"-"/ Users/konsy/Development/volume/zkcluster/zook2/datalog:/datalog"-"/ Users/konsy/Development/volume/zkcluster/zook2/logs:/logs" port: ZOO_MY_ID: 2 # is the node value of zookeeper Also the brokerid value of kafka ZOO_SERVERS: server.1=zook1:2888:3888 2181 server.2=zook2:2888:3888;2181 server.3=zook3:2888:3888 2181 networks: docker-net: ipv4_address: 172.20.10.12 zook3: image: zookeeper:latest restart: always hostname: zook3 container_name: zook3 # Container name It is convenient to display the meaningful name ports:-2185 Users/konsy/Development/volume/zkcluster/zook3/data:/data 2181 # to map the default port number of this container to volumes:-"/ Users/konsy/Development/volume/zkcluster/zook3/data:/data"-"/ Users/konsy/Development/volume/zkcluster/zook3/datalog:/datalog"-"/ Users/konsy/Development/volume/zkcluster/zook3/logs:/logs" port: ZOO_MY_ID: 3 # is the node value of zookeeper Also the brokerid value of kafka ZOO_SERVERS: server.1=zook1:2888:3888 2181 server.2=zook2:2888:3888;2181 server.3=zook3:2888:3888;2181 networks: docker-net: ipv4_address: 172.20.10.13networks: docker-net: external: name: docker-net

two。 Execute the script to deploy zookeeper to Docker:

Docker compose-f. / zk-docker-compose.yml up-d V, Kafka cluster installation

1. Create a new file kafka-docker-compose.yml

Version: '2'services: kafka1: image: docker.io/wurstmeister/kafka restart: always hostname: kafka1 container_name: kafka1 ports:-9093 kafka1 9193 environment: KAFKA_BROKER_ID: 1 KAFKA_LISTENERS: INSIDE://:9093,OUTSIDE://:9193 # KAFKA_ADVERTISED_LISTENERS=INSIDE://:9092,OUTSIDE://:9094 SKAFKA_ADVERTISED_LISTENERS: INSIDE://kafka1:9093 OUTSIDE://localhost:9193 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE KAFKA_ZOOKEEPER_CONNECT: zook1:2181,zook2:2181,zook3:2181 ALLOW_PLAINTEXT_LISTENER: 'yes' JMX_PORT: 9999 # Open JMX monitoring port To monitor the cluster data volumes:-/ Users/konsy/Development/volume/kafka/kafka1/wurstmeister/kafka:/wurstmeister/kafka-/ Users/konsy/Development/volume/kafka/kafka1/kafka:/kafka external_links:-zook1-zook2-zook3 networks: docker-net: ipv4_address: 172.20.10.14 kafka2: image: docker.io/wurstmeister/kafka restart: Always hostname: kafka2 container_name: kafka2 ports:-9094 KAFKA_LISTENERS 9194 environment: KAFKA_BROKER_ID: 2 KAFKA_LISTENERS: INSIDE://:9094 OUTSIDE://:9194 # KAFKA_ADVERTISED_LISTENERS=INSIDE://:9092,OUTSIDE://:9094 KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka2:9094,OUTSIDE://localhost:9194 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE KAFKA_ZOOKEEPER_CONNECT: zook1:2181,zook2:2181,zook3:2181 ALLOW_PLAINTEXT_LISTENER: 'yes' JMX_PORT: 9999 # Open JMX monitoring port To monitor the cluster data volumes:-/ Users/konsy/Development/volume/kafka/kafka2/wurstmeister/kafka:/wurstmeister/kafka-/ Users/konsy/Development/volume/kafka/kafka2/kafka:/kafka external_links:-zook1-zook2-zook3 networks: docker-net: ipv4_address: 172.20.10.15 kafka3: image: docker.io/wurstmeister/kafka restart: Always hostname: kafka3 container_name: kafka3 ports:-9095 environment: KAFKA_BROKER_ID: 3 KAFKA_LISTENERS: INSIDE://:9095 OUTSIDE://:9195 # KAFKA_ADVERTISED_LISTENERS=INSIDE://:9092,OUTSIDE://:9094 KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka3:9095,OUTSIDE://localhost:9195 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE KAFKA_ZOOKEEPER_CONNECT: zook1:2181,zook2:2181,zook3:2181 ALLOW_PLAINTEXT_LISTENER: 'yes' JMX_PORT: 9999 # Open JMX monitoring port To monitor the cluster data volumes:-/ Users/konsy/Development/volume/kafka/kafka3/wurstmeister/kafka:/wurstmeister/kafka-/ Users/konsy/Development/volume/kafka/kafka3/kafka:/kafka external_links:-zook1-zook2-zook3 networks: docker-net: ipv4_address: 172.20.10.16networks: docker-net: external: name: docker-net

two。 Execute the script to deploy kafka to Docker:

Docker compose-f. / kafka-docker-compose.yml up-d

3. Listeners and advertised.listeners

Listeners: the scientific name listener is to tell external connectors what protocol to use to access Kafka services with a specified hostname and port open.

Advertised.listeners: there is an extra advertised compared to listeners. The meaning of Advertised means declared and published, that is to say, this group of listeners are used by Broker for public release.

For example:

Listeners: INSIDE://172.17.0.10:9092,OUTSIDE://172.17.0.10:9094 advertised_listeners: INSIDE://172.17.0.10:9092,OUTSIDE://: port kafka_listener_security_protocol_map: "INSIDE:SASL_PLAINTEXT,OUTSIDE:SASL_PLAINTEXT" kafka_inter_broker_listener_name: "INSIDE"

Advertised_listeners listeners will be registered in zookeeper

When we request to establish a connection to 172.17.0.10 INSIDE 9092, the kafka server will find the INSIDE listener through the listener registered in zookeeper, and then find the corresponding communication ip and port in listeners

Similarly, when we request to establish a connection to: port, the kafka server will find the OUTSIDE listener through the listener registered in zookeeper, and then find the corresponding communication ip and port 172.17.0.10 OUTSIDE 9094 in listeners.

Summary: advertised_listeners is an exposed service port, and listeners is really used to establish a connection.

4. Install kafka-manager

4.1 create a new file kafka-manager-docker-compose.yml

Version: '2'services: kafka-manager: image: scjtqs/kafka-manager:latest restart: always hostname: kafka-manager container_name: kafka-manager ports:-9000 zook1 9000 external_links: # Connect to container-zook1-zook2-zook3-kafka1-kafka2-kafka3 environment: ZK_HOSTS: zook1:2181,zook2:2181,zook3:2181 KAFKA_BROKERS: kafka1:9093 outside this compose file Kafka2:9094,kafka3:9095 APPLICATION_SECRET: letmein KM_ARGS:-Djava.net.preferIPv4Stack=true networks: docker-net: ipv4_address: 172.20.10.10networks: docker-net: external: name: docker-net

4.2 execute the script to deploy kafka-manager to Docker:

Docker compose-f. / kafka-manager-docker-compose.yml up-d

4.3 configure Cluster

5. Test kafka

5.1 connecting the container

Enter the command line of the kafka container:

Docker exec-ti kafka / bin/bash

Enter the directory where kafka is located:

Cd opt/kafka_2.13-2.8.1 /

5.2 create a topic with a Replication of 2 and Partition 2

Bin/kafka-topics.sh-create-zookeeper zook1:2181-replication-factor 2-partitions 2-topic partopic

5.3 View the status of topic

Enter under the opt/kafka_2.12-1.1.0 / directory in the kafka container

Bin/kafka-topics.sh-- describe-- zookeeper zook1:2181-- topic partopic has read this article "how to build kafka clusters in docker". If you want to master the knowledge points of this article, you still need to practice and use it. If you want to know more about related articles, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.