Shulou(Shulou.com)06/02 Report--

Recently, when I was learning Kafka, I felt that it was too troublesome to turn on three virtual machines or to open three different port numbers in one virtual machine when I was ready to test the status of the cluster. Mainly lazy).

Environmental preparation

A computer with Internet access and a CentOS7 virtual machine

Why use a virtual machine? Because of the notebook used, the IP will change every time I connect to the network, and the configuration file will always be modified, which is too tedious to test. (this problem can be avoided through the Docker virtual network, which was not learned during the experiment.)

Docker installation

Ignore this step if Docker is already installed

Docker supports the following CentOS versions: CentOS 7 (64-bit): requires the system to be 64-bit and the system kernel version to be above 3.10. CentOS 6.5 (64-bit) or later: requires the system to be 64-bit and the system kernel version 2.6.32-431 or higher. Only kernels in CentOS distributions support Docker.

Yum installation

Docker requires the kernel version of the CentOS system to be higher than 3.10. check the prerequisites above to verify that your version of CentOS supports Docker.

# check the kernel version $uname-a # install Docker$ yum-y install docker# to start the Docker background service $service docker start# since there is no hello-world image locally, it will download an image of hello-world and run it in the container. $docker run hello-world

Script installation

Log in to Centos with sudo or root privileges.

Make sure the yum package is up to date.

$sudo yum update

Get and execute the Docker installation script.

$curl-fsSL https://get.docker.com-o get-docker.sh# executes this script to add the docker.repo source and install Docker. $sudo sh get-docker.sh

Start Docker

Sudo systemctl start docker# verifies that docker is installed successfully and executes a test image in the container. $sudo docker run hello-world$ docker ps

Mirror acceleration

When I first asked me to configure the domestic mirror source, I refused, but after using it, I found that the download speed duang~ went up immediately. Therefore, it is strongly recommended that we configure domestic mirror sources.

Open / create the / etc/docker/daemon.json file and add the following:

{"registry-mirrors": ["http://hub-mirror.c.163.com"]}"

Zookeeper cluster building

Zookeeper image: zookeeper:3.4

Mirror preparation

$docker pull zookeeper:3.4

You can go to https://hub.docker.com/ to find an image.

Docker pull images:TAG / / represents pulling the image image of the TAG version

Create a separate Zookeeper container

We first create a separate Zookeeper node in the simplest way, and then we create other nodes based on this example.

$docker run-- name zookeeper-p 2181-d zookeeper:3.4

By default, the configuration file in the container is / conf/zoo.cfg, and the data and log directories are / data and / datalog by default. You can map the above directories to the host if necessary.

Parameter interpretation

-- name: specify the container name

-p: assign a port number to the port exposed by the container

-d: run the container in the background and print the container ID

Cluster building

The creation of Zookeeper containers for other nodes is similar to creating independent containers. It is important to note that to specify the id of nodes and modify the configuration of multiple nodes in the file, the corresponding creation commands are as follows:

Create a new docker network

$docker network create zoo_kafka$ docker network ls

Zookeeper Container 1

$docker run-d\-- restart=always\-v / opt/docker/zookeeper/zoo1/data:/data\-v / opt/docker/zookeeper/zoo1/datalog:/datalog\-e ZOO_MY_ID=1\-p 2181 opt/docker/zookeeper/zoo1/data:/data 2181\-e ZOO_SERVERS= "server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888"\-- name=zoo1\-- net=viemall-zookeeper\-- privileged\ zookeeper:3.4

Zookeeper Container 2

$docker run-d\-- restart=always\-v / opt/docker/zookeeper/zoo2/data:/data\-v / opt/docker/zookeeper/zoo2/datalog:/datalog\-e ZOO_MY_ID=2\-p 2182 opt/docker/zookeeper/zoo2/data:/data 2181\-e ZOO_SERVERS= "server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888"\-- name=zoo2\-- net=viemall-zookeeper\-- privileged\ zookeeper:3.4

Zookeeper Container 3

$docker run-d\-- restart=always\-v / opt/docker/zookeeper/zoo3/data:/data\-v / opt/docker/zookeeper/zoo3/datalog:/datalog\-e ZOO_MY_ID=3\-p 2183purl 2181\-e ZOO_SERVERS= "server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888"\-- name=zoo3\-- net=viemall-zookeeper\-- privileged\ zookeeper:3.4

Although this approach also achieved what we wanted, but the steps are too cumbersome and troublesome to maintain (late stage of lazy cancer), so we use the docker-compose approach to achieve.

Setting up zookeeper Cluster by docker-compose

Create a new docker network

$docker network create viemall-zookeeper$ docker network ls

Write docker-compose.yml scripts

Mode of use:

Install docker-compose

# get the script $curl-L https://github.com/docker/compose/releases/download/1.25.0-rc2/docker-compose-`uname-s`-`uname-m`-o / usr/local/bin/docker-compose# to grant the execution permission $chmod + x / usr/local/bin/docker-compose

Create a new docker-compose.yml file in any directory and copy the following

Execute the command docker-compose up-d

Command comparison

| | Command | explanation |

| |-|-|

| | docker-compose up | start all containers |

| | docker-compose up-d | starts and runs all containers in the background |

| | docker-compose up-- no-recreate-d | do not recreate stopped containers |

| | docker-compose up-d test2 | only launch the test2 container | |

| | docker-compose stop | stop the container |

| | docker-compose start | launch container |

| | docker-compose down | stop and destroy the container |

Docker-compose.yml download address: https://github.com/JacianLiu/docker-compose/tree/master/zookeeper

Docker-compose.yml details

Version: '2'services: zoo1: image: zookeeper:3.4 # Image name restart: always # automatically restart hostname: zoo1 container_name: zoo1 privileged: true ports: # Port-2181 volumes: # Mount data volume -. / zoo1/data:/data -. / zoo1/datalog:/datalog environment: TZ: Asia/Shanghai ZOO_MY_ID: 1 # Node ID ZOO_PORT: 2181 # zookeeper Port number ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888 # zookeeper Node list networks: default: ipv4_address: 172.23.0.11 zoo2: image: zookeeper:3.4 restart: always hostname: zoo2 container_name: zoo2 privileged: true ports:-2182purl 2181 volumes: -. / zoo2/data:/data -. / zoo2/datalog:/datalog environment : TZ: Asia/Shanghai ZOO_MY_ID: 2 ZOO_PORT: 2181 ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888 networks: default: ipv4_address: 172.23.0.12 zoo3: image: zookeeper:3.4 restart: always hostname: zoo3 container_name: zoo3 privileged: true ports:-2181 volumes: -. / zoo3/data:/data -. / zoo3/datalog: / datalog environment: TZ: Asia/Shanghai ZOO_MY_ID: 3 ZOO_PORT: 2181 ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888 networks: default: ipv4_address: 172.23.0.13networks: default: external: name: zoo_kafka

Verification

From the picture, we can see that there is one Leader and two Flower, so our Zookeeper cluster has been set up.

Kafka cluster building

With the above foundation, is it still a problem to build a Kafka cluster? It's just that several variables have different values.

With the above example, do not bother to do single-node Kafka, directly use the way of docker-compose, deploy three nodes, in fact, the way is more or less the same, as mentioned above, it is just some different attributes; at this time, we do not need to build a new Docker network, just use the network created when we built the Zookeeper cluster!

Environmental preparation

Kafka image: wurstmeister/kafka

Kafka-Manager image: sheepkiller/kafka-manager

# do not specify a version to pull the latest version of the image docker pull wurstmeister/kafkadocker pull sheepkiller/kafka-manager by default

Write docker-compose.yml scripts

Mode of use:

Install docker-compose

# get the script $curl-L https://github.com/docker/compose/releases/download/1.25.0-rc2/docker-compose-`uname-s`-`uname-m`-o / usr/local/bin/docker-compose# to grant the execution permission $chmod + x / usr/local/bin/docker-compose

Create a new docker-compose.yml file in any directory and copy the following

Execute the command docker-compose up-d

Command comparison

| | Command | explanation |

|-|

| | docker-compose up | start all containers |

| | docker-compose up-d | starts and runs all containers in the background |

| | docker-compose up-- no-recreate-d | do not recreate stopped containers |

| | docker-compose up-d test2 | only launch the test2 container | |

| | docker-compose stop | stop the container |

| | docker-compose start | launch container |

| | docker-compose down | stop and destroy the container |

Docker-compose.yml download address: https://github.com/JacianLiu/docker-compose/tree/master/zookeeper

Docker-compose.yml details

Version: '2'services: broker1: image: wurstmeister/kafka restart: always hostname: broker1 container_name: broker1 privileged: true ports:-"9091 KAFKA_BROKER_ID: 1 KAFKA_LISTENERS: PLAINTEXT://broker1:9092 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker1:9092 KAFKA_ADVERTISED_HOST_NAME: broker1 KAFKA_ADVERTISED_PORT: 9092 KAFKA_ZOOKEEPER_CONNECT: zoo1:2181/kafka1,zoo2:2181/kafka1 Zoo3:2181/kafka1 JMX_PORT: 9988 volumes:-/ var/run/docker.sock:/var/run/docker.sock -. / broker1:/kafka/kafka\-logs\-broker1 external_links:-zoo1-zoo2-zoo3 networks: default: ipv4_address: 172.23.0.14 broker2: image: wurstmeister/kafka restart: always hostname: broker2 container_name: broker2 privileged: true ports:-"9092 environment: KAFKA _ BROKER_ID: 2 KAFKA_LISTENERS: PLAINTEXT://broker2:9092 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker2:9092 KAFKA_ADVERTISED_HOST_NAME: broker2 KAFKA_ADVERTISED_PORT: 9092 KAFKA_ZOOKEEPER_CONNECT: zoo1:2181/kafka1 Zoo2:2181/kafka1 Zoo3:2181/kafka1 JMX_PORT: 9988 volumes:-/ var/run/docker.sock:/var/run/docker.sock -. / broker2:/kafka/kafka\-logs\-broker2 external_links: # Connect container-zoo1-zoo2-zoo3 networks: default: ipv4_address: 172.23.0.15 broker3: image: wurstmeister/kafka restart: always hostname: broker3 container_name: broker3 privileged: true ports: -environment: KAFKA_BROKER_ID: 3 KAFKA_LISTENERS: PLAINTEXT://broker3:9092 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker3:9092 KAFKA_ADVERTISED_HOST_NAME: broker3 KAFKA_ADVERTISED_PORT: 9092 KAFKA_ZOOKEEPER_CONNECT: zoo1:2181/kafka1 Zoo2:2181/kafka1 Zoo3:2181/kafka1 JMX_PORT: 9988 volumes:-/ var/run/docker.sock:/var/run/docker.sock -. / broker3:/kafka/kafka\-logs\-broker3 external_links: # Connect container-zoo1-zoo2-zoo3 networks: default: ipv4_address: 172.23.0.16 kafka-manager: image: sheepkiller/kafka-manager:latest restart: always container_name: kafka-manager hostname outside this compose file : kafka-manager ports:-"9000 broker1 9000" links: # Connect container-broker1-broker2-broker3 external_links created in this compose file: # connect container-zoo1-zoo2-zoo3 environment: ZK_HOSTS: zoo1:2181/kafka1 outside this compose file Zoo2:2181/kafka1,zoo3:2181/kafka1 KAFKA_BROKERS: broker1:9092,broker2:9092,broker3:9092 APPLICATION_SECRET: letmein KM_ARGS:-Djava.net.preferIPv4Stack=true networks: default: ipv4_address: 172.23.0.10networks: default: external: # use the created network name: zoo_kafka

Verification

We go to the management page of kafka-manager, and the access path is host ip:9000.

If shown, fill in the address of the Zookeeper cluster, click save at the bottom

Click on the cluster you just added and you can see that there are three nodes in the cluster

Problems encountered in the process of construction

To mount an unlimited restart of data volumes, check the log prompt: chown: changing ownership of'/ var/lib/mysql/....': Permission denied

Solution:

Add-- privileged=true to the container to add specific permissions to docker run temporarily close selinux: setenforce 0 add selinux rule, change the security text of the directory to be mounted

Kafka-manager reported jmx related error

Solution:

After adding the environment variable JMX_PORT= port to each kafka node, it is found that it is not connected, which is the problem of network connection again, so each jmx port is exposed again, and then fire-wall is released to solve the problem. KAFKA_ADVERTISED_HOST_NAME, it is best to set the ip of the host, other than the host code or tools to connect, and the back port also needs to be set to expose the port.

[error] k.m.j.KafkaJMX $- Failed to connect to service:jmx:rmi:///jndi/rmi://9.11.8.48:-1/jmxrmi java.lang.IllegalArgumentException: requirement failed: No jmx port but jmx polling enabled!

View the following errors in the topic Times in the container (not just topic commands, as if all will go wrong)

$bin/kafka-topics.sh-- list-- zookeeper zoo1:2181/kafka1,zoo2:2181/kafka1,zoo3:2181/kafka1# the following is the error Error: Exception thrown by the agent: java.rmi.server.ExportException: Port already in use: 7203; nested exception is: java.net.BindException: Address already in use

Solution:

Add the unset JMX_PORT; instruction before the command, and the above command is changed to:

$unset JMX_PORT;bin/kafka-topics.sh-list-zookeeper zoo1:2181/kafka1,zoo2:2181/kafka1,zoo3:2181/kafka1

Attached: Docker common instructions

# View all mirrored docker images# View all running Container docker ps# View all Container docker ps-a # get all Container ip$ docker inspect-- format=' {{.Name}-{{range .NetworkSettings.Networks}} {{.IPAddress}} {{end}}'$(docker ps-aq) # View container internal log $docker logs-f # enter container internal $docker exec-it / bin/ Basj# creates container-d means to start docker run in the background-- name-e-v # restart container docker restart # close container docker stop # run container docker start

The above is the whole content of this article, I hope it will be helpful to your study, and I also hope that you will support it.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.