Shulou(Shulou.com)06/03 Report--

In the process of supporting customers, EMQ X learned that some customers use Nginx for load balancing and the Docker container joins the cluster manually to run the EMQ cluster. Now record the main process.

Business requirements using Nginx as the reverse proxy Nginx needs to assign the address of the proxy server in advance and use the Docker container to run EMQEMQ automatic restart EMQ restart after automatic cluster configuration Nginx configuration $cat / etc/nginx/tcpstream.conf## tcp LB and SSL passthrough for backend # # stream {upstream mqtt_broker {server 127.0.1 upstream mqtt_broker 21871; # max_fails=5 fail_timeout=30s; server 127.0.0.1 Docker 21872; # max_fails=5 fail_timeout=30s Server 127.0.0.1 server 21873; # max_fails=5 fail_timeout=30s; server 127.0.1 max_fails=5 fail_timeout=30s; server 21874; # max_fails=5 fail_timeout=30s; server 127.0.1 max_fails=5 fail_timeout=30s; server 21875; # max_fails=5 fail_timeout=30s; server 127.0.1 max_fails=5 fail_timeout=30s; server 21881; # max_fails=5 fail_timeout=30s; server 127.0.0.1 max_fails=5 fail_timeout=30s; server 21891; # max_fails=5 fail_timeout=30s Server 127.0.0.1 server 21882; # max_fails=5 fail_timeout=30s; server 127.0.1 max_fails=5 fail_timeout=30s; server 21892; # max_fails=5 fail_timeout=30s; server 127.0.1 max_fails=5 fail_timeout=30s; server 21883; # max_fails=5 fail_timeout=30s; server 127.0.1 max_fails=5 fail_timeout=30s; server 21893; # max_fails=5 fail_timeout=30s; server 127.0.0.1 max_fails=5 fail_timeout=30s; server 21884; # max_fails=5 fail_timeout=30s Server 127.0.0.1 max_fails=5 fail_timeout=30s; server 21894; # max_fails=5 fail_timeout=30s; server 127.0.1 max_fails=5 fail_timeout=30s; server 21895; # max_fails=5 fail_timeout=30s } log_format basic'$proxy_protocol_addr-$remote_addr [$time_local]'$protocol $status $bytes_sent $bytes_received'$session_time "$upstream_addr"'"$upstream_bytes_sent" $upstream_bytes_received "$upstream_connect_time"; access_log / var/log/nginx/access.log basic; error_log / var/log/nginx/error.log Server {listen 8884 ssl; # proxy_protocol; proxy_next_upstream on; # proxy_bind $remote_addr transparent; proxy_ssl off; proxy_pass mqtt_broker; proxy_protocol on; # ssl_on; # adding some extra proxy settings proxy_timeout 350s; # proxy_buffer_size 128k; # ssl_certificate / etc/nginx/certs/solace.pem # ssl_certificate_key / etc/nginx/certs/solace.pem; ssl_certificate / etc/nginx/certs/cert.pem; ssl_certificate_key / etc/nginx/certs/key.pem; # ssl_verify_client off; ssl_protocols TLSv1 TLSv1.1 TLSv1.2; ssl_ciphers HIGH / etc/nginx/certs/cert.pem; ssl_certificate_key / MD5;}} Docker configuration

The Docker image compiled by the customer does not use the official image provided by EMQ.

The Dockerfile directory is as follows:

Total amount of ll / opt/Docker/ 28 StartEmqInstance.sh-rwxr-xr-x StartEmqInstance.sh-rwxr-xr-x 1 alexeyp alexeyp 26 October 22 17:26 Dockerfilelrwxrwxrwx 1 alexeyp emq 13 October 24 13:59 emqttd-> emqttd.2.3.11drwxr-xr-x 10 alexeyp emq 24 14:27 emqttd.2.3.11-rwxr-xr-x 1 alexeyp emq 3463 October 26 05:03 StartEmqInstance.sh-rwxr-xr-x 1 alexeyp alexeyp 27 October 25 10:46 status.sh

Dockerfile:

$cat DockerfileFROM centos:latestRUN yum-y updateEXPOSE 60000-65000WORKDIR / opt/emqttdADD. / vsparc.rpm / tmp/vsparc.rpmADD. / StartEmqInstance.sh / opt/emqttd/StartEmqInstance.shRUN yum install-y epel-releaseRUN yum install-y which less sed net-tools telnet gtest / tmp/vsparc.rpmENV TZ Australia/MelbourneCMD bash / opt/emqttd/StartEmqInstance.sh & & bash

You can see that after the Docker container starts, a script for StartEmqInstance.sh will be executed. View the script:

$cat StartEmqInstance.shiny re binbinBash function adjust_instance $(dirname $0) HOSTNAME=$ (hostname-s) function adjust_instance () {local INST=$1 local INST_ROOT=$2 cat $INST_ROOT/etc/emq.conf |\ sed-re "s / ^ node\ .name\ s*=.*$/node.name = emq$INST@127.0.0.1/" |\ # sed-re "s / ^ cluster\ .name\ s*=.*$/cluster.name = $HOSTNAME/" |\ Sed-re "s / ^ listener\ .tcp\ .external\ s*=.*$/listener.tcp.external = 0.0.0.0s*=.*$/listener.tcp.external 6188$ INST/" |\ sed-re "s / ^ listener\ .tcp\ .external1\ s*=.*$/listener.tcp.external1 = 0.0.0.0llistener 6189$ INST/" |\ sed-re "s / ^ l istener\ .tcp\ .external2\ s*=.*$/listener.tcp.external2 = 0.0. 0.0re 6187$ INST/ "|\ sed-re" s / ^ listener\ .tcp\ .internal\ s*=.*$/listener.tcp.internal = 127.0.0.1Part 6298$ INST/ "|\ sed-re" s / ^ listener\ .ssl\ .external\ s*=.*$/listener.ssl.external = 6288$ INST/ "|\ sed-re" s / ^ listener\ .ws\ .external\ s*=.*$/listener.ws.external = 6208$ INST/ " | |\ sed-re "s / ^ listener\ .wss\ .external\ s*=.*$/listener.ws.external = 6308$ INST/" |\ sed-re "s / ^ listener\ .API\ .mgmt\ s*=.*$/listener.api.mgmt = 6408$ INST/" |\ sed-re "s / ^ (# #\ s)? listener\ .tcp\ .external\ .proxy _ protocol\ s=.*$/listener.tcp.external.proxy_protocol = on/" | |\ sed-re "s / ^ (# #\ s)? listener\ .tcp\ .external1\ .proxy _ protocol\ s=.*$/listener.tcp.external1.proxy_protocol = on/" |\ sed-re "s / ^ (# #\ s)? listener\ .tcp\ .external2\ .proxy _ protocol\ s=.*$/listener.tcp.external2.proxy_protocol = on/" |\ sed-re "s / ^ (# #\ s)? listener \ .TCP\ .external\ .proxy _ protocol_timeout\ s=.*$/listener.tcp.external.proxy_protocol_timeout = 30s/ "|\ sed-re" s / ^ (# #\ s)? listener\ .tcp\ .external1\ .proxy _ protocol_timeout\ s=.*$/listener.tcp.external1.proxy_protocol_timeout = 30s/ "|\ sed-re" s / ^ (# #\ s)? listener\ .tcp\ .external2\ .proxy _ protocol_ Timeout\ s=.*$/listener.tcp.external2.proxy_protocol_timeout = 30s/ "|\ sed-re" s / ^ (# #\ s)? node.dist_listen_min\ s*=.*$/node.dist_listen_min = 6000$ INST/ "|\ sed-re" s / ^ (# #\ s)? node.dist_listen_max\ s*=.*$/node.dist_listen_max = 6000$ INST/ "|\ cat-> $INST_ROOT/etc/emq.conf.new mv $INST_ROOT/etc/emq.conf.new $INST_ROOT/etc/emq.conf} function cluster_instance () {local INST=$1 for DEST in 1 2 3 4 5 Do if [$DEST = = $INST]; then continue Fi DEST_NODE= "emq$DEST@127.0.0.1" RESULT=$ (/ opt/emqttd/bin/emqttd_ctl cluster join $DEST_NODE 2 > & 1) echo "$RESULT" echo "$RESULT" | grep-E 'successfully | already' > / dev/null RC=$? [$RC= = 0] & & break done} cd "$DIR" if ["$EMQ_INSTANCE_NUMBER" = ""] Then echo "Environment variable EMQ_INSTANCE_NUMBER (1.. 10) is not set." Echo "eMQ instance name is not configured." Exit 1else adjust_instance $EMQ_INSTANCE_NUMBER $DIRfifunction run_application () {local CMD= "$1" local RC=1 while [$RC! = 0]; do $CMD RC=$? Echo "# Exited: $CMD" echo "# rc = $RC" # [$RC! = 0] & & sleep 3 RC=1 done echo "# Done: $CMD"} function start_node () {bin/emqttd start STARTED=0 while [$STARTED= = 0] Do sleep 1 / opt/emqttd/bin/emqttd_ctl status | grep "is running" [$? = = 0] & & break done cluster_instance $EMQ_INSTANCE_NUMBER > / tmp/cluster_instance.log} start_nodesleep 5run_application "/ usr/local/bin/emqtt-stats-collector" & # waitIDLE_TIME=0while [[$IDLE_TIME-lt 5]] do IDLE_TIME=$ ((IDLE_TIME+1)) if [[!-z] "$(/ opt/emqttd/bin/emqttd_ctl status | grep'is running' | awk'{print $1}')"] Then IDLE_TIME=0 else echo "['$(date-u +"% Y-%m-%dT%H:%M:%SZ ")]: emqttd not running, waiting for recovery in $((60-IDLE_TIME*5)) seconds" fi sleep 5doneecho "['$(date-u +"% Y-%m-%dT%H:%M:%SZ ")']: emqttd exit abnormally" exit 1

The content of the script is a little more and a little complicated, so you need to look at it together with start.sh script and etc/emq.conf.

$cat start.shrunken binapash 5do docker ps for INST in 1 2 3 4 bash | grep-E "\ sinstance_$INST$" if [$?! = 0] The full text of then # docker run-itd-- ulimit nofile=1048576-restart=always-v / opt/Docker/emqtt/emq$INST/data/mnesia:/opt/emqttd/data/mnesia-e EMQ_INSTANCE_NUMBER=$INST-- name=instance_$INST-- network host emq:test & docker run-itd-- ulimit nofile=1048576-e EMQ_INSTANCE_NUMBER=$INST-- name=instance_$INST-- network host emq:latest & fidonewaitEMQ configuration etc/ emq.conf` will not be posted, mainly due to the addition of two tcp listening ports And close `listener.tcp.external.tune_buffer$ cat etc/emq.conf.##----listener.tcp.external = 0.0.0.0:21881listener.tcp.external.acceptors = 16listener. Tcp.external.max_clients = 512000listener.tcp.external.access.1 = allow alllistener.tcp.external.proxy_protocol = onlistener.tcp.external.proxy_protocol_timeout = 30slistener.tcp.external.backlog = 1024listener.tcp.external.send_timeout = 15slistener.tcp.external.send_timeout_close = on## listener.tcp.external.tune_buffer = onlistener.tcp.external.nodelay = truelistener.tcp.external.reuseaddr = true##- -listener.tcp.external1 = 0.0.0.0:21891listener.tcp.external1.acceptors = 16listener.tcp.external1.max_clients = 512000listener.tcp.external1.access.1 = allow alllistener.tcp.external1.proxy_protocol = onlistener.tcp.external1.proxy_protocol_timeout = 30slistener.tcp. External1.backlog = 1024listener.tcp.external1.send_timeout = 15slistener.tcp.external1.send_timeout_close = on## listener.tcp.external1.tune_buffer = onlistener.tcp.external1.nodelay = truelistener.tcp.external1.reuseaddr = true## -listener.tcp.external2 = 0.0.0.0:21871listener.tcp.external2.acceptors = 16listener.tcp.external2.max_clients = 512000listener.tcp.external2.access.1 = allow alllistener.tcp.external2.proxy_protocol = onlistener.tcp.external2.proxy_protocol_timeout = 30slistener.tcp.external2.backlog = 1024listener.tcp.external2.send_timeout = 15slistener.tcp.external2.send_timeout_close = on## listener.tcp.external2.tune_buffer = onlistener.tcp.external2. Nodelay = truelistener.tcp.external2.reuseaddr = true. Business Analysis Docker Container initialization

After the Docker container is created, StartEmqInstance.sh executes adjust_instance () to change the listening port in etc/emq.conf to Nginx's proxy server

Sed-re "s / ^ node\ .name\ s*=.*$/node.name = emq$INST@127.0.0.1/" |\ sed-re "s / ^ listener\ .tcp\ .external\ s*=.*$/listener.tcp.external = 0.0.0.0pur6188 $INST/" sed-re "s / ^ listener\ .tcp\ .external1\ s*=.*$/listener.tcp.external1 = 0.0.0.0pur6189 $INST/" sed-re "s / ^ listener \ .tcp\ .external2\ s*=.*$/listener.tcp.external2 = 0.0.0.0 sed 6187$ INST/ "sed-re" s / ^ listener\ .tcp\ .internal\ s*=.*$/listener.tcp.internal = 127.0.0.1 sed 6298$ INST/ "

And realize the cluster function through the join command.

Function cluster_instance () {local INST=$1 for DEST in 1 2 3 4 5; do if [$DEST = = $INST]; then continue Fi DEST_NODE= "emq$DEST@127.0.0.1" RESULT=$ (/ opt/emqttd/bin/emqttd_ctl cluster join $DEST_NODE 2 > & 1) echo "$RESULT" echo "$RESULT" | grep-E 'successfully | already' > / dev/null RC=$? [$RC= = 0] & & break done}

Loop to check the status of EMQ and exit the container when EMQ is stopped

IDLE_TIME=0while [[$IDLE_TIME-lt 5]] do IDLE_TIME=$ ((IDLE_TIME+1)) if [!-z "$(/ opt/emqttd/bin/emqttd_ctl status | grep'is running' | awk'{print $1}')"]] Then IDLE_TIME=0 else echo "['$(date-u +"% Y-%m-%dT%H:%M:%SZ ")]: emqttd not running, waiting for recovery in $((60-IDLE_TIME*5)) seconds" fi sleep 5doneecho "['$(date-u +"% Y-%m-%dT%H:%M:%SZ ")']: emqttd exit abnormally" exit 1 access

The client connects to the address in SSL, and Nginx loads the connection to the EMQ node in TCP.

PS: for settings on how Nginx reverse proxies tcp and ssl, please refer to EMQ X message server Nginx reverse proxy

Automatic restart and automatic clustering

After the container starts, query the status of EMQ through the StartEmqInstance.sh script, exit the container when EMQ stops, and work with-restart=always to restart the container.

EMQ stores the cluster information in data/mnesia and maps the directory in the container to the host. When the container is restarted, it will read the relevant directory mapped by the host to achieve automatic clustering after restart.

There is a problem that the host network mode of Docker uses the host network. When the host has other services performing, it is prone to port conflict resolution modification / proc/sys/net/ipv4/ip_local_port_range specifies that the port assigned by the system is 1024 60000, and then assigns the service port of EMQ to the port after 60000.

It is recommended that you use kubernetes to orchestrate docker containers:

EMQ can realize the function of automatic clustering through kube-apiserver. Currently, the customer only deploys docker clusters on a stand-alone machine, and it is easy to deploy clusters between multiple nodes using kubernetes. Kubernetes's deployment can monitor the status of emqx pod and achieve automatic restart, elastic expansion and other functions. Each emqx pod has its own virtual IP, and there is no port collision problem. The Service of kubernetes can meet the requirements of fixed IP and cloud load balancer. In the request created by Service, you can specify your cluster IP address by setting the spec.clusterIP field, and set the proxy server of Nginx to clusterIP, and Service can achieve load balancer on its own.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.