Shulou(Shulou.com)05/31 Report--

This article introduces the knowledge of "how to build a K8S cluster of EMQ X MQTT server from scratch". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

EMQ X Team provides Helm chart to make it easy for users to deploy EMQ X MQTT servers on kubernetes clusters with one click, which is EMQ X Team's most recommended way to deploy EMQ X MQTT servers on kubernetes or K3s clusters.

Deploy a single EMQ X MQTT server node on K8S directly deploy EMQ X Broker using Pod

In Kubernetes, the smallest administrative element is not individual containers, but Pod,Pod is the basic unit of execution of a Kubernetes application, that is, it is the smallest and simplest unit created or deployed in the Kubernetes object model. Pod represents the processes running on the cluster.

EMQ X Broker provides mirroring on docker hub, so you can easily deploy EMQ X Broker on a single pod and use the kubectl run command to create a Pod running EMQ X Broker:

$kubectl run emqx-image=emqx/emqx:v4.1-rc.1-generator=run-pod/v1pod/emqx created

View the status of EMQ X Broker:

$kubectl get pods-o wideNAME READY STATUS RESTARTS AGEemqx 1amp 1 Running 0 3m13s $kubectl exec emqx-- emqx_ctl statusNode 'emqx@192.168.77.108' is startedemqx 4.1-rc.1 is running

Delete Pod:

$kubectl delete pods emqxpod "emqx" deleted

Pod is not designed to be a persistent resource, it does not survive scheduling failures, node crashes, or other reclamation (such as lack of resources, or other maintenance), so a controller is also needed to manage Pod.

Deploy Pod using Deoloyment

Deployment provides a declarative definition (declarative) method for Pod and ReplicaSet to replace the previous ReplicationController to facilitate the management of applications. Typical application scenarios include:

Define Deployment to create Pod and ReplicaSet

Rolling upgrade and rollback application

Capacity expansion and reduction

Pause and resume Deployment

Deploy an EMQ X Broker Pod using Deployment:

Define Deployment:

$cat deployment.yamlapiVersion: apps/v1kind: Deploymentmetadata: name: emqx-deployment labels: app: emqxspec: replicas: 1 selector: matchLabels: app: emqx template: metadata: labels: app: emqxspec: containers:-name: emqx image: emqx/emqx:v4.1-rc.1 ports:-name: mqtt containerPort: 1883-name: mqttssl ContainerPort: 8883-name: mgmt containerPort: 8081-name: ws containerPort: 8083-name: wss containerPort: 8084-name: dashboard containerPort: 18083

Deploy Deployment:

$kubectl apply-f deployment.yamldeployment.apps/emqx-deployment created

View the deployment:

$kubectl get deploymentNAME READY UP-TO-DATE AVAILABLE AGEdeployment.apps/emqx-deployment 3ax 3 3 3 74s $kubectl get podsNAME READY STATUS RESTARTS AGEpod/emqx-deployment-7c44dbd68-8j77l 1 Acer 1 Running 0 74s $kubectl exec pod/emqx-deployment-7c44dbd68-8j77l-- emqx_ctl statusNode' Emqx-deployment-7c44dbd68-8j77lass 192.168.77.117' is startedemqx 4.1-rc.1 is running

Attempt to delete Pod manually

$kubectl delete pods emqx-deployment-7c44dbd68-8j77lpod "emqx-deployment-7c44dbd68-8j77l" deleted$ kubectl get podsNAME READY STATUS RESTARTS AGEemqx-deployment-68fcb4bfd6-2nhh7 1 Running 0 59s

The output shows that EMQ X Broker Pod is successfully deployed with Deployment, and even if the Pod is terminated unexpectedly, Deployment will recreate a new Pod.

Expose EMQ X Broker Pod services using Services

Kubernetes Pods has a life cycle. They can be created, and destruction will no longer be started. If you use Deployment to run the application, it can create and destroy the Pod dynamically.

Each Pod has its own IP address, but in Deployment, the collection of Pod running at the same time may be different from the collection of Pod that runs the application later.

This leads to a question: if you use EMQ X Broker Pod to serve the MQTT client, how should the client find and track the IP address to connect to so that the client can use the EMQ X Broker service?

The answer is: Service

Service is an abstract method that exposes applications running on a set of Pods as network services.

Expose EMQ X Broker Pod as a network service using Service:

Define Service:

$cat service.yamlapiVersion: v1kind: Servicemetadata: name: emqx-servicespec: selector: app:-name: mqtt port: 1883 protocol: TCP targetPort: mqtt-name: mqttssl port: 8883 protocol: TCP targetPort: mqttssl-name: mgmt port: 8081 protocol: TCP targetPort: mgmt-name: ws port: 8083 protocol: TCP targetPort: ws-name: wss port: 8084 protocol: TCP targetPort: wss-name: dashboard port: 18083 protocol: TCP targetPort: dashboard

Deploy Service:

$kubectl apply-f service.yamlservice/emqx-service created

View deployment

$kubectl get svcNAME TYPE CLUSTER-IP EXTERNAL-IP PORT (S) AGEemqx-service ClusterIP 10.96.54.205 1883 Universe TCPWE8883Universe TCPWE8081Universe TCPP8083Universe TCPJ08084max TCP18083Universe TCP58s

Use IP provided by Service to view the API of EMQ X Broker

$curl 10.96.54.205:8081/statusNode emqx-deployment-68fcb4bfd6-2nhh7@192.168.77.120 is startedemqx is running

At this point, a single EMQ X Broker node is deployed on the kubernetes, manages the EMQ X Broker Pod through Deployment, and exposes the EMQ X Broker service through Service.

Automatically cluster EMQ X MQTT servers through kubernetes

A single EMQ X Broker Pod is deployed through Deployment above, and it is very convenient to expand the number of Pod through Deployment. Execute the command kubectl scale deployment ${deployment_name}-- replicas ${numer} to expand the number of Pod. Expand the EMQ X Broker Pod to 3 below:

$kubectl scale deployment emqx-deployment--replicas 3deployment.apps/emqx-deployment scaled$ kubectl get podsNAME READY STATUS RESTARTS AGEemqx-deployment-68fcb4bfd6-2nhh7 1 18memqx-deployment-68fcb4bfd6-mpvch 1 Running 0 18memqx-deployment-68fcb4bfd6-mpvch 1 Running 0 6semqx-deployment-68fcb4bfd6-mx55q 1 pound 1 Running 0 6s $kubectl exec emqx-deployment-68fcb4bfd6- 2nhh7-- emqx_ctl statusNode 'emqx -deployment-68fcb4bfd6-2nhh7 / 192.168.77.120' is startedemqx 4.1-rc.1 is running$ kubectl exec emqx-deployment-68fcb4bfd6-- emqx_ctl cluster statusCluster status: # {running_nodes = > ['emqx-deployment-68fcb4bfd6-2nhh7 / 192.168.77.120'] Stopped_nodes = > []}

You can see that the number of EMQ X Broker Pod has been expanded to 3, but each Pod is independent and there is no cluster, and then try to automatically cluster EMQ X Broker Pod through kubernetes.

Modify the configuration of EMQ X Broker

Looking at the contents of automatic clustering in the EMQ X Broker documentation, you can see that the configuration of EMQ X Broker needs to be modified:

Cluster.discovery = kubernetescluster.kubernetes.apiserver = http://10.110.111.204:8080cluster.kubernetes.service_name = ekkacluster.kubernetes.address_type = ipcluster.kubernetes.app_name = ekka

Where cluster.kubernetes.apiserver is the address of kubernetes apiserver, which can be obtained through the kubectl cluster-info command, cluster.kubernetes.service_name is the name of Service above, and cluster.kubernetes.app_name is the part before the @ symbol in the node.name of EMQ X Broker, so you also need to set EMQ X Broker in the cluster to the prefix of unified node.name.

The docker image of EMQ X Broker provides the ability to modify the configuration through environment variables, such as docker hub or Github.

Modify the yaml file of Deployment to add environment variables:

$cat deployment.yamlapiVersion: apps/v1kind: Deploymentmetadata: name: emqx-deployment labels: app: emqxspec: replicas: 3 selector: matchLabels: app: emqx template: metadata: labels: app: emqxspec: containers:-name: emqx image: emqx/emqx:v4.1-rc.1 ports:-name: mqtt containerPort: 1883-name: mqttssl ContainerPort: 8883-name: mgmt containerPort: 8081-name: ws containerPort: 8083-name: wss containerPort: 8084-name: dashboard containerPort: 18083 env:-name: EMQX_NAME value: emqx-name: EMQX_CLUSTER__DISCOVERY value: K8s-name: EMQX_CLUSTER_ _ K8S__APP_NAME value: emqx- name: EMQX_CLUSTER__K8S__SERVICE_NAME value: emqx-service-name: EMQX_CLUSTER__K8S__APISERVER value: "https://kubernetes.default.svc:443"-name: EMQX_CLUSTER__K8S__NAMESPACE value: default

Because the command `kubectl scale deployment ${deployment_name}-- replicas ${numer} does not modify the yaml file, you need to set spec.replicas: 3 when modifying yaml.

Pod has built-in DNS rules for kubernetes, so https://kubernetes.default.svc:443 will be resolved to the address of kubernetes apiserver.

Delete the previous Deployment and redeploy:

$kubectl delete deployment emqx-deploymentdeployment.apps "emqx-deployment" deleted$ kubectl apply-f deployment.yamldeployment.apps/emqx-deployment created gives Pod access to kubernetes apiserver

After deploying Deployment above, looking at the status of EMQ X Broker, you can see that although EMQ X Broker has been started successfully, the cluster is still not successful. Check the log of EMQ X Broker Pod:

$kubectl get podsNAME READY STATUS RESTARTS AGEemqx-deployment-5c8cfc4d75-67lmt 1 Running 0 5semqx-deployment-5c8cfc4d75-r6jgb 1 kubectl exec emqx-deployment-5c8cfc4d75- 67lmt 1 Running 0 5semqx-deployment-5c8cfc4d75-wv2hj 1 kubectl exec emqx-deployment-5c8cfc4d75- 67lmt-- emqx_ctl statusNode 'emqx@192.168.87.150' is startedemqx 4.1-rc. 1 is running$ kubectl exec emqx-deployment-5c8cfc4d75-67lmt-- emqx_ctl cluster statusCluster status: # {running_nodes = > ['emqx@192.168.87.150'] Stopped_nodes = > []} $kubectl logs emqx-deployment-76f6895c46-4684f (emqx@192.168.87.150) 1 > 2020-05-20 01 Purse 4814 [error] Ekka (AutoCluster): Discovery error: {403, "{\" kind\ ":\" Status\ ",\" apiVersion\ ":\" v1\ ",\" metadata\ ": {} \ "status\":\ "Failure\",\ "message\":\ "endpoints\\" emqx-service\ "is forbidden: User\\" system:serviceaccount:default:default\\ "cannot get resource\\" endpoints\\ "in API group\\"\\ "in the namespace\\" default\\ ",\" reason\ ":\" Forbidden\ ",\" details\ ": {\" name\ ":\" emqx-service\ " \ "kind\":\ "endpoints\"},\ "code\": 403}\ n "}

Pod was denied access to kubernetes apiserver due to permission issues and returned HTTP 403, so the cluster failed.

Ordinary Pod cannot access kubernetes apiserver. There are two ways to solve this problem, one is to open the http interface of kubernetes apiserver, but this method has some security risks, and the other is to configure RBAC authentication through ServiceAccount, Role and RoleBinding.

Define ServiceAccount, Role, and RoleBinding:

Cat rbac.yamlapiVersion: v1kind: ServiceAccountmetadata: namespace: default name: emqx---kind: RoleapiVersion: rbac.authorization.kubernetes.io/v1beta1metadata: namespace: default name: emqxrules:- apiGroups:-"" resources:-endpoints verbs:-get-watch-list---kind: RoleBindingapiVersion: rbac.authorization.kubernetes.io/v1beta1metadata: namespace: default name: emqxsubjects:-kind: ServiceAccount name: emqx namespace: defaultroleRef: kind: Role name: emqx apiGroup: rbac.authorization.kubernetes.io

Deploy the appropriate resources:

$kubectl apply-f rbac.yamlserviceaccount/emqx createdrole.rbac.authorization.kubernetes.io/emqx createdrolebinding.rbac.authorization.kubernetes.io/emqx created

Modify Deployment's yaml file, add spec.template.spec.serviceAccountName, and redeploy:

$cat deployment.yamlapiVersion: apps/v1kind: Deploymentmetadata: name: emqx-deployment labels: app: emqxspec: replicas: 3 selector: matchLabels: app: emqx template: metadata: labels: app: emqxspec: serviceAccountName: emqx containers:-name: emqx image: emqx/emqx:v4.1-rc.1 ports:-name: mqtt containerPort: 1883- Name: mqttssl containerPort: 8883-name: mgmt containerPort: 8081-name: ws containerPort: 8083-name: wss containerPort: 8084-name: dashboard containerPort: 18083 env:-name: EMQX_NAME value: emqx-name: EMQX_CLUSTER__DISCOVERY value: kubernetes-name: EMQX_CLUSTER__K8S__APP_NAME value: emqx- name: EMQX_CLUSTER__K8S__SERVICE_NAME value: emqx-service-name: EMQX_CLUSTER__K8S__APISERVER value: "https://kubernetes.default.svc:443"-name: EMQX_CLUSTER__K8S__NAMESPACE value: default $kubectl delete deployment emqx-deploymentdeployment.apps" emqx- Deployment "deleted$ kubectl apply-f deployment.yamldeployment.apps/emqx-deployment created

View status:

$kubectl get podsNAME READY STATUS RESTARTS AGEemqx-deployment-6b854486c-dhd7p 1 Running 0 10semqx-deployment-6b854486c-psv2r 1 + 1 Running 0 10semqx-deployment-6b854486c-tdzld 1 + 1 Running 0 10s $kubectl exec emqx-deployment-6b854486c-dhd7p-- emqx_ctl statusNode 'emqx@192.168.77.92' is startedemqx 4.1-rc. 1 is running$ kubectl exec emqx-deployment-6b854486c-dhd7p-- emqx_ctl cluster statusCluster status: # {running_nodes = > ['emqx@192.168.77.115' 'emqx@192.168.77.92',' emqx@192.168.87.157'], stopped_nodes = > []}

Abort a Pod:

$kubectl delete pods emqx-deployment-6b854486c-dhd7ppod "emqx-deployment-6b854486c-dhd7p" deleted$ kubectl get podsNAME READY STATUS RESTARTS AGEemqx-deployment-6b854486c-846v7 1 emqx_ctl cluster 1 Running 0 56semqx-deployment-6b854486c-psv2r 1 Running 0 3m50semqx-deployment-6b854486c-tdzld 1 Running 0 3m50s $kubectl exec emqx-deployment-6b854486c-846v7-- emqx_ctl cluster StatusCluster status: # {running_nodes = > ['emqx@192.168.77.115' 'emqx@192.168.77.84',' emqx@192.168.87.157'], stopped_nodes = > ['emqx@192.168.77.92']}

The output shows that EMQ X Broker will correctly display the stopped Pod and add the new Pod created by Deployment to the cluster.

So far, EMQ X Broker has successfully established the cluster on kubernetes.

Persistent EMQ X Broker cluster

Deployment is used to manage Pod above, but the network of Pod is constantly changing, and when Pod is destroyed and rebuilt, the data and configuration stored in EMQ X Broker will disappear, which is unacceptable in production. Then try to persist the EMQ X Broker cluster. Even if Pod is destroyed and rebuilt, EMQ X Broker data can still be preserved.

ConfigMap

ConfigMap is a configMap is an API object that is used to save non-confidential data to a healthy value pair. It can be used as an environment variable, a command line parameter, or a configuration file in a storage volume.

ConfigMap decouples the configuration information of your environment from the container image to facilitate the application of configuration modifications.

ConfigMap does not provide secrecy or encryption. If you want to store data that is confidential, use Secret, or use other third-party tools to keep your data private instead of using ConfigMap.

Next, use ConfigMap to record the configuration of EMQ X Broker and import them into Deployment as environment variables.

Define Configmap and deploy:

$cat configmap.yamlapiVersion: v1kind: ConfigMapmetadata: name: emqx-configdata: EMQX_CLUSTER__K8S__ADDRESS_TYPE: "hostname" EMQX_CLUSTER__K8S__APISERVER: "https://kubernetes.default.svc:443" EMQX_CLUSTER__K8S__SUFFIX:" svc.cluster.local "$kubectl apply-f configmap.yamlconfigmap/emqx-config created

Configure Deployment to use Configmap

$cat deployment.yamlapiVersion: apps/v1kind: Deploymentmetadata: name: emqx-deployment labels: app: emqxspec: replicas: 3 selector: matchLabels: app: emqx template: metadata: labels: app: emqxspec: serviceAccountName: emqx containers:-name: emqx image: emqx/emqx:v4.1-rc.1 ports:-name: mqtt containerPort: 1883- Name: mqttssl containerPort: 8883-name: mgmt containerPort: 8081-name: ws containerPort: 8083-name: wss containerPort: 8084-name: dashboard containerPort: 18083 envFrom:-configMapRef: name: emqx-config

Redeploy Deployment to check status

$kubectl delete-f deployment.yamldeployment.apps "emqx-deployment" deleted$ kubectl apply-f deployment.yamldeployment.apps/emqx-deployment created$ kubectl get podsNAME READY STATUS RESTARTS AGEemqx-deployment-5c7696b5d7-k9lzj 1 Running 0 3semqx-deployment-5c7696b5d7-mdwkt 1 Running 0 3semqx-deployment-5c7696b5d7-z57z7 1 kubectl exec emqx-deployment 1 Running 0 3s $kubectl exec emqx-deployment -5c7696b5d7-k9lzj-- emqx_ctl statusNode 'emqx@192.168.87.149' is startedemqx 4.1-rc.1 is running$ kubectl exec emqx-deployment-5c7696b5d7-k9lzj-- emqx_ctl cluster statusCluster status: # {running_nodes = > [' emqx@192.168.77.106' 'emqx@192.168.77.107',' emqx@192.168.87.149'], stopped_nodes = > []}

The configuration file of EMQ X Broker has been decoupled into Configmap, and if necessary, you are free to configure one or more Configmap and introduce them into Pod as environment variables or files.

StatefulSet

StatefulSet is designed to solve the problem of stateful service (corresponding to Deployments and ReplicaSets are designed for stateless service), and its application scenarios include

Stable persistent storage, that is, Pod can still access the same persistent data after rescheduling, which is based on PVC.

Stable network sign, that is, its PodName and HostName remain unchanged after Pod rescheduling, which is based on Headless Service (that is, Service without Cluster IP).

Orderly deployment, orderly expansion, that is, Pod is sequentially deployed or expanded according to the defined order (that is, from 0 to Nmuri 1, all previous Pod must be Running and Ready states before the next Pod runs), which is implemented based on init containers.

Orderly contraction, orderly deletion (I. e., from Nmuri 1 to 0)

From the above application scenario, we can see that StatefulSet consists of the following parts:

Headless Service used to define the network flag (DNS domain)

VolumeClaimTemplates for creating PersistentVolumes

Define the application-specific StatefulSet

The DNS format of each Pod in StatefulSet is statefulSetName- {0..N-1} .serviceName.namespace.svc.cluster.local, where

ServiceName is the name of Headless Service

0..N-1 is the serial number where Pod is located, starting from 0 to NMUI 1

StatefulSetName is the name of StatefulSet

The namespace,Headless Servic and StatefulSet where the namespace is located must be on the same namespace

.cluster.local is Cluster Domain

Next, use StatefulSet instead of Deployment to manage Pod.

Delete Deployment:

$kubectl delete deployment emqx-deploymentdeployment.apps "emqx-deployment" deleted

Define StatefulSet:

$cat statefulset.yamlapiVersion: apps/v1kind: StatefulSetmetadata: name: emqx-statefulset labels: app: emqxspec: emqx-headless updateStrategy: type: RollingUpdate replicas: 3 selector: matchLabels: app: emqx template: metadata: labels: app: emqxspec: serviceAccountName: emqx containers:-name: emqx image: emqx/emqx:v4.1-rc.1 ports:-name: Mqtt containerPort: 1883-name: mqttssl containerPort: 8883-name: mgmt containerPort: 8081-name: ws containerPort: 8083-name: wss containerPort: 8084-name: dashboard containerPort: 18083 envFrom:-configMapRef: name: emqx-config

Note that StatefulSet needs Headless Service to achieve a stable network flag, so you need to define another Service

$cat headless.yaml apiVersion: v1 kind: Service metadata: name: emqx-headless spec: type: None selector: app: emqx ports:-name: mqtt port: 1883 protocol: TCP targetPort: 1883-name: mqttssl port: 8883 protocol: TCP targetPort: 8883-name: mgmt port: 8081 protocol: TCP targetPort: 8081-name: websocket port: 8083 protocol: TCP targetPort: 8083-name: wss port: 8084 protocol: TCP targetPort: 8084-name: dashboard port: 18083 protocol: TCP targetPort: 18083

Because IP is not required for Headless Service, clusterIP: None is configured.

Deploy the appropriate resources:

$kubectl apply-f headless-service.yamlservice/emqx-headless created$ kubectl apply-f statefulset.yamlstatefulset.apps/emqx-deployment created$ kubectl get podsNAME READY STATUS RESTARTS AGEemqx-statefulset-0 1 to 1 Running 0 2m59semqx-statefulset-1 1 to 1 Running 0 2m57semqx-statefulset-2 1 to 1 Running 0 2m54s $kubectl exec emqx-statefulset-0-- emqx_ctl cluster statusCluster status: # {running_nodes = > ['emqx@192.168.77.105' 'emqx@192.168.87.153',' emqx@192.168.87.155'], stopped_nodes = > []}

Update Configmap:

StatefulSet provides stable network flags. EMQ X Broker supports using hostname and dns rules to carry out IP clusters. Take hostname as an example, you need to modify emqx.conf:

Cluster.kubernetes.address_type = hostnamecluster.kubernetes.suffix = "svc.cluster.local"

The DNS rules of Pod in kubernetes cluster can be customized by users, and EMQ X Broker provides cluster.kubernetes.suffix to facilitate users to match custom DNS rules. This article uses the default DNS rule: statefulSetName- {0..N-1} .serviceName.namespace.svc.cluster.local, and the serviceName in DNS rule is the Headless Service used by StatefulSet, so you also need to change cluster.kubernetes.service_name to Headless Service Name.

To change the configuration item to an environment variable, you need to configure it in Configmap:

EMQX_CLUSTER__K8S__ADDRESS_TYPE: "hostname" EMQX_CLUSTER__K8S__SUFFIX: "svc.cluster.local" EMQX_CLUSTER__K8S__SERVICE_NAME: emqx-headless

Configmap provides a hot update function, which executes $kubectl edit configmap emqx-config to hot update Configmap.

Redeploy StatefulSet:

Pod will not restart after Configmap update. We need to update StatefulSet manually.

$kubectl delete statefulset emqx-statefulsetstatefulset.apps "emqx-statefulset" deleted$ kubectl apply-f statefulset.yamlstatefulset.apps/emqx-statefulset created$ kubectl get podsNAME READY STATUS RESTARTS AGEemqx-statefulset-0 1 Running 0 115semqx-statefulset-1 1 112semqx-statefulset-2 0 112semqx-statefulset-2 1 Running 0 110s $kubectl exec emqx-statefulset-2-- emqx_ctl cluster statusCluster status: # { Running_nodes = > ['emqx@emqx-statefulset-0.emqx-headless.default.svc.cluster.local' 'emqx@emqx-statefulset-1.emqx-headless.default.svc.cluster.local',' emqx@emqx-statefulset-2.emqx-headless.default.svc.cluster.local'], stopped_nodes = > []}

You can see that the new EMQ X Broker cluster has been successfully established.

Abort a Pod:

After the Pod in StatefulSet is rescheduled, its PodName and HostName remain unchanged. Let's try this:

$kubectl get podskuNAME READY STATUS RESTARTS AGEemqx-statefulset-0 1 + 1 Running 0 6m20semqx-statefulset-1 1 + 1 Running 0 6m17semqx-statefulset-2 1 + 1 Running 0 6m15s $kubectl delete pod emqx-statefulset-0pod "emqx-statefulset-0" deleted$ kubectl get podsNAME READY STATUS RESTARTS AGEemqx-statefulset-0 1 + 1 Running 0 27semqx-statefulset-1 1 Running 0 9m45semqx-statefulset-2 1 kubectl exec emqx-statefulset-2 1 Running 0 9m43s $kubectl exec emqx-statefulset-2-- emqx_ctl cluster statusCluster status: # {running_nodes = > ['emqx@emqx-statefulset-0.emqx-headless.default.svc.cluster.local' 'emqx@emqx-statefulset-1.emqx-headless.default.svc.cluster.local',' emqx@emqx-statefulset-2.emqx-headless.default.svc.cluster.local'], stopped_nodes = > []}

As expected, StatefulSet rescheduled a Pod,Pod with the same network flag and successfully joined the cluster.

StorageClasses, PersistentVolume and PersistentVolumeClaim

PersistentVolume (PV) is the storage set up by the administrator and is part of the cluster. Just as a node is a resource in a cluster, PV is a resource in a cluster. PV is a volume plug-in such as Volume, but has a lifecycle independent of Pod that uses PV. This API object contains the details of the storage implementation, that is, NFS, iSCSI, or cloud vendor-specific storage systems.

PersistentVolumeClaim (PVC) is a request stored by the user. It is similar to Pod. Pod consumes node resources and PVC consumes PV resources. Pod can request specific levels of resources (CPU and memory). The declaration can request a specific size and access mode (for example, it can be mounted in read / write once or read-only multiple modes).

StorageClass provides administrators with a way to describe the storage of "class". Different class may map to different quality of service levels or backup policies, or to any policies determined by the cluster administrator. Kubernetes itself doesn't know what the various class stands for. This concept is sometimes referred to as a "profile" in other storage systems.

When deploying EMQ X Broker, you can create PV or StorageClass in advance, and then use PVC to mount the / opt/emqx/data/mnesia directory of EMQ X Broker. When Pods is rescheduled, EMQ X will get the data from the / opt/emqx/data/mnesia directory and recover it, thus achieving EMQ X Broker persistence.

Define StatefulSet

$cat statefulset.yamlapiVersion: apps/v1kind: StatefulSetmetadata: name: emqx-statefulset labels: app: emqxspec: replicas: 3 serviceName: emqx-headless updateStrategy: type: RollingUpdate selector: matchLabels: app: emqx template: metadata: labels: app: emqxspec: volumes:-name: emqx-data persistentVolumeClaim: claimName: emqx-pvc serviceAccountName: emqx containers:-name: emqx Image: emqx/emqx:v4.1-rc.1 ports:-name: mqtt containerPort: 1883-name: mqttssl containerPort: 8883-name: mgmt containerPort: 8081-name: ws containerPort: 8083-name: wss containerPort: 8084-name: dashboard containerPort: 18083 envFrom: -configMapRef: name: emqx-config volumeMounts:-name: emqx-data mountPath: "/ opt/emqx/data/mnesia" volumeClaimTemplates:-metadata: name: emqx-pvc annotations: volume.alpha.kubernetes.io/storage-class: manual spec: accessModes: ["ReadWriteOnce"] resources: requests: storage: 1Gi

The file first specifies through volumeClaimTemplates that the name of StorageClass is used to create a PVC resource named emqx-pvc for the storage class of manual. The read and write mode of PVC resource is ReadWriteOnce, and the space of 1Gi is needed. Then the PVC is defined as volumes with name as emqx-data, and the volumes is mounted in the / opt/emqx/data/mnesia directory in Pod.

Deployment resources:

Before deploying StatefulSet, users or kubernetes cluster administrators are required to create their own storage classes.

$kubectl apply-f statefulset.yamlstatefulset.apps/emqx-statefulset created$ kubectl get podsNAME READY STATUS RESTARTS AGEemqx-statefulset-0 1 * 1 Running 0 27semqx-statefulset-1 1 * CAPACITY ACCESS MODES STORAGECLASS AGEemqx-data-emqx-statefulset-0 Bound pvc-8094cd75-adb5-11e9-80cc-0697b59e8064 1Gi RWO gp2 2m11semqx-data-emqx-statefulset-0 Bound pvc-9325441d-adb5-11e9-80cc-0697b59e8064 1Gi RWO gp2 99semqx-data-emqx-statefulset-0 Bound pvc-ad425e9d-adb5-11e9-80cc-0697b59e8064 1Gi RWO gp2 56s

The output results show that the state of the PVC has been successfully established for the Bound,PVC storage. When the Pod is rescheduled, the EMQ X Broker will read the data mounted in the PVC to achieve persistence.

MQTT Quick experience MQTT online Server

The EMQ X MQTT Internet of things Cloud Service provides an online public MQTT 5.0server that allows you to quickly start learning, testing, or prototyping the MQTT protocol without any installation.

For more information about the access to the MQTT server, please see the EMQ official website page: free online MQTT server.

MQTT online client

EMQ also provides a MQTT online client tool that supports browser access, which supports connecting to the MQTT server through a normal or encrypted WebSocket port, as well as caching connections for next access.

This is the end of the introduction of "how to build a K8S cluster of EMQ X MQTT servers from scratch". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.