Shulou(Shulou.com)05/31 Report--

How to remotely process the Kafka startup program through the Apache Kafka plug-in in Kubernetes? aiming at this problem, this article introduces the corresponding analysis and solution in detail, hoping to help more partners who want to solve this problem to find a more simple and feasible method.

Project introduction

The current version of Remoting over Apache Kafka plugin remoting requires users to manually configure the entire system, including zookeeper, kafka, and remoting agents. It also does not support dynamic proxy configuration, so it is difficult to implement scalable extensions. My project aims to solve two problems: 1. Provide off-the-shelf solutions for Apache-Kafka clusters. 2. Dynamic proxy configuration in Kubernetes cluster.

Current state

A Kubernetes connector that supports credentials.

The ApacheKafka preconfiguration function in the Kubernetes function has been fully implemented.

Helm chart part of the implementation.

Apache-Kafka configuration in Kubernetes

This feature is part of version 2.0, so it hasn't been officially released yet. You can try this feature by updating to the 2.0.0-alpha version using Experimental Update Center or building directly from the master branch:

Git clone https://github.com/jenkinsci/remoting-kafka-plugin.gitcd remoting-kafka-plugin/pluginmvn hpi:run

On the global configuration page, users can enter Kubernetes server information and credentials. Then they can start ApacheKafka at the click of a button.

When the user clicks the Start Kafka on Kubernetes button, Jenkins creates a Kubernetes client based on the information, and then applies the zookeeper and kafka yaml specification files from resources.

Helm Chart

Remotely processed Helm charts on the Apache-Kafka plug-in are based on stable/jenkins charts and incubator/kafka charts. As of now, the diagram is still under development because it is still waiting for the Cloud API implementation of Phase 2. However, you can view the demo diagram using a separate remote Kafka agent:

Git clone-b demo-helm-phase-1 https://github.com/longngn/remoting-kafka-plugin.gitcd remoting-kafka-pluginK8S_NODE=. / helm/jenkins-remoting-kafka/do.sh start

The command do.sh start performs the following steps: * install the chart (for use with Jenkins and Kafka). * start a Kafka computer on Jenkins master by applying the JCasC below.

Jenkins: nodes:-permanent: name: "test" remoteFS: "/ home/jenkins" launcher: kafka: {}

Start a single Remoting Kafka Agent pod. You can check the chart status by running kubectl, for example: ```$kubectl get all-n demo-helm NAME READY STATUS RESTARTS AGE pod/demo-jenkins-998bcdfd4-tjmjs 2 "2 Running 0 6m30s pod/demo-jenkins-remoting-kafka-agent 1" 1 Running 0 4m10s pod/demo-kafka-0 1 "1 Running 0 6m30s pod/demo-zookeeper-0 1" 1 Running 0 6m30s

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT (S) AGE service/demo-0-external NodePort 10.106.254.187 19092:31090/TCP 6m30s service/demo-jenkins NodePort 10.101.84.33 8080:31465/TCP 6m31s service/demo-jenkins-agent ClusterIP 10.97.169.65 50000/TCP 6m31s service/demo-kafka ClusterIP 10.106.248.10 9092/TCP 6m30s service/demo-kafka-headless ClusterIP None 9092/TCP 6m30s service/demo-zookeeper ClusterIP 10.109.222.63 2181/TCP 6m30s service/demo-zookeeper-headless ClusterIP None 2181/TCP 3888/TCP,2888/TCP 6m31s

NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/demo-jenkins 1 # 1 1 1 6m30s

NAME DESIRED CURRENT READY AGE replicaset.apps/demo-jenkins-998bcdfd4 1 1 1 6m30s

NAME READY AGE statefulset.apps/demo-kafka 1 "1 6m30s statefulset.apps/demo-zookeeper 1" 1 6m30s ```

This is the answer to the question about how to remotely handle the Kafka startup program through the Apache Kafka plug-in in Kubernetes. I hope the above content can be of some help to you. If you still have a lot of doubts to be solved, you can follow the industry information channel to learn more about it.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.