Shulou(Shulou.com)06/01 Report--

This article mainly explains "the way of building spark cluster in kubernetes". The content of the article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "the way of building spark cluster in kubernetes".

Applicable scenarios: linux system, clusters with kubernetes1.4 or above have been built, authentication such as CA has not been configured, DNS has been built. Other scenarios are for reference only.

If the DNS has not been built, please refer to deploying DNS scaffolding in kubernetes.

The relevant yaml files have been uploaded to my github, and the foreign images I need have also been replaced by Aliyun images, which can be downloaded and used directly.

Setting up a spark cluster in kubernetes in the previous section (1) it is mentioned that the ui of spark mater and worker nodes is separate and cannot be viewed in a UI. Clicking the = = back to master== in worker UI cannot return the UI of master. And our spark cluster cannot be accessed outside the cluster at this time, so this article will focus on how to merge these UI.

Solve the problem of multi-UI merger and opening to the outside world

Here, take the spark cluster that has been started in my cluster as an example.

As shown in the following figure, the external spark cluster in the cluster cannot be accessed properly, and each spark node has its own UI user interface, so it is impossible to implement one port to access all UI.

In this case, a proxy is used to implement the merge, and a lightweight HTTP server is created through a Python script that proxies all requests to your Spark Master and Spark Workers. All you need to do is create a single SSH channel to the proxy, which will forward all requests to the appropriate node.

The specific implementation process is as follows:

1. Create a proxy PODa. File spark-ui-proxy-controller.yaml contents kind: ReplicationControllerapiVersion: v1metadata: name: spark-ui-proxy-controllerspec: replicas: 1 selector: component: spark-ui-proxy template: metadata: labels: component: spark-ui-proxy spec: containers:-name: spark-ui-proxy image: registry.cn-hangzhou.aliyuncs.com/sjq-study/spark-ui-proxy:1.0 Ports:-containerPort: 80 resources: requests: cpu: 100m args:-spark-master:8080 livenessProbe: httpGet: path: / port: 80 initialDelaySeconds: 120 timeoutSeconds: 5

The relevant images have been replaced with Ali Cloud images, which can be downloaded and used directly.

b. Create RC$ kubectl create-f spark-ui-proxy-controller.yamlc. Check to verify $kubectl get pods | grep spark-uispark-ui-proxy-controller-gkx2v 1 Running 1 Running 0 5h

Already running!

d. Create services

The contents of the file spark-ui-proxy-service.yaml:

Kind: ServiceapiVersion: v1metadata: name: spark-ui-proxyspec: type: NodePort ports:-port: 80 targetPort: 80 nodePort: 8082 selector: component: spark-ui-proxy

Here I have mapped the port to port 8082 of the host so that the working interface of the spark cluster can be accessed externally through that port.

Create

$kubectl create-f spark-ui-proxy-service.yamle. View Verification $kubectl get svc | grep spark-uispark-ui-proxy 192.168.3.158 80:8082/TCP 4h

You can see that the cluster has assigned a cluster IP of 192.168.3.158 to services, so the user interface of the spark cluster can be accessed through this Ip+port within the cluster.

The actual construction and verification a. Access outside the cluster through exposed ports

Here my host IP is 192.168.122.10 and the port mapped from ui-proxy is 8082, then access http://192.168.122.10:8082

Click any worker node in master to jump to the Ui of the corresponding worker node, and click = = back to master== in worker to successfully jump to master

b. Intra-cluster access

The port mapped by the host's Ip+ in the cluster can also be accessed directly, or the normal access can be achieved through ui-proxy-service 's IP (192.168.3.158) + port.

to be continued!

Thank you for your reading. the above is the content of "how to build spark clusters in kubernetes". After the study of this article, I believe you have a deeper understanding of the way of building spark clusters in kubernetes, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.