Shulou(Shulou.com)06/01 Report--

This article mainly introduces the installation example based on k8s in storm, which has a certain reference value, and interested friends can refer to it. I hope you will gain a lot after reading this article.

First, let's write a Dockerfile.

FROM ip:80/bigdata/centos7-jdk:1.8

# config storm env

ENV STORM_USER=storm\

STORM_INSTALL_BASE=/home/frank/platform\

STORM_HOME=/home/frank/platform/storm\

STORM_CONF_DIR=/home/frank/conf/storm\

STORM_DIR=/home/frank/hard_disk/storm\

STORM_BLOBS=/home/frank/hard_disk/storm/blobs\

STORM_NIMBUS=/home/frank/hard_disk/storm/nimbus\

STORM_LOG_DIR=/home/frank/logs/storm\

STORM_ROOT=/home/frank\

STORM_LOG=/home/frank/logs

ENV PATH $PATH:$STORM_HOME/bin

# add FLINK user and make dirs

RUN set-x\

& & adduser "$STORM_USER"\

& & mkdir-p "$STORM_INSTALL_BASE"$STORM_CONF_DIR"$STORM_LOG"$STORM_LOG_DIR"$STORM_BLOBS"$STORM_NIMBUS"

# install storm libs

ADD apache-storm-1.2.2.tar.gz $STORM_INSTALL_BASE

RUN cd $STORM_INSTALL_BASE & & ln-s-f apache-storm-1.2.2 storm & & mkdir-p / home/storm

RUN chown-R "$STORM_USER:$STORM_USER"$STORM_ROOT"

COPY docker-entrypoint.sh / home/storm

RUN chmod 755 / home/storm/docker-entrypoint.sh

USER storm

ENTRYPOINT ["/ home/storm/docker-entrypoint.sh"]

The script is as follows:

#! / bin/sh

# If unspecified, the hostname of the container is taken as the JobManager address

ACTION_CMD= "$1"

Source / etc/profile

Export STORM_CONF_DIR=/home/frank/conf/storm

Export STORM_CONF_FILE=/home/frank/conf/storm/storm.yaml

Sed-I-e "s/storm.zookeeper.servers:\ [localhost\] / storm.zookeeper.servers:\ [${STORM_ZK_QUORUM}\] /" / home/frank/conf/storm/storm.yaml

Sed-I-e "s/storm.zookeeper.root:\"\ / storm\ "/ storm.zookeeper.root:\ / ${STORM_ZK_ROOT} /" / home/frank/conf/storm/storm.yaml

# if use cluster model, pod ${JOB_CLUSTER_NAME}-0J ${JOB_CLUSTER_NAME}-1 as jobmanager

If [${ACTION_CMD} = = "cluster"]; then

Nimbuses= (${NIMBUS_HOSTS//,/})

ACTION_CMD= "supervisor"

For i in ${! nimbuses [@]}

Do

If ["$(hostname-s)" = "${nimbuses [I]}"]; then

NIMBUS_ADDRESS=$ {NIMBUS_ADDRESS:-$ (hostname-f)}

ACTION_CMD= "nimbus"

Sed-I-e "s/nimbus.seeds:\ [localhost\] / nimbus.seeds:\ [" ${NIMBUS_ADDRESS} "\] /" / home/frank/conf/storm/storm.yaml

Echo "pod hostname match nimbus config host, change action to nimbus."

Fi

Done

Fi

# if ha model, replace ha configuration

If [${ACTION_CMD} = = "help"]; then

Echo "Usage: $(basename" $0 ") (cluster | nimbus | supervisor | ui | logviewer | help)"

Exit 0

Elif [${ACTION_CMD} = = "nimbus"]; then

Echo "Starting nimbus ui logviewer supervisor"

Nohup / home/frank/platform/storm/bin/storm nimbus > > / tmp/nimbus.log 2 > & 1 &\

Nohup / home/frank/platform/storm/bin/storm ui > > / tmp/ui.log 2 > & 1 &\

/ home/frank/platform/storm/bin/storm logviewer

Elif [${ACTION_CMD} = = "supervisor"]; then

Echo "Starting supervisor"

/ home/frank/platform/storm/bin/storm supervisor

Fi

The image building command is as follows:

Docker build-- network=host-f Dockerfile-t ip:80/bigdata/storm:1.2.2.

Let's provide the creation file of service.

ApiVersion: v1

Kind: Service

Metadata:

Name: storm-cluster

Labels:

App: storm-cluster

Namespace: storm

Spec:

ClusterIP: None

Ports:

-port: 8080

Name: ui

Selector:

App: storm-cluster

ApiVersion: apps/v1

Kind: StatefulSet

Metadata:

Name: storm-cluster

Namespace: storm

Spec:

Selector:

MatchLabels:

App: storm-cluster

ServiceName: storm-cluster

Replicas: 4

PodManagementPolicy: Parallel

Template:

Metadata:

Labels:

App: storm-cluster

Spec:

TerminationGracePeriodSeconds: 2

Containers:

-name: storm-cluster

ImagePullPolicy: Always

Image: ip:80/bigdata/storm:1.2.2

Command: ["/ bin/sleep"]

Args: ["30d"]

VolumeMounts:

-name: storm-conf

MountPath: / home/conf/storm

-name: storm-log

MountPath: / home/logs/storm

Resources:

Requests:

Memory: "4096Mi"

Cpu: 4

Limits:

Memory: "4096Mi"

Cpu: 4

Env:

-name: NIMBUS_HOSTS

Value: "storm-cluster-2,storm-cluster-3"

-name: STORM_ZK_QUORUM

Value: "ip"

-name: STORM_ZK_ROOT

Value: "stormNew"

Ports:

-containerPort: 6627

Name: nimbus

-containerPort: 8000

Name: logviewer

-containerPort: 8080

Name: storm-ui

-containerPort: 6700

Name: sv0

-containerPort: 6701

Name: sv1

-containerPort: 6702

Name: sv2

-containerPort: 6703

Name: sv3

-containerPort: 22

Name: ssh

Volumes:

-name: storm-conf

ConfigMap:

Name: storm-conf

-name: storm-log

HostPath:

Path: / stormtmp

Type: Directory

The creation script for service is as follows:

Kubectl create-f storm-svc.yaml

The creation script for statefulSet is as follows:

Kubectl create-f storm-statefulSet.yaml

This creation script should pay attention to the following problems:

Command:/bin/sleep

The program needs to be started manually

/ home/vipshop/hard_disk/storm

Drwxr-xr-x 2 storm storm 4096 Jul 12 12:20 blobs

Drwxr-xr-x 2 storm storm 4096 Jul 12 12:20 nimbus

Drwxrwxr-x 5 storm storm 4096 Jul 29 00:00 supervisor

This hard_disk directory requires storm permission

Note: nimbus.thrift.max_buffer_size: 10485760

This parameter needs to be a multiple of 2

Thank you for reading this article carefully. I hope the article "K8s-based installation examples in storm" shared by the editor will be helpful to you. At the same time, I also hope you will support us and pay attention to the industry information channel. More related knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.