K8s-based installation example in storm
This article mainly introduces the installation example based on k8s in storm, which has a certain reference value, and interested friends can refer to it. I hope you will gain a lot after reading this article.
First, let's write a Dockerfile.
FROM ip:80/bigdata/centos7-jdk:1.8
# config storm env
ENV STORM_USER=storm\
STORM_INSTALL_BASE=/home/frank/platform\
STORM_HOME=/home/frank/platform/storm\
STORM_CONF_DIR=/home/frank/conf/storm\
STORM_DIR=/home/frank/hard_disk/storm\
STORM_BLOBS=/home/frank/hard_disk/storm/blobs\
STORM_NIMBUS=/home/frank/hard_disk/storm/nimbus\
STORM_LOG_DIR=/home/frank/logs/storm\
STORM_ROOT=/home/frank\
STORM_LOG=/home/frank/logs
ENV PATH $PATH:$STORM_HOME/bin
# add FLINK user and make dirs
RUN set-x\
& & adduser "$STORM_USER"\
& & mkdir-p "$STORM_INSTALL_BASE"$STORM_CONF_DIR"$STORM_LOG"$STORM_LOG_DIR"$STORM_BLOBS"$STORM_NIMBUS"
# install storm libs
ADD apache-storm-1.2.2.tar.gz $STORM_INSTALL_BASE
RUN cd $STORM_INSTALL_BASE & & ln-s-f apache-storm-1.2.2 storm & & mkdir-p / home/storm
RUN chown-R "$STORM_USER:$STORM_USER"$STORM_ROOT"
COPY docker-entrypoint.sh / home/storm
RUN chmod 755 / home/storm/docker-entrypoint.sh
USER storm
ENTRYPOINT ["/ home/storm/docker-entrypoint.sh"]
The script is as follows:
#! / bin/sh
# If unspecified, the hostname of the container is taken as the JobManager address
ACTION_CMD= "$1"
Source / etc/profile
Export STORM_CONF_DIR=/home/frank/conf/storm
Export STORM_CONF_FILE=/home/frank/conf/storm/storm.yaml
Sed-I-e "s/storm.zookeeper.servers:\ [localhost\] / storm.zookeeper.servers:\ [${STORM_ZK_QUORUM}\] /" / home/frank/conf/storm/storm.yaml
Sed-I-e "s/storm.zookeeper.root:\"\ / storm\ "/ storm.zookeeper.root:\ / ${STORM_ZK_ROOT} /" / home/frank/conf/storm/storm.yaml
# if use cluster model, pod ${JOB_CLUSTER_NAME}-0J ${JOB_CLUSTER_NAME}-1 as jobmanager
If [${ACTION_CMD} = = "cluster"]; then
Nimbuses= (${NIMBUS_HOSTS//,/})
ACTION_CMD= "supervisor"
For i in ${! nimbuses [@]}
Do
If ["$(hostname-s)" = "${nimbuses [I]}"]; then
NIMBUS_ADDRESS=$ {NIMBUS_ADDRESS:-$ (hostname-f)}
ACTION_CMD= "nimbus"
Sed-I-e "s/nimbus.seeds:\ [localhost\] / nimbus.seeds:\ [" ${NIMBUS_ADDRESS} "\] /" / home/frank/conf/storm/storm.yaml
Echo "pod hostname match nimbus config host, change action to nimbus."
Fi
Done
Fi
# if ha model, replace ha configuration
If [${ACTION_CMD} = = "help"]; then
Echo "Usage: $(basename" $0 ") (cluster | nimbus | supervisor | ui | logviewer | help)"
Exit 0
Elif [${ACTION_CMD} = = "nimbus"]; then
Echo "Starting nimbus ui logviewer supervisor"
Nohup / home/frank/platform/storm/bin/storm nimbus > > / tmp/nimbus.log 2 > & 1 &\
Nohup / home/frank/platform/storm/bin/storm ui > > / tmp/ui.log 2 > & 1 &\
/ home/frank/platform/storm/bin/storm logviewer
Elif [${ACTION_CMD} = = "supervisor"]; then
Echo "Starting supervisor"
/ home/frank/platform/storm/bin/storm supervisor
Fi
The image building command is as follows:
Docker build-- network=host-f Dockerfile-t ip:80/bigdata/storm:1.2.2.
Let's provide the creation file of service.
ApiVersion: v1
Kind: Service
Metadata:
Name: storm-cluster
Labels:
App: storm-cluster
Namespace: storm
Spec:
ClusterIP: None
Ports:
-port: 8080
Name: ui
Selector:
App: storm-cluster
ApiVersion: apps/v1
Kind: StatefulSet
Metadata:
Name: storm-cluster
Namespace: storm
Spec:
Selector:
MatchLabels:
App: storm-cluster
ServiceName: storm-cluster
Replicas: 4
PodManagementPolicy: Parallel
Template:
Metadata:
Labels:
App: storm-cluster
Spec:
TerminationGracePeriodSeconds: 2
Containers:
-name: storm-cluster
ImagePullPolicy: Always
Image: ip:80/bigdata/storm:1.2.2
Command: ["/ bin/sleep"]
Args: ["30d"]
VolumeMounts:
-name: storm-conf
MountPath: / home/conf/storm
-name: storm-log
MountPath: / home/logs/storm
Resources:
Requests:
Memory: "4096Mi"
Cpu: 4
Limits:
Memory: "4096Mi"
Cpu: 4
Env:
-name: NIMBUS_HOSTS
Value: "storm-cluster-2,storm-cluster-3"
-name: STORM_ZK_QUORUM
Value: "ip"
-name: STORM_ZK_ROOT
Value: "stormNew"
Ports:
-containerPort: 6627
Name: nimbus
-containerPort: 8000
Name: logviewer
-containerPort: 8080
Name: storm-ui
-containerPort: 6700
Name: sv0
-containerPort: 6701
Name: sv1
-containerPort: 6702
Name: sv2
-containerPort: 6703
Name: sv3
-containerPort: 22
Name: ssh
Volumes:
-name: storm-conf
ConfigMap:
Name: storm-conf
-name: storm-log
HostPath:
Path: / stormtmp
Type: Directory
The creation script for service is as follows:
Kubectl create-f storm-svc.yaml
The creation script for statefulSet is as follows:
Kubectl create-f storm-statefulSet.yaml
This creation script should pay attention to the following problems:
Command:/bin/sleep
The program needs to be started manually
/ home/vipshop/hard_disk/storm
Drwxr-xr-x 2 storm storm 4096 Jul 12 12:20 blobs
Drwxr-xr-x 2 storm storm 4096 Jul 12 12:20 nimbus
Drwxrwxr-x 5 storm storm 4096 Jul 29 00:00 supervisor
This hard_disk directory requires storm permission
Note: nimbus.thrift.max_buffer_size: 10485760
This parameter needs to be a multiple of 2
Thank you for reading this article carefully. I hope the article "K8s-based installation examples in storm" shared by the editor will be helpful to you. At the same time, I also hope you will support us and pay attention to the industry information channel. More related knowledge is waiting for you to learn!
