Shulou(Shulou.com)06/03 Report--

A brief introduction to Storm:

Storm is an open source distributed real-time computing framework originally developed by Twitter and is now one of Apache's top projects. Storm is suitable for dealing with potentially unlimited streaming data. Storm and real-time data computing are equivalent to Hadoop and batch processing.

Storm features:

Simple programming model: similar to MapReduce reduces the complexity of parallel batch processing, Storm reduces the complexity of real-time processing.

Fault tolerance: Storm manages worker processes and node failures.

Horizontal scaling: computing is performed in parallel between multiple threads, processes, and servers.

Reliable message processing: Storm ensures that each message is fully processed at least once. When a task fails, it is responsible for retrying the message from the message source.

Fast: the design of the system ensures that messages can be processed quickly, and MQ is used as its underlying message queue.

Multilingual support: you can use a variety of programming languages on top of Storm. Clojure, Java, Ruby and Python are supported by default. To add support for other languages, you only need to implement a simple Storm communication protocol.

Storm processing flow:

Applicable scenarios for Storm:

(1) Information flow processing (Stream Processing): Storm can be used to process new data and update database in real time, with both fault tolerance and expansibility.

(2) continuous computing (Continuous Computation): Storm can make continuous queries and immediately feed back the results to the customer, such as sending hot topics on Twitter to the client.

(3) distributed remote procedure call (Distributed RPC): Storm can be used to process dense queries in parallel. The topology of Storm (described later) is a distribution function waiting for call information. When it receives a call message, it calculates the query and returns the query results.

Storm deployment:

Build steps:

-install zookeeper

-install storm

-start storm

-access storm ui

Install zookeeper

Installation script:

#! / bin/bashname=zookeeper1SERVER_ID=1HOST1=192.168.1.1port=22181p_w_picpath=docker.ttxsgoto.com:5000/zk.ttxsgoto.com:v20160712run () {docker run-d\-name $name\-e "SERVER_ID=$SERVER_ID"\-e "ADDITIONAL_ZOOKEEPER_1=server.1=$HOST1:2181:3181"\-p ${port}: 2181\-v / opt/zookeeper/data:/opt/zookeeper/data\ $p_w_picpath} pullp_w_picpath () {docker pull $p_w_picpathif [$? -ne 0]; thenecho "Docker pull $p_w_picpath fail." exit 1fi} case $1 indeploy) pullp_w_picpathrun;;stop) docker stop $name;;rm) docker stop $namedocker rm $name;;start) docker start $name;;redeploy) pullp_w_picpathdocker stop $namedocker rm $namerun;;*) echo "Usage $0 [deploy | start | stop | rm | redeploy]"; esac;

1. You need to modify the SERVER_ID,HOST1,port in the script

two。 Add execute permissions and execute chmod + x zkstart.sh & &. / zkstart.sh deploy

Install storm

Installation script:

#! / bin/bashif [- z "$1"]; thenecho "Usage error exit" echo ". / build.sh [appname] [ver]" exit 2fiif [- z "$2"]; thenecho "Usage error please input docker version. Exit "echo". / build.sh [appname] [ver] "exit 2fidocker build-- no-cache=true-t docker.ttxsgoto.com:5000/$1.ttxsgoto.com:$2 .docker push docker.ttxsgoto.com:5000/$1.ttxsgoto.com:$2d= `date" + [% Y-%m-%d% H:%M:%S] "`echo-e" $d\ tasking * "> > build.txt

1. Add execution permission chmod + x build.sh

two。 Pass parameters to execute the script (eg:. / build.sh storm v20160712)

Before PS:deploy/redeploy, you need to modify the / etc/hosts file to update the mapping relationship between hostname and ip of the nimbus node.

Storm start

Start the script:

#! / bin/bashp_w_picpath= "docker.ttxsgoto.com:5000/$1.ttxsgoto.com:$2" log= "/ var/logs/$1.ttxsgoto.com.$2.$3" port=$5name=$3mode=$4run () {docker run-net=host-it-d\-name $name\-v $log:/opt/apache-storm-1.0.1/logs\ "$p_w_picpath" / bin/sh-c "/ opt/apache-storm-1.0.1/bin/storm $mode"} pullp_w_ Picpath () {docker pull $p_w_picpathif [$?-ne 0] Thenecho "Docker pull $p_w_picpath fail." exit 1fi} case $5 indeploy) pullp_w_picpathrun;;redeploy) pullp_w_picpathdocker stop $namedocker rm $namerun;;*) echo "Usage: $0 [appname version container_name mode deploy | redeploy]" esac

1. Modify the storm.yaml file to configure zookeeper node information (storm.zookeeper.servers+storm.zookeeper.port), such as:

Storm.zookeeper.servers:- "192.168.1.1" storm.zookeeper.port: 22181storm.local.dir: "/ opt" nimbus.seeds: ["ub"] supervisor.slots.ports:- 6700-6701-6702-6703

two。 Add execution permission chmod + x storm.sh

3. Pass parameters to execute storm.sh

Eg:./storm.sh storm v20160712 ui ui deploy./storm.sh storm v20160712 nimbus nimbus deploy./storm.sh storm v20160712 supervisor supervisor deploy

Access the storm UI interface

Eg: http://192.168.1.1:8080

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.