Shulou(Shulou.com)05/31 Report--

This article introduces how to understand Storm and JStorm. The content is very detailed. Interested friends can use it for reference. I hope it will be helpful to you.

A simple overview of Storm is: JStorm is more stable, more powerful and faster than Storm. Programs running on Storm can run on JStorm with a single line of code unchanged. To put it bluntly, JStorm is the secondary development product of Alibaba's team based on Storm, which is equivalent to their Tengine development based on Ngix.

The existing Storm cannot meet some needs

The existing storm scheduling is too simple and rough to be customized.

Storm task allocation is not balanced

RPC OOM has not been solved.

Monitoring is too simple.

Frequent access to ZK

JStorm is more stable than Storm.

Nimbus implements HA: when a nimbus is down, it is automatically eager to backup nimbus.

Native Storm RPC:Zeromq uses out-of-heap memory, resulting in insufficient OS memory. Netty causes the underlying RPC of OOM;JStorm to adopt netty + disruptor to ensure that the sending speed matches the receiving speed.

The new tasks will not impact the old tasks: the new scheduling allocates the tasks from the four angles of cpu,memory,disk,net, and the new tasks that have been assigned do not need to preempt the cpu,memory,disk and net of the old tasks.

Supervisor mainline

Open/prepar of Spout/Bolt

All IO, serialization, deserialization

Reduce the number of visits to ZK: remove a large number of useless watch;task and double the heartbeat time; Task heartbeat detection does not require a full ZK scan.

JStorm scheduling is more powerful than Storm scheduling.

Completely solved the problem of uneven task allocation in storm.

Assign tasks from four dimensions: CPU, Memory, Disk, Net

The default is a task and a cpu slot. When task consumes more cpu, you can apply for more cpu slot

The default is a task and a memory slot. When task needs more memory, you can request more memory slot

Default task, do not apply for disk slot. When the task disk IO is heavy, you can apply for disk slot

You can force the task of a component to run on different nodes

You can force topology to run on a single node

You can customize the task assignment, make a reservation in advance on which machine, which port, how many cpu slot, how much memory, and whether to apply for disk

You can book the task assignment for the last successful run, and what resources were allocated by task last time, and use these resources this time.

JStorm has better performance than Storm.

The performance of JStorm 0.9.0 is very good. The maximum sending speed of a single worker is 110000 QPS when using netty and 120000 QPS when using zeromq.

JStorm 0.9.0 is 10% faster than Storm 0.9.0 using netty when using Netty, and JStorm netty is stable while Storm's Netty is unstable.

With ZeroMQ, JStorm 0.9.0 is 30% faster than Storm 0.9.0.

Reasons for performance improvement:

Zeromq reduces one memory copy

Add deserialization thread

Rewrite the sampling code to greatly reduce the impact of sampling

Optimize ack code

Optimize buffered map performance

Java is lower than clojure.

Other optimization points of JStorm

Resource isolation. Different departments use different group names, and each group has its own Quato; for resource isolation. Cgroups hard isolation is adopted.

Classloader . Resolve the conflict between the applied class and the Jstorm class, and the applied class is in its own class space.

Task is asynchronized internally. Worker internal full pipeline mode, Spout nextTuple and ack/fail run on different threads

About how to understand Storm and JStorm to share here, I hope the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.