Shulou(Shulou.com)05/31 Report--

This article to share with you is about JStorm principle and application scenarios is what, Xiaobian think quite practical, so share to everyone to learn, I hope you can read this article after some gains, not much to say, follow Xiaobian to see it.

JStorm is a system similar to Hadoop MapReduce. Users implement a task according to the specified interface, and then submit this task to JStorm system. JStorm runs this task and runs it for 7 * 24 hours. Once an unexpected failure occurs in the middle Worker, the scheduler immediately assigns a new Worker to replace the failed Worker.

Thus, from an application perspective, a JStorm application is a distributed application that adheres to a programming specification. From a system perspective, JStorm is a MapReduce like scheduling system. From a data perspective, JStorm is a pipelined messaging mechanism.

Real-time computing is now one of the hottest directions in the field of big data, because people's requirements for data are getting higher and higher, and real-time requirements are getting faster and faster. Traditional Hadoop MapReduce gradually cannot meet the needs, so there is constant demand in this field.

Storm components versus Hadoop components

StormHadoop Role NimbusJobTrackerSupervisor TaskTrackerWorkerChild Application Name TopologyJob Programming Interface Spout/BoltMapper/Reducer Advantages

Before Storm and JStorm appeared, there were many real-time computing engines on the market, but since Storm and JStorm appeared, they can basically be said to unify the Jianghu:

Development is very fast: the interface is simple and easy to use. As long as you follow the programming specifications of Topology, Spout and Bolt, you can develop an application with excellent scalability. The redundancy between the underlying RPC and Worker, and the actions such as data diversion are completely ignored.

Excellent scalability: when the speed of the first-level processing unit, directly configure the concurrency number, you can linearly expand the performance

Robust: When a Worker fails or a machine fails, a new worker is automatically assigned to replace the failed worker.

Data accuracy: Ack mechanism can be used to ensure that data is not lost. If there is more requirement for precision, transaction mechanism is adopted to ensure data accuracy.

application scenarios

JStorm processes data in a message-based pipelined way, and is therefore particularly suited for stateless computing, where the dependent data of a compute unit can all be found in the received message, and preferably one data stream does not depend on another data stream.

Therefore, it is often used

Log analysis, analyzing specific data from logs and storing the results of analysis in external memory such as databases. Currently, mainstream log analysis techniques use JStorm or Storm

Pipeline systems, which transfer data from one system to another, such as synchronizing databases to Hadoop

Message converter converts received messages into a certain format and stores them in another system such as message middleware

A statistical analyzer extracts a field from a log or message, then does a count or sum calculation, and finally stores the statistical value in an external memory. Intermediate processing may be more complex.

The above is what the JStorm principle and application scenarios are. Xiaobian believes that some knowledge points may be seen or used in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.