Shulou(Shulou.com)05/31 Report--

This article mainly shows you "how to use Twitter Storm to deal with real-time big data", the content is simple and clear, hoping to help you solve your doubts, let the editor lead you to study and learn "how to use Twitter Storm to deal with real-time big data" this article.

How to use Twitter Storm to deal with real-time big data

Hadoop (the indisputable king of big data's analytical field) focuses on batch processing. This model is sufficient for many situations, such as indexing web pages, but there are other usage models that require real-time information from highly dynamic sources. To solve this problem, you have to rely on Nathan Marz's Storm (now called BackType in Twitter). Storm does not handle static data, but it does handle stream data that is expected to be continuous. Given that Twitter users generate 140 million tweet a day, it's easy to see the great usefulness of this technology.

But Storm is not just a traditional big data analysis system: it is an example of a complex event processing (CEP) system. CEP systems are usually classified as computing and detection-oriented, in which each system can be implemented in Storm through user-defined algorithms. For example, CEP can be used to identify meaningful events in the event torrent and then process them in real time.

Nathan Marz provides a large number of examples of using Storm in Twitter. One of the most interesting examples is to generate trend information. Twitter extracts emerging trends from massive tweets and maintains them at the local and national levels. This means that when a case begins to emerge, Twitter's trend topic algorithm identifies the topic in real time. This real-time algorithm is implemented as a continuous analysis of Twitter data in Storm.

What is "big data"?

Big data refers to a huge amount of data that cannot be managed in traditional ways. Internet-wide data is driving the creation of new architectures and applications that can handle such new data. These architectures are highly scalable and can process data in parallel and efficiently across an unlimited number of servers.

Big data realized

The core of Hadoop is to use Java? Language, but supports data analysis applications written in various languages. The latest application implementations take a more esoteric approach to take full advantage of modern languages and their features. For example, Spark at the University of California, Berkeley (UC) is implemented in the Scala language, while Twitter Storm is implemented in the Clojure (pronounced closure) language.

Clojure is a modern dialect of the Lisp language. Similar to Lisp,Clojure, Clojure supports a functional programming style, but Clojure also introduces features to simplify multithreaded programming (a feature that is useful for creating Storm). Clojure is a virtual machine (VM)-based language that runs on Java virtual machines. However, although Storm is developed in the Clojure language, you can still write applications in almost any language in Storm. All that is needed is an adapter that connects to the architecture of the Storm. There are already adapters for Scala, JRuby, Perl, and PHP, but there are also structured query language adapters that support streaming to Storm topologies.

How to use Twitter Storm to deal with real-time big data

Key attributes of Storm

Some features of Storm implementation determine its performance and reliability. Storm uses ZeroMQ to send messages, which eliminates the intermediate queuing process and allows messages to flow directly between the tasks themselves. Behind the message is an automated and efficient mechanism for serializing and deserializing primitive types of Storm.

One of the most interesting things about Storm is that it focuses on fault tolerance and management. Storm implements guaranteed message processing, so each tuple is fully processed through this topology; if a tuple is found to be unprocessed, it is automatically replayed from the nozzle. Storm also implements task-level fault detection, and when a task fails, messages are automatically reassigned to quickly restart processing. Storm contains more intelligent process management than Hadoop, and the process is managed by the supervisor to ensure that resources are fully utilized.

Storm model

Storm implements a data flow model in which data flows continuously through a network of transformed entities (see figure 1). The abstraction of a data stream is called a stream, which is an infinite sequence of tuples. Tuples are like a structure that uses some additional serialization code to represent standard data types (such as integers, floating-point, and byte arrays) or user-defined types. Each flow is defined by a unique ID that can be used to build the topology of the data source and sink. The flow originates from the nozzle, which flows data from external sources into the Storm topology.

The above is all the contents of the article "how to use Twitter Storm to deal with Real-time big data". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.