Shulou(Shulou.com)06/02 Report--

This article mainly introduces the relevant knowledge of "what are the five mainstream big data computing frameworks". The editor shows you the operation process through an actual case, and the operation method is simple, fast and practical. I hope this article "what are the five mainstream big data computing frameworks" can help you solve the problem.

Hadoop framework

When it comes to big data, the first thing that comes to mind must be Hadoop, because Hadoop is the most widely used big data tool in the world. With his high fault tolerance rate and low hardware price, he has gained success in the big data market. Hadoop is also the first batch framework that has attracted a lot of attention in the open source community, and his computing models for Map and Reduce are simple and elegant. So far, Hadoop has become a vast ecosystem, implementing a large number of algorithms and components. Because the computing task of Hadoop needs to read and write multiple times on multiple nodes of the cluster, it will show a slight disadvantage in speed, but its throughput is also unmatched by other frameworks.

Storm framework

Unlike Hadoop's batch mode, Storm uses a streaming computing framework, which is open source by Twitter and hosted on GitHub. Similar to Hadoop, Storm also proposes two computing roles, Spout and Bolt. If Hadoop is a bucket, you can only carry it in the well, then Storm is the faucet, as long as you open it, you can keep coming out of the water. Storm also supports many languages, such as Java, Ruby, Python and so on. Because Storm is a streaming computing framework, it uses memory and has a great advantage in latency, but Storm does not persist data.

Samza framework

Smaza is also a streaming computing framework, but it currently only supports the JVM language, which is slightly less flexible, and Samza must be used in conjunction with Kafka. But in response, it also inherits the advantages of Kafka, such as low latency, partitioning, avoiding backpressure and so on. Samza is a good choice for teams that already have a Hadoop+Kafka work environment, and Samza shows good performance when used by multiple teams.

Spark framework

Spark is a hybrid computing framework, which belongs to the collection of the first two kinds of frames. It not only has its own real-time stream processing tools, but also can be integrated with Hadoop to replace MapReduce, and even Spark can be used to deploy clusters separately, but also with the help of distributed storage systems such as HDFS. The power of Spark lies in its computing speed, similar to Storm, Spark is also based on memory, and when the memory is full, the hard disk can also operate. The results show that the speed of Spark is about 100 times that of Hadoop, and its cost may be lower than Hadoop. However, at present, Spark does not have a cluster of tens of thousands of levels like Hadoop, so it is more appropriate to use Spark and Hadoop at this stage.

Flink framework

Flink is also a hybrid computing framework, but at the beginning of the design, Fink focused on dealing with streaming data, which is contrary to the original intention of Spark, and driven by market demand, both are moving towards more compatibility. Flink is not very mature at present, and Flink still plays a reference role in more cases.

This is the end of the introduction of "what are the five mainstream big data computing frameworks". Thank you for your reading. If you want to know more about the industry, you can follow the industry information channel. The editor will update different knowledge points for you every day.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.