Shulou(Shulou.com)05/31 Report--

This article mainly introduces "what are the shortcomings of spark". In daily operation, I believe that many people have doubts about the shortcomings of spark. The editor consulted all kinds of materials and sorted out simple and easy-to-use methods of operation. I hope it will be helpful for you to answer the doubts about "what are the shortcomings of spark?" Next, please follow the editor to study!

Can Q1:spark streaming join with different data streams?

Different data streams of Spark Streaming can be operated by join

Spark Streaming is an extension of the core Spark API that allows enables high-throughput, fault-tolerant stream processing of live data streams. Data can be ingested from many sources like Kafka, Flume, Twitter, ZeroMQ or plain old TCP sockets and be processed using complex algorithms expressed with high-level functions like map, reduce, join and window

Join (otherStream, [numTasks]): When called on two DStreams of (K, V) and (K, W) pairs, return a new DStream of (K, (V, W)) pairs with all pairs of elements for each key.

Are Q2:flume and spark streaming suitable for cluster mode?

Flume and Spark Streaming are created for clusters.

For input streams that receive data over the network (such as, Kafka, Flume, sockets, etc.), the default persistence level is set to replicate the data to two nodes for fault-tolerance.

Does Q3:spark have any shortcomings?

The core disadvantage of Spark is that it takes up a lot of memory.

In previous versions, the processing of data by Spark was mainly coarse-grained, so it was difficult to control it finely.

Later, with the addition of Fair mode, fine-grained processing can be done.

There is no need to deploy, just install Spark, and follow Spark Streaming

Spark Streaming is used in China, such as Pippi net.

At this point, the study of "what are the shortcomings of spark" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.