Shulou(Shulou.com)06/01 Report--

In this issue, the editor will bring you about the principles of Spark Streaming and Kafka Stream. The article is rich in content and analyzes and narrates it from a professional point of view. I hope you can get something after reading this article.

Here are two commonly used streaming frameworks, Spark Streaming and Kafka Stream, and their respective characteristics are described in detail to help readers choose the framework in different scenarios. The following is a translation. The demand for streaming processing is increasing every day, and it is not enough to process a large amount of data. Data must be processed quickly so that enterprises can respond to the changing business environment in real time. Streaming is the continuous and concurrent real-time processing of data. Streaming is an ideal platform for processing data streams or sensor data, while complex event processing (CEP) uses techniques such as event-by-event processing and aggregation. For real-time data processing functions, we have many options, such as Spark, Kafka Stream, Flink, Storm and so on. In this blog, I will discuss the difference between Apache Spark and Kafka Stream.

Apache Spark

Apache Spark is a general framework for large-scale data processing, supporting many different programming languages and concepts, such as MapReduce, memory processing, streaming, graphics processing, and machine learning. It can also be used at the top level of Hadoop. Data can be obtained from a variety of sources (such as Kafka, Flume, Kinesis, or TCP sockets) and processed using complex algorithms such as mapping, reduction, joins, and windows.

Inside the framework, how it works is shown in the following figure. Spark Streaming receives the real-time input data stream and divides the data into multiple batches, which are then processed by the Spark engine to generate the final result stream.

Spark Streaming provides a high-level abstraction called discretized stream (DStream), which represents a continuous data flow. DStream can be created from input data streams from sources such as Kafka, Flume, or Kinesis, or by performing advanced operations on other DStream. Within the framework, DStream can be thought of as a series of RDD (Resilient Distributed Datasets, flexible distributed datasets).

Kafka Stream

Kafka Streams is a client library for processing and analyzing data. It first processes and analyzes the data stored in Kafka, and then writes back the final data result to Kafka or sends it to an external system. It is based on some very important streaming concepts, such as proper distinction between event time and processing time, window support, and simple (efficient) management of application state. At the same time, it is also based on many concepts in Kafka, such as extending by dividing topics. In addition, for this reason, it can be integrated into the application as a lightweight library. The application can be run independently as needed, in an application server, as a Docker container, or through a resource manager such as Mesos.

Kafka Streams directly solves many difficult problems in streaming:

Event-by-event processing with millisecond delay.

Stateful processing, including distributed connections and aggregation.

Convenient DSL.

Windowing unordered data using a model like DataFlow.

It has the distributed processing and fault tolerance of fast failover.

Rolling deployment without downtime.

Apache Spark can be used with Kafka to transfer data, but if you are deploying a Spark cluster for a new application, this is definitely a complex problem.

To overcome this complexity, we can use a complete streaming framework, and Kafka streams is the choice to do this.

Our goal is to simplify streaming and make it the mainstream application programming model for asynchronous services. This is the * * library I know that makes full use of Kafka, not just Kafka as an information intermediary.

Streams builds on the concepts of KTables and KStreams, which helps them provide event time processing.

This paper presents a processing model which is highly integrated with the core abstraction of Kafka, which can reduce the total number of mobile parts in the streaming architecture.

Fully integrating state tables with event flows and providing both in a single conceptual framework makes Kafka Streams a completely embedded library rather than a streaming cluster (just Kafka and your application). When you add a new instance to the application, or when an existing instance crashes, it automatically balances the load and maintains the local state of the table, allowing the system to recover from the failure.

Kafka Streams has low latency and supports easy-to-use event time. It is a very important library and is well suited for certain types of tasks. This is why some designs can be deeply optimized for how Kafka works. You don't need to set up any kind of Kafka Streams cluster, and there is no cluster manager. If you need to implement a simple Kafka topic-to-topic conversion, count elements by keyword, load data from another topic into a stream, or run aggregations or just perform real-time processing, then Kafka Streams is for you.

If the event time is irrelevant and the second delay is acceptable, then Spark is your choice. It is quite stable and can be easily integrated into almost any type of system. In addition, it is included in every Hadoop distribution. Also, the code for batch applications can also be used for streaming applications because the API is the same.

Conclusion

In my opinion, Kafka Streams works best in "Kafka > Kafka" scenarios, while Spark Streaming can be used in scenarios such as "Kafka > database" or "Kafka > data science model".

These are the principles of Spark Streaming and Kafka Stream shared by the editor. If you happen to have similar doubts, you might as well refer to the above analysis to understand. If you want to know more about it, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.