Shulou(Shulou.com)06/01 Report--

This article analyzes "how to compare kappa and lambda". The content is detailed and easy to understand. Friends who are interested in "how to compare kappa and lambda" can follow the editor's idea to read it slowly and deeply. I hope it will be helpful to you after reading. Let's learn more about "how to compare kappa and lambda" with the editor.

First of all, we will explain the two architectures in detail, the technical tools to implement them, and how to decide to use them.

How to build a real-time processing system architecture has been debated continuously. A good real-time processing system must be fault-tolerant and scalable. Batch and incremental updates must be supported and must be extensible.

An important milestone in these discussions is that the founder of storm, Nathan Marz, describes the lambda architecture as we know it so far. The Lambda architecture has many use cases, and a large number of companies are using it in real time, such as Yahoo and Netflix. Of course, the lambda architecture is not all praised, there is also some criticism, is that it brings the burden of coding. (original: But of course, Lambda is not a silver bullet and has received some fair criticism on the coding overhead it can create.)

In the summer of 2014, LinkedIn's Jay Kreps published an article describing the Kappa architecture and addressing some of the pitfalls of the Lambda architecture. The Kappa architecture is not an alternative to the Lambda architecture, because some Lambda schemas are not suitable for migration to the Kappa architecture.

For a given case, it is challenging to accurately assess which architect is the best, and wrong design decisions can have a serious impact on the implementation of data analysis projects.

Now, go into the details to learn about the two data processing architectures.

1lambda architecture

The Lambda architecture consists of three layers: batch,speed,serving.

There are two main tasks at the Batch level:

1. Manage historical data.

two。 Re-settle the results, such as retraining the model.

The Batch layer accepts the new data, merges the new data with historical data, and then recalculates the results. The Batch layer calculates all the data, which enables the system to produce relatively accurate results. However, due to the long calculation time, the delay of the result will be relatively large.

The Speed layer mainly provides low latency, near real-time computing results. The Speed layer receives the data and incrementally updates the results of the batch layer. Due to the incremental algorithm of the speed layer, the computational cost is greatly reduced.

Serving provides a variety of queries using the results calculated by the batch layer and the speed layer.

2kappa architecture

One of the most important motivations for creating a kappa architecture is to avoid maintaining separate code for the batch and speed layers. A core idea is to use a single stream processing engine to deal with real-time computing and continuous data repetition. Changes in the code have a big impact on the results, so the data must be recalculated. As a result, the kappa architecture consists of only two parts: stream processing and serving. The flow processing layer runs the flow processing task. Run a streaming job to enable real-time data processing. The data is reprocessed only after the streaming job has changed some code. You can process all previous data by restarting a stream processing job after an outline code.

The Serving layer also provides data query.

On how to compare kappa and lambda to share here, I hope that the above content can make you improve. If you want to learn more knowledge, please pay more attention to the editor's updates. Thank you for following the website!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.