Shulou(Shulou.com)06/02 Report--

This article mainly introduces the concept and application of big data Lambda architecture example analysis, the article is very detailed, has a certain reference value, interested friends must read it!

Big data's concept and Application of Lambda Architecture

Big data's platform includes Batch Layer for batch computing and Speed Layer for real-time computing, which integrates batch computing and stream computing in one platform.

For example, HadoopMapReduce and Spark are used for batch data processing, and ApacheStorm and Spark Streaming are used for real-time data processing.

This architecture solves the problems of different computing types to a certain extent, but the problem is that there are too many frameworks, which will lead to high complexity of the platform and high success of operation and maintenance.

The main idea of the Lambda architecture is to build the big data system into multiple levels, as shown in the following figure:

Let's sort out how they divide the work and help:

First of all, new data as the data source of the whole data system, Batch Layer as the batch processing level of the data to process and process the original data, and input the Batch View of the processed data to Serving Layer. (this corresponds to full data.)

Speed Layer processes the real-time added data and generates the Real-time View of the calculation result of the incremental data. (here corresponds to incremental data)

The end-user query presents the final result through the combination of Batch View and Real-time View.

Based on the Lambda architecture, once the data enters the Serving layer through Batch layer, the corresponding results in Real-time view are no longer needed.

The above is all the contents of the article "big data's sample Analysis of the concept and Application of Lambda Architecture". Thank you for reading! Hope to share the content to help you, more related knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.