Shulou(Shulou.com)05/31 Report--

This article mainly introduces "how much memory is needed for spark to run 1T data". In daily operation, I believe many people have doubts about how much memory is needed for spark to run 1T data. Xiaobian consulted all kinds of information and sorted out simple and easy operation methods. I hope to help you answer the question of "how much memory is needed for spark to run 1T data"! Next, please follow the small series to learn together!

Q1: What is the use of observer enterprises?

There is a video site in China that has been using JobServer for more than half a year;

JobServer was strongly recommended at both the 2013 and 2014 Spark Summits;

Q2: Is Jobserver suitable for internal or external customers (possibly concurrent, security requirements), or both OK?

The enterprise use cases currently visible are used internally;

If it is external to the enterprise, it can be used as a cloud service or a big data resource pool;

Q3: How much memory does spark need to run 1T of data to run quickly?

This first relates to the memory and CPU used on each Worker when the program is running, which can be manually configured when the program is submitted;

Secondly, it has something to do with bandwidth. Shuffle should minimize data.

The configuration of the machine where the Driver is located is also extremely important. Generally speaking, the memory and CPU of the Client where the Driver is located should be configured as high as possible according to the actual situation. At the same time, it is also crucial that the Driver and Spark clusters should be in the same network environment. The Driver should constantly task the Executor on the Worker and accept the data of the Driver at the same time.

Q4: I am currently solving stackoverflow Error, which is to use CheckPoint to solve the problem of too long lineup, but this will affect efficiency. How to balance efficiency and error?

:StackOverflow can be mitigated by configuring BlockManager memory management policies;

Checkpoint should be adjusted according to actual situation. For example, Spark Streaming defaults to having two copies of data in memory. At this time, if the processing capacity cannot consume real-time stream data in time, StackOverflow will be extremely easy to occur. At this time, the time window and checkpoint should be adjusted according to actual situation.

At this point, the study of "how much memory is needed for spark to run 1T data" is over, hoping to solve everyone's doubts. Theory and practice can better match to help you learn, go and try it! If you want to continue learning more relevant knowledge, please continue to pay attention to the website, Xiaobian will continue to strive to bring more practical articles for everyone!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.