Shulou(Shulou.com)06/03 Report--

When using Apache Spark, jobs run on different nodes in a distributed manner; especially when the size of the cluster is very large, it is very common for the nodes of the cluster to have various problems, such as problems with a certain disk. We all know that ApacheSpark is a high-performance, fault-tolerant distributed computing framework. Once it knows that a computing machine has a problem (such as a disk failure), it will reschedule the Task based on the previously generated lineage.

Let's now consider the following scenario:

The disk on one node failed intermittently for some reason, so that some sectors could not be read. Assuming that the data needed for our Spark job happens to be on these sectors, this will cause the Task to fail.

The Driver of the job gets this information and knows that Task has failed, so it will resubmit the Task.

After Scheduler gets this request, it takes into account the locality of the data, so it is likely to distribute the Task to the above machine because it does not know that there is a problem with the disk of the machine mentioned above.

This Task may also fail because there is a problem with the disk of this machine. Then Driver repeats these operations, resulting in the failure of the Spark job!

The scenarios mentioned above can actually be avoided by some measures for us. But it was inevitable before version 2.2.0 of Apache Spark, but happily, engineers from Cloudera solved the problem by introducing the blacklisting mechanism Blacklist (see SPARK-8425 for details, see Design Doc for Blacklist Mechanism for specific design documentation) and with the release of version 2.2.0 of Apache Spark, but it is still in the experimental stage.

The blacklist mechanism actually works by maintaining records of previously problematic actuators (Executors) and nodes (Hosts). When a task (Task) fails, the blacklist mechanism will track the executors and hosts associated with the task and record this information; when the number of failures in scheduling tasks at this node exceeds a certain number (default is 2), then the scheduler will no longer distribute tasks to that node. The scheduler can even kill the corresponding executor of that machine, which can be done through the corresponding configuration.

We can see the Status of the actuator through the Apache Spark WEB UI interface: if the actuator is blacklisted, you can see it on the page as Blacklisted, otherwise as Active. As shown in the following figure:

With the blacklist mechanism, the problems in the above scenarios can be well solved.

At present, the blacklist mechanism can be controlled by a series of parameters, mainly as follows:

Because the blacklisting mechanism is currently in an experimental state, some of the above parameters may be modified later in Spark.

Conclusion

Thank you for watching. If there are any deficiencies, you are welcome to criticize and correct them.

If you have a partner who is interested in big data or a veteran driver who works in big data, you can join the group:

658558542

Welcome everyone to exchange and share, study and exchange, and make common progress. There are also a lot of free materials to help you overcome difficulties on your way to becoming big data engineers and even architects! )

Finally, I wish all the big data programmers who encounter bottlenecks to break through themselves and wish you all the best in the future work and interview.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.