Shulou(Shulou.com)06/01 Report--

This article mainly introduces "what is the cluster deploy mode of Spark". In the daily operation, I believe that many people have doubts about the cluster deploy mode of Spark. The editor has consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful to answer the questions of "what is the cluster deploy mode of Spark?" Next, please follow the editor to study!

Spark has three cluster deployment modes, or cluster management modes. They are standalone,YARN and Mesos. All three modes are actually master/slave modes.

Overview of three kinds of Cluster Resource Management

Spark Standalone

As part of Spark, Standalone is a simple cluster manager. It has the HA of master, the flexibility to respond to WorkerFailures, the ability to manage resources for each application, and the ability to run and access HDFS data together with existing Hadoop. The distribution includes scripts that can be easily deployed locally or in AmazonEC2 cloud computing. It can be run on Linux,Windows or Mac OSX.

Apache Mesos

Apache Mesos, distributed system kernel, with HA masters and slaves, can manage the resources of each application, and has good support for Docker containers. It can run Spark work, Hadoop's MapReduce, or any other service application. It has Java, Python and C + + API. It can be run on Linux or Mac OSX.

Hadoop YARN

Hadoop YARN, a distributed computing framework for job scheduling and cluster resource management, with HA of masters and slaves, Docker container in non-secure mode, Linux and Windows container executors in secure mode, and pluggable scheduler. It can run on Linux and Windows.

Comparison of Cluster Resource scheduling capability

Mode resource scheduling capability is compared to SecurityHA

Standalone only supports FIFO scheduler, single user serial. By default, all resources of all nodes are available to applications. Limitations of node nodes, cpu memory and other limitations can be used to control single point recovery of shared secret, SSL for data encryptionStandby Masters with ZooKeeper and local file system through SparkConf.

YARN supports Resource Scheduler Scheduler and Application Manager ApplicationsManager. CapacityScheduler and FairScheduler are within the scope of the queue and resources are shared. Kerberos . HA mode of SSL for data encryption master / slave switching, which depends on zookeeper, but does not require a separate zkfc process.

Mesos look below: plug-in security module, default Cyrus SASL,SSL for data encryption one master and multi-slave, leader election based on zookeeper

Resource scheduling capability description of Mesos

Coarse-grained pattern (Coarse-grained Mode): the running environment of each application consists of a Dirver and several Executor, where each Executor takes up several resources, and multiple Task can be run internally (corresponding to how many "slot"). Before each task of the application program is officially run, all the resources in the running environment need to be applied for, and these resources should be occupied all the time in the process of running. Even if they are not used, these resources will be recycled after the program runs. For example, when you submit an application, you specify five executor to run your application, each executor takes up 5GB memory and five CPU, and each executor sets up five slot internally, then Mesos needs to allocate resources for executor and start them, and then start scheduling tasks. In addition, in the process of running the program, the master and slave of mesos do not know the operation of each task within executor, and executor reports the task status to Driver directly through the internal communication mechanism. to some extent, it can be considered that each application uses mesos to build a virtual cluster for its own use.

Fine-grained pattern (Fine-grained Mode): in view of the fact that coarse-grained patterns waste a lot of resources, Spark On Mesos also provides another scheduling mode: fine-grained mode, which is similar to today's cloud computing, with the idea of distribution on demand. Like the coarse-grained mode, when the application starts, executor starts first, but each executor takes up resources only for its own operation, without considering the tasks to be run in the future. After that, mesos dynamically allocates resources for each executor, and each allocation can run a new task, and the corresponding resources can be released immediately after a single Task is run. Each Task reports the status to Mesos slave and Mesos Master to facilitate more fine-grained management and fault tolerance. This scheduling mode is similar to the MapReduce scheduling mode. Each Task is completely independent. The advantage is that it is easy to control and isolate resources, but the disadvantage is also obvious. Short jobs run with large delays.

At this point, the study on "what is the cluster deploy mode of Spark" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.