Shulou(Shulou.com)06/03 Report--

The contents of this issue:

Dynamic allocation of Spark Streaming resources

Spark Streaming dynamically controls consumption rate

Why do you need dynamic?

Spark is coarse-grained by default, and resources are allocated before calculation. While Spark Streaming has high peak and low peak, but the resources they need are not the same, if according to the high peak point of view, there will be a lot of waste of resources.

With the continuous operation of Spark Streaming, the consumption and management of resources are also factors that we have to consider.

There are challenges when Spark Streaming resources are dynamically adjusted:

Spark Streaming runs according to Batch Duration, and Batch Duration needs a lot of resources. Next time, Batch Duration will not need so many resources. The Batch Duration operation has expired before the resources are adjusted. Adjust the time interval at this time.

Dynamic request for Spark Streaming resources

1. Dynamic resource allocation is not enabled by default in SparkContext, but can be configured manually in SparkConf.

/ / Optionally scale number of executors dynamically based on workload. Exposed for testing.val dynamicAllocationEnabled = Utils.isDynamicAllocationEnabled (_ conf) if (! dynamicAllocationEnabled & & / / whether the parameter configuration enables dynamic resource allocation _ conf.getBoolean ("spark.dynamicAllocation.enabled", false)) {logWarning ("DynamicAllocation and num executors both set, thus dynamic allocation disabled.")} _ executorAllocationManager = if (dynamicAllocationEnabled) {Some (new ExecutorAllocationManager (this, listenerBus, _ conf))} else {None} _ executorAllocationManager.foreach (_ .start ())

ExecutorAllocationManager: there is a timer that will constantly scan the Executor. If you want to run the Stage in a different Executor, either increase the Executor or decrease it.

The schedule method in ExecutorAllocationManager will be triggered periodically for resource dynamic adjustment.

/ * * This is called at a fixed interval to regulate the number of pending executor requests * and number of executors running. * First, adjust our requested executors based on the add time and our current needs. * Then, if the remove time for an existing executor has expired, kill the executor. * * This is factored out into its own method for testing. * / private def schedule (): Unit = synchronized {val now = clock.getTimeMillis updateAndSyncNumExecutorsTarget (now) removeTimes.retain {case (executorId, expireTime) = > val expired = now > = expireTime if (expired) {initializing = false removeExecutor (executorId)}! expired}}

In ExecutorAllocationManager, the timer will continue to run schedule in the thread pool.

/ * * Register for scheduler callbacks to decide when to add and remove executors, and start * the scheduling task. * / def start (): Unit = {listenerBus.addListener (listener) val scheduleTask = new Runnable () {override def run (): Unit = {try {schedule ()} catch {case ct: ControlThrowable = > throw ct case t: Throwable = > logWarning (s "Uncaught exception in thread ${Thread.currentThread (). GetName}", t)}} / / intervalMillis timer trigger time executor.scheduleAtFixedRate (scheduleTask 0, intervalMillis, TimeUnit.MILLISECONDS)}

Dynamic control of consumption rate: Spark Streaming provides a flexible mechanism, the relationship between the speed of flow and processing speed, whether there is time to process data. If it is not in time, it will automatically dynamically control the speed of the data flow in and the spark.streaming.backpressure.enabled parameter setting.

The principle of dynamic control of consumption rate can be found in the paper Adaptive Stream Processing using Dynamic Batch Sizing

Note:

1. DT big data DreamWorks Wechat official account DT_Spark

2. IMF 8: 00 p.m. Big data actual combat YY live broadcast channel number: 68917580

3. Sina Weibo: http://www.weibo.com/ilovepains

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.