Shulou(Shulou.com)06/03 Report--

Spark sets setMaster=local, does not submit clusters, and enables multithreaded simulation to run locally

Object SparkUtil {privateval logger = Logger.getLogger (getClass.getName, true) def getSparkContext (appName:String, local:Boolean=false, threadNum:Int=4): SparkContext = {val conf = new SparkConf (). SetAppName (appName) if (local) conf.setMaster (s "local [$threadNum]") val sc = new SparkContext (conf) return sc} SparkUtil.getSparkContext (" testApp ", true) / / start 4 threads locally to simulate running

Running in local mode can test the main program logic and may not run correctly in the cluster.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.