Shulou(Shulou.com)06/01 Report--

The editor will share with you how spark merges github. I hope you will get something after reading this article. Let's discuss it together.

Recently, we are working on an internal spark version upgrade, which involves merging pr on github. Specifically, when upgrading from spark 2.x to spark 3.0.1, it is compatible with hdfs cdh-2.6.0-5.13.1 and a compilation error is reported.

[INFO] Compiling 25 Scala sources to / Users/libinsong/Documents/codes/tongdun/spark-3.0/resource-managers/yarn/target/scala-2.12/classes... [ERROR] [Error] resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:298: value setRolledLogsIncludePattern is not a member of org.apache.hadoop.yarn.api.records.LogAggregationContext [ERROR] [Error] resource-managers/yarn/src/main/scala/org/apache / spark/deploy/yarn/Client.scala:300: value setRolledLogsExcludePattern is not a member of org.apache.hadoop.yarn.api.records.LogAggregationContext [ERROR] [Error] resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:551: not found: value isLocalUri [ERROR] [Error] resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:1367: not found: value isLocalUri [ERROR] four errors found

The specific solution github spark pr has given a solution, just modify the corresponding code, but if only modify, we can choose a more elegant way (in git cherry-pick way)

Now simply share:

Go directly to the setRolledLogsIncludePattern line.

SparkConf.get (ROLLED_LOG_INCLUDE_PATTERN). Foreach {includePattern = > try {val logAggregationContext = Records.newRecord (includePattern) sparkConf.get (ROLLED_LOG_EXCLUDE_PATTERN). Foreach {excludePattern = > logAggregationContext.setRolledLogsExcludePattern (excludePattern)} appContext.setLogAggregationContext (logAggregationContext)} catch {case NonFatal (e) = > logWarning (s "Ignoring ${ ROLLED_LOG_INCLUDE_PATTERN.key} because the version of YARN "+" does not support it " E)} appContext.setUnmanagedAM (isClientUnmanagedAMEnabled) sparkConf.get (APPLICATION_PRIORITY). Foreach {appPriority = > appContext.setPriority (Priority.newInstance (appPriority))} appContext}

When we find that the code on master is not what we want, we can use git blame on github for

In this way, we can find that the code has been modified many times and find the corresponding [SPARK-19545] [YARN] Fix compile issue for Spark on Yarn when building. Click in

Find the corresponding commitId

Execute the command git cherry-pick 8e8afb3a3468aa743d13e23e10e77e94b772b2ed to append the commit to your working directory

In this way, there is no need to modify the code manually, and the information of the original commit can be well preserved for tracking.

After reading this article, I believe you have a certain understanding of "how spark merges github". If you want to know more about it, you are welcome to follow the industry information channel. Thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.