Shulou(Shulou.com)06/02 Report--

This article shows you how to customize functions in Spark, which is concise and easy to understand, which will definitely brighten your eyes. I hope you can get something through the detailed introduction of this article.

UDF function

Def myUdf (id: Int) = s "tag$id" sqlContext.udf.register ("myUdf", myUdf (_: Int)) df.registerTempTable ("tb1") sqlContext.sql ("select myUdf (col1) from tb1") val myUdf = udf ((vec: Int) = > s "tag$vec") df.withColumn ("newId", myUdf ($"item"). Show () spark.udf.register ("myfunction", (str: String) = > {...})

SparkSessionUDF

Val spark: SparkSession = SparkSession.builder () .master ("local") .enableHiveSupport () .getOrCreate () import org.apache.spark.sql.functions._val userData = Array (("A", 16), ("B", 21), ("C", 14), ("D", 18)) val userDF = spark.createDataFrame (userData) .toDF ("name") "age") val strLen = udf ((str: String) = > str.length ()) userDF.withColumn ("name_len", strLen (col ("name")) userDF.show (false) the above is how to customize the function in Spark. Have you learned any knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.