Shulou(Shulou.com)05/31 Report--

This article shows you how to solve all kinds of serialization errors in big data. The content is concise and easy to understand, which will definitely brighten your eyes. I hope you can get something through the detailed introduction of this article.

What kind of serialization will you see that causes an error?

Use client mode to submit spark jobs and observe the locally printed log. If there is an error log with words like Serializable, Serialize, and so on, congratulations, you will encounter an error caused by a serialization problem. Although it is an error report, serialization should be relatively simple and easy to deal with.

Serialization error reporting points to pay attention to:

If an external custom type variable is used in your operator function, then your custom type must be serializable at this point.

Final Teacher teacher = new Teacher ("SET"); studentsRDD.foreach (new VoidFunction () {public void call (Row row) throws Exception {String teacherName = teacher.getName ();. }}); public class Teacher implements Serializable {}

If you want to use a custom type as an element type of RDD, then the custom type must also be serializable

JavaPairRDD teacherRDDJavaPairRDD studentRDDstudentRDD.join (teacherRDD) public class Teacher implements Serializable {} public class Student implements Serializable {} the above content is how to solve the errors caused by various serialization in big data. Have you learned the knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.