In addition to Weibo, there is also WeChat
Please pay attention
WeChat public account
Shulou
2025-02-23 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > Development >
Share
Shulou(Shulou.com)06/02 Report--
Today, I would like to talk to you about how to learn spark. Many people may not know much about it. In order to make you understand better, the editor has summarized the following for you. I hope you can get something from this article.
Now when you think of cloud computing big data, you will think of spark. How to learn spark? First of all, we need to take a brief look at it: Spark is an open source cluster computing environment similar to Hadoop, even more superior in some workloads. Spark uses a unified technology stack to solve all the core problems of cloud computing big data, such as flow processing, graph technology, machine learning, NoSQL query and so on. It has a very perfect ecosystem, which directly established its dominant position in the field of unified cloud computing big data. Here we share how to learn spark!
Stage 1: master the Scala language
The Spark framework is written in Scala. So if you want to be a master of Spark, you must read the source code of Spark, so mastering Scala is a must. Although today's Spark can use multi-language Java, Python, etc., for application development, the fastest and best development API is still and will always be Scala API, so you must master Scala to write complex and high-performance Spark distributed programs; in particular, you must be proficient in Scala's trait, apply, functional programming, generics, inverter and covariant, etc.
Stage 2: proficient in the corresponding API
For example, RDD-oriented development model, mastering the use of various transformation and action functions; wide and narrow dependencies in Spark and lineage mechanism; mastering the computing flow of RDD, such as the division of Stage, the basic process of submitting Spark applications to the cluster and the basic working principle of Worker nodes, etc.
Phase 3: go deep into the Spark kernel
This stage is mainly through the source code study of the Spark framework to go deep into the Spark kernel: to master the task submission process of Spark and the task scheduling of the Spark cluster through the source code; especially to be proficient in the details of every step of the work within the DAGScheduler, TaskScheduler and Worker nodes.
Stage 4: master the use of the core framework
When we use Spark, we spend most of our time using the framework on it. For example, Spark Streaming is an excellent real-time stream processing framework, to master its DStream, transformation and checkpoint; version 1.0.0 of Spark launched Spark SQL on the basis of Shark, and the efficiency of offline statistical analysis has been significantly improved, which needs to be mastered; it is necessary to master the principle and usage of Spark machine learning and GraphX.
Stage 5: do actual combat projects
Through a complete and representative Spark project to run through all aspects of Spark, including the architecture design of the project, analysis of the technology used, development and implementation, operation and maintenance, etc., a complete grasp of each stage and details, so that you can calmly face the vast majority of Spark projects in the future.
After reading the above, do you have any further understanding of how to learn spark? If you want to know more knowledge or related content, please follow the industry information channel, thank you for your support.
Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.
Views: 0
*The comments in the above article only represent the author's personal views and do not represent the views and positions of this website. If you have more insights, please feel free to contribute and share.
Continue with the installation of the previous hadoop.First, install zookooper1. Decompress zookoope
"Every 5-10 years, there's a rare product, a really special, very unusual product that's the most un
© 2024 shulou.com SLNews company. All rights reserved.