Shulou(Shulou.com)05/31 Report--

This article mainly introduces "which version is good for spark online". In daily operation, I believe many people have doubts about which version to use on spark online. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful to answer the question of "which version is good to use on spark online"! Next, please follow the editor to study!

Q1: which version should I use on the spark line?

It is recommended that you start with the minimum version of Spark 1.0.0, and the core API has been stable since Spark 1.0.0.

From a functional point of view, it is also very good to use the latest version of Spark 1.0.2. Spark 1.0.2 makes a lot of improvements on the basis of Spark 1.0.1.

Q2: I hope to talk about the recommendation system in detail.

The recommendation system is the main opportunity to show your talents in machine learning. Spark Asia-Pacific Research Institute wins the 100th public welfare lecture hall of big data era and will open at least three special topics in detail.

Q3: which is better to use online in these ways of yarn mesos standalone? Which version should I use online spark?

If no other big data clusters have been deployed before, only Spark is the computing framework in the cluster. It is recommended to use Standalone directly, which is concise and efficient, which helps to maximize the efficiency of cluster execution.

If Hadoop's MapReduce, Storm and other frameworks are running along with the Spark computing platform in the cluster, it is recommended to use mesos or yarn

Yarn is recommended in China, because Taobao has used Yarn on a large scale in the production environment, and Yarn has very Chinese materials.

Q4: does machine learning require deep math skills or is it possible for others to ok when they can run?

Spark's MLLib greatly simplifies the use of machine learning libraries. If it's simple to use, don't have math skills, just follow the official example and use it directly.

If the algorithm for replication is implemented, it requires mathematical skills, such as linear algebra, statistics, etc.

Q5: or do you want to learn more about machine learning algorithms?

From the point of view of practical application, the most important machine learning algorithm is collaborative filtering, and the recommendation system based on collaborative filtering is widely used in application systems.

Classification, clustering and linear regression are also very common and important.

Q6: if the current application is mainly structured data ORCALE and the language is PLSQL, is it difficult to convert to SPARKSQL, requiring complete code rewriting?

In the actual production environment, data and big data system exist in parallel, the database is generally directly responsible for online interaction, big data system is responsible for data analysis, real-time stream processing, interactive query and so on.

If you are proficient in using PLSQL, you can easily master Spark SQL

At this point, the study of "which version is better to use online with spark" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.