Network Security Internet Technology Development Database Servers Mobile Phone Android Software Apple Software Computer Software News IT Information

In addition to Weibo, there is also WeChat

Please pay attention

WeChat public account

Shulou

China Azure uses the Spark function in HDinsight

2025-04-04 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > Internet Technology >

Share

Shulou(Shulou.com)06/03 Report--

The international version of Azure can support Spark applications normally, but when we are using the Chinese version of Azure, we find an introduction to the Spark service on the official website of the Chinese version of Azure, but when we actually use it, we find that it cannot be created. Only Storm,HBase and Hadoop. As shown in the figure:

So the question is, does HDInsight of the Chinese version of Azure support Spark?

Of course the answer is yes, but you need to customize it:

First, we need to create a PAAS layer SQL Database to store the metadata of Hive and Oozie.

Create a HADOOP cluster based on the PAAS tier:

HDInsight 3.1 clusters will install Spark 1.0.2

HDInsight 3.2 clusters will install Spark 1.2.0

Configure cluster nodes and virtual machines:

Configure access users, and check configure Oozie Cloud Storage:

Configure Oozie to store the SQL Database database:

Configure storage:

Use a script to install Spark:

Insert Spark configuration steps:

Https://hdiconfigactions.blob.core.windows.net/sparkconfigactionv03/spark-installer-v03.ps1 . Select "header node only" for node type

Finally, click the "create" button.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.

Views: 0

*The comments in the above article only represent the author's personal views and do not represent the views and positions of this website. If you have more insights, please feel free to contribute and share.

Share To

Internet Technology

Wechat

© 2024 shulou.com SLNews company. All rights reserved.

12
Report