Shulou(Shulou.com)05/31 Report--

This article mainly explains "how to install the software needed by the Spark cluster". The content in the article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "how to install the software needed by the Spark cluster".

Software required for Spark clusters

Build a Hadoop cluster on the basis of the Spark cluster built from scratch in lectures 1 and 2. Here, we use Spark version 1.0.0 released on May 30, 2014, which is the latest version of Spark. To build a Spark cluster based on Spark 1.0.0, the software required is as follows:

1.Spark 1.0.0, the author uses spark-1.0.0-bin-hadoop1.tgz here

As shown in the following figure:

The author saved it in the location of the Master node as shown in the following figure:

two。 Download the Scala version corresponding to Spark 1.0.0. The official requirement is that Scala must be Scala 2.10.x:

What the author downloaded is "Scala 2.10.4". The specific official download address is http://www.scala-lang.org/download/2.10.4.html and saved on the Master node as:

Install each softwar

Install Scala

Open the terminal and create a new directory "/ usr/lib/scala", as shown in the following figure:

two。 Extract the Scala file, as shown in the following figure:

Move the extracted Scala to the "/ usr/lib/scala" you just created, as shown in the following figure

3. Modify the environment variable:

Enter the configuration file shown in the following figure:

Press "I" to enter INSERT mode and add the Scala environment authoring information to it, as shown in the following figure:

As you can see from the configuration file, we set "SCALA_HOME" and set the bin directory of Scala to PATH.

Press the esc key to return to normal mode, save and exit the configuration file:

Execute the following command for the changes to the configuration file to take effect:

4. Display the newly installed version of Scala in the terminal, as shown in the following figure

It is found that the version is "2.10.4", which is exactly what we expect.

When we enter the command "scala", we can go directly to the command line interface of Scala:

At this point, we enter the expression "9: 9":

At this point, we found that Scala correctly helped us to calculate the results.

At this point, we have completed the installation of Scala on Master

Since our Spark is running on Master, Slave1 and Slave2 machines, we need to install the same Scala on Slave1 and Slave2. Use the scp command to copy the Scala installation directory and "~ / .bashrc" to the same directory as Slave1 and Slave2. Of course, you can also install it manually on Slave1 and Slave2 as the Master node does.

After Scala is installed on Slave1, the test results are as follows:

After Scala is installed on Slave2, the test results are as follows:

So far, we have successfully deployed Scala on three machines: Master, Slave1 and Slave2.

Thank you for reading, the above is the content of "how to install the software needed by Spark cluster". After the study of this article, I believe you have a deeper understanding of how to install the software needed by Spark cluster, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.