Shulou(Shulou.com)06/02 Report--

This article introduces the relevant knowledge of "Hadoop windows local environment installation tutorial". In the actual case operation process, many people will encounter such difficulties. Next, let Xiaobian lead you to learn how to handle these situations! I hope you can read carefully and learn something!

Sometimes you need to debug hadoop programs, if you want to transfer to linux, very inconvenient. We can build a standalone version of windows hadoop environment to test our code.

1. Download the required documents

JDK download address, jdk1.8 download

Hadoop download, hadoop download, go in and find a version and click Binary download to download locally.

Windows Hadoop environment compiler tools, winutils download, download and extract

II. Installation documents

jdk installation, omitted

hadoop installation, because the downloaded hadoop file is not installed, we only need to extract it to extract the directory D:\dev-tools\hadoop\hadoop-2.8.3

III. Configuration of environmental variables

jdk, omitted

IV. Configure hadoop files

Configure the core-site.xml file

hadoop.tmp.dir /D:/data/hadoop fs.default.name hdfs://localhost:9000

2. Configure mapred-site.xml

mapreduce.framework.name yarn

Configure hdfs-site.xml

dfs.replication 1

Configure yarn-site.xml

yarn.nodemanager.aux-services mapreduce_shuffle yarn.nodemanager.aux-services.mapreduce.shuffle.class org.apache.hadoop.mapred.ShuffleHandler

Find the hadoop-env.cmd file and open it, adding the following configuration.

set JAVA_HOME=%JAVA_HOME%set HADOOP_HOME=D:\dev-tools\hadoop\hadoop-2.8.3rem %HADOOP_HOME%\bin Add to runtime path set PATH=%PATH%;%HADOOP_HOME%\bin

V. Replacement of documents

Extract the winutils-master file we downloaded earlier, find the bin directory that matches your version, copy winutils.exe and hadoop.dll to the bin directory

VI. Operation

1. Run cmd window, execute hdfs name-format, format hdfs file system.

2. Run cmd window, switch to sbin directory of hadoop, execute start-all.cmd to start hadoop, including yarn/hdfs

VII. Testing

Open the following page to view hdfs status

In hadoop 2.x, the admin interface is http://localhost:50070

In hadoop 3.0, it has been changed to http://localhost:9870

Open the following page to view the status of Yarn

http://localhost:8088/cluster

The configured hadoop 2.8.3 has been uploaded to the network disk. You can download it yourself and configure the environment variables.

The content of "Hadoop Windows Local Environment Installation Tutorial" is introduced here. Thank you for reading it. If you want to know more about industry-related knowledge, you can pay attention to the website. Xiaobian will output more high-quality practical articles for everyone!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.