Shulou(Shulou.com)06/01 Report--

This article introduces how to run Eclipse Ubuntu hadoop cluster service built by virtualbox on Windows. The content is very detailed. Interested friends can use it for reference. I hope it will be helpful to you.

Use eclipse to run programs in the ubuntu cluster on the windows side

Copy the hadoop of the master node of ubuntu to a path in windows, for example: e:\ Spring\ Hadoop\ hadoop\ hadoop-2.7.1

Eclipse installs the corresponding version of the hadoop plug-in, and sets the path to the hadoop directory in windows-preference-mapreduce

First: null pointer exception

Exception in thread "main" java.lang.NullPointerException

At java.lang.ProcessBuilder.start (ProcessBuilder.java:441)

At org.apache.hadoop.util.Shell.runCommand (Shell.java:445)

At org.apache.hadoop.util.Shell.run (Shell.java:418)

At org.apache.hadoop.util.Shell$ShellCommandExecutor.execute (Shell.java:650)

At org.apache.hadoop.util.Shell.execCommand (Shell.java:739)

At org.apache.hadoop.util.Shell.execCommand (Shell.java:722)

At org.apache.hadoop.fs.RawLocalFileSystem.setPermission (RawLocalFileSystem.java:633)

At org.apache.hadoop.fs.RawLocalFileSystem.mkdirs (RawLocalFileSystem.java:421)

At org.apache.hadoop.fs.FilterFileSystem.mkdirs (FilterFileSystem.java:281)

At org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir (JobSubmissionFiles.java:125)

At org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal (JobSubmitter.java:348)

At org.apache.hadoop.mapreduce.Job$10.run (Job.java:1285)

At org.apache.hadoop.mapreduce.Job$10.run (Job.java:1282)

At java.security.AccessController.doPrivileged (Native Method)

At javax.security.auth.Subject.doAs (Subject.java:396)

At org.apache.hadoop.security.UserGroupInformation.doAs (UserGroupInformation.java:1548)

At org.apache.hadoop.mapreduce.Job.submit (Job.java:1282)

At org.apache.hadoop.mapreduce.Job.waitForCompletion (Job.java:1303)

At WordCount.main (WordCount.java:89)

Source:

Reason: there is no permission to read and write files on the windows platform, so put the corresponding version of winutils.exe and hadoop.dll in hadoop\ bin and System32, add the environment variable HADOOP_HOME, the value is E:\ Spring\ Hadoop\ hadoop\ hadoop-2.7.1, add% HADOOP_HOME%\ bin in Path, restart eclipse (otherwise it will not take effect), the operation will not report this exception

The second kind: Permission denied

Caused by: org.apache.hadoop.ipc.RemoteException (org.apache.hadoop.security.AccessControlException): Permission denied: user=cuiguangfan, access=WRITE, inode= "/ tmp/hadoop-yarn/staging/cuiguangfan/.staging": linux1:supergroup:drwxr-xr-x

Refer to the following link (inspiration):

Http://www.huqiwen.com/2013/07/18/hdfs-permission-denied/

So, set System.setProperty ("HADOOP_USER_NAME", "linux1") when the program is running

Note that setting the system environment variable in windows does not work

The third kind: no job control

2014-05-28 17 container_1401177251807_0034_01_000001 and exit code 32 container_1401177251807_0034_01_000001 and exit code 1976 61 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: 1

Org.apache.hadoop.util.Shell$ExitCodeException: / bin/bash: line 0: fg: no job control

At org.apache.hadoop.util.Shell.runCommand (Shell.java:505)

At org.apache.hadoop.util.Shell.run (Shell.java:418)

At org.apache.hadoop.util.Shell$ShellCommandExecutor.execute (Shell.java:650)

At org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer (DefaultContainerExecutor.java:195)

At org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call (ContainerLaunch.java:300)

At org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call (ContainerLaunch.java:81)

At java.util.concurrent.FutureTask.run (FutureTask.java:262)

At java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1145)

At java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:615)

At java.lang.Thread.run (Thread.java:744)

Cause: caused by incorrect setting of system environment variables in hadoop

Solution: rewrite YARNRunner

Reference link: http://blog.csdn.net/fansy1990/article/details/27526167

Replace all% XX% with $XX, and replace\\ with /. For a more complete picture here, take a screenshot:

The fourth kind: Invalid host name: local host is: (unknown); destination host is

Reason: the port that should be running on the remote master is not fully set

Resolve:

Explicitly set the properties in hdfs-site.xml, mapred-site.xml, yarn-site.xml to the master node in the program (IP setting)

Screenshot:

Fifth: in 192.168.99.101 Application 8088 (view all datanode), click on a datanode node and cannot find it.

Reason: because linuxX-clound is clicked on, the system cannot find the IP address corresponding to linuxX-clound. Here, set the hosts file of windows and copy the hosts set in master or slaves.

That is:

192.168.99.101 linux0-cloud192.168.99.100 linux1-cloud192.168.99.102 linux2-cloud192.168.99.103 linux3-cloud

Thus, after modifying the hosts, we can change the set remote address in conf to linux0-cloud (master node)

Add: after resolving the third error, this error should disappear, if not, add the following to both Mapreduce-site.xml and Yarn-site.xml:

Mapreduce.application.classpath / home/linux1/hadoop/hadoop-2.7.1/etc/hadoop, / home/linux1/hadoop/hadoop-2.7.1/share/hadoop/common/*, / home/linux1/hadoop/hadoop-2.7.1/share/hadoop/common/lib/*, / home/linux1/hadoop/hadoop-2.7.1/share/hadoop/hdfs/*, / home/linux1/hadoop/hadoop-2.7.1/share/hadoop/hdfs/lib/* / home/linux1/hadoop/hadoop-2.7.1/share/hadoop/mapreduce/*, / home/linux1/hadoop/hadoop-2.7.1/share/hadoop/mapreduce/lib/*, / home/linux1/hadoop/hadoop-2.7.1/share/hadoop/yarn/*, / home/linux1/hadoop/hadoop-2.7.1/share/hadoop/yarn/lib/*

The sixth kind: java.lang.RuntimeException:java.lang.ClassNotFoundException

Reason: how the mapreduce program runs in hadoop: when the mapreduce framework runs Job, in order to enable each slave node to perform task tasks (that is, map and reduce functions), the resources needed to run the job, including job jar files, configuration files and calculated inputs, will be copied to a directory named job ID on HDFS, and there will be more copies of job jar when the job is submitted. To ensure that tasktracker can access the copy and execute the program when running task. The program does not run as jar, so it will not upload jar to HDFS, so that all nodes outside the node cannot find the map and reduce classes when performing task tasks, so there will be an error when running task.

Solution: temporarily generate jar package, set path

This is the end of the hadoop cluster service about how to run Eclipse on Windows using Ubuntu built by virtualbox. I hope the above content can be helpful to you and learn more. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.