Shulou(Shulou.com)05/31 Report--

This article introduces the knowledge of "how eclipse starts Debug mode under Windows or Linux". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

The Debug mode of MR program in other words is the running mode of MR program. There are three running modes of MR program: local mode (local) and cluster mode (cluster).

In Linux environment:

1. Local mode: right click on eclipse to Debug directly.

2. Cluster mode:

2.1.copy the hadoop configuration file: core-site.xml/hdfs-site.xml/mapreduce-site.xml/yarn.xml to the project's src directory, so that the eclipse will be submitted through the cluster

2.2.When submitting, you need to type the jar package of the project to the directory of the project.

At the same time, you need to set the environment variable: conf.set ("mapreduce.job.jar", "wordcount.jar")

In Windows environment:

First of all, Windows is really not suitable to modulate MR programs, not only the version incompatibility problem, but also various environmental errors caused by the modulation are also puzzling.

1. Local mode:

1.1. first, you need to download the windows version of the hadoop program, and you need to specify HADOOP_HOME under the PATH path

1.2.Now you can use DEBUG directly by right-clicking on eclipse (because I failed to run the local mode in WINDOWS and encountered the following exception:)

2015-06-03 00 WARN 52V 10649 [main] util.NativeCodeLoader (NativeCodeLoader.java: (62))-Unable to load native-hadoop library for your platform... Using builtin-java classes where applicable2015-06-03 00 using builtin-java classes where applicable2015 52 main 15730 INFO [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated (1009))-session.id is deprecated. Instead, use dfs.metrics.session-id2015-06-03 00 INFO 15730 INFO [main] jvm.JvmMetrics (JvmMetrics.java:init (76))-Initializing JVM Metrics with processName=JobTracker SessionId=Exception in thread "main" org.apache.hadoop.util.Shell$ExitCodeException: at org.apache.hadoop.util.Shell.runCommand (Shell.java:505) at org.apache.hadoop.util.Shell.run (Shell.java:418) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute (Shell.java:650) at org.apache.hadoop.util.Shell.execCommand (Shell.java:739) at org.apache.hadoop.util.Shell.execCommand (Shell.java:722 ) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission (RawLocalFileSystem.java:633) at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs (RawLocalFileSystem.java:421) at org.apache.hadoop.fs.FilterFileSystem.mkdirs (FilterFileSystem.java:281) at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir (JobSubmissionFiles.java:125) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal (JobSubmitter.java:348) at org.apache.hadoop.mapreduce.Job$10.run (Job.java:1285) at org.apache.hadoop.mapreduce.Job$10.run (Job.java:1282) at java.security.AccessController.doPrivileged (Native Method) at javax.security.auth.Subject.doAs (Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs (UserGroupInformation.java:1556) at org.apache.hadoop.mapreduce.Job.submit (Job.java:1282) at org.apache.hadoop.mapreduce.Job.waitForCompletion (Job. Java:1303) at com.itheima.hadoop.runner.WordCountRunner.main (WordCountRunner.java:38)

2. Cluster mode

2.1. copy the configuration file core-site.xml/hdfs-site.xml/mapreduce-site.xml/yarn.xml of hadoop to the src directory of the project

2.2.Modify the Hadoop source code: YARNRunner.java file. YARNRunner contains a large number of configurations related to the Linux environment, which needs to be replaced with the Windows environment configuration and recompiled. This is also the reason why it is not recommended to lower the WINDOWS mode, because it is very troublesome to modify.

That's all for "how eclipse starts Debug mode under Windows or Linux". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.