Shulou(Shulou.com)06/03 Report--

Execute wordcount error java.lang.ClassNotFoundException under eclipse

17-08-29 07:52:54 INFO Configuration.deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS

17-08-29 07:52:54 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... Using builtin-java classes where applicable

17-08-29 07:52:55 INFO client.RMProxy: Connecting to ResourceManager at / 192.168.93.130:8032

17-08-29 07:52:56 WARN mapreduce.JobResourceUploader: No job jar file set. User classes may not be found. See Job or Job#setJar (String).

17-08-29 07:52:56 INFO input.FileInputFormat: Total input paths to process: 2

17-08-29 07:52:56 INFO mapreduce.JobSubmitter: number of splits:2

17-08-29 07:52:56 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address

17-08-29 07:52:56 INFO Configuration.deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS

17-08-29 07:52:56 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1503848790903_0013

17-08-29 07:52:56 INFO mapred.YARNRunner: Job jar is not present. Not adding any jar to the list of resources.

17-08-29 07:52:56 INFO impl.YarnClientImpl: Submitted application application_1503848790903_0013

07:52:56 on 17-08-29 INFO mapreduce.Job: The url to track the job: http://master:18088/proxy/application_1503848790903_0013/

17-08-29 07:52:56 INFO mapreduce.Job: Running job: job_1503848790903_0013

17-08-29 07:53:04 INFO mapreduce.Job: Job job_1503848790903_0013 running in uber mode: false

17-08-29 07:53:04 INFO mapreduce.Job: map 0 reduce 0

17-08-29 07:53:09 INFO mapreduce.Job: Task Id: attempt_1503848790903_0013_m_000001_0, Status: FAILED

Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.asiainfo.demo.count.WorldTest$MyWorld not found

At org.apache.hadoop.conf.Configuration.getClass (Configuration.java:2195)

At org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass (JobContextImpl.java:186)

At org.apache.hadoop.mapred.MapTask.runNewMapper (MapTask.java:745)

At org.apache.hadoop.mapred.MapTask.run (MapTask.java:341)

At org.apache.hadoop.mapred.YarnChild$2.run (YarnChild.java:164)

At java.security.AccessController.doPrivileged (Native Method)

At javax.security.auth.Subject.doAs (Subject.java:422)

At org.apache.hadoop.security.UserGroupInformation.doAs (UserGroupInformation.java:1698)

At org.apache.hadoop.mapred.YarnChild.main (YarnChild.java:158)

Caused by: java.lang.ClassNotFoundException: Class com.asiainfo.demo.count.WorldTest$MyWorld not found

At org.apache.hadoop.conf.Configuration.getClassByName (Configuration.java:2101)

At org.apache.hadoop.conf.Configuration.getClass (Configuration.java:2193)

... 8 more

17-08-29 07:53:09 INFO mapreduce.Job: Task Id: attempt_1503848790903_0013_m_000000_0, Status: FAILED

Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.asiainfo.demo.count.WorldTest$MyWorld not found

At org.apache.hadoop.conf.Configuration.getClass (Configuration.java:2195)

At org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass (JobContextImpl.java:186)

At org.apache.hadoop.mapred.MapTask.runNewMapper (MapTask.java:745)

At org.apache.hadoop.mapred.MapTask.run (MapTask.java:341)

At org.apache.hadoop.mapred.YarnChild$2.run (YarnChild.java:164)

At java.security.AccessController.doPrivileged (Native Method)

At javax.security.auth.Subject.doAs (Subject.java:422)

At org.apache.hadoop.security.UserGroupInformation.doAs (UserGroupInformation.java:1698)

At org.apache.hadoop.mapred.YarnChild.main (YarnChild.java:158)

Caused by: java.lang.ClassNotFoundException: Class com.asiainfo.demo.count.WorldTest$MyWorld not found

At org.apache.hadoop.conf.Configuration.getClassByName (Configuration.java:2101)

At org.apache.hadoop.conf.Configuration.getClass (Configuration.java:2193)

... 8 more

17-08-29 07:53:14 INFO mapreduce.Job: Task Id: attempt_1503848790903_0013_m_000001_1, Status: FAILED

Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.asiainfo.demo.count.WorldTest$MyWorld not found

At org.apache.hadoop.conf.Configuration.getClass (Configuration.java:2195)

At org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass (JobContextImpl.java:186)

At org.apache.hadoop.mapred.MapTask.runNewMapper (MapTask.java:745)

At org.apache.hadoop.mapred.MapTask.run (MapTask.java:341)

At org.apache.hadoop.mapred.YarnChild$2.run (YarnChild.java:164)

At java.security.AccessController.doPrivileged (Native Method)

At javax.security.auth.Subject.doAs (Subject.java:422)

At org.apache.hadoop.security.UserGroupInformation.doAs (UserGroupInformation.java:1698)

At org.apache.hadoop.mapred.YarnChild.main (YarnChild.java:158)

Caused by: java.lang.ClassNotFoundException: Class com.asiainfo.demo.count.WorldTest$MyWorld not found

At org.apache.hadoop.conf.Configuration.getClassByName (Configuration.java:2101)

At org.apache.hadoop.conf.Configuration.getClass (Configuration.java:2193)

... 8 more

17-08-29 07:53:14 INFO mapreduce.Job: Task Id: attempt_1503848790903_0013_m_000000_1, Status: FAILED

Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.asiainfo.demo.count.WorldTest$MyWorld not found

At org.apache.hadoop.conf.Configuration.getClass (Configuration.java:2195)

At org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass (JobContextImpl.java:186)

At org.apache.hadoop.mapred.MapTask.runNewMapper (MapTask.java:745)

At org.apache.hadoop.mapred.MapTask.run (MapTask.java:341)

At org.apache.hadoop.mapred.YarnChild$2.run (YarnChild.java:164)

At java.security.AccessController.doPrivileged (Native Method)

At javax.security.auth.Subject.doAs (Subject.java:422)

At org.apache.hadoop.security.UserGroupInformation.doAs (UserGroupInformation.java:1698)

At org.apache.hadoop.mapred.YarnChild.main (YarnChild.java:158)

Caused by: java.lang.ClassNotFoundException: Class com.asiainfo.demo.count.WorldTest$MyWorld not found

At org.apache.hadoop.conf.Configuration.getClassByName (Configuration.java:2101)

At org.apache.hadoop.conf.Configuration.getClass (Configuration.java:2193)

... 8 more

17-08-29 07:53:19 INFO mapreduce.Job: Task Id: attempt_1503848790903_0013_m_000001_2, Status: FAILED

Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.asiainfo.demo.count.WorldTest$MyWorld not found

At org.apache.hadoop.conf.Configuration.getClass (Configuration.java:2195)

At org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass (JobContextImpl.java:186)

At org.apache.hadoop.mapred.MapTask.runNewMapper (MapTask.java:745)

At org.apache.hadoop.mapred.MapTask.run (MapTask.java:341)

At org.apache.hadoop.mapred.YarnChild$2.run (YarnChild.java:164)

At java.security.AccessController.doPrivileged (Native Method)

At javax.security.auth.Subject.doAs (Subject.java:422)

At org.apache.hadoop.security.UserGroupInformation.doAs (UserGroupInformation.java:1698)

At org.apache.hadoop.mapred.YarnChild.main (YarnChild.java:158)

Caused by: java.lang.ClassNotFoundException: Class com.asiainfo.demo.count.WorldTest$MyWorld not found

At org.apache.hadoop.conf.Configuration.getClassByName (Configuration.java:2101)

At org.apache.hadoop.conf.Configuration.getClass (Configuration.java:2193)

... 8 more

17-08-29 07:53:20 INFO mapreduce.Job: Task Id: attempt_1503848790903_0013_m_000000_2, Status: FAILED

Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.asiainfo.demo.count.WorldTest$MyWorld not found

At org.apache.hadoop.conf.Configuration.getClass (Configuration.java:2195)

At org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass (JobContextImpl.java:186)

At org.apache.hadoop.mapred.MapTask.runNewMapper (MapTask.java:745)

At org.apache.hadoop.mapred.MapTask.run (MapTask.java:341)

At org.apache.hadoop.mapred.YarnChild$2.run (YarnChild.java:164)

At java.security.AccessController.doPrivileged (Native Method)

At javax.security.auth.Subject.doAs (Subject.java:422)

At org.apache.hadoop.security.UserGroupInformation.doAs (UserGroupInformation.java:1698)

At org.apache.hadoop.mapred.YarnChild.main (YarnChild.java:158)

Caused by: java.lang.ClassNotFoundException: Class com.asiainfo.demo.count.WorldTest$MyWorld not found

At org.apache.hadoop.conf.Configuration.getClassByName (Configuration.java:2101)

At org.apache.hadoop.conf.Configuration.getClass (Configuration.java:2193)

... 8 more

Container killed by the ApplicationMaster.

Container killed on request. Exit code is 143

Container exited with a non-zero exit code 143

17-08-29 07:53:25 INFO mapreduce.Job: map 100 reduce 100%

17-08-29 07:53:25 INFO mapreduce.Job: Job job_1503848790903_0013 failed with state FAILED due to: Task failed task_1503848790903_0013_m_000001

Job failed as tasks failed. FailedMaps:1 failedReduces:0

17-08-29 07:53:25 INFO mapreduce.Job: Counters: 13

Job Counters

Failed map tasks=8

Killed reduce tasks=1

Launched map tasks=8

Other local map tasks=6

Data-local map tasks=2

Total time spent by all maps in occupied slots (ms) = 28393

Total time spent by all reduces in occupied slots (ms) = 0

Total time spent by all map tasks (ms) = 28393

Total time spent by all reduce tasks (ms) = 0

Total vcore-milliseconds taken by all map tasks=28393

Total vcore-milliseconds taken by all reduce tasks=0

Total megabyte-milliseconds taken by all map tasks=29074432

Total megabyte-milliseconds taken by all reduce tasks=0

one

The solution is to import your project into a jar package, and then set conf.set ("mapred.jar", "/ eclipse-jee/eclipse/demo.jar") on the Configuration object.

Or set the jar package path job.setJar ("/ eclipse-jee/eclipse/demo.jar") executed by the Job object

Here's an example I wrote.

Configuration conf = new Configuration ()

Conf.set ("fs.default.name", "hdfs://192.168.93.130:9000")

Conf.set ("mapred.job.tracker", "9001")

Conf.set ("hadoop.job.user", "root")

Conf.set ("mapreduce.framework.name", "yarn")

Conf.set ("yarn.resourcemanager.address", "192.168.93.130pur8032")

Conf.set ("mapreduce.jobtracker.address", "192.168.93.130VR 9001")

Conf.set ("yarn.resourcemanager.hostname", "192.168.93.130")

Conf.set ("yarn.resourcemanager.admin.address", "192.168.93.130pur8033")

Conf.set ("yarn.resourcemanager.address", "192.168.93.130pur8032")

Conf.set ("yarn.resourcemanager.resource-tracker.address", "192.168.93.130pur8036")

Conf.set ("yarn.resourcemanager.scheduler.address", "192.168.93.130pur8030")

Conf.set ("mapred.jar", "/ eclipse-jee/eclipse/demo.jar")

String [] otherArgs = new GenericOptionsParser (conf, args) .getRemainingArgs

If (otherArgs.length! = 2) {

System.out.println ("stage not exis input output")

System.exit (2)

}

Job job = new Job (conf, "job" + Calendar.getInstance () .getTimeInMillis ())

/ / job.setJar ("/ eclipse-jee/eclipse/demo.jar")

Job.setMapperClass (MyWorld.class)

Job.setCombinerClass (MyReduce.class)

Job.setReducerClass (MyReduce.class)

Job.setOutputKeyClass (Text.class)

Job.setOutputValueClass (IntWritable.class)

FileInputFormat.addInputPath (job, new Path (otherArgs [0]))

FileOutputFormat.setOutputPath (job, new Path (otherArgs [1]))

System.out.println (job.waitForCompletion (true)? 0: 1)

Just re-execute your code and the effect is as follows

You can find the execution history http://master:19888/jobhistory/app in the jobhistory address. [the mr-jobhistory-daemon.sh service should be enabled on the master node first]

You can also view http://192.168.93.130:18088/cluster/apps/FINISHED on yarn

17-09-02 23:17:24 INFO Configuration.deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS

17-09-02 23:17:24 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... Using builtin-java classes where applicable

17-09-02 23:17:25 INFO client.RMProxy: Connecting to ResourceManager at / 192.168.93.130:8032

17-09-02 23:17:27 INFO input.FileInputFormat: Total input paths to process: 2

17-09-02 23:17:27 INFO mapreduce.JobSubmitter: number of splits:2

17-09-02 23:17:27 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address

17-09-02 23:17:27 INFO Configuration.deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS

17-09-02 23:17:27 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1504364765355_0001

17-09-02 23:17:28 INFO impl.YarnClientImpl: Submitted application application_1504364765355_0001

23:17:28 on 17-09-02 INFO mapreduce.Job: The url to track the job: http://master:18088/proxy/application_1504364765355_0001/

17-09-02 23:17:28 INFO mapreduce.Job: Running job: job_1504364765355_0001

17-09-02 23:17:37 INFO mapreduce.Job: Job job_1504364765355_0001 running in uber mode: false

17-09-02 23:17:37 INFO mapreduce.Job: map 0 reduce 0

17-09-02 23:17:47 INFO mapreduce.Job: map 100% reduce 0

17-09-02 23:17:53 INFO mapreduce.Job: map 100 reduce 100%

17-09-02 23:17:53 INFO mapreduce.Job: Job job_1504364765355_0001 completed successfully

0

17-09-02 23:17:54 INFO mapreduce.Job: Counters: 49

File System Counters

FILE: Number of bytes read=247

FILE: Number of bytes written=357769

FILE: Number of read operations=0

FILE: Number of large read operations=0

FILE: Number of write operations=0

HDFS: Number of bytes read=383

HDFS: Number of bytes written=157

HDFS: Number of read operations=9

HDFS: Number of large read operations=0

HDFS: Number of write operations=2

Job Counters

Launched map tasks=2

Launched reduce tasks=1

Data-local map tasks=2

Total time spent by all maps in occupied slots (ms) = 14625

Total time spent by all reduces in occupied slots (ms) = 3482

Total time spent by all map tasks (ms) = 14625

Total time spent by all reduce tasks (ms) = 3482

Total vcore-milliseconds taken by all map tasks=14625

Total vcore-milliseconds taken by all reduce tasks=3482

Total megabyte-milliseconds taken by all map tasks=14976000

Total megabyte-milliseconds taken by all reduce tasks=3565568

Map-Reduce Framework

Map input records=13

Map output records=30

Map output bytes=293

Map output materialized bytes=253

Input split bytes=209

Combine input records=30

Combine output records=21

Reduce input groups=21

Reduce shuffle bytes=253

Reduce input records=21

Reduce output records=21

Spilled Records=42

Shuffled Maps = 2

Failed Shuffles=0

Merged Map outputs=2

GC time elapsed (ms) = 386

CPU time spent (ms) = 2480

Physical memory (bytes) snapshot=620863488

Virtual memory (bytes) snapshot=1672093696

Total committed heap usage (bytes) = 497025024

Shuffle Errors

BAD_ID=0

CONNECTION=0

IO_ERROR=0

WRONG_LENGTH=0

WRONG_MAP=0

WRONG_REDUCE=0

File Input Format Counters

Bytes Read=174

File Output Format Counters

Bytes Written=157

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.