Shulou(Shulou.com)06/03 Report--

Let's talk about the environment first. there are two clusters, one is new and the other is old. We are going to debug the new one and then turn off the old one.

New: Cloudera Express 5.6.0cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc

Old: Cloudera Express 5.0.5 cdccccmcccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc

A problem is found in the process of building a new cluster. Execute the following command to create an index on the lzo file. Job cannot be submitted to the specified queue in the new cluster. The same command is normal in the old cluster:

Hadoop jar hadoop-lzo.jar com.hadoop.compression.lzo.DistributedLzoIndexer-D mapreduce.job.queuename=copy / data/lzodata/

The configuration difference between the two clusters is that the configuration of the old cluster lzo is manual, while the new cluster is configured with the parcel package of GPLEXTRAS.

The new cluster configured with GPLEXTRAS defaults to loading lzo's jar package into the hadoop classpath path, resulting in DistributedLzoIndexer not being submitted to the correct queue.

Because in the new version of hadoop-lzo.jar

Com.hadoop.compression.lzo.DistributedLzoIndexer modified the way to get Configuration

Configuration conf = new Configuration (); Job job = new Job (conf)

In the old version,

Job job = new Job (getConf ())

The source code is modified and then re-uploaded to the server that submitted the job, which only works when the job is submitted.

Modify the location of the script. Please make a backup before modifying it:

/ opt/cloudera/parcels/CDH-5.6.0-1.cdh6.6.0.p0.45/lib/hadoop/bin

Modify export CLASSPATH=/opt/cloudera/userlib/hadoop-lzo.jar:$CLASSPATH in script

The / opt/cloudera/userlib/hadoop-lzo.jar above is the path you want to load

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.