Shulou(Shulou.com)05/31 Report--

This article mainly introduces "how to compile Oozie-4.1.0 and hadoop-2.7.1". In daily operation, I believe many people have doubts about how to compile Oozie-4.1.0 and hadoop-2.7.1. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful to answer the question of "how to compile Oozie-4.1.0 and hadoop-2.7.1"! Next, please follow the editor to study!

I. Environment

Maven-3.3.0

Hadoop-2.7.1

II. Compilation

[root@hftclclw0001 opt] # pwd/opt [root@hftclclw0001 opt] # wget http://apache.mirrors.pair.com/oozie/4.1.0/oozie-4.1.0.tar.gz[root@hftclclw0001 opt] # tar-zxvf oozie-4.1.0.tar.gz [root@hftclclw0001 opt] # cd oozie-4.1.0# default # sqoop.version=1.4.3 # hive.version=0.13.1 = > modified to something else Compilation error # hbase.version=0.94.2 = > modified to something else Compilation error # pig.version=0.12.1 # hadoop.version=2.3.0 = > latest version 2.3.0 but supports 2.7.1#tomcat.version=6.0.43 [root@hftclclw0001 opt] #. / bin/mkdistro.sh-DskipTests-Phadoop-2-Dsqoop.version=1.4.6.INFO] BUILD SUCCESS [INFO] -[INFO] Total time: 07:25 min [INFO] Finished at: 2016-06-19T12:46:07+00:00 [INFO] Final Memory: 128M/1178M [INFO]- -- Oozie distro created DATE [June 19, 2016-12:38:39GMT] VC-REV [unavailable], available at [/ opt/oozie-4.1.0/distro/target]

III. Configuration

[root@hftclclw0001 opt] # pwd/opt [root@hftclclw0001 opt] # mkdir Oozie [root@hftclclw0001 opt] # cd Oozie [root@hftclclw0001 Oozie] # pwd/opt/Oozie [root@hftclclw0001 Oozie] # cp.. / oozie-4.1.0/distro/target/oozie-4.1.0-distro.tar.gz. / [root@hftclclw0001 Oozie] # tar-zxvf oozie-4.1.0-distro.tar.gz [root@hftclclw0001 Oozie] # cd oozie-4.1.0 [root@hftclclw0001 Oozie-4.1.0] # pwd/opt/Oozie/oozie-4.1.0 [root@hftclclw0001 oozie-4.1.0] # mkdir libext [root@hftclclw0001 oozie-4.1.0] # cp / opt/oozie-4.1.0/hadooplibs/hadoop-2/target/hadooplibs/hadooplib-2.3.0.oozie-4.1.0/*. / libext [root@hftclclw0001 oozie-4.1.0] # cd libext [root@hftclclw0001 libext] # curl-O http: / / archive.cloudera.com/gplextras/misc/ext-2.2.zip download mysql driver and put it into libext Because using mysql as the Metabase The default is Derby [root@hftclclw0001 libext] # lltotal 26452. Murray RW-1 root root 848401 Jun 19 13:41 mysql-connector-java-5.1.25-bin.jar... [root@hftclclw0001 libext] # cd. [root@hftclclw0001 oozie-4.1.0] # pwd/opt/Oozie/oozie-4.1.0 [root@hftclclw0001 oozie-4.1.0] #. / bin/oozie-setup.sh prepare-war [root@hftclclw0001 oozie- 4.1.0] #. / bin/oozie-setup.sh sharelib create-fs hdfs://localhost:9000 create Oozie database [root@hftclclw0001 oozie-4.1.0] # mysql-uroot-pmysql > CREATE DATABASE OOZIEDB Mysql > GRANT ALL PRIVILEGES ON OOZIEDB.* TO oozie IDENTIFIED BY "oozie"; mysql > FLUSH PRIVILEGES Configure conf/oozie-site.xmloozie.service.JPAService.jdbc.driveroozie.service.JPAService.jdbc.urloozie.service.JPAService.jdbc.usernameoozie.service.JPAService.jdbc.password [root@hftclclw0001 oozie-4.1.0] #. / bin/ooziedb.sh create db-run configure etc/hadoop/core-site.xml, configure proxyuser hadoop.proxyuser.$USER.hosts * hadoop.proxyuser.$USER.groups * / value > $USER of oozie to replace with oozie service user, or oozie Or root et al. [root@hftclclw0001 oozie-4.1.0] #. / oozied.sh start

IV. Examples

Job.properties

NameNode=hdfs://nameservice1#nameNode=hdfs://nameservice1 = > HA#nameNode=hdfs://$ {namenode}: 8020 = > single namenodejobTracker=dapdevhmn001.qa.webex.com:8032#jobTracker=rm1,rm2 = = > HA#jobTracker (yarn.resourcemanager.address) = 10.224.243.124virtual 8032queueNamenames defaultexintesRootpaths examplestrueozie.wf.application.pathways ${nameNode} / user/$ {user.name} / ${examplesRoot} / apps/map-reduceoutputDir=map-reduce

Workflow.xml

${jobTracker} ${nameNode} mapred.job.queue.name ${queueName} mapred.mapper.class org. Apache.oozie.example.SampleMapper mapred.reducer.class org.apache.oozie.example.SampleReducer mapred.map.tasks 1 Mapred.input.dir / user/$ {wf:user ()} / ${examplesRoot} / input-data/text mapred.output.dir / user/$ {wf:user ()} / ${examplesRoot} / output-data/$ {outputDir} Map/Reduce failed Error message [${wf:errorMessage (wf:lastErrorNode ())}]

Lib/oozie-examples-4.1.0.jar

Hadoop fs-mkdir-p / user/root/examples/apps/map-reduce

Hadoop fs-put. / job.properties / user/root/examples/apps/map-reduce

Hadoop fs-put. / workflow.xml / user/root/examples/apps/map-reduce

Hadoop fs-put. / lib/oozie-examples-4.1.0.jar / user/root/examples/apps/map-reduce

Job.properties = = > need a copy not only in HDFS, but also locally. Execute command-config is to point to a local file

Oozie job-oozie ${OOZIE_URL}-config job.properties-run

Oozie job-oozie ${OOZIE_URL}-info ${oozie_id}

# oozie job-oozie ${OOZIE_URL}-info 0000001-170206083712434-oozie-oozi-W

Oozie job-oozie ${OOZIE_URL}-log ${oozie_id}

# oozie job-oozie ${OOZIE_URL}-log 0000001-170206083712434-oozie-oozi-W

5. Distcp

Job.properties

NameNode=hdfs://$ {sourceNameNode}: 8020destNameNodeparts hdfspaste ${destNameNode}: 8020jobTrackerparts ${RM}: 8032examples yarn. Resourcemanager.addressflowers ${RM}: 8032queueNameNodefaultexsRootbacks examplesoozie.use.system.libpathways trueoozie.wf.application.pathways ${nameNode} / user/$ {user.name} / ${examplesRoot} / apps/distcp_2outputDir=distcp

Workflow.xml

${jobTracker} ${nameNode} mapred.job.queue.name ${queueName} ${nameNode} / user/$ {wf:user ()} / ${examplesRoot} / input -data/text/data.txt ${destNameNode} / tmp/data.txt DistCP failed Error message [${wf:errorMessage (wf:lastErrorNode ())}] so far The study on "how to compile Oozie-4.1.0 and hadoop-2.7.1" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.