Network Security Internet Technology Development Database Servers Mobile Phone Android Software Apple Software Computer Software News IT Information

In addition to Weibo, there is also WeChat

Please pay attention

WeChat public account

Shulou

One of big data: Hadoop installation

2025-01-24 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > Internet Technology >

Share

Shulou(Shulou.com)06/03 Report--

First, install Hadoop1. First extract the hadoop to the relevant folder

two。 After entering the extracted folder, modify the relevant files

3. Configure hadoop-env.shvi hadoop-env.sh:export= JAVA_HOME=/opt/inst/jdk181

4. Configure core-site.xmlvi core-site.xml fs.defaultFS hdfs://bigdata:9000 hadoop.tmp.dir / opt/hadoopdata hadoop.proxyuser.root.users * hadoop .proxyuser.root.groups *

5. Configure hdfs-site.xmlvi hdfs-site.xml dfs.replication 1

6. Configure cp mapred-site.xml.template mapred-site.xmlcp mapred-site.xml.template mapred-site.xml

7.vi mapred-site.xml mapreduce.framework.name yarn

8.vi yarn-site.xmlvi

9.vi / etc/profileexport HADOOP_HOME=/opt/bigdata/hadoop260export HADOOP_MAPRED_HOME=$HADOOP_HOMEexport HADOOP_COMMON_HOME=$HADOOP_HOMEexport HADOOP_HDFS_HOME=$HADOOP_HOMEexport YARN_HOME=$HADOOP_HOMEexport HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/nativeexport PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/binexport HADOOP_INSTALL=$HADOOP_HOME

10. Update and format source / etc/profilehdfs namenode-format

11. Start start-all.sh

Jps# View process

twelve。 Import linux into hadoop hdfs dfs-put / opt/a.txt / cm/hdfs dfs-ls / cm

Hadoop understands the three core 1.HDFS: Hadoop Distribute File System distributed file system

​ NameNode: master node, directory

​ DataNode: slave nodes, data

​ SecondaryNameNode: backup of the primary node

2.YARN:Yet Another Resource Negotiator Resource Management and scheduling system

​ dispatches memory resources and CPU computing power.

​ is dispatched through ResourceManager (only one)

The main functions of ​ ResourceManager are:

​ 1. Processing client request

​ 2. Monitoring NodeManager

​ 3. Start or monitor ApplicationMaster ()

​ 4. Allocation or scheduling of resources

​ NodeManager (multiple)

The main functions of ​ NodeManager are:

​ 1. Manage resources on a single node

​ 2. Processing commands from ResourceManager

​ 3. Processing commands from ApplicationMaster

3.MapReduce: distributed Computing Framework

​ operation

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.

Views: 0

*The comments in the above article only represent the author's personal views and do not represent the views and positions of this website. If you have more insights, please feel free to contribute and share.

Share To

Internet Technology

Wechat

© 2024 shulou.com SLNews company. All rights reserved.

12
Report