Shulou(Shulou.com)06/03 Report--

Scenario description:

The login information of the user is stored in the table of mysql (version 5.1.73) in the way of record. Now we intend to use sqoop to extract and store the login record on hdfs, and use mapreduce program to count the login times of each user.

Environment description:

Deploy a fully distributed environment of hadoop2.6.5 with four virtual machines, and install mysql-server and sqoop on one of the hosts that act as namenode nodes to store and extract source data. In addition, use a different machine to write mapreduce programs and publish jar packages for hadoop to use.

The information of the four deployed hadoop machines and the roles played in the cluster are as follows:

IP hostname hadoop cluster plays the role 10.0.1.100 hadoop-test-nn NameNode,ResourceManager10.0.1.101 hadoop-test-snn SecondaryNameNode10.0.1.102 hadoop-test-dn1 DataNode,NodeManager10.0.1.103 hadoop-test-dn2 DataNode,NodeManager

Software involved:

1. Jdk1.7.0_45-hadoop and sqoop must be run.

2. Hadoop2.6.5-download address: http://apache.fayea.com/hadoop/common/hadoop-2.6.5/hadoop-2.6.5.tar.gz

3. Sqoop1.4.6-- download address: https://mirrors.tuna.tsinghua.edu.cn/apache/sqoop/1.4.6/ Note: please select sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz to download. Other versions may be incompatible with hadoop2.6.5, resulting in an exception when extracting data.

4. Mysql-connector-java-5.1.32-bin.jar, an executable JAR package for connecting to mysql, which is provided to sqoop to connect to the mysql database, which can be obtained from the unzipped directory in the tar package downloaded by https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.32.tar.gz.

Preparatory work:

1. Modify the hostname, modify the / etc/hosts file, and configure the hostname and IP mapping so that the hostnames can be resolved between the VM:

Add the following records to the / etc/hosts file for each VM:

10.0.1.100 hadoop-test-nn nn10.0.1.101 hadoop-test-snn snn10.0.1.102 hadoop-test-dn1 dn110.0.1.103 hadoop-test-dn2 dn2

two。 The hadoop user is established on each VM, and the future configuration work is carried out by using the hadoop user, and the ssh-keygen and ssh-copy-id commands are used to configure that the hadoop user of each VM can have SSH mutual trust access.

3. Install JDK and configure environment variables: extract the jar package of jdk for each VM to / usr/local/java, install JDK, and configure the JAVA_HOME variable, and add the following record in / etc/profile:

Export JAVA_HOME=/usr/local/java/jdk1.7.0_45export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jarexport PATH=$JAVA_HOME/bin:$PATH

The next section describes how to build a fully distributed environment in hadoop.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.