Shulou(Shulou.com)06/03 Report--

Big data is defined as a large amount of unstructured data. The quantity should be large and unstructured.

Hadoop is composed of three parts, 1.hdfs hadoop distributed file system. 2.MapReduce distributed computing. 3.hive distributed storage.

Operating system: centos6.5 64

Environment building: 1. Install Hadoop,2. Install mysql, 3. Install hive,4, install jdk

Hadoop installation:

Download Hadoop1.0.4,hadoop-1.0.4.tar.gz.

Wget http://archive.apache.org/dist/hadoop/core/hadoop-1.0.4/hadoop-1.0.4.tar.gz

Decompress:

Tar xzvf hadoop-1.0.4.tar.gz-C / usr

Modify 3 configuration files

Cd / usr/hadoop-1.0.4/conf/

Vi core-site.xml

Fs.default.name

Hdfs://localhost:9000

Vi mapred-site.xml

Mapred.job.tracker

Localhost:9001

Vi hdfs-site.xml

Dfs.name.dir

/ home/longlong/temp/log1,/home/longlong/temp/log2

Dfs.data.dir

/ home/longlong/temp/data1,/home/longlong/temp/data2

Dfs.replication

two

Configure the environment variable, where the environment variable of java,hive,hadoop is pasted

Vi / etc/profile

JAVA_HOME=/home/Hadoop/jdk1.6.0_45

CLASSPATH=$JAVA_HOME/jre/lib/rt.jar

HADOOP_HOME=/usr/hadoop-1.0.4

HIVE_HOME=/usr/hive

PATH=$HADOOP_HOME/bin:$PATH:$JAVA_HOME/bin:$HIVE_HOME/bin

Install jdk:

Download jdk

Wget http://download.oracle.com/otn-pub/java/jdk/6u45-b06/jdk-6u45-linux-x64.bin

Chmod + x jdk-6u45-linux-x64.bin

. / jdk-6u45-linux-x64.bin

Configure vi hadoop-env.sh and add jdkhome at the end

Export JAVA_HOME=/home/Hadoop/jdk1.6.0_45

Enter the bin directory to configure vi hadoop-config.sh

Export HADOOP_HOME=$ {HADOOP_PREFIX}

Export HADOOP_HOME_WARN_SUPPRESS=1

Formatting

. / hadoop namenode-format

2.mysql installation

3.hive installation

download

Wget http://mirrors.cnnic.cn/apache/hive/hive-0.13.1/apache-hive-0.13.1-bin.tar.gz

Tar-xf apache-hive-0.13.1-bin.tar.gz

Mv apache-hive-0.13.1-bin hive

Replace Metabase

Cd conf/

Touch hive-site.xml

Vi hive-site.xml

Javax.jdo.option.ConnectionURL

Jdbc:mysql://localhost:3306/hive

Javax.jdo.option.ConnectionDriverName

Com.mysql.jdbc.Driver

Javax.jdo.option.ConnectionUserName

Root

Javax.jdo.option.ConnectionPassword

Root

Download the mysql driver

Wget http://cdn.mysql.com/Downloads/Connector-J/mysql-connector-java-5.0.8.tar.gz

Tar-xzvf mysql-connector-java-5.0.8.tar.gz

Cd mysql-connector-java-5.0.8

Cp mysql-connector-java-5.0.8-bin.jar / usr/hive/lib/

Start

1.hdfs start

Cd / usr/hadoop-1.0.4/bin

. / start-all.sh

Password/ operating system password

2.mysql start

Service mysqld start

Mysql-uroot-proot

3.hive start

Cd / usr/hive/bin

. / hive

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.