Shulou(Shulou.com)06/03 Report--

Premise:

Install jdk

Non-secret communication

Install zookeeper

Install hadoop

Install mysql

Yum installation

Mysql is used as a Metabase, and mysql is installed on the mater host

Sudo yum install-y mysql mysql-server

Sudo service mysqld start

Sudo mysql

Compile and install mysql

Download the mysql binary installation package first, and choose the version at will.

Download address of mysql installation package: http://mirrors.sohu.com/mysql/

Reference document: http://blog.csdn.net/t1dmzks/article/details/71374740

# create a data storage directory # extract the file tar axf mysql-boost-5.7.18.tar.gz-C / usr/local/#, compile and install cd mysql-5.7.18cmake-DCMAKE_INSTALL_PREFIX=/usr/local/mysql\-DMYSQL_UNIX_ADDR=/usr/local/mysql/mysql.sock-DDEFAULT_CHARSET=utf8\-DWITH_BOOST=boost-DDEFAULT_COLLATION=utf8_general_ci\-DWITH_INNOBASE_STORAGE_ENGINE=1-DWITH_ARCHIVE_STORAGE_ENGINE=1\-DWITH_BLACKHOLE_STORAGE _ ENGINE=1-DMYSQL_DATADIR=/data/mysqldb\-DMYSQL_TCP_PORT=3306-DENABLE_DOWNLOADS=1# install make & & make install & & echo $? # modify permissions chown-R mysql.mysql / data/mysqldbchown-R mysql.mysql / usr/local/mysql initialize mysql database cd / usr/local/mysql./bin/mysqld-- initialize-- user=mysql-- basedir=/usr/local/mysql

Notice that the part that is framed is your user's password. Now mysql initially has a password.

Delete all files under / data/mysqldb if you want to reinitialize the database

Rm-rf / data/mysqldb

Modify

Environment variable

Add the bin directory of mysql to path

Vim / etc/profile

Export PATH=$PATH:/usr/local/mysql/bin

Source / etc/profile

View mysql version information

Mysql-V

Modify the my.cnf main configuration file

[mysqld]

Datadir=/data/mysqldb

Socket=/usr/local/mysql/mysql.sock

User=mysql

# Disabling symbolic-links is recommended to prevent assorted security risks

Symbolic-links=0

[mysqld_safe]

Log-error=/var/log/mysqld.log

Pid-file=/var/run/mysqld/mysqld.pid

Copy startup script

Cd / usr/local/mysql

Cp support-files/mysql.server / etc/init.d/mysqld

Start mysql

Service mysqld start

Set up boot boot

Chkconfig-level 2345 mysqld on

Set environment variabl

Vim / etc/profile

Export PATH=$PATH:/usr/local/mysql/bin

Source / etc/profile

Log in to mysql and change the password (login password is prompted when initializing data)

Mysql-uroot-pendant 8XeyEBgesteryLj6'

Grant cannot be used

Mysql > SET PASSWORD = PASSWORD ('mysql')

Ok, log in again

Mysql-uroot-pmysql

There is no password field

View table structure

Mysql > desc mysql.user

View

Select HOst,User,authentication_string from mysql.user

The mysql.sys user seems to be the system user of the newly added mysql. The password * THISISNOTAVALIDPASSWORDTHATCANBEUSEDHERE is translated as almost no password is set and needs to be set.

Multiple instances of mysql can be easily implemented through support-files/mysqld_multi.server.

Reference: http://blog.csdn.net/miyatang/article/details/56846283

Create a hive database

Create a hive database

Create a hadoop user and authorize access to the database (here permissions are given to those with high permissions, and only to hive libraries)

Download and install hive

If the download reports an error, please use your browser to check whether the version under http://mirror.bit.edu.cn/apache/hive has changed.

Mkdir / data

Cd / data

Wget http://mirrors.hust.edu.cn/apache/hive/hive-1.2.2/apache-hive-1.2.2-bin.tar.gz

Tar axf apache-hive-1.2.2-bin.tar.gz

Modify environment variabl

Vim / etc/profile

Export HIVE_HOME=/data/apache-hive-1.2.2-bin

Export PATH=$PATH:$ {HIVE_HOME} / bin

Source / etc/profile

Chown-R hadoop:hadoop / data/apache-hive-1.2.2-bin

Create database hive

Create a corresponding mysql account for hive and grant permissions (above)

Create a hive-specific Metabase (with the hadoop user above)

Generate the hive-site.xml master configuration file

Cp hive-default.xml.template hive-site.xml

Modify the database connection parameters

Javax.jdo.option.ConnectionDriverName

Com.mysql.jdbc.Driver

Driver class name for a JDBC metastore

Javax.jdo.option.ConnectionURL

Jdbc:mysql://192.168.3.58:3306/hive?characterEncoding=UTF-8

JDBC connect string for a JDBC metastore

Javax.jdo.option.ConnectionUserName

Hadoop

Username to use against metastore database

Javax.jdo.option.ConnectionPassword

Hadoop

Password to use against metastore database

Create a temporary storage directory for IO files

Mkdir ${HIVE_HOME} / iotmp

Modify parameters in hive-site.xml

Hive.querylog.location

/ data/apache-hive-1.2.2-bin/iotmp

Location of Hive run time structured log file

Hive.exec.local.scratchdir

/ data/apache-hive-1.2.2-bin/iotmp

Local scratch space for Hive jobs

Hive.downloaded.resources.dir

/ data/apache-hive-1.2.2-bin/iotmp

Temporary local directory for added resources in the remote file system.

Copy the mysql-connector-java-5.1.21.jar driver package to the $HIVE_HOME/lib directory.

Mysql-connector-java download address

Http://mvnrepository.com/artifact/mysql/mysql-connector-java

Start the hadoop cluster (requires datanode and namenode)

Start mysql

Start hive

Logging initialized using configuration in jar Vele fileVlue Log 4j.properties is the initialization log module, so you need to create a hive-log4j.properties file.

Cd ${HIVE_HOME} / conf

Cp hive-log4j.properties.template hive-log4j.properties

Cp hive-exec-log4j.properties.template hive-exec-log4j.properties

Modify the hive-log4j.properties file to specify the log storage path

Vim hive-log4j.properties

Hive.log.dir=/data/apache-hive-1.2.2-bin/log

Create a directory

Mkdir / data/apache-hive-1.2.2-bin/log

Multi-node hive construction

Hive stores data in mysql and hdfs, using only the same metadata

Install hive with a single node first

Initialize hive metadata

Schematool-dbType mysql-initSchema

Cd ${HIVE_HOME} / conf

Cp hive-env.sh.template hive-env.sh

Specify JAVA_HOME, HADOOP_HOME, HIVE_HOME, and HIVE_CONF_DIR in hive-env.sh

# Set HADOOP_HOME to point to a specific hadoop install directory

# HADOOP_HOME=$ {bin} /.. /.. / hadoop

HADOOP_HOME=/data/hadoop

JAVA_HOME=/usr/local/jdk

HIVE_HOME=/data/apache-hive-1.2.2-bin

# Hive Configuration Directory can be controlled by:

# export HIVE_CONF_DIR=

Export HIVE_CONF_DIR=$HIVE_HOME/con

Copy files to other machines

[hadoop@master data] $scp-r apache-hive-1.2.2-bin slave1:/data

[hadoop@master data] $scp-r apache-hive-1.2.2-bin slave2:/data

Other nodes also need to modify the / etc/profile file

[root@master hadoop] # scp-r / etc/profile slave1:/etc/

[root@master hadoop] # scp-r / etc/profile slave2:/etc/

As soon as the node starts the metastore service (it will occupy the window, nohup is available)

[hadoop@master data] $hive-- service metastore

Node 2 starts the metastore service

[hadoop@slave1 /] $source / etc/profile

[hadoop@slave1 /] $hive-- service metastore

Node 3 starts the metastore service

[hadoop@slave1 /] $source / etc/profile

[hadoop@slave1 /] $hive-- service metastore

Access test

Create a test table

Create table test_hive (id bigint, name string) row format delimited fields terminated by'\ t'

to be continued

Reference documentation:

Installation and use of hive (recommended): http://www.cnblogs.com/zlslch/p/6039901.html

Hive installation and deployment: http://www.cnblogs.com/zlslch/p/6700695.html

Detailed explanation of the latest data operation of Hive (super detailed): http://blog.csdn.net/yhao2014/article/details/46312469

Hive import and export usage details: http://www.cnblogs.com/wcwen1990/p/6845850.html

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.