Shulou(Shulou.com)06/03 Report--

Building data Warehouse (Slave1)

1. Decompress the database

Tar-zxvf apache-hive-1.2.2-bin.tar.gz / usr/local/

Cd / usr/local/

Mv apache-hive-1.2.2 hive

2. Add environment variables to hive

Edit the / etc/profile file to add hive-related environment variable configuration

After editing the profile file, execute the following command to make the configuration effective. The command is

3. Configure hive-site.xml

Hive-site.xml related configuration

Cd / usr/local/hive/conf

Cp hive-default.xml.template hive-site.xml

Then create a directory with hadoop

Hadoop fs-mkdir-p / user/hive/warehouse

Hadoop fs-mkdir-p / tmp/hive/

Give read and write access to the directory you just created, and execute the command:

Hadoop fs-chmod 777 / user/hive/warehouse

Hadoop fs-chmod 777 / tmp/hive

The directory was created because there is such a configuration in hive-site-xml

Check whether the directory is created successfully

Hadoop fs-ls / user/hive/

Hadoop fs-ls / tmp/

Modify the temporary directory in hive-site.xml

Replace ${system:java.io.tmpdir} in the hive-site.xml file with the temporary directory of hive, for example, I replace / usr/local/hive/tmp, replace ${system:user.name} with root, and give read and write permission

Be careful to replace all of them, here with regular ones.

Sed-I's usr/local/hive/tmp/@g' hive-site.xml ${system:java.io.tmpdir} @ /

Sed-I's customers ${system:user.name} @ root@g' hive-site.xml

Modify the configuration related to hive-site.xml database

Search for javax.jdo.option.ConnectionURL and change the value corresponding to the name to the address of MySQL

Search for javax.jdo.option.ConnectionDriverName and change the value corresponding to the name to the MySQL driver classpath

Search for javax.jdo.option.ConnectionUserName and change the corresponding value to the MySQL database login:

Search for javax.jdo.option.ConnectionPassword and change the corresponding value to the login password of the MySQL database:

Search for hive.metastore.schema.verification and change the corresponding value to false:

4. Load mysql driver

Upload the MySQL driver package to the lib directory

Cd / usr/local/hive/lib/

Wget ftp://172.18.79.77/mysql-connector-java-5.1.5-bin.jar

Note that the address here is my private address, you can go to Baidu to find the mysql driver package, or download it in my blog (because the repeated materials on 51cto cannot be uploaded, it is very speechless, just add a txt file and the driver package into a compressed package and upload it again).

5. Modify hive-env.sh

Create a new hive-env.sh file and modify it

Cd / usr/local/hive/conf

Make a copy of the hive-env.sh.template file and rename it hive-env.sh

Cp hive-env.sh.template hive-env.sh

Open the hive-env.sh configuration and add the following:

Export HADOOP_HOME=/usr/local/hadoop

Export HIVE_CONF_DIR=/usr/local/hive/conf

Export HIVE_AUX_JARS_PATH=/usr/local/hive/lib

6. Initialize mysql

Initialize the MySQL database

Go to mysql database and create hive user and hive database

CREATE USER 'hive'@'localhost' IDENTIFIED BY "hive"

Grant all privileges on. To hive@localhost identified by 'hive'

Create database hive

Refresh the database

Flush privileges

Quit

Exit

Test whether the hive user is created successfully

Mysql-uhive-p

Enter passwd: hive

Go to the bin directory of hive and execute the command:

Cd / usr/local/hive/bin

Initialize the database and execute the command:

Schematool-initSchema-dbType mysql

After the successful execution, there are already a bunch of tables created in the hive database.

7. Test hive

Start hive and test

Go to the bin directory of hive and execute the command:

Cd / usr/local/hive/bin

Execute the hive script to start and execute the command:

. / hive

Execute the command to view the details of the sum function:

Desc function sum

Execute the hive command for the new database:

Create database db_hive

Create a data table in the database you just created and execute the hive command:

Use db_hive

Create table student (id int,name string) row format delimited fields terminated by'\ t'

Write file data to the table

Create a new file in the / usr/local/hive directory

Execute the Linux command (preferably by reopening a terminal to execute):

Touch / opt/hive/student.txt

Note: ID and name are directly TAB keys, not spaces, because terminated by'\ t'is used in the above statement to create the table, so the division of id and name in this text must use the TAB key (copy and paste if there is a problem, manually hit the TAB key), and there can be no blank lines between lines, otherwise load will be executed below, and the NULL will be stored in the table. The file should be in unix format. If you are editing with a txt text editor on windows and uploading to the server, you need to use a tool to convert the windows format to unix format, for example, you can use Notepad++ to convert it.

After completing the above steps, the / opt/hive/student.txt file on disk has been created successfully, and there is already content in the file. Execute the hive command to load the data on the hive command line:

Load data local inpath'/ usr/local/hive/std.txt' into table db_hive.std

Execute the command to see if the data just in the file has been written successfully

View the page

Http://172.18.74.105:50070/explorer.html#/user/hive/warehouse/db_hive.db/student

Click std.txt

Execute the select statement in the MySQL database to view the table created by hive

Select * from hive.TBLS

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.