Shulou(Shulou.com)06/01 Report--

Introduction to the hadoop environment:

Master service: node1

Slave server: node2,node3,node4

Mysql server: node29

Thrift is installed on the node1 server!

Related software version:

Hadoop version: hadoop-0.20.2

Sqoop version: sqoop-1.2.0-CDH3B4

Java version: jdk1.7.0_67

Mysql version: 5.1.65

Thrift version: thrift-0.9.0

Thrift installation link: http://thrift.apache.org/download/

Python version: 2.7.3

There is a problem with using thrift in ps:python2.5 version.

One: preparation before testing

1) first, load the data from the mysql database into hbase:

The mysql data are as follows:

The format of the command to import data from mysql into hbase is:

Sqoop import-connect jdbc:mysql://mysqlserver_IP/databaseName-username-password password-table datatable-hbase-create-table-hbase-table hbase_tablename-column-family col_fam_name-hbase-row-key key_col_name

Description: databaseName and datatable are the database and table names of mysql, hbase_tablename is the table name to be imported into hbase, and key_col_name can specify which column in datatable is the hbase new table rowkey,col_fam_name is the column family name of all columns except rowkey

2) load mysql data (node29) on node1 into hbase:

Sqoop import-connect jdbc:mysql://172.16.41.29/sqoop-username sqoop-password routon-table students-hbase-create-table-hbase-table students-column-family stuinfo-hbase-row-key id

Verify that the load is successful in hbase:

II. Installation of Thrift software

Python version: 2.7.3

The steps are:

1) install python2.7.3

Description: there is no problem with the combination of python2.7.3 and thrift, but the python2.5 version does not seem to work!

Syntax in the generated Hbase.py file rhel5 comes with python2.4 that is not supported

Tar fvxj Python-2.7.3.tar.bz2

. / configure-- prefix=/usr/local/python2.7

Make & & make install

The python2.7.3 path is:

/ usr/local/python2.7/bin/python

Change the default python version to 2.7

Set python2.7 as the environment variable, and the default python version of the system is 2.4

Rm-rf / usr/bin/python

Ln-s / usr/local/python2.7/bin/python / usr/bin/python

[root@node1 thrift-0.9.0] # python-V

Python 2.7.3

2) install thrift

Tar fvxz thrift-0.9.0.tar.gz

Cd thrift-0.9.0

. / configure

Make & & make install

Thrift 0.9.0

Building C++ Library. : no

Building C (GLib) Library.... : yes

Building Java Library. : no

Building C# Library. : no

Building Python Library. : yes

Building Ruby Library. : no

Building Haskell Library. : no

Building Perl Library. : no

Building PHP Library. : no

Building Erlang Library. : no

Building Go Library. : no

Building D Library. : no

Python Library:

Using Python. : / usr/bin/python

You can see that thrift supports many languages, according to the current needs, just support python!

View the Thrift version:

[root@node1 thrift-0.9.0] # thrift- version

Thrift version 0.9.0

3) Let thrift support hbase

Execute the following command:

Thrift-gen py / usr/local/hbase-0.90.5/src/main/resources/org/apache/hadoop/hbase/thrift/Hbase.thrift

A directory is created under the current situation, and the directory name is:

[hadoop@node1 ~] $ll

Total 7056

-rw-rw-r-- 1 hadoop hadoop 3045 Oct 14 13:55 access_log2.txt

-rw-r--r-- 1 hadoop hadoop 7118627 Feb 1 2012 access_log.txt

-rw-rw-r-- 1 hadoop hadoop 3500 Oct 22 10:17 derby.log

Drwxrwxr-x 3 hadoop hadoop 4096 Oct 24 15:28 gen-py

-rw-rw-r-- 1 hadoop hadoop 3551 Oct 13 11:21 pig_1413170429087.log

The gen-py directory structure is as follows:

[hadoop@node1 ~] $tree gen-py/

Gen-py/

| |-_ _ init__.py |

`--hbase

|-- Hbase-remote

|-- Hbase.py

| |-_ _ init__.py |

|-- constants.py

`--ttypes.py

1 directory, 6 files

4) copy the gen-py directory to the python-related directory:

Cp-r gen-py/hbase/ / usr/local/python2.7/lib/python2.7/site-packages/.

5) allow python to import thrift the module:

[root@node1] # ln-s / usr/lib/python2.7/site-packages/thrift* / usr/local/python2.7/lib/python2.7/site-packages/.

[root@node1] # ls-l / usr/local/python2.7/lib/python2.7/site-packages/

Total 12

Drwxr-xr-x 2 root root 4096 Oct 24 15:32 hbase

-rw-r--r-- 1 root root 119 Oct 24 11:30 README

Lrwxrwxrwx 1 root root 39 Oct 24 15:50 thrift-> / usr/lib/python2.7/site-packages/thrift

Lrwxrwxrwx 1 root root 60 Oct 24 15:50 thrift-0.9.0-py2.7.egg-info-> / usr/lib/python2.7/site-packages/thrift-0.9.0-py2.7.egg-info

6) start the thrift service:

Hbase thrift-p 9090 start

7) write a python script on node1 to see which tables are in hbase:

The py file generated by #! / usr/bin/env python#coding=utf-8import sys#Hbase.thrift is put here sys.path.append ('/ usr/local/lib/python2.7/site-packages/hbase') from thrift import Thriftfrom thrift.transport import TSocketfrom thrift.transport import TTransportfrom thrift.protocol import TBinaryProtocolfrom hbase import Hbase#, such as ColumnDescriptor, define from hbase.ttypes import * # Make socket# in hbase.ttypes. Here you can modify the address and port transport = TSocket.TSocket ('172.16.41.26, 9090) # Buffering is critical. Raw sockets are very slow# can also use TFramedTransport, which is also an efficient transmission mode transport = TTransport.TBufferedTransport (transport) # Wrap ina protocol# transport protocol and transmission process is separate, can support multi-protocol protocol = TBinaryProtocol.TBinaryProtocol (transport) # client on behalf of a user client = Hbase.Client (protocol) # Open connection transport.open () # print table name print (client.getTableNames ())

Execute the script:

At this point, through the thrift plug-in, python can communicate with hbase!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.