Shulou(Shulou.com)06/01 Report--

What this article shares to you is about how python connects to hive through thrift. The editor thinks it is very practical, so I share it with you to learn. I hope you can get something after reading this article.

After the hive installation is complete, if it is for local use only, enable

Nohup hive--service metastore & [hadoop@master1 usr] $hiveLogging initialized using configuration in file:/data/usr/hive/conf/hive-log4j.propertieshive > use fmcm;OKTime taken: 0.874 seconds

If you want to make a script call, you need to enable HiveServer2 to make sure that port 10000 is already listening (you can modify the port in hive-site.xml)

Nohup hive-- service hiveserver2 colors & [hadoop@master1 usr] $netstat-an | grep 10000 tcp 00 0.0.0.0VOUR 10000 0.0.0.0VOV * LISTEN

HiveServer2 provides an interface for clients to execute hive queries remotely, which is realized through Thrift RPC. It also provides multi-user concurrency and authentication functions. At present, python can connect to HiveServer2 through the module pyhs2 to query and retrieve the results.

However, pyhs2 is no longer under maintenance, so you can refer to the other two good python package for updates (pyhs2 has been proved to have performance bottlenecks, so it is best to switch to pyhive as soon as possible)

Https://github.com/dropbox/PyHive

Https://github.com/cloudera/impyla

If you fail to install sasl, install: yum install gcc-c++ python-devel.x86_64 cyrus-sasl-devel.x86_64 first

Pyhs2's project is hosted on github at https://github.com/BradRuderman/pyhs2 or downloaded directly from https://pypi.python.org/pypi/pyhs2/0.2

If the installation is not successful, you can try installing the following components first:

Yum install cyrus-sasl-plain

Yum install cyrus-sasl-devel

If you encounter an error during installation:

Error: sasl/sasl.h: No such file or directory

You can try to install sasl first, ubantu can use sudo apt-get install libsasl2-dev, CentOS can use anaconda's pip installation, or follow these steps to install:

Curl-O-L ftp://ftp.cyrusimap.org/cyrus-sasl/cyrus-sasl-2.1.26.tar.gztar xzf cyrus-sasl-2.1.2.26.tar.gzcd cyrus-sasl-2.1.26.tar.gz./configure & & make install ends with the test code: #-*-coding:utf-8-*''connect to the database by Hive and thrift' import pyhs2import sysreload (sys) sys.setdefaultencoding ('utf8 ') class HiveClient: def _ _ init__ (self Db_host, user, password, database, port=10000, authMechanism= "PLAIN"): self.conn = pyhs2.connect (host=db_host, port=port, authMechanism=authMechanism, user=user, password=password, database=database ) def query (self Sql): with self.conn.cursor () as cursor: cursor.execute (sql) return cursor.fetch () def close (self): self.conn.close () def main (): "" main process @ rtype: @ return: @ note: "hive_client = HiveClient (db_host='10.24.33.3', port=10000, user='hadoop', password='hadoop') Database='fmcm', authMechanism='PLAIN') result = hive_client.query ('select * from fm_news_newsaction limit 10') print result hive_client.close () if _ _ name__ = =' _ main__': main () above is how python connects hive through thrift The editor believes that there are some knowledge points that we may see or use in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.