Shulou(Shulou.com)06/01 Report--

When the business needs a large number of connections to redis or hbase, a large number of connections will cause a large number of socket occupation, resulting in the server has no more ports to allocate, the best solution in this case is to implement the client connection single-instance mode, keep the connection always the same. Speaking of which, maybe you haven't experienced it before, if you insert 4000 pieces of data per second, this phenomenon is very obvious. The following is a Python implementation of redis+hbase single-case mode, there are many improvements, according to their own business adjustments, you can verify by printing the ID of the instance:

import happybaseimport redisclass _RedisMgrSingleton(type): ''redis singleton '' def __init__(self, name, bases, dict): super(_RedisMgrSingleton, self).__ init__(name, bases, dict) self._ instance = {} def __call__(self, host,port,db): if not self._ instance.has_key((host,port,db)): self._ instance[(host,port,db)] = super(_RedisMgrSingleton, self).__ call__(host,port,db) return self._ instance[(host,port,db )]class HbaseSingleton(type): Singleton of ''hbase '' def __init__(self, name, bases, dict): super(HbaseSingleton, self).__ init__(name, bases, dict) self._ instance = {} def __call__(self, host,table): if not self._ instance.has_key((host,table)): self._ instance[(host,table)] = super(HbaseSingleton, self).__ call__(host,table) return self._ instance[(host,table)]class RedisMgr: "redis operation specific class" def __init__(self,host,port,db,max_connections=3): "eg: host '192.168.2.184' port 6379 db 14" self.host=host self.port=port self.db=db self.conn=redis.Redis(connection_pool= redis.ConnectionPool(host=host,port=port,db=db,max_connections=max_connections)) def run_redis_fun(self,funname,*args): fun=getattr(self.conn,funname) print args return fun(*args) def pipe(self): return self.conn.pipeline(transaction=False) __metaclass__ = _RedisMgrSingleton #metaclass implementation singleton class HbaseOperate(object): def __init__(self,host,table): self.host = host self.table = table self.conn = happybase.Connection(self.host) self.table = self.conn.table(self.table) def run(self,fun,*args): # result =self.table.row(*args) funname = getattr(self.table,fun) return funname(*args) def cells(self,column,info,version): return self.table.cells(column,info,versions=version) __metaclass__ = HbaseSingleton #metaclass implementation singleton conn = HbaseOperate ('xxx.xx.11.8',"history_visitor_product")result = conn.cells ('chenhuachao ','info:visiotr',3)print resultprint "first time",id(conn)conn1 = HbaseOperate ('xxx.xxx.11.8',"history_visitor_product")result1= conn1.cells ('chenhuachao','info:visiotr',6)print result1print "second time",id(conn1)#output <$'test10 ', ' test9','test97'] first 38014896 <$'test10', 'test9',' test97','test17',' test1345\n'] Second 38014896

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.