Shulou(Shulou.com)05/31 Report--

This article will explain in detail how Redis obtains a large key value. The editor thinks it is very practical, so I share it with you as a reference. I hope you can get something after reading this article.

1. Preface

In the work, there are often some reasons such as inappropriate use of Redis instances, or inaccurate business estimates, or lack of timely processing of key, and so on, resulting in some KEY is quite large.

So what are the problems with a big Key?

If the load balance cannot be achieved in the cluster mode, the request will be skewed to an instance, which will have a larger QPS and more memory. For the Redis single-threaded model, it is easy to have CPU bottlenecks. When there is a memory bottleneck, you can only use vertical storage capacity and use more powerful servers.

When it comes to the operation of a large key, especially when using hgetall, lrange 0-1, get, hmget, etc., the Nic may become a bottleneck, or it may cause other operations to block, and the qps may suddenly drop or rise. The trend may not be very smooth. In serious cases, it will cause the application to be disconnected, and the instance or cluster will be unavailable for a certain period of time.

If the key needs to be deleted, and if the DEL operation is performed directly, the operated instance will be occupied by the Block, resulting in inability to respond to the application request, and the Block time will become longer as the key grows.

In redis, it is very slow to look at the value of a large key, such as the hash type, directly, so you want to write a script to get it through incremental iteration, so let's take a look at the detailed introduction.

2. The specific script is as follows:

Function: scan all the elements in a key of redis

Usage: python bigkey_save_values.py "zyyset" * "100

3. The python script is implemented as follows:

#! / usr/bin/env python#-*-coding: UTF-8-*-# function: scan all the elements in a key # how to use: python bigkey_save_values.py "zyyset"m *" 10 elements if you need to get all the elements: python bigkey_save_values.py "zyyset"*" 100 written authorizations _ = "lcl" import sysimport redis import os pool=redis.ConnectionPool (host='192.168.225.128',port=6379 Db=0) r = redis.StrictRedis (connection_pool=pool) # specify keykey = sys.argv [1] # scan match value match = sys.argv [2] # number of matches per match count = sys.argv [3] # Total number total = get the current path path = os.getcwd () keytype = r.type (key) print "key of type% s"% (keytype) # the key output file txt = path+ "/% s.txt" scanned % (key) # Open file # f = open (txt "w") def list_iter (name): list_count = r.llen (key) for index in range (list_count): yield r.lindex (key, index) if keytype=='list': f = open (txt, "w") for item in list_iter (key): # output member/score matching to the corresponding key to f.write ("% s% s"% (item, "\ n")) total = total+1 f.closeelif keytype=='hash': f = open (txt) "w") for item in r.hscan_iter (key,match = match,count = count): # output the member/score matching to the corresponding key to the file f.write ("% s% s"% (item, "\ n")) total = total+1 f.closeelif keytype=='set': f = open (txt, "w") for item in r.sscan_iter (key,match = match,count = count): f.write ("% s% s"% (item) "\ n") total = total+1 f.closeelif keytype=='zset': f = open (txt, "w") for item in r.zscan_iter (key,match = match,count = count): f.write ("% s% s"% (item, "\ n") total = total+1 f.closeelse: print ("key is of type string" Value is: "+ r.get (key)) print" key:%s the number of match:%s is:% d "% (key,match,total) on" how to get a large key value of Redis "this article is shared here, I hope the above content can be of some help to you, so that you can learn more knowledge, if you think the article is good, please share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.