Shulou(Shulou.com)06/01 Report--

Redis data types and Application scenarios

Redis supports five data types: string (string), hash (hash), list (list), set (collection) and zset (sorted set: ordered set).

1. String

Summary: the Strings data type is the most commonly used and simple key-value type, and ordinary key/ value storage can be classified into this category. Value can be not only a string, but also a number. Because it is binary secure, you can store the contents of an image file as a string. Redis's string can fully realize the functions of current memcached, and it is more efficient. In addition to providing the same get, set, incr, decr, and so on operations as Memcached, Redis also provides the following additional operations:

1. Get string length

two。 To the string append content

3. Set and get a segment of a string

4. Set and get a bit of a string (bit)

5. Set the contents of a series of strings in batch

Common commands: set,get,decr,incr,mget, etc.

Application scenarios:

1. All scenarios where Memcached and CKV are applied. Direct access to strings and numbers. Structured data needs to be serialized first, then set to value;, and then deserialized after get to value.

two。 You can use redis's INCR, INCRBY, DECR, DECRBY and other instructions to achieve the effect of atomic counting. That is, it can be used to realize the statistical counting requirements on the business. It can also be used to implement idmaker, that is, to generate globally unique id.

3. Store session key to realize a distributed session system. Redis's key can easily set the expiration time, which is used to realize the automatic expiration of session key. When verifying the skey, first route to the corresponding redis according to the uid. If the skey cannot be obtained, the skey has expired and you need to log in again. If the skey is obtained and the verification is passed, upgrade the expiration time of the skey.

4. Set nx or SetNx, Set only if key does not exist. Can be used to elect Master or implement distributed locks: all Client keep trying to use SetNx master myName to preempt Master, and the successful one keeps using Expire to refresh its expiration time. If the Master fails, the key will fail, and a new round of grabbing will take place in the remaining nodes.

5. With the lua scripts supported by redis2.6, two more secure distributed locks can be implemented: a scenario in which each process competes but is always acquired and processed by a single process. The lock will not be acquired by other processes unless the original process hangs up and the lock expires. There is no need to unlock it voluntarily. Implemented through lua scripts of get, expire/pexpire, setnx ex | px; a scenario where processes compete to acquire locks and process them. The lock is acquired through set nx ex | px. You need to release the lock after get judgment, otherwise the lock cannot be acquired before the lock expires.

6. GetSet, set the new value and return the old value. For example, to implement a counter, you can use GetSet to get the count and reset it to 0.

7. GetBit/SetBit/BitOp/BitCount, the method of BitMap. For example, when counting the number of unique visitors today, each registered user has an offset. If he comes in today, set his bit to 1, and use BitCount to get the total number of people today.

8. Append/SetRange/GetRange/StrLen, which extends, replaces, intercepts, and lengths text is useful for specific data formats.

Implementation: String is stored in redis as a string by default, which is referenced by redisObject. When it encounters operations such as incr,decr, it will be converted to numeric calculation. In this case, the encoding field of redisObject is int.

II. Hash

Summary: Hash stores the mapping between strings and string values. Hash stores the properties of an object in Map and can read / update only some of the properties of the object. In this way, some properties are too long to leave it motionless, and different modules can only update the properties they care about without cocurring with each other.

Common commands: hget,hset,hgetall, etc.

Application scenarios:

1. Store structured data, such as user information. In Memcached or CKV, for user information such as user's nickname, age, gender, points, etc., we need to serialize and then store the value as a string. When we need to modify one of the items, we usually need to take out and deserialize all the values, modify the value of a certain item, and then serialize and store it back. This not only increases the overhead, but also does not apply to situations where concurrent operations are possible (for example, two concurrent operations need to modify the integral). The Hash structure of Redis allows you to modify only one attribute value as if you were Update an attribute in the database. As shown below:

2. Key is the user ID, value is a Map, the key of this Map is the attribute name of the member, and value is the attribute value, so the data can be modified and accessed directly through the Key of the internal Map (the key of the internal Map is called field in the Redis), that is, the corresponding attribute data can be manipulated through key (user ID) + field (attribute tag). There is no need to store the data repeatedly. Nor does it cause problems with serialization and concurrency modification control.

3. However, it should be noted that Redis provides an hgetall to fetch all the attribute data directly, but if there are many members of the internal Map, it involves traversing the entire internal Map. Due to the Redis single-threaded model, this traversal operation may be time-consuming and completely unresponsive to requests from other clients, which requires special attention.

4. Can be used to build an index. For example, for User objects, in addition to id and sometimes query by name, you can build a Hash with Key as user:name:id. When inserting User objects (set user:101 {"id": 101, "name": "calvin"}), insert a bar into this hash (hset user:name:id calvin 101and calvin as a key in hash, with a value of 101l). When you press name to query, you can use hgetuser:name:id calvin to extract id from the key named calvin. If you need to use multiple indexes to find a piece of data, you can use one hash key to avoid using multiple string key to store index values.

5. HINCRBY can also be used to implement idmaker. Relative to the idmaker of string type, each type requires a key,hash type to use a key.

Implementation method:

Redis Hash is actually a HashMap inside the Value. There are two different implementations. When the number of members of this Hash is small, the Redis will use a compact storage method similar to an one-dimensional array to save memory, instead of using the real HashMap structure. The encoding of the corresponding value redisObject is zipmap. When the number of members increases, it will be automatically converted to the real HashMap, and the encoding will be ht.

III. List

Introduction: List is a two-way linked list, supports two-way Pop/Push, quack rules generally from the left end Push, right end Pop--LPush/RPop, and there is a version of Blocking BLPop/BRPop, the client can block there until a message arrives. And RPopLPush/ BRPopLPush, which pops up and returns to client while pushing itself into another list,LLen to get the length of the list. There are also operations by value: LRem (delete elements by value), LInsert (insert before and after elements of a value), complexity is O (N), N is the length of List, because the value of List is not unique, so all elements are traversed, while Set only needs O (log (N)).

Operation by subscript: the subscript starts at 0, the queue is counted from left to right, and when the subscript is negative, it is from right to left. LSet, press the subscript to set the element value. LIndex, press the subscript to return the element. LRange, unlike POP directly popping away elements, only returns a subscript element in the list, which is a favorite for paging. LTrim, limit the size of the List, such as keeping only the latest 20 messages. Complexity is also O (N), where N of LSet is the length of List, N of LIndex is the value of subscript, N of LRange is the value of start + the number of elements listed, because it is a linked list rather than an array, so clicking the subscript actually traverses the linked list, unless the subscript happens to be the head and end of the line. The N of LTrim is the number of elements removed.

Common commands: lpush,rpush,lpop,rpop,lrange, etc.

Application scenarios:

1. Various lists, such as twitter's follow list, fan list, ranking of the latest news, comments on each article, etc., can also be implemented using Redis's list structure.

2 message queue, you can use the PUSH operation of Lists to store the task in Lists, and then the worker thread uses the POP operation to take out and execute the task. There is no ack mechanism in the message queue here. What if the consumer leaves the task to Pop and crashes before it is finished? One of the solutions is to add an additional sorted set, and send it to both list and sorted set at the same time, so that the distribution time is score. After the user has finished the task, he / she needs to use ZREM to eliminate the job in the sorted set, and regularly take out the unfinished tasks from the sorted set and put them back into the list. Another way is to add an extra list for each worker, use RPopLPush when popping up tasks, put job in worker's own list at the same time, and eliminate it with LREM when finished. If the cluster management (such as zookeeper) finds that the worker is dead, it puts the list content of the worker back into the main list.

(3) the function of list content paging can be easily realized by using LRANGE.

4. Operation to take the latest N pieces of data: LPUSH is used to insert a content ID, which is stored as a keyword in the header of the list. LTRIM is used to limit the number of items in the list to a maximum of 5000. If the amount of data retrieved by the user exceeds the cache capacity, the request needs to be sent to the database.

Implementation method:

The implementation of Redis list is a two-way linked list, that is, it can support reverse search and traversal, which is more convenient to operate, but it brings some additional memory overhead. Many implementations within Redis, including send buffer queues, also use this data structure.

IV. Set

Introduction: it is a kind of unordered collection in which the elements in the collection are not sequenced and do not repeat. Putting duplicate elements into Set will automatically remove the weight.

Common commands:

Sadd,spop,smembers,sunion et al.

Application scenarios:

1. There are some lists that need to be deduplicated, and set provides an important interface to determine whether a member is in an set collection, which list cannot provide.

two。 You can store some collective data, such as in Weibo applications, you can store all the followers of a user in a collection and all their fans in a collection. Redis also provides intersection, union, difference and other operations for the collection, which is very convenient to achieve, such as common concern, common preferences, second-degree friends and other functions. For all the above collection operations, you can also use different commands to choose whether to return the results to the client or save to a new collection. For example, QQ has a social function called "friend tag". People can tag your friends, such as "Beauty", "Tuhao", "Brother" and so on. Here, you can store each user's tag in a collection.

3. If you want to know how many registered users or IP addresses have visited a page, you can do this: SADD page:day1:. To know the number of specific users, use SCARD page:day1:. Need to test whether a particular user has visited this page? SISMEMBER page:day1: .

Implementation method:

The internal implementation of set is a HashMap in which value is always null. In fact, the weight is arranged quickly by calculating hash, which is why set can determine whether a member is in the collection or not.

5. Sorted Set

Introduction: ordered collection, compared to set, the element into the collection, but also provide the score of the element, can be sorted automatically according to the score.

Common commands:

Zadd,zrange,zrem,zcard et al.

Use the scene:

1. Store an ordered and non-repeating list of collections, such as twitter's public timeline, which can be stored as a score at the time of publication, so that the fetches are automatically sorted in time.

two。 You can do weighted queues, such as score 1 for ordinary messages and score 2 for important messages, and then the worker thread can choose to get work tasks in reverse order of score. Give priority to important tasks.

3. Ranking related: ZADD leaderboard. Getting the top 100 high-scoring users is simple: ZREVRANGE leaderboard 099. The global ranking of users is similar, only need to execute: ZRANK leaderboard.

4. The news is sorted by user's vote and time, score = points / time ^ alpha in ZADD, so that the user's vote will dig up the news accordingly, but the time will bury the news according to a certain index.

5. Overdue items processing: use unix time as the keyword to keep the list sorted by time. Search current_time and time_to_live to complete the arduous task of finding overdue items. Another background task uses ZRANGE...WITHSCORES to query and delete expired entries.

Implementation method:

Redis sorted set internal use HashMap and jump table (SkipList) to ensure data storage and order, HashMap is put in the member to score mapping, while the jump table is stored in all the members, sorting according to the score stored in HashMap, the use of jump table structure can achieve higher search efficiency, and relatively simple in implementation.

These are all the relevant knowledge points of this introduction. Thank you for your study and support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.