Shulou(Shulou.com)06/01 Report--

What are the main application scenarios of redis? In response to this problem, the editor summed up this article today, hoping to help more friends who want to solve this problem to find a more simple and feasible way.

Redis pioneered a new idea of data storage. With Redis, we don't have to focus on how to put elephants in refrigerators when facing monotonous databases. Instead, we use Redis's flexible data structure and data manipulation to build different refrigerators for different elephants. I hope you like this metaphor.

1. Redis common data types

The following five data types are most commonly used in Redis:

String 、 Hash 、 List 、 Set 、 Sorted set

Before describing these data types in detail, let's take a look at how these different data types are described in Redis internal memory management:

First of all, Redis uses a redisObject object to represent all the main information of key and value,redisObject, as shown in the figure above: type represents the specific data type of a value object, and encoding is how different data types are stored within redis.

For example, type=string represents value to store an ordinary string, then the corresponding encoding can be raw or int, and if int represents the actual redis, the string is stored and represented as a numeric class internally, of course, provided that the string itself can be represented by a numeric value, such as a string like "123,456".

The vm field needs to be specified here. Only when the virtual memory feature of Redis is enabled, this field will actually allocate memory, which is turned off by default.

From the figure above, we can find that it is a waste of memory for Redis to use redisObject to represent all key/value data. Of course, these memory management costs are mainly to provide a unified management interface for different data types of Redis. The actual author also provides a variety of methods to help us save memory as much as possible, which we will discuss in detail later.

II. Application and implementation of various data types

Let's first analyze the use and internal implementation of these five data types one by one:

1 、 String

The String data structure is a simple key-value type, and value is not only a String, but also a number.

Common commands: get, set, incr, decr, mget, etc.

Application scenario: String is the most commonly used data type, ordinary key/ value storage can be classified into this category, that is, it can fully achieve the current Memcached functions, and more efficient. You can also enjoy the regular persistence of Redis, operation log and Replication and other functions. In addition to providing the same get, set, incr, decr, and so on operations as Memcached, Redis also provides the following operations:

Get string length

To the string append content

Set and get a segment of a string

Set and get a bit of a string (bit)

Set the contents of a series of strings in batch

Usage scenarios: regular key-value caching applications. Regular count: Weibo, fans.

Implementation: String is stored in redis as a string by default, which is referenced by redisObject. When it encounters operations such as incr,decr, it will be converted to numeric calculation. In this case, the encoding field of redisObject is int.

2 、 Hash

Common commands: hget,hset,hgetall, etc.

Application scenarios:

Let's give a simple example to describe the application scenario of Hash. For example, we want to store a user information object data that contains the following information:

The user ID is the found key, and the stored value user object contains name, age, birthday and other information. If you use the normal key/value structure to store it, there are two main storage methods:

The first method takes the user ID as a lookup key and encapsulates other information into an object to store in a serialized way. the disadvantage of this method is that it increases the cost of serialization / deserialization, and when one of the information needs to be modified, the whole object needs to be retrieved, and the modification operation needs to protect concurrency and introduce complex problems such as CAS.

The second method is to save as many key-value pairs as the number of members of the user information object, and use the name of the attribute corresponding to the user ID+ as the unique identification to get the value of the corresponding attribute. Although the serialization overhead and concurrency problems are eliminated, the user ID is stored repeatedly. If there is a large amount of such data, the memory waste is still considerable.

Then the Hash provided by Redis solves this problem very well. The Hash of Redis is actually an internally stored Value as a HashMap, and provides an interface to directly access this Map member, as shown below:

In other words, Key is still the user ID, value is a Map, the key of this Map is the attribute name of the member, and value is the attribute value, so the data can be modified and accessed directly through the Key of the internal Map (the key of the internal Map is called field in the Redis), that is, the corresponding attribute data can be manipulated through key (user ID) + field (attribute tag). There is no need to store the data repeatedly. Nor does it cause problems with serialization and concurrency modification control. Solved the problem very well.

It should also be noted that Redis provides hgetall to fetch all attribute data directly, but if there are many members of the internal Map, then the operation of traversing the entire internal Map is involved. Because of the Redis single-thread model, this traversal operation may be time-consuming, while requests from other clients are completely unresponsive, which requires special attention.

Usage scenarios: store some change data, such as user information, etc.

Implementation method:

It has been mentioned above that Redis Hash corresponds to Value is actually an internal HashMap. In fact, there are two different implementations. When the members of this Hash are relatively small, the Redis will use a compact storage method similar to an one-dimensional array to save memory, instead of using the real HashMap structure. The encoding of the corresponding value redisObject is zipmap. When the number of members increases, it will automatically be converted to the real HashMap, and the encoding will be ht.

3 、 List

Common commands: lpush,rpush,lpop,rpop,lrange, etc.

Application scenarios:

There are many application scenarios of Redis list, and it is also one of the most important data structures of Redis. For example, twitter's watch list, fan list and so on can be implemented by Redis's list structure.

List is a linked list, and anyone with a little knowledge of data structures should be able to understand its structure. Using the List structure, we can easily achieve functions such as ranking the latest messages. Another application of List is message queuing

You can take advantage of the PUSH operation of List to store the task in List, and then the worker thread uses the POP operation to fetch the task for execution. Redis also provides api for manipulating a certain segment of List. You can directly query and delete the elements of a certain segment of List.

Implementation method:

The implementation of Redis list is a two-way linked list, that is, it can support reverse search and traversal, which is more convenient to operate, but it brings some additional memory overhead. Many implementations within Redis, including send buffer queues, also use this data structure.

The list of Redis is a bi-directional linked list with each child element of type String. You can add or remove elements from the head or tail of the list through push and pop operations, so that List can be used as either a stack or a queue.

Use the scene:

Message queuing system

You can build a queue system using list, and you can even build a queue system with priority using sorted set.

For example, using Redis as a log collector

It is actually a queue, where multiple endpoints write log information to Redis, and then a worker writes all logs to disk.

The operation of taking the latest N pieces of data

Record the Id list of the first N newly logged-in users, which can be obtained from the database.

/ / add the current login to the linked list ret = r.lpush ("login:last_login_times", uid) / / keep the linked list with only N bits ret = redis.ltrim ("login:last_login_times", 0, NMUE 1) / / get the first N newly logged in users Id list last_login_list = r.lrange ("login:last_login_times", 0, NMu1)

For example, sina Weibo:

Our latest Weibo ID in Redis uses the resident cache, which is always updated. But we have set a limit of no more than 5000 ID, so our get ID function keeps asking Redis. You need to access the database only if the start/count parameter is out of this range.

Our system does not "flush" the cache in the traditional way, and the information in the Redis instance is always consistent. The SQL database (or other type of database on the hard drive) is triggered only when the user needs to obtain "very remote" data, while the home page or the first comment page will not bother the database on the hard disk.

4 、 Set

Common commands:

Sadd,spop,smembers,sunion et al.

Application scenarios:

Redis set external function is similar to list is a list function, the special feature is that set can automatically arrange weight, when you need to store a list of data, but do not want to duplicate data, set is a good choice, and set provides an important interface to determine whether a member is in a set collection, which list can not provide.

Set is a collection, and the concept of a set is a combination of unrepeated values. Using the Set data structure provided by Redis, some collective data can be stored.

Case study:

In the Weibo application, you can store all the followers of a user in a collection and all their fans in a collection. Redis also provides intersection, union, difference and other operations for the collection, which is very convenient to achieve, such as common concern, common preferences, second-degree friends and other functions. For all the above collection operations, you can also use different commands to choose whether to return the results to the client or save to a new collection.

Set is a set, an unordered set of String types, and set is realized through hashtable. The concept is basically similar to the collection in mathematics, which can be intersected, union, difference, and so on. The elements in set are out of order.

Implementation method:

The internal implementation of set is a HashMap in which value is always null. In fact, the weight is arranged quickly by calculating hash, which is why set can determine whether a member is in the collection or not.

Use the scene:

Intersection, Union, difference: (Set)

/ / book tables store book names set book:1:name "The Ruby Programming Language" set book:2:name "Ruby on rail" set book:3:name "Programming Erlang" / / tag tables use collections to store data, because collections are good at finding intersections, merging sadd tag:ruby 1sadd tag:ruby 2sadd tag:web 2sadd tag:erlang 3gambles / books that belong to both ruby and web? Inter_list = redis.sinter ("tag.web", "tag:ruby") / / that is, books that belong to ruby but not to web? Inter_list = redis.sdiff ("tag.ruby", "tag:web") / / A collection of books belonging to ruby and web? Inter_list = redis.sunion ("tag.ruby", "tag:web")

Get all the data to be deduplicated in a certain period of time

This set data structure using Redis is the most appropriate, just keep throwing the data into the set. Set means collection, so it will be automatically weighed.

5 、 Sorted Set

Common commands:

Zadd,zrange,zrem,zcard et al.

Use the scene:

The usage scenario of Redis sorted set is similar to that of set, except that set is not automatically ordered, while sorted set can sort members by providing an additional parameter of score, and it is inserted in order, that is, automatic sorting. When you need an ordered and non-repeating list of collections, you can choose the sorted set data structure. For example, twitter's public timeline can be stored as a score with the publication time, so that the acquisition is automatically sorted according to time.

Compared with Set, Sorted Set adds a weight parameter score, so that the elements in the collection can be arranged in order by score, such as a Sorted Set that stores the scores of the whole class, the collection value can be the student number of the students, and the score can be the test score, so that when the data is inserted into the collection, it has been sorted naturally.

In addition, Sorted Set can be used to make weighted queues, such as the score of ordinary messages is 1 and the score of important messages is 2, and then the worker thread can choose to get work tasks in reverse order of score. Give priority to important tasks.

Implementation method:

Redis sorted set internal use HashMap and jump table (SkipList) to ensure data storage and order, HashMap is put in the member to score mapping, while the jump table is stored in all the members, sorting according to the score stored in HashMap, the use of jump table structure can achieve higher search efficiency, and relatively simple in implementation.

III. Practical application scenarios of Redis

1. Display the latest list of projects

The following statement is often used to show the latest projects, and as there is more data, the query will no doubt get slower and slower.

SELECT * FROM foo WHERE... ORDER BY time DESC LIMIT 10

In Web applications, queries such as "list the latest responses" are common, which usually leads to scalability problems. This is frustrating because projects are created in this order, but sort operations have to be performed to output this order. Similar problems can be solved with Redis. For example, one of our Web apps wants to list the latest 20 comments posted by users.

We have a "Show all" link next to the latest comments, and you can get more comments after clicking on it. Let's assume that each comment in the database has a unique incremental ID field. We can use pagination to make home and comment pages, use Redis's template, and each time a new comment is posted, we add its ID to a Redis list:

LPUSH latest.comments

We cut the list to a specified length, so Redis only needs to save the latest 5000 comments:

LTRIM latest.comments 0 5000

Every time we need to get the project scope of the latest comment, we call a function to do it (using pseudo code):

FUNCTION get_latest_comments (start, num_items): id_list = redis.lrange ("latest.comments", start,start+num_items-1) IF id_list.length

< num_items id_list = SQL_DB("SELECT ... ORDER BY time LIMIT ...") END RETURN id_list END 这里我们做的很简单。在Redis中我们的最新ID使用了常驻缓存，这是一直更新的。但是我们做了限制不能超过5000个ID，因此我们的获取ID函数会一直询问Redis。只有在start/count参数超出了这个范围的时候，才需要去访问数据库。 我们的系统不会像传统方式那样"刷新"缓存，Redis实例中的信息永远是一致的。SQL数据库（或是硬盘上的其他类型数据库）只是在用户需要获取"很远"的数据时才会被触发，而主页或第一个评论页是不会麻烦到硬盘上的数据库了。 2、排行榜应用，取TOP N操作 这个需求与上面需求的不同之处在于，取最新N个数据的操作以时间为权重，这个是以某个条件为权重，比如按顶的次数排序，这时候就需要我们的sorted set出马了，将你要排序的值设置成sorted set的score，将具体的数据设置成相应的value，每次只需要执行一条ZADD命令即可。 热门，排行榜应用： //将登录次数和用户统一存储在一个sorted set里zadd login:login_times 5 1zadd login:login_times 1 2zadd login:login_times 2 3//当用户登录时，对该用户的登录次数自增1ret = r.zincrby("login:login_times", 1, uid)//那么如何获得登录次数最多的用户呢，逆序排列取得排名前N的用户ret = r.zrevrange("login:login_times", 0, N-1) 另一个很普遍的需求是各种数据库的数据并非存储在内存中，因此在按得分排序以及实时更新这些几乎每秒钟都需要更新的功能上数据库的性能不够理想。典型的比如那些在线游戏的排行榜，比如一个Facebook的游戏，根据得分你通常想要： - 列出前100名高分选手 - 列出某用户当前的全球排名 这些操作对于Redis来说小菜一碟，即使你有几百万个用户，每分钟都会有几百万个新的得分。模式是这样的，每次获得新得分时，我们用这样的代码： ZADD leaderboard 你可能用userID来取代username，这取决于你是怎么设计的。得到前100名高分用户很简单： ZREVRANGE leaderboard 0 99 用户的全球排名也相似，只需要： ZRANK leaderboard 3、删除与过滤 我们可以使用LREM来删除评论。如果删除操作非常少，另一个选择是直接跳过评论条目的入口，报告说该评论已经不存在。 有些时候你想要给不同的列表附加上不同的过滤器。如果过滤器的数量受到限制，你可以简单的为每个不同的过滤器使用不同的Redis列表。毕竟每个列表只有5000条项目，但Redis却能够使用非常少的内存来处理几百万条项目。 4、按照用户投票和时间排序 排行榜的一种常见变体模式就像Reddit或Hacker News用的那样，新闻按照类似下面的公式根据得分来排序：score = points / time^alpha 因此用户的投票会相应的把新闻挖出来，但时间会按照一定的指数将新闻埋下去。下面是我们的模式，当然算法由你决定。 模式是这样的，开始时先观察那些可能是最新的项目，例如首页上的1000条新闻都是候选者，因此我们先忽视掉其他的，这实现起来很简单。每次新的新闻贴上来后，我们将ID添加到列表中，使用LPUSH + LTRIM，确保只取出最新的1000条项目。 有一项后台任务获取这个列表，并且持续的计算这1000条新闻中每条新闻的最终得分。计算结果由ZADD命令按照新的顺序填充生成列表，老新闻则被清除。这里的关键思路是排序工作是由后台任务来完成的。 5、处理过期项目 另一种常用的项目排序是按照时间排序。我们使用unix时间作为得分即可。 模式如下： - 每次有新项目添加到我们的非Redis数据库时，我们把它加入到排序集合中。这时我们用的是时间属性，current_time和time_to_live。 - 另一项后台任务使用ZRANGE…SCORES查询排序集合，取出最新的10个项目。如果发现unix时间已经过期，则在数据库中删除条目。 6、计数 Redis是一个很好的计数器，这要感谢INCRBY和其他相似命令。我相信你曾许多次想要给数据库加上新的计数器，用来获取统计或显示新信息，但是最后却由于写入敏感而不得不放弃它们。 好了，现在使用Redis就不需要再担心了。有了原子递增（atomic increment），你可以放心的加上各种计数，用GETSET重置，或者是让它们过期。例如这样操作： INCR user: EXPIRE 你可以计算出最近用户在页面间停顿不超过60秒的页面浏览量，当计数达到比如20时，就可以显示出某些条幅提示，或是其它你想显示的东西。 7、特定时间内的特定项目 另一项对于其他数据库很难，但Redis做起来却轻而易举的事就是统计在某段特点时间里有多少特定用户访问了某个特定资源。比如我想要知道某些特定的注册用户或IP地址，他们到底有多少访问了某篇文章。每次我获得一次新的页面浏览时我只需要这样做： SADD page:day1: 当然你可能想用unix时间替换day1，比如time()-(time()%3600*24)等等。 想知道特定用户的数量吗？只需要使用 SCARD page:day1: 需要测试某个特定用户是否访问了这个页面？ SISMEMBER page:day1: 8、查找某个值所在的区间(区间无重合) ：(Sorted Set) 例如有下面两个范围，10－20和30－40 A_start 10, A_end 20 B_start 30, B_end 40 我们将这两个范围的起始位置存在Redis的Sorted Sets数据结构中，基本范围起始值作为score，范围名加start和end为其value值： redis 127.0.0.1:6379>

Zadd ranges 10 A_start (integer) 1redis 127.0.0.1 1redis 6379 > zadd ranges 20 A_end (integer) 1redis 127.0.0.1 1redis 127.0.0.1 1redis > zadd ranges 40 B_end (integer) 1

In this way, after the data is inserted into the Sorted Sets, it is equivalent to arranging these starting positions in order. Now I need to find out which range the value of 15 is in, and I only need to do the following zrangbyscore lookup:

Redis 127.0.0.1 inf LIMIT 6379 > zrangebyscore ranges (15 + inf LIMIT 0 11) "A_end"

This command means to look for the first value greater than 15 in Sorted Sets. (+ inf stands for positive infinity in Redis, and parentheses before 15 indicate > 15 rather than > = 15) the result of the search is A_end, and since all values are arranged in order, 15 can be determined to be in the range from A_start to A_end, that is to say, 15 is in the range of A. So far, it's done.

9. Intersection, Union, difference: (Set)

/ / book tables store book names set book:1:name "The Ruby Programming Language" set book:2:name "Ruby on rail" set book:3:name "Programming Erlang" / / tag tables use collections to store data, because collections are good at finding intersections, merging sadd tag:ruby 1sadd tag:ruby 2sadd tag:web 2sadd tag:erlang 3gambles / books that belong to both ruby and web? Inter_list = redis.sinter ("tag.web", "tag:ruby") / / that is, books that belong to ruby but not to web? Inter_list = redis.sdiff ("tag.ruby", "tag:web") / / A collection of books belonging to ruby and web? Inter_list = redis.sunion ("tag.ruby", "tag:web") the main application scenarios of redis are shared here. I hope the above content can be helpful to you and learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.