Shulou(Shulou.com)06/02 Report--

Today, I will talk to you about how to carry out in-depth analysis of Redis. Many people may not know much about it. In order to make you understand better, the editor has summarized the following contents for you. I hope you can get something according to this article.

0. Foundation: 1) when the length of a string is less than 1m, the capacity expansion will double the existing space. If it exceeds 1m, only 1m more space will be expanded at a time. It should be noted that the maximum length of the string is 512m. 2) if the value value is an integer, you can also increment it. Self-increment has a range, its range is the maximum and minimum value of signed long, beyond this value, Redis will report an error, the maximum value 9223372036854775807 3) the list of Redis is equivalent to LinkedList in Java language, note that it is a linked list rather than an array. The list structure of Redis is often used as an asynchronous queue. The task structure that needs to be deferred is serialized into a string into a list of Redis from which another thread polls the data for processing. Right in, left out-> queue right in right out-> stack if you go a little deeper, you will find that the underlying storage of Redis is not a simple linkedlist, but a structure called fast linked list quicklist. First of all, a contiguous piece of memory storage is used when there are fewer list elements, and this structure is ziplist, that is, compressed list. It stores all the elements next to each other and allocates a continuous piece of memory. When the amount of data is relatively large, it will be changed to quicklist. Because the additional pointer space required by an ordinary linked list is too large, it will waste space and aggravate the fragmentation of memory. For example, this list only stores data of type int, and structurally requires two additional pointers prev and next. So Redis combines linked lists with ziplist to form quicklist. That is, multiple ziplist are strung together with two-way pointers. This not only meets the fast insertion and deletion performance, but also does not have too much spatial redundancy. 4) compressed list ziplist is a sequential data structure developed to save memory, which is used as one of the underlying implementations of list keys and hash keys. 5) the value of Hash,Redis dictionary can only be a string, and Redis adopts progressive Rehash strategy. 6) the collection of set,Redis is equivalent to HashSet in Java language, the internal key-value pair is unordered and unique, its internal implementation is equivalent to a special dictionary, and all value in the dictionary is a value NULL. The set structure can be used to store the ID of the winning user in the activity, because of the deduplication function, it can ensure that the same user will not win twice. 7) zset, which is similar to the combination of Java's SortedSet and HashMap, on the one hand, it is a set, which ensures the uniqueness of the internal value, on the other hand, it can give each value a score, representing the ranking weight of the value. Its internal implementation uses a data structure called a jump list. Zset can also be used to store students' scores, and the value is the student's ID,score is his test score. We can rank the grades according to their scores to get his ranking. Zset can be used to store fan lists, and the value value is the fan's user ID,score is the follow time. We can sort the fan list by follow time. / / https://yq.aliyun.com/articles/666398typedef struct zset {/ / Dictionary, the key is a member, the value is score / / used to support O (1) complexity by member operation dict * dict; / / jump table, sort members by score / / to support the average complexity of O (log N) to locate members by score operation / / and range operation zskiplist * zsl } zset;1, Qianfan Race: Redis distributed Lock 1) using setnx command to achieve distributed lock timeout? Redis's distributed locks cannot solve the timeout problem, and problems will arise if the logic between locking and releasing locks is executed so long that the timeout limit of the lock is exceeded. Because the lock expires, the second thread re-holds the lock, but then the first thread finishes executing the business logic and releases the lock, and the third thread gets the lock between the second thread's logic execution. Solution: 1. 1) Redis distributed locks should not be used for longer tasks. If it does happen occasionally, the wavelet confusion of data may need to be solved by human intervention. A more secure solution is to set the value parameter of the set instruction to a random number, and then delete the key after matching the random number when releasing the lock. But matching value and deleting key is not an atomic operation, which needs to be handled using Lua scripts, because Lua scripts can guarantee the atomic execution of multiple instructions in succession. 2) the defect of distributed locking implemented by stand-alone Redis, for example, in Sentinel cluster, when the master node dies, the slave node will take its place, but there is no obvious perception on the client. The first client successfully applied for a lock in the master node, but before the lock was synchronized to the slave node, the master node suddenly hung up. Then the slave node becomes the master node, and there is no lock inside the new node, so when another client comes to request a lock, it is approved immediately. This will cause the same lock in the system to be held by two clients at the same time, resulting in insecurity. However, this kind of insecurity only occurs in the case of master-slave failover, and the duration is very short, and the business system can tolerate it in most cases. Solution: when RedLock locks, it sends set (key, value, nx=True, ex=xxx) instructions to more than half of the nodes. As long as the set of more than half of the nodes is successful, it is considered to be successful. When you release the lock, you need to send a del instruction to all nodes. However, the Redlock algorithm also needs to consider many details, such as error retry, clock drift and so on. At the same time, because Redlock needs to read and write to multiple nodes, it means that the performance of Redis is lower than that of single instance. If you care about high availability and want to hang up a redis that will not be affected at all, then you should consider redlock. However, there are costs. More redis instances are needed, performance is degraded, additional library needs to be introduced in the code, and special treatment is needed in operation and maintenance. These are all costs that need to be considered. Please think twice before using them. 3) Lock conflict handling We talked about the problem of distributed locks, but did not mention what if the client failed to lock the request. There are generally three strategies to deal with locking failures: 3.1) throwing an exception directly and notifying the user to retry later is more suitable for requests initiated directly by the user. After seeing the error dialog box, the user will first read the contents of the dialog box and click to retry. In this way, it can have the effect of manual delay. If the user experience is taken into account, the front-end code can be used to delay the retry control instead of the user. It is essentially an abandonment of the current request, and it is up to the user to decide whether to re-initiate the new request. 3.2) sleep will try again later. It is not recommended to transfer the request to the delay queue. Try again later. This method is more suitable for asynchronous message processing, and throws the conflicting request to another queue to postpone processing to avoid conflict. 2, delay tactics: delay queue 1) Asynchronous message queue Redis message queue is not a professional message queue, it does not have a lot of advanced features, there is no ack guarantee, if you have the ultimate pursuit of message reliability, then it is not suitable to use. Redis's list (list) data structure is commonly used as an asynchronous message queue, using rpush/lpush operations to enter the queue and lpop and rpop to exit the queue. What if the queue is empty? But if the queue is empty, the client will fall into an endless loop of pop, constantly pop, no data, and then pop, no data. This is an empty poll that wastes life. Empty polling pulls up not only the CPU,redis QPS of the client, but also the QPS of the client. If there are dozens of empty polling clients, the slow query of Redis may increase significantly. Solution: 1.1) use sleep to solve this problem, let threads sleep for a while. 2) use blpop/brpop to block reads. When the queue has no data, it goes to sleep immediately, and wakes up as soon as the data arrives. The delay of the message is almost zero. What if the idle connection is automatically disconnected? Solution: if the thread has been blocked somewhere, the Redis client connection will become an idle connection, idle for too long, the server will generally take the initiative to disconnect, reducing the occupation of idle resources. At this point, blpop/brpop will throw an exception. So be careful when writing client consumers, catch exceptions, and try again. 2) delay queue delay queue can be implemented through zset (ordered list) of Redis. We serialize the message into a string as the value of zset, and the processing time of this message is taken as score, and then multiple threads are used to poll zset to get expired tasks for processing. Multiple threads are used to ensure availability, and there are other threads that can continue to process if one thread is hung up. Because there are multiple threads, you need to consider concurrent contention tasks to ensure that tasks cannot be executed multiple times.

/ / Redis implements delay queue public class RedisDelayingQueue {static class TaskItem {public String id; public T msg;} private Type TaskType = new TypeReference () {} .getType (); private Jedis jedis; private String queueKey; public RedisDelayingQueue (Jedis jedis, String queueKey) {this.jedis = jedis; this.queueKey = queueKey;} public void delay (T msg) {TaskItem taskItem = new TaskItem () TaskItem.id = UUID.randomUUID (). ToString (); taskItem.msg = msg; String s = JSON.toJSONString (taskItem); jedis.zadd (queueKey, System.currentTimeMillis () + 5000, s);} public void loop () {while (! Thread.interrupted ()) {/ / take only one piece of data Set values = jedis.zrangeByScore (queueKey,0,System.currentTimeMillis (), 0mem1) If (values.isEmpty ()) {try {Thread.sleep;} catch (InterruptedException e) {break;} continue;} String s = (String) values.iterator () .next () / / zrem is used to remove one or more members of an ordered set if (jedis.zrem (queueKey, s) > 0) {TaskItem task = JSON.parseObject (s, TaskType); this.handleMsg (task.msg);}} private void handleMsg (Object msg) {System.out.println (msg) } public static void main (String [] args) {Jedis jedis = new Jedis (); RedisDelayingQueue queue = new RedisDelayingQueue (jedis, "test-queue"); Thread producer = new Thread () {@ Override public void run () {for (int I = 0; I)

< 10; i++){ queue.delay("lwh" + i); } } }; Thread consumer = new Thread(){ @Override public void run() { queue.loop(); } }; producer.start(); consumer.start(); try { producer.join(); Thread.sleep(6000); consumer.interrupt(); consumer.join(); } catch (InterruptedException e) { e.printStackTrace(); } }}3、节衣缩食: 位图1) 位图 位图不是特殊的数据结构，它的内容其实就是普通的字符串，也就是 byte 数组。我们可以使用普通的 get/set 直接获取和设置整个位图的内容，也可以使用位图操作 getbit/setbit等将 byte 数组看成「位数组」来处理。 redis位图可以实现零存整取、零存零取等。「零存」就是使用 setbit 对位值进行逐个设置，「整存」就是使用字符串一次性填充所有位数组，覆盖掉旧值。 命令形如 1) setbit s 1 1 2) getbit s 3) get s 4) set s h 2) 统计和查找 Redis提供了位图统计指令bitcount和位图查找指令bitpos，bitcount用来统计指定位置范围内1的个数，bitpos用来查找指定范围内出现的第一个0或1。比如我们可以通过bitcount统计用户一共签到了多少天，通过bitpos指令查找用户从哪一天开始第一次签到。如果指定了范围参数[start,end]，就可以统计在某个时间范围内用户签到了多少天，用户自某天以后的哪天开始签到。 3) bitfield4、四两拨千斤: HyperLogLog1) HyperLogLog使用 统计PV：每个网页一个独立的Redis计数器 统计UV?解决办法：1.1) set去重，页面访问量大的情况下，耗费太多存储空间1.2) 使用HyperLogLog，不精确去重，标准误差0.81% HyperLogLog提供了两个指令pfadd和pfcount，根据字面意义很好理解，一个是增加计数，一个是获取计数。pfadd用法和set集合的sadd是一样的，来一个用户ID，就将用户ID塞进去就是。pfcount和scard用法是一样的，直接获取计数值。 pfadd test-log user1 pfadd test-log user2 pfcount test-log HyperLogLog 除了上面的pfadd和pfcount之外，还提供了第三个指令pfmerge，用于将多个pf计数值累加在一起形成一个新的pf值。比如在网站中我们有两个内容差不多的页面，运营说需要这两个页面的数据进行合并。其中页面的UV访问量也需要合并，那这个时候pfmerge就可以派上用场了。5、层峦叠嶂：布隆过滤器1) 布隆过滤器 讲个使用场景，比如我们在使用新闻客户端看新闻时，它会给我们不停地推荐新的内容，它每次推荐时要去重，去掉那些已经看过的内容。问题来了，新闻客户端推荐系统如何实现推送去重的？ 当布隆过滤器说某个值存在时，这个值可能不存在；当它说不存在时，那就肯定不存在。 基本指令：布隆过滤器有二个基本指令，bf.add添加元素，bf.exists查询元素是否存在，它的用法和set集合的sadd和sismember差不多。注意bf.add只能一次添加一个元素，如果想要一次添加多个，就需要用到bf.madd指令。同样如果需要一次查询多个元素是否存在，就需要用到bf.mexists指令。 bf.add test-filter user1 bf.add test-filter user2 bf.exists test-filter user1 Redis其实还提供了自定义参数的布隆过滤器，需要我们在add之前使用bf.reserve指令显式创建。如果对应的 key已经存在，bf.reserve会报错。bf.reserve有三个参数，分别是key,error_rate和initial_size。错误率越低，需要的空间越大。initial_size参数表示预计放入的元素数量，当实际数量超出这个数值时，误判率会上升。2) 布隆过滤器的原理 每个布隆过滤器对应到Redis的数据结构里面就是一个大型的位数组和几个不一样的无偏hash函数。所谓无偏就是能够把元素的hash值算得比较均匀。向布隆过滤器中添加key时，会使用多个hash函数对 key进行hash算得一个整数索引值然后对位数组长度进行取模运算得到一个位置，每个hash函数都会算得一个不同的位置。再把位数组的这几个位置都置为1就完成了add操作。3) 布隆过滤器的其他应用 在爬虫系统中，我们需要对URL进行去重，已经爬过的网页就可以不用爬了。但是URL太多了，几千万几个亿，如果用一个集合装下这些URL地址那是非常浪费空间的。这时候就可以考虑使用布隆过滤器。它可以大幅降低去重存储消耗，只不过也会使得爬虫系统错过少量的页面。布隆过滤器在NoSQL数据库领域使用非常广泛，我们平时用到的HBase、Cassandra还有LevelDB、RocksDB内部都有布隆过滤器结构，布隆过滤器可以显著降低数据库的IO请求数量。当用户来查询某个row时，可以先通过内存中的布隆过滤器过滤掉大量不存在的row请求，然后再去磁盘进行查询。 邮箱系统的垃圾邮件过滤功能也普遍用到了布隆过滤器，因为用了这个过滤器，所以平时也会遇到某些正常的邮件被放进了垃圾邮件目录中，这个就是误判所致，概率很低。6、断尾求生：简单限流 除了控制流量，限流还有一个应用目的是用于控制用户行为，避免垃圾请求。比如在UGC社区，用户的发帖、回复、点赞等行为都要严格受控，一般要严格限定某行为在规定时间内允许的次数，超过了次数那就是非法行为。对非法行为，业务必须规定适当的惩处策略。

/ / there is a sliding time window in this current restriction requirement. Think about whether the score value of the zset data structure can be circled by score. And we only need to keep this time window, and all the data outside the window can be cut off. What is more appropriate to fill in the value of this zset? It only needs to be unique. Using uuid will waste / / space, so use millisecond timestamps instead. / / but this scheme also has shortcomings, because it records all the behavior records in the time window. If this amount is large, for example, no more than 100w operations within 60s, it / / is not suitable for such a current limit, because it consumes a lot of storage space. Public class SimpleRateLimiter {private final Jedis jedis; public SimpleRateLimiter (Jedis jedis) {this.jedis = jedis;} public boolean isActionAllowed (String userId, String actionKey, int period, int maxCount) throws IOException {String key = String.format ("hist:%s:%s", userId, actionKey); long nowTs = System.currentTimeMillis (); Pipeline pipeline = jedis.pipelined (); / / Open a transaction pipeline.multi () / / both value and score timestamp pipeline.zadd (key, nowTs, "" + nowTs) in milliseconds; / / remove the behavior records outside the time window, and the rest are pipeline.zremrangeByScore in the time window (key, 0, nowTs-period * 1000); / / obtain the number of key of [nowTs-period * 1000, nowTs] Response count = pipeline.zcard (key) / / update the expiration time pipeline.expire (key, period) of key every time you set it; / / execute the above command pipeline.exec (); pipeline.close () in the transaction Return count.get () returns cursor13796 as the cursor returned by the next traversal of the cursor scan 13976 match key99* count 1000 scan instruction is the position index of the first dimension array, which we call a slot. If you do not consider the expansion and reduction of the dictionary, just traverse one by one according to the array subscript. The limit parameter indicates the number of slots that need to be traversed. The reason why the results returned may be more or less is that linked lists may not be attached to all slots, some slots may be empty, and some slots may have multiple elements on the linked list. Each traversal will return all the linked list elements attached to the slot of the number of limit to the client at once after pattern matching filtering. The traversal order of scan is very special. Instead of traversing from bit 0 of the first dimension array to the end, it uses high carry addition to traverse. The reason for using such a special way to traverse is to avoid repetition and omission of traversing slots when expanding and reducing the capacity of the dictionary. In the usual business development, we should try to avoid the generation of large key. Sometimes because of improper use by business people, large objects are formed in Redis instances, such as a large hash and a large zset. Such objects bring great problems to the cluster data migration of Redis, because in a cluster environment, if a key is too large, the data will lead to migration stutters. In addition, in terms of memory allocation, if a key is too large, it will apply for a larger piece of memory at one time when it needs to expand, which will also lead to stutters. If the big key is deleted, the memory will be recycled at one time, and the stutter phenomenon will occur again. However, Redis officials have provided a large key scan function in the redis-cli instruction. The second instruction hibernates 0.1s scan every 100 instructions so that it doesn't rise sharply, but it takes longer to scan. Redis-cli-h 127.0.0.1-p 7001-- bigkeys redis-cli-h 127.0.0.1-p 7001-- bigkeys-I 0.1

10.Thread IO model 1) the timed task server handles other things in addition to responding to IO events. For example, scheduled tasks are very important. If the thread blocks on the select system call, the timed task will not be scheduled on time. So how does Redis solve this problem? Redis's scheduled tasks are recorded in a data structure called the minimum heap. In this heap, the fastest task is at the top of the heap. In each cycle, Redis immediately processes the tasks that have arrived in the smallest heap. After processing, record the time required for the fastest task to be executed, which is the timeout parameter of the select system call. Because Redis knows that there are no other scheduled tasks to deal with in the future timeout time, it can rest assured that it is time to sleep timeout. The event handling principle of Nginx and Node is similar to that of Redis. 11. Whisper: the author of the communication protocol Redis believes that the bottleneck of the database system generally does not lie in the network traffic, but in the internal logic processing of the database itself. So even if Redis uses a text protocol that wastes traffic, it can still achieve extremely high access performance. RESP (Redis Serialization Protocol). RESP is short for Redis serialization protocol. It is an intuitive text protocol, which has the advantages of simple implementation and excellent parsing performance. 12. Prepare in advance: when the parent process modifies the data of one of the pages, it will copy a copy of the shared page and separate it, and then modify the copied page. At this time, the corresponding page of the child process does not change, or the data at the moment when the process is generated. Because the data of the child process does not change, the data it can see in memory is frozen in the moment the process is generated, and will never change again, which is why the persistence of Redis is called "snapshot". Next, the child process can traverse the data with great peace of mind and serialize the write disk. When restarting Redis with Redis4.0 mixed persistence, we rarely use rdb to restore the memory state because a large amount of data is lost. We usually use AOF log playback, but the performance of replaying AOF logs is much slower than rdb, so it takes a long time to start when the Redis instance is large. To solve this problem, Redis4.0 has introduced a new persistence option-hybrid persistence. Store the contents of the rdb file with the incremental AOF log file. The AOF log here is no longer a full log, but an incremental AOF log that occurs during the period from the beginning of the persistence to the end of the persistence, which is usually very small. Therefore, when Redis is restarted, the contents of rdb can be loaded first, and then the incremental AOF logs can be replayed, which can completely replace the previous AOF full file replay, thus greatly improving the restart efficiency. 13. Open source and reduce expenditure: small object compression 1) small object compression if the collection data structure managed by Redis is very small, it will use compact storage to compress storage. If it stores a hash structure, then key and value will exist as two entry adjacent to each other. If it stores zset, then value and score exist as two entry adjacent to each other. Storage limits when the number of elements of a collection object increases, or when a value value is too large, this small object storage is also upgraded to a standard structure. 2) the memory recovery mechanism Redis does not always return free memory to the operating system immediately. If the current Redis memory has 10 gigabytes, when you delete the key of 1GB, and then look at the memory, you will find that the memory will not change much. The reason is that the operating system reclaims memory in pages, and if there is as long as a key on this page is still in use, it cannot be reclaimed. Redis removes 1GB's key, but the key is scattered across many pages, and there is another key on each page, which results in memory that is not immediately reclaimed. However, if you execute flushdb and then look at the memory, you will find that the memory is indeed reclaimed. The reason is that all the key has been killed, and most of the previously used pages are completely clean and will be immediately recycled by the operating system. Although Redis cannot guarantee that key memory that has been deleted will be reclaimed immediately, it will reuse free memory that has not been reclaimed. This is like in the cinema, although people have left, but the seats are still there, the next wave of audience is coming, just sit down. The operating system reclaiming memory is like moving all the seats away. Is this metaphor very 6? 14. Precaution: master-slave synchronization 1) CAP theory C:Consistent, consistency A:Availbility, availability P:Partition tolerance, partition tolerance distributed distributed systems nodes are often distributed on different machines for network isolation, which means that there is bound to be the risk of network disconnection. The technical term for this network disconnection scenario is "network partition". When the network partition occurs, there is no communication between the two distributed nodes, and our modifications to one node will not be synchronized to the other, so the "consistency" of the data will not be satisfied. because the data of the two distributed nodes are no longer consistent. Unless we sacrifice the "availability", that is, suspend the distributed node service, when the network partition occurs, we will no longer provide the function to modify the data, and continue to provide services until the network condition fully returns to normal. In a word, the principle of CAP is that when network partitions occur, there is a dilemma between consistency and availability. After reading the above, do you have any further understanding of how to conduct Redis in-depth analysis? If you want to know more knowledge or related content, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.