Shulou(Shulou.com)05/31 Report--

How to solve the Redis problem, many novices are not very clear, in order to help you solve this problem, the following editor will explain for you in detail, people with this need can come to learn, I hope you can gain something.

Redis is essentially an in-memory database of Key-Value type. The whole database is loaded and operated in memory, and the database data is saved to the hard disk regularly through asynchronous operation.

Because of its pure memory operation, Redis has excellent performance, which can handle more than 100000 read and write operations per second, which is a known performance.

The fastest Key-Value DB.

The excellence of Redis is not just about performance, the greatest charm of Redis is that it supports the preservation of multiple data structures, in addition to a single

The maximum limit of value is 1GB, unlike memcached, which can only save 1MB data, so Redis can use the

To achieve many useful functions, such as using his List to do a FIFO two-way linked list, to achieve a lightweight high performance

Message queuing service, using its Set can do high-performance tag system and so on.

In addition, Redis can also set the expire time for the stored Key-Value, so it can also be used as an enhanced version of memcached.

The main disadvantage of Redis is that the database capacity is limited by physical memory and cannot be used for high-performance reading and writing of massive data, so the suitable scenarios for Redis are mainly limited to high-performance operations and operations with a small amount of data.

What are the advantages over memcached?

(1) all values of memcached are simple strings, and Redis, as its replacement, supports richer data types.

(2) Redis is much faster than memcached.

(3) Redis can persist its data

What is the full name of Redis?

Remote Dictionary Server .

Which data types are supported?

String 、 List 、 Set 、 Sorted Set 、 hashes

What kinds of data elimination strategies does Redis have?

Hongmeng official Strategic Cooperation to build HarmonyOS Technology Community

Noeviction: returns an error when the memory limit is reached and the client tries to execute commands that allow more memory to be used (most write instructions, but DEL and a few exceptions)

Allkeys-lru: try to recycle the least used key (LRU) so that the newly added data has room to store.

Volatile-lru: try to recycle the least used keys (LRU), but only those keys in expired collections, so that newly added data has room to store.

Allkeys-random: reclaim random keys to make room for newly added data.

Volatile-random: reclaiming random keys gives room for newly added data, but only for keys in expired collections.

Volatile-ttl: reclaim keys in expired collections, and give priority to keys with short TTL, so that newly added data has room to store.

Why does Redis use a jump table instead of a red-black tree?

When doing range lookups, balanced trees are more complex than skiplist operations. On the balanced tree, after we find the small value of the specified range, we need to continue to look for other nodes that do not exceed the large value in the order of traversing in the middle order. If the balanced tree is not modified, the mid-order traversal here is not easy to achieve. Range lookup on skiplist is very simple, as long as you find a small value and traverse the layer 1 linked list in several steps.

The insertion and deletion of the balanced tree may lead to the adjustment of the subtree, which is logically complex, while the insertion and deletion of the skiplist only need to modify the pointers of the adjacent nodes, which is simple and fast.

In terms of memory footprint, skiplist is more flexible than balanced trees. Generally speaking, each node of the balanced tree contains two pointers (pointing to the left and right subtrees respectively), while the average number of pointers per node of the skiplist is 1 / (1murp), depending on the size of the parameter p. If, as in the implementation in Redis, we take pendant 1Universe 4, then on average each node contains 1.33 pointers, which is more advantageous than the balanced tree.

The time complexity of searching a single key,skiplist and a balanced tree is O (log n), which is about the same, while the hash table has a lower hash value conflict probability, the search time complexity is close to O (1), and the performance is higher. So most of the Map or dictionary structures we use are based on hash tables.

In terms of the difficulty of implementing the algorithm, skiplist is much simpler than balanced tree.

Can you introduce HyperLogLog?

HyperLogLog is a probabilistic data structure used to estimate the cardinality of data. The dataset can be the IP address of the site visitor, the E-mail mailbox, or the user ID.

The cardinality refers to the number of different values in a set, for example, the cardinality of a, b, c, d is 4, the cardinality of a, b, c, d, an is still 4. Although an appears twice, it will only be counted once.

There are generally three ways to count the cardinality of a collection using Redis, which is HashMap,BitMap and HyperLogLog using Redis. As the first two data structures grow by the order of magnitude of the collection, the memory consumed increases significantly, but HyperLogLog does not.

Redis's HyperLogLog reduces memory space consumption at the expense of accuracy, requiring only 12K of memory, and can count 2 ^ 64 data with a standard error of 0.81%. So whether HyperLogLog is suitable for scenarios with low accuracy, such as counting the number of daily active users and monthly active users.

This is an amazing result to record such a large amount of data cardinality with such a small amount of memory.

Why does Redis need to put all the data in memory?

In order to achieve the fastest reading and writing speed, Redis reads all the data into memory and writes the data to disk asynchronously.

So Redis has the characteristics of high speed and data persistence. If you do not put the data in memory, the speed of the disk Icando O is serious.

Affect the performance of Redis. Today, when memory is getting cheaper and cheaper, Redis will become more and more popular.

What data types are supported by Redis?

String string:

Format: set key value

The string type is binary safe. It means that the string of redis can contain any data. Such as jpg images or serialized objects.

String type is the most basic data type of Redis, and a key can store 512MB as much as possible.

Hash (hash)

Format: hmset name key1 value1 key2 value2

Redis hash is a collection of key-value pairs (key= > value).

Redis hash is a mapping table for field and value of type string, and hash is particularly suitable for storing objects.

List (list)

The Redis list is a simple list of strings sorted in the order in which they are inserted. You can add an element to the head (left) or tail (right) of the list.

Format: lpush name value

Add a string element to the header of the corresponding list of key

Format: rpush name value

Add a string element to the tail of the corresponding list of key

Format: lrem name index

Delete count and value elements from list corresponding to key.

Format: llen name

Returns the length of the list corresponding to key

Set (collection)

Format: sadd name value

The Set of Redis is an unordered collection of type string.

The collection is implemented through a hash table, so the complexity of adding, deleting, and finding is all O (1).

Zset (sorted set: ordered set)

Format: zadd name score value

Redis zset, like set, is a collection of elements of type string, and duplicate members are not allowed.

The difference is that each element is associated with a score of type double. Redis sorts the members of the collection from small to large by scores.

The members of the zset are unique, but the score can be repeated.

What is the improvement of sds over c?

Get length: C string does not record its own length, so getting length can only traverse the string once, and redis can read len directly.

Buffer security: C strings can easily cause buffer overflows, such as programmers performing stitching operations without allocating enough space. Redis will first check whether the space of the sds meets the required requirements, and will automatically expand if it does not.

Memory allocation: because c does not record the length of the string, for a string containing n characters, the bottom layer is always an array of length nim1, and every time the length changes, there is always a memory reallocation for this array. Because memory allocation involves complex algorithms and may require system calls, it is usually a time-consuming operation.

Redis linked list source code? What are the characteristics?

Double-ended, acyclic, with length recording,

Polymorphism: use void* pointers to save node values. You can set type-specific functions for node values through dup, free, and match, and you can save different types of values.

How is the dictionary implemented?

In fact, dictionaries are built into many high-level languages, but c doesn't have them, so redis implements them on its own.

It is also widely used, for example, the database of redis is realized by dictionary. Not only that, when a hash key contains many key-value pairs, or are very long strings, redis will use a dictionary as the underlying implementation of the hash key.

LRU? The concrete implementation in Redis?

The full name of LRU is Least Recently Used, which means that it has not been used for the longest time recently.

The design principle of the LRU algorithm is that if a data has not been accessed recently, it is unlikely to be accessed in the future. In other words, when the limited space is full of data, the data that has not been accessed for the longest time should be eliminated.

Redis's original elimination algorithm is simple to implement: when you need to eliminate a key, randomly select 3 key to eliminate the key with the longest interval. * * basically, we randomly select key to eliminate key. Later, three random key were changed into a configuration item "N random key". However, the effect is greatly improved after changing the default value to 5. Considering its effect, you don't have to modify it at all.

Persistence of Redis?

Hongmeng official Strategic Cooperation to build HarmonyOS Technology Community

RDB persistence can be performed manually or periodically, and the data state at a certain time can be saved to a RDB file. On the contrary, we can restore the database state with a RDB file.

AOF persistence records status by saving commands executed by the server. You can do it again when you restore it.

How to choose the appropriate persistence method?

In general, if you want to achieve data security comparable to PostgreSQL, you should use both persistence

Chemical function. If you are very concerned about your data, but can still withstand data loss within minutes, then you can

Use only RDB persistence.

Many users only use AOF persistence, but this is not recommended: because RDB snapshots are generated on a regular basis

(snapshot) is very convenient for database backup, and RDB restores datasets faster than AOF does.

In addition, using RDB can also avoid the bug of the AOF program mentioned earlier.

What should the Redis cluster scheme do? What are the plans?

The general concept of 1.twemproxy is that it is similar to a proxy method, and there is no difference between it and ordinary Redis.

After setting up multiple Redis instances to which it belongs, it can be changed to a connection where the Redis should be connected.

Twemproxy, which receives requests as an agent and uses the consistent hash algorithm to transfer requests to the

Body Redis, and return the result to twemproxy. Easy to use (only need to modify the connection port relative to Redis), for

The first choice for old project expansion. Problem: the pressure of twemproxy's own single-port instance, after using consistent hash, to

With the change of the calculated value when the number of Redis nodes changes, the data cannot be moved to the new node automatically.

2. Codis, the most widely used cluster scheme at present, has the same effect as twemproxy, but it supports nodes.

When the quantity changes, the data of the old node can be restored to the new hash node.

3. Redis cluster3.0 's own cluster, which is characterized by that his distributed algorithm is not consistent hash, but hash.

The concept of slot, as well as its own support node setting slave node. Take a look at the official document introduction.

4. In the business code layer, several unrelated Redis instances are implemented, and in the code layer, the hash calculation of key is performed.

Then go to the corresponding Redis instance to manipulate the data. This approach requires a high level of hash code, and some of the considerations include

Alternative algorithm after node failure, automatic script recovery after data concussion, instance monitoring, and so on.

There are 2000w data in MySQL and only 20w in Redis.

How to ensure that the data in Redis is hot data?

When the Redis in-memory dataset size rises to a certain size, the data elimination strategy will be implemented.

What are the suitable scenarios for Redis?

(1), session cache (Session Cache)

One of the most common scenarios where Redis is used is session caching (session cache). Caching sessions with Redis is better than others

The advantage of storage (such as Memcached) is that Redis provides persistence. When maintaining a consistency that is not strictly required

If all the users' shopping cart information is lost, most people will be unhappy, and now they will continue to do so.

Is it?

Fortunately, as Redis has improved over the years, it is easy to find text on how to properly use Redis to cache sessions

File. Even the well-known business platform Magento provides plug-ins for Redis.

(2), full-page cache (FPC)

In addition to the basic session token, Redis provides a very simple FPC platform. Back to the consistency issue, even if you restart

Redis instance, because of the persistence of the disk, users will not see the decline in page loading speed, which is an extreme

Big improvement, similar to PHP native FPC.

Again, take Magento as an example. Magento provides a plug-in to use Redis as the full-page cache backend.

In addition, for WordPress users, Pantheon has a very good plug-in wp-Redis, which is

It can help you load the pages you have visited as quickly as possible.

(3), queue

One of the advantages of Reids in the field of memory storage engines is that it provides list and set operations, which enables Redis to act as a

A good message queuing platform to use. The operation used by Redis as a queue is similar to the native programming language, such as

Python) push/pop operation on list.

If you quickly search for "Redis queues" in Google, you will find a large number of open source projects right away.

The goal of the project is to use Redis to create very good back-end tools to meet a variety of queue needs. For example, Celery

There is a background that uses Redis as broker, which you can check from here.

(4), ranking / counter

Redis does a great job of incrementing or decrementing numbers in memory. Set and ordered set (Sorted)

Set) also makes it very easy for us to perform these operations. Redis just happens to provide these two kinds of data.

structure. So, we need to get the top 10 users from the sorted set-- we call it

"user_scores", we just need to execute as follows:

Of course, this assumes that you are sorting incrementally according to the scores of your users. If you want to return the user and the user's score, you

You need to do this:

ZRANGE user_scores 0 10 WITHSCORES

Agora Games is a good example, implemented in Ruby, and its ranking is to use Redis to store

Data, you can see it here.

(5), publish / subscribe

Finally, there is the publish / subscribe function of Redis. The usage scenario of publish / subscribe is really not

Changduo. I've seen people use it in social networking connections, as a publish / subscribe-based script trigger, or even

Use the publish / subscribe function of Redis to build a chat system.

Tell me about the concept of Redis hash slot?

Redis cluster does not use consistent hash, but introduces the concept of hash slot. Redis cluster has 16384 hash slots.

Each key passes the CRC16 check and takes the module 16384 to decide which slot to place, and each node of the cluster is responsible for part of it.

Hash slot

Why does the Redis cluster have 16384 slots

(1) if the slot is 65536, the message header for sending heartbeat information reaches 8k, and the heartbeat packet sent is too large.

As mentioned above, the one that takes up the most space in the header is myslots [cluster _ SLOTS/8]. When the slot is 65536, the size of this block is: 65536 / 8 / 1024=8kb because every second, the redis node needs to send a certain number of ping messages as heartbeats. If the slot is 65536, the header of this ping message is too large and a waste of bandwidth.

(2) the number of cluster master nodes in redis can hardly exceed 1000.

As mentioned above, the more nodes in the cluster, the more data in the message body of the heartbeat packet. If there are more than 1000 nodes, it will also cause network congestion. Therefore, the redis author does not recommend that the number of redis cluster nodes exceed 1000. Then, for redis cluster clusters with less than 1000 nodes, 16384 slots will be sufficient. There is no need to expand to 65536.

(3) the smaller the slot and fewer nodes, the higher the compression ratio.

In the configuration information of the master node of Redis, the hash slot it is responsible for is saved in the form of a bitmap. In the process of transmission, bitmap will be compressed, but if the filling ratio of bitmap slots / N is very high (N represents the number of nodes), the compression ratio of bitmap is very low. If the number of nodes is small and the number of hash slots is large, the compression ratio of bitmap is very low.

Will any write operations be lost in the Redis cluster? Why?

Redis does not guarantee strong consistency of data, which means that in practice, clusters may lose write operations under certain conditions.

Do.

What should the Redis cluster scheme do? What are the plans?

The general concept of 1.twemproxy is that it is similar to a proxy method, which is used to connect to the redis where it is needed to connect to the twemproxy. It receives the request as an agent and uses the consistent hash algorithm to transfer the request to a specific redis and return the result to twemproxy.

Disadvantages: due to the pressure of twemproxy's own single-port instance, after using consistent hash, the data cannot be moved to the new node automatically due to the change of the calculated value when the number of redis nodes is changed.

2.codis, the most widely used cluster scheme at present, has the same effect as twemproxy, but it supports the recovery of old node data to new hash nodes when the number of nodes changes.

3.redis cluster3.0 's own cluster is characterized by the fact that his distributed algorithm is not consistent hash, but the concept of hash slots, as well as its own support nodes to set slave nodes. Take a look at the official document introduction.

Why do you want to do Redis partition?

Partitioning will allow Redis to manage more memory, and Redis will be able to use the memory of all machines. If there is no partition, you

You can only use the memory of one machine at most. Partitioning multiplies the computing power of Redis by simply increasing the computer.

The network bandwidth of Redis will also increase exponentially with the increase of computers and network cards.

What are the disadvantages of Redis partitioning?

Operations that involve multiple key are usually not supported. For example, you can't intersect two sets because they may be saved.

Save to different Redis instances (in fact, there is a way to do this, but you cannot use the intersection instruction directly).

If you operate on multiple key at the same time, you cannot use Redis transactions.

Partitions use a granularity of key and cannot use a very long sorted key to store a dataset (The partitioning

Granularity is the key, so it is not possible to shard a dataset with a single huge

Key like a very big sorted set).

When using partitions, data processing can be very complex, for example, in order to back up, you have to get data from different Redis instances and hosts

The computer also collects RDB / AOF files.

Dynamic capacity expansion or reduction during partitioning may be very complex. Redis cluster adds or deletes Redis nodes at run time, and can

Achieve maximum transparent data rebalancing to users, but some other client or proxy partitioning methods do not support

This trait. However, there is a pre-slicing technology that can also solve this problem.

How is Redis different from other key-value stores?

Redis has more complex data structures and provides atomic operations on them, which is different from other databases

The evolutionary path. The data types of Redis are based on basic data structures while being transparent to programmers without additional requirements.

The abstraction.

Redis runs in memory but can be persisted to disk, so there is a tradeoff when reading and writing different datasets at high speed.

Memory, the amount of data should not be larger than the hardware memory. Another advantage of in-memory databases is that compared to on disk

The same complex data structure is very simple to operate in memory, so Redis can do a lot of internal complexity.

It's a strong thing. At the same time, they are compact and appended in terms of disk format, because they do not need

To have random access.

What happens when Redis runs out of memory?

If the upper limit is reached, Redis's write command returns an error message (but the read command returns normally. ) or

You can use Redis as a cache to use the configuration elimination mechanism, and the old content will be washed out when the Redis reaches the upper limit of memory.

Redis is single-threaded, how to improve the utilization of multicore CPU?

You can deploy multiple instances of Redis on the same server and use them as different servers, at some point

Wait, anyway, one server is not enough.

So, if you want to use multiple CPU, you can consider shard.

How much keys can be stored in a Redis instance? List, Set, Sorted Set, how many elements can they store at most?

Redis can handle up to 232 keys in theory and has been tested in practice, with at least 250 million keys stored in each instance. We are testing some larger values.

Any list, set, and sorted set can put 232 elements.

In other words, the storage limit of Redis is the available memory values in the system.

Will it take effect in real time if I modify the configuration and do not restart Redis?

For a running instance, there are many configuration options that can be modified through the CONFIG SET command without performing any

A formal restart. Starting with Redis 2.2, you can switch from AOF to RDB snapshot persistence or other ways

There is no need to restart Redis. Retrieve the 'CONFIG GET *' command for more information.

But an occasional restart is necessary, such as to upgrade the Redis program to a new version, or when you need to modify some current

When a configuration parameter that is not supported by the CONFIG command

Sentinel

Redis sentinel is a distributed system that monitors redis master and slave servers and automatically fails over when the master server goes offline. Three of these features:

Monitoring (Monitoring): Sentinel will constantly check whether your master server and slave server are working properly.

Reminder (Notification): when there is a problem with a monitored Redis server, Sentinel can send a notification to the administrator or other application through API.

Automatic failover (Automatic failover): when a primary server does not work properly, Sentinel starts an automatic failover operation.

Features:

1. Ensure high availability

2. Monitor each node

3. Automatic failover

Disadvantages: master-slave mode, switching takes time to lose data

Did not solve the pressure of master writing

Cache penetration

In general, caching systems cache queries according to key. If there is no corresponding value, go to the back-end system to find it (such as DB).

Some malicious requests will deliberately query the key that does not exist, and the number of requests is large, which will put a lot of pressure on the back-end system. This is called cache traversal.

How to avoid it?

1: the case where the query result is empty is also cached, so that when accessed again, the cache layer will directly return a null value. The cache time is set to be shorter, or the cache is cleaned after the data corresponding to the key has been insert.

2: filter key that must not exist. For details, please see the Bloom filter.

Cache breakdown

It is for data that is not in the cache but in the database.

The scenario is that when the Key expires, if a large number of requests suddenly pour in to request the same Key, these requests will not hit the Redis, but will be requested to the DB, resulting in too much pressure on the database, or even hang up.

Solution.

1. Set the hotspot Key, automatically detect the hotspot Key, and increase the expiration time of the hotspot Key to never expire, or logically never expire.

2. Add mutex. When it is found that the Redis has not been hit, when the database is checked, the operation of updating the cache is locked. When one thread accesses, another thread waits. After this thread accesses, the data in the cache is rebuilt so that other threads can take values from the cache.

Cache avalanche

It means that a large number of Key fails at the same time, and the requests for these Key will be called to DB, which will also lead to excessive pressure on the database and even hang up.

Solution.

1) if the failure time of Key is dispersed, a random value can be added to the uniform failure time, or a more advanced algorithm can be used to spread the failure time.

2) build multiple redis instances, and others can be used when individual nodes fail.

3) Multi-level cache: for example, increase local cache to reduce redis pressure.

4) add current-limiting measures to the storage layer, and provide degraded service when the request exceeds the limit (usually an error is returned)

Why is single-threaded redis so fast?

(1) Pure memory operation

(2) single-thread operation to avoid frequent context switching

(3) the non-blocking Iripple O multiplexing mechanism is adopted.

As a matter of fact, it is a problem left over by history, and it has to be played so well. )

Deletion strategy adopted by Redis

Redis uses a periodic delete + lazy delete strategy.

Why not delete policies regularly?

Delete regularly, use a timer to monitor the key, and delete automatically when it expires. Although memory is released in time, it consumes CPU resources. In the case of large concurrent requests, CPU uses time to process requests instead of deleting key, so it does not adopt this strategy.

How does regular deletion + lazy deletion work?

Delete regularly. By default, redis checks each 100ms to see if there is an expired key, and if there is an expired key, delete it. It is important to note that redis does not check all the key once for every 100ms, but is randomly selected for inspection (if every other 100ms, all key are checked, the redis will not be stuck). Therefore, if you only use the periodic deletion policy, it will result in a lot of key not being deleted by the time.

As a result, lazy deletion comes in handy. That is to say, when you get a key, redis will check whether the key has expired if the expiration time is set. If it expires, it will be deleted.

Why is the operation of Redis atomic, and how can atomicity be guaranteed?

For Redis, the atomicity of a command means that an operation can no longer be divided, and the operation is either executed or not executed.

The operation of Redis is atomic because Redis is single-threaded.

All the API provided by Redis itself are atomic operations, and the transactions in Redis are actually meant to ensure the atomicity of batch operations.

Are multiple commands atomic in concurrency?

Not necessarily, change get and set to single command operation, incr. Transactions using Redis, or implemented using Redis+Lua==.

Message queue

Do not use redis for message queuing, which is not the design goal of redis. But too many people use redis to do message queuing, and the authors of redis can't stand it.

Is it helpful for you to read the above content? If you want to know more about the relevant knowledge or read more related articles, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.