Shulou(Shulou.com)06/01 Report--

This article is about how Redis saves memory. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

First of all, the application that checks the user's UID through the image ID has the following requirements:

Query speed should be fast enough.

Data can be stored in memory, preferably on an EC2 high-memory model (17GB or 34GB, 68GB is too wasteful)

Persistence is supported so that there is no need to warm up after the server is restarted

First of all, they negate the scheme of database storage, and they maintain the KISS principle (Keep It Simple and Stupid), because this application simply does not use the update function, transaction function and associated query function of the database, so it is not necessary to choose to maintain a database for these unnecessary functions.

So they chose Redis,Redis as an in-memory database that supports persistence, and all the data is stored in memory (forget VM), and the simplest implementation is to use Redis's String structure to do a key-value storage. Like this:

SET media:1155315 939GET media:1155315 > 939

Among them, 1155315 is the picture ID,939 is the user ID, we take each picture ID as key, user uid as value to save as key-value pair. Then they tested and stored the data according to the above method. 1000000 of the data would use up 70MB memory and 300000000 photos would use 21GB memory. The 17GB compared to the budget is still overspent.

(NoSQLFan: in fact, we can see an optimization point here. We can remove the same media before the key value and save only numbers, so that the length of the key is reduced and the memory overhead of the key value is reduced. [note: the key value of Redis does not convert strings to numbers, so what is saved here is only media: the cost of these six bytes]. After experiments, the memory footprint will be reduced to 50MB, and the total memory footprint is 15GB, which meets the requirements, but the improvements after Instagram are still necessary.

So the developer of Instagram asked Pieter Noordhuis, one of the developers of Redis, about the optimization plan, and the reply was to use the Hash structure. The specific approach is to segment the data, each segment using a Hash structure storage, because the Hash structure will be a single Hash element in less than a certain number of compressed storage, so you can save a lot of memory. This does not exist in the above String structure. This amount is controlled by the hash-zipmap-max-entries parameter in the configuration file. After the developers' experiments, when the hash-zipmap-max-entries is set to 1000, the performance is better, and when the HSET command exceeds 1000, the CPU consumption will become very large.

So they changed the plan and stored the data in the following structure:

HSET "mediabucket:1155"1155315"939" HGET "mediabucket:1155"1155315" > "939"

By taking the first four bits of the 7-bit picture ID as the key values of the Hash structure, it is guaranteed that each Hash contains only 3-bit key, that is, 1000.

Do another experiment, and the result is that only 16MB memory is consumed for every 1000000 key. The total memory usage is also reduced to 5GB, which meets the application requirements.

(NoSQLFan: again, we can still optimize here, first by changing the key value of the Hash structure to a pure number, so that the key length is reduced by 12 bytes, and then by changing the subkey value in the Hash structure to three digits, which reduces the overhead by another 4 bytes, as shown below. After experiments, the memory footprint will be reduced to 10MB, and the total memory footprint will be 3GB)

HSET "1155" 315,939 HGET "1155" 315,939 Thank you for reading! This is the end of this article on "how to save memory in Redis". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it out for more people to see!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.