Shulou(Shulou.com)06/01 Report--

The case of bulk writing by Redis? This problem may be often seen in our daily study or work. I hope you can gain a lot from this question. The following is the reference content that the editor brings to you, let's take a look at it!

Recently, to test the performance of redis, you need to insert more than 10 million pieces of data in batch.

A simple study has been carried out, and there are probably the following methods:

One: java program call, simple for loop, through the method of Jedis, directly inserted

As for speed, you don't have to look at it, you don't have to try, it doesn't work at all, and it won't be realized.

Second: through the shell script, it is also relatively simple, but also because of the form of the port port.

Connect to redis, the speed is also slow, give up.

Third: through the redis to provide pipeline, feel that this method is the most reliable, the following is achieved:

Step 1: first simply write a shell script:

For ((item1witi > / tmp/_t.txt done

The aim is to generate a script file that is inserted in bulk

Step 2: here the files generated on linux or windows cannot be run directly as redis statements

Simply put, it's because the line breaks of linux,windows and dos are not the same.

So you need to make a simple conversion:

There are many ways to convert:

1: this gadget is generally available in Linux distributions. You can only convert DOS to UNIX files. The command is as follows:

# unix2dos dosfile.txt

The above command removes the ^ M symbol from the end of the line. (if nuix2dos is not installed on the machine, just use method 2)

2: using vim,vim is an improved version of vi, how to use it:

# vim file.txt:set fileformat=dos::wq

Just one line of command, save and exit

Step 3: just run the script, through the pipe provided by redis-- pipe form

Cat redisTest.txt | redis-cli-h reids.aliyuncs.com-p 6379-a xxxx-- pipe

It takes three steps to complete the batch insertion of redis. In the case of 20 million entries, the speed of generating script is a little slower, and the speed of insertion is still a few minutes.

The speed can be said to be very fast.

Thank you for reading! After reading the above, do you have a general understanding of the case of batch writing by Redis? I hope the content of the article will be helpful to all of you. If you want to know more about the relevant articles, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.