Shulou(Shulou.com)05/31 Report--

It is believed that many inexperienced people do not know what to do about how to efficiently migrate to Redis under the order of millions of MySQL data. therefore, this paper summarizes the causes and solutions of the problem. I hope you can solve this problem through this article.

Redis protocol learning

The format of the agreement is:

*\ r\ n

$\ r\ n

\ r\ n

...

$\ r\ n

\ r\ n

For example: insert a piece of data of type hash

HSET id book1 book_description1

According to the Redis protocol, there are four parts in total, so it starts with * 4, and the rest is explained as follows:

Content length protocol command HSET4 $4id2 $2book15 $5book_description117 $17

Note that the HSET command itself is sent as one of the parameters of the protocol.

The constructed protocol data structure:

* 4\ r\ nroom4\ r\ nHSET\ r\ nroom2\ r\ nid\ r\ nroom5\ r\ nbook1\ r\ nroom17\ r\ nbook_description1\ r\ n

Format it:

* 4\ r\ n

$4\ r\ n

HSET\ r\ n

$2\ r\ n

Idvvvv\ r\ n

$5\ r\ n

Book1\ r\ n

$17\ r\ n

Book_description1\ r\ nRESP protocol bulk

The Redis client communicates with the Redis server using a protocol called RESP (Redis serialization Protocol).

Redis-cli pipe mode needs to be as fast as the nc command, and solves the problem that the nc command does not know when the command ends.

While sending the data, it also reads the response and tries to parse it.

Once no more data is read in the input stream, it will send a special 20-bit echo command indicating that the last command has been sent. If the same data is matched in the response result, the batch delivery is successful.

Using this technique, we don't need to parse the protocol sent to the server to see how many commands we sent, just parse the response.

When parsing the reply, redis makes a count of the parsed reply, and in the end can tell the user the number of commands transmitted to the server by a large number of insert sessions. This is the result of the actual operation we did using pipe mode above.

Change the input data source to mysql

In the above example, we use an txt text as the input data source and use the pipe schema to import the data.

Based on the study and understanding of the above protocol, we only need to import the data from mysql into Redis through pipe schema according to the established protocol.

Actual case-- importing million-level data from Mysql to Redis to create data first

Due to environmental limitations, there is no real data for import, so let's first use a stored procedure to create 1 million pieces of data. Use stored procedures as follows:

DELIMITER $$

USE `cb_ mon` $$

DROP PROCEDURE IF EXISTS `test_ insert` $$

CREATE DEFINER= `root` @ `% `PROCEDURE `test_ insert` ()

BEGIN

DECLARE i INT DEFAULT 1

WHILE i

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.