Shulou(Shulou.com)06/01 Report--

Let's talk about a simple way to deal with the deduplication of a large amount of data in MySQL. The secret of the text is to be close to the topic. So, no gossip, let's go straight to the following, I believe you will benefit from reading this article on the simple way to deal with a large amount of data in MySQL.

Because there is a large amount of duplicate data, the chosen method of deduplication is to create a new table through the aggregate function and then rename it. The sql code is as follows:

Create table tmp select * from table_name group by column1,column2

Then, because a large amount of data has been deleted, the id is almost 600000. It's too obnoxious to look at it, so I changed id to start with 1. The method is to delete the id field first, and then add the id field and set it to self-growth. Well, it finally looks comfortable. More than 300,000 data after about 18 pieces of data, it takes 70 seconds, estimated tens of millions of data, I should cry, anyway, then continue to think about what good way.

Note: if it was sorted by id before, it will not be in the same order as before when it starts with 1. It will be reordered according to the second field.

Is there anything you don't understand about the above simple method of deduplicating a large amount of data in MySQL? Or if you want to know more about it, you can continue to follow our industry information section.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.