Shulou(Shulou.com)06/02 Report--

This article mainly explains "PHP combined with MySQL to process tens of millions of data". The explanation in this article is simple and clear, easy to learn and understand. Please follow the ideas of Xiaobian slowly and deeply to study and learn "PHP combined with MySQL to process tens of millions of data" together.

mysql sub-table thinking

A 100 million order table can be divided into five tables, so that each table has only 20 million data, sharing the pressure of the original table. The tables need to be divided according to certain conditions. Here, tables can be divided according to regions. A middleware is needed to control which table to find the data you want.

Middleware: according to the self-increasing id of the main table as middleware (what kind of fields are suitable for middleware? to be unique)

How? After the main table is inserted, an id is returned, modulo is taken according to the id and the number of tables, and the remainder is a few to insert data into which table.

Note: The id in the child table should be consistent with the id in the main table.

In the future, only insert operations will use the main table. Modify, delete, and read operations will not use the main table.

redis message queue

1 What is a message queue?

A container that holds messages as they propagate

2. Historical reasons for message queues

Message queue characteristics: first-in, first-out

Save the sql statement to be executed in the message queue, and then insert it into the database asynchronously.

Application: Sina, put the instant comments into the message queue first, and then insert the sql statements in the message queue into the database through the timed task.

modify

Modify the operation sub-table

There is a problem with this modification. The data in the main table and the sub-table will be inconsistent. How to make the data in the main table and the sub-table consistent?

Redis queue keeps the data of main table and child table consistent

After modification, the data to be modified in the main table is stored in the redis queue.

Then linux timed task (contble) loop execution redis queue sql statements, synchronous update the contents of the main table

mysql distributed table (check, delete)

Query only needs to query sub-tables, do not query the total table

Delete, first find the sub-table to delete according to id, then delete, and then press a sql statement to delete the total table data into the message queue.

Then perform a timed task to delete the summary table data

Timed Mission:

The thought of mysql distributed library division

Schematic diagram of sub-warehouse:

mysql distributed sub-library (increase)

Note: After operating a database, be sure to close the database connection, otherwise mysql will think that the same database has been connected.

Or modulo determines which profile to load and which database to connect to

mysql distributed library (change)

Principle same as new

mysql distributed library (check, delete)

principle is similar

delete

execution queue

Application of mysql distributed memcache

Put the data into the cache, save database overhead, first go to the cache to check, if there is direct extraction, if not, go to the database to check, and then store it in the cache

You need to delete the cache after editing information, otherwise you will always read cached data instead of modified data.

Thank you for reading, the above is the content of "PHP combined with MySQL processing tens of millions of data", after learning this article, I believe that everyone has a deeper understanding of PHP combined with MySQL processing tens of millions of data, the specific use of the situation also needs to be verified by practice. Here is, Xiaobian will push more articles related to knowledge points for everyone, welcome to pay attention!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.