Shulou(Shulou.com)06/01 Report--

This article talks about my understanding of ddbs from my personal project experience, including the reasons for the birth of ddbs, the basic principles of ddbs, what problems it has solved, and what are its shortcomings. Welcome to discuss and correct the axe.

When the business scale is small, stand-alone mysql is used for storage. However, with the development of business, there will be bottlenecks in storage capacity and concurrency capacity.

First of all, assuming that the stand-alone hard disk is 1.8T, you can also mount a larger capacity hard disk, but it is still limited.

Secondly, the read-write concurrency capability of the stand-alone machine is limited. It is assumed that the peak write qps1000, the peak read qps3000, and the network card also have requirements on the traffic when reading, so the read volume of a single access should not be too large.

The number of links on a single machine is also limited.

So, when the business development using stand-alone mysql suffers from the above bottlenecks, what is the general idea? Not one machine, use two, no more, expand more.

One is expanded into two, the disk capacity is expanded, through sub-tables, the tables are scattered on different machines, share the writing task, and the concurrency is also improved. I feel that this idea is correct.

So what do we need to do in the process?

The business has become unbearable on a single machine, and even on a single machine, many watches should be made too much. Generally, the subtable key will be selected according to the business. The size of a single table mysql also has certain requirements, generally the storage capacity is not more than 1G, the single record is smaller, generally no more than 1k, the number of entries is generally no more than 10 million, the maximum is not more than 50 million, otherwise the use and maintenance efficiency of the table is very low. Assume that the business has done enough sub-tables to meet the data growth needs of three years. After the first year, the number of each sub-table reaches 2 million, and half of the storage capacity of the whole machine is used. At this time, we want to split it into two machines.

At this time, we can synchronize part of the table data on the original machine to the new machine, and abstract a routing layer in the model layer to send the operation of the database to different machines. The upper layer business can still be considered to be using a single machine. At this point, you can delete the tables on the original machine that do not belong to your own management scope to make room.

One became two, taking a step towards distribution. At this point, the storage capacity and concurrency are increased, and the routing layer manages the two machines. What if the number of concurrency on two or more machines in the future is higher than the processing capacity of the routing layer? It is also necessary to expand the routing layer machines and write down all the routing rules, so that everyone will act according to the same rule.

After a lot of trouble above, the database machine is expanded horizontally, which solves some problems existing in the stand-alone machine. Will there be any disruption to the business during this expansion?

It's gonna have a little impact. At least when the routing layer changes the routing table, the writing of the database will be interrupted, and the reading can be uninterrupted at this time.

Many machines used in ddbs are called sharding. Slicing improves the storage capacity and concurrency of the system, and the introduction of slicing also increases the complexity of the system, which requires the introduction of routing layer machines, routing machines may also need to be extended, complex operations, and need to add more logic functions. But at least the business logic can be distinguished, and the business can use ddbs as a stand-alone machine.

So what are the shortcomings of ddbs?

Ddbs is still based on sub-table, shard implementation. Then the first condition for any operation on the database is the need to specify the partition key of the operation. Without the exact value of this dimension, you cannot operate on the database. Of course, unless it is an alternate database, you can scan the table freely, because the alternate database can be converted to redundant security, do not run online traffic, and can do statistical tasks.

It is not enough to specify the table key alone, but also note that the operational data may be distributed in different tables and shards. This operation will cause a large number of concurrent operations in ddbs, and a business request will take up multiple links on multiple machines, which will greatly reduce the concurrency capability of ddbs. Such as the 'where subtable key in ()' operation, this operation should be cautious, the number of in should not be too many.

It is best to choose × × ×, string type, which may result in uneven distribution and uneven table size after hash.

The implementation of transaction operation in ddbs consumes system performance very much. Transaction operations require the routing control layer to control the whole operation process, which may involve the operation of multiple fragments and different tables, which requires high availability of the system as a whole.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.