Network Security Internet Technology Development Database Servers Mobile Phone Android Software Apple Software Computer Software News IT Information

In addition to Weibo, there is also WeChat

Please pay attention

WeChat public account

Shulou

How to solve the problem of unlimited expansion of database and table

2025-01-17 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > Development >

Share

Shulou(Shulou.com)06/02 Report--

This article mainly explains "how to solve the problem of unlimited expansion of database sub-table". The content of the explanation in this article is simple and clear, and it is easy to learn and understand. let's study and learn "how to solve the problem of unlimited expansion of database sub-database and sub-table".

Monomer application

Every startup is basically built from a structure like SSM and SSH. There's nothing to talk about. Almost every programmer has experienced it.

RPC application

When the business is getting larger and larger, we need to expand the service horizontally, which is very simple, as long as we ensure that the service is stateless, as shown below:

When the business is getting bigger and bigger, our service relationship is complicated. At the same time, there are many service accesses that do not need to connect to DB, but only need connection cache, so it can be made separate and reduce the valuable connections of DB.

As shown below:

I believe most companies are at this stage. Dubbo is born to solve this problem.

Sub-database sub-table

If your company's products are popular, your business continues to grow at a high speed, there is more and more data, and SQL operations are getting slower and slower, then the database will become a bottleneck.

Then you will definitely think of sub-database and sub-table, no matter through ID Hash or Range, as shown below:

This should be all right. No matter how many users and no matter how high the concurrency is, I just need to expand the database indefinitely and expand the application indefinitely.

This is also the title of this article. Can unlimited expansion be solved by sub-library and sub-table? In fact, an architecture like the one above does not solve the problem.

In fact, this problem is somewhat similar to that of RPC: too many database connections!

Usually, because our RPC application uses middleware to access the database, the application actually does not know which database to access. The rules of accessing the database are determined by the middleware, such as Sharding JDBC.

As a result, the application must connect to all databases, and as in our architecture diagram above, a RPC application needs to connect to three MySQL.

For 30 RPC applications, the database connection pool size for each RPC is 8, and each MySQL needs to maintain 240 connections.

We know that the default number of connections in MySQL is 100, and the maximum number of connections is 16384, that is, assuming that the connection pool size of each application is 8, more than 2048 applications cannot continue to connect and expand.

Note that because there are many logical libraries in each physical library, and the micro-service movement is in full swing, 2048 is not as big as it seems.

You may say that I can solve the problem of the number of connections by adding a Proxy in front of it. In fact, the performance of the agent will also be a problem. Why?

The number of connections to the agent cannot exceed 16384, and if the concurrency exceeds 16384 and becomes 163840, then Proxy will not solve the problem.

What shall I do? Let's take a look at the architecture diagram above:

We found that the problem is that "every RPC application has to connect to all libraries", resulting in an increase in the number of connections to each database while expanding the capacity of the application.

Even if you increase the database, it will not solve the problem of the number of connections. What are we going to do?

Unitization

Unitalization, which sounds high-end, usually occurs at some XXX conferences when sharing powerful terms such as "about two places and three centers", "three places and five centers", "living in different places" and so on.

Here we do not discuss so powerful, just say "too many database connections" problem. In fact, the idea is simple: we just don't want the application to connect to all databases.

Suppose we divide 10 libraries according to Range, and now there are 10 applications. We let each application connect to only one library. When the number of applications increases to 20, and the database connections are insufficient, we divide 10 libraries into 20 libraries.

In this way, no matter how many applications you expand to, you can solve the problem of too many database connections.

Note: the premise of doing this is that you must make sure that the database requested by your application's Request must be accessed in this application.

In other words, when users come in from DNS, they know which app they are going to, so the rules are set before DNS. Although this is a bit of an exaggeration, they must know which library to go to before they enter the app.

Therefore, this usually requires a rule, for example, through the user ID Hash, which broadcasts the Hash rule by the configuration center.

In this way, all components can maintain consistent rules to correctly access the database, as shown in the following figure:

Thank you for your reading. The above is the content of "how to solve the problem of unlimited expansion of database sub-database and sub-table". After the study of this article, I believe you have a deeper understanding of how to solve the problem of unlimited expansion of database sub-database and sub-table. the specific use of the situation also needs to be verified by practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.

Views: 0

*The comments in the above article only represent the author's personal views and do not represent the views and positions of this website. If you have more insights, please feel free to contribute and share.

Share To

Development

Wechat

© 2024 shulou.com SLNews company. All rights reserved.

12
Report