Shulou(Shulou.com)05/31 Report--

Why to introduce database middleware? in view of this problem, this article introduces the corresponding analysis and answer in detail, hoping to help more partners who want to solve this problem to find a more simple and feasible method.

Many friends often ask me the following questions:

58 do you use database middleware when you get home?

What kind of database middleware is used? is it self-developed or a third party?

How to implement it? is it based on client-side middleware or server-based middleware?

How to solve the problems of join/ subquery / set function / transaction after using middleware

Do you have similar questions?

However, "Why on earth is the introduction of database middleware" is rarely asked. The idea of the "architect's Road" article, which gives priority to solving "why", talks about this core issue by writing a series of articles on the hierarchical architecture of the Internet in the near future:

Why on earth should we introduce database middleware?

After the evolution of continuous and hierarchical architecture, DAO layer, basic data service, general business service, and the separation of front and rear, the back-end structure of a business system is as follows:

The web-view layer obtains json data from web-data through http interface (front and back ends are separated)

The web-data layer acquires data from biz-service through the RPC interface (common business service)

The biz-service layer obtains data from base-service through RPC interface (basic data service).

The base-service layer obtains data from db through DAO (DAO)

Db stores data

With the passage of time, the amount of data will become larger and larger, and the performance of base-service accessing db through DAO will become lower and lower. You need to consider horizontal sharding of db. Once db is split horizontally, many features that SQL can support require special processing by base-service layer:

Some data needs to be routed to a specific horizontal sharding library

If some data is not sure which horizontal sharding library it falls on, you need to access all the libraries.

Some data need to access the global library, get the global view of the data, and go to the service layer for additional processing.

...

More specifically, for the business with high concurrency in the foreground, after db horizontal segmentation, there are several typical business scenarios and solutions. In particular, we are dealing with the scenario of "foreground", "high concurrency" and "db horizontal split". The needs of the background will be dealt with through the architecture of the separation of foreground and background, and will not be discussed here.

One: single-line query on partition key

Typical scenario: query user through uid

Scene features:

Query via patition key

Only one row of records is returned at a time

Solution: the base-service layer routes libraries through patition key

As shown above:

User-service underlying user library. Sub-library patition key is uid.

For queries on uid, user-service can navigate directly to the library

Second, single-line query on non-patition key

Typical scenario: query user through login_name

Scene features:

Through non-patition key query

Only one row of records is returned at a time

Solution 1:base-service layer accesses all libraries

As shown above:

User-service checks the whole database first through login_name.

The result sets are merged in user-service, and a record is returned.

Solution 2:base-service checks the mapping library before routing through patition key

As shown above:

Create a new mapping library to record the mapping from login_name to uid

When there is a non-patition key query, first query uid through login_name

It is then routed through patition key, and a record is returned.

Solution 3: genetic approach

For more information on "genetic method" to solve the query requirements on non-patition key, see "multiple solutions for access to non-patition key after sub-database".

3. Batch query on patition key

Typical scenario: IN query on user list uid

Scene features:

Query via patition key

Return multiple rows of records at a time

The solution 1:base-service layer accesses all libraries, and the result set is merged into base-service

Solution 2:base-service analyzes routing rules and accesses them on demand

As shown above:

According to the analysis of routing rules, base-service judges that some data is in database 1 and some data is in database 2.

Base-service accesses related libraries on demand instead of accessing the entire library

Base-service merges result sets and returns list data

Fourth, Kuaku paging requirements on non-patition key

For more information on the query requirements of Kuaku paging after database division, see "Industry problems, four schemes of Kuaku paging".

5. Other needs.

At the end of this article, the first, second, third, fourth and fifth mentioned above are not the key points. Through a variety of tricks and tricks, the base-service layer can solve the data access problem after db horizontal sharding, but:

The complexity of the base-service layer has increased

The efficiency of data acquisition is reduced.

As more and more base-service need to be split horizontally for db, the hierarchical architecture looks like this:

The underlying complexity spreads to each base-service, and all base-service should pay attention to:

Patition key routing

Non-patition key query, mapping first, then route

First the whole library, then merge

Analyze first, then route on demand

Kuaku paging processing

...

Does this architecture diagram look awkward? How to make the acquisition of data more efficient and faster?

The introduction of database middleware is imperative.

This is a server-based database middleware architecture diagram:

The base-service layer, just like accessing db, accesses db-proxy to obtain data efficiently

All the underlying complexity is shielded at the db-proxy layer.

This is a "client-based" database middleware architecture diagram:

Base-service layer, efficient access to data through db-proxy.jar

All the underlying complexity is shielded at the db-proxy.jar layer.

When the database is split horizontally and the db data obtained by the base-service layer is too complex to become a general pain point, database middleware should be abstracted to simplify the data acquisition process, improve the efficiency of data acquisition and shield the complexity of the bottom layer to the upstream.

The answer to the question about why to introduce database middleware is shared here. I hope the above content can be of some help to you. If you still have a lot of doubts to be solved, you can follow the industry information channel to learn more about it.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.