Shulou(Shulou.com)05/31 Report--

This article introduces what the database design of micro-service is like, the content is very detailed, interested friends can refer to it, I hope it can be helpful to you.

Separate database:

A key to microservice design is database design, the basic principle is that each service has its own separate database, and only the microservice itself can access this database. It is based on three reasons.

Optimize the service interface: the smaller the interface between microservices, the better, preferably only the service invocation interface (RPC or message), and no other interfaces. If the microservice cannot own its own database, then the database becomes part of the interface, which greatly expands the scope of the interface.

Error diagnosis: most of the errors in the production environment are related to the database, either because of the data or the way the database is used. There are all kinds of errors when you don't have complete control over the access to the database. It may be that other programs are directly connected to your database or other departments directly use the client to access the data of the database, which can not be found in the program, increasing the difficulty of error troubleshooting. If it is a problem in the program, as long as the code is modified, then the error will not occur again. And the mistakes mentioned above, you can never predict when they will happen again.

Performance tuning: the same is true of performance tuning. You need full control over the database to ensure its performance. If other departments have to access the database and are just querying, you can create a separate read-only database and let them query in another library so that your library will not be affected.

The ideal design is that only your service can access your database, you can only invoke the data in your own database, and all access to other micro-services is realized through service invocation. Of course, in practical applications, simple service invocation may not meet performance or other requirements, and different microservices need to share some data more or less.

Shared data:

There are four ways to share data between microservices.

Static table:

Some static database tables, such as countries, may be used by many programs, and the program needs to join the country table to generate end-user presentation data, so the way of micro-service invocation is inefficient and affects performance. One way is to configure such a table in each microservice, which is read-only, so that you can make a database connection. Of course, you need to keep your data synchronized. This option is acceptable in most cases because of the following two points:

Hongmeng official Strategic Cooperation to build HarmonyOS Technology Community

The static database table structure is basically the same: because once the table structure changes, you have to change not only the database tables of all microservices, but also the programs of all microservices.

The data in the database table does not change frequently: this does not require much work for data synchronization. In addition, there is always a delay when you synchronize the database, and if the data does not change frequently, then you have many synchronization methods to choose from.

Read-only business data access:

If you need to read dynamic business data from another database, the ideal way is service invocation. If you just call other microservices to do some calculations, performance is generally acceptable. If you need to make a data connection, you can do it in program code instead of SQL statements. If the performance does not meet the requirements after the test, you can consider building a set of read-only tables in your own database. There are roughly two ways to synchronize data. If it is event-driven, synchronize by sending messages, and if it is RPC, use the synchronization provided by the database itself or third-party synchronization software.

In general, you may only need a few tables from other databases, with only a few fields for each table. At this time, other databases are the ultimate source of data, controlling all write operations and the corresponding business verification logic, which we call the master table. Your read-only library can be called from the table. When a piece of data is written to the master table, a broadcast message is sent, and all microservices that own the slave table listen for the message and update the data in the read-only table. But you have to be very careful at this time, because it is much more dangerous than a static watch. First, its table structure changes more frequently, and its changes are completely out of your control. The second business data is not like a static table, it is updated frequently, so the requirement of data synchronization is higher. It is acceptable to decide how much delay is based on specific business needs.

In addition, it has two problems:

Hongmeng official Strategic Cooperation to build HarmonyOS Technology Community

Data capacity: the amount of data in the database is the main factor affecting performance. Because this data is external, it is not conducive to grasp its traffic law, it is difficult to carry out capacity planning, and can not better optimize the performance.

Interface leakage: the interface between microservices is originally only the service invocation interface, so you can make any changes to the internal program and database without affecting other services. Now the database table structure has become part of the interface. Once the interface is released, it is basically unchangeable, which greatly limits your flexibility. Fortunately, because another set of tables is built, there is a buffer, and when the master table is modified, the slave table may not need to be updated synchronously.

Unless you can do all the functionality with service invocation (without a local read-only database), the problems mentioned above are inevitable whether you integrate microservices in RPC or event-driven mode. But you can reduce the impact of the above problems by reasonably planning database changes, which will be explained in more detail below.

Read and write business data access:

This is the most complicated situation. In general, you have one table that is the master table and the other tables are slave tables. The master table contains primary information, and this master information is copied to the slave table, but the microservice will have additional fields to write to the slave table. In this way, the local microservice can read and write to the slave table. And there is a priority relationship between master table and slave table. The primary key of the slave table comes from the master table, so there must be a master table first and then a slave table.

The picture above is an example. Suppose we have two movie-related microservices, one is a movie forum, where users can comment on movies. The other is the movie store. "movie" is the shared table, the one on the left is the movie forum library, and its "movie" table is the main table. On the right is the movie store library, whose "movie" table is the slave table. They share the "id" field (primary key). The main table is the main source of data, but not from the "quantity" and "price" fields in the table. After the main table inserts the data, it sends a message, receives the message from the table, and inserts a piece of data into the local "movie" table. And the "quantity" and "price" fields in the table are modified from the table. In this case, to assign a unique source (micro-service) to each field, only the source has the right to actively change the field, and other micro-services can only be changed passively (after receiving the change message from the source). In this example, the source of the "quantity" and "price" fields is the table on the right, and the source of the other fields is the table on the left. In this example, "quantity" and "price" exist only in the slave table, so the data is written one-way, from the master table to the slave table. If these fields are also needed in the main table, they will also be written back, and the data will be written in both directions.

Direct access to other databases:

This way is absolutely prohibited. Many program errors and performance problems in a production environment are caused in this way. The above three methods create a new local read-only database table, resulting in physical isolation of the database, so that the performance problems of one database will not affect the other. In addition, when the table structure in the master library changes, you can temporarily leave the tables in the slave library unchanged so that the program can run. If you directly access someone else's library, as soon as the main library is modified, other micro-service programs will immediately report an error.

Backward compatible database updates:

As can be seen from the above discussion, the modification of the database table structure is a matter with a wide range of influence. In a microservice architecture, shared tables also have a read-only copy in other services. Now when you want to change the table structure, you also need to consider the impact on other micro-services. When in a Monolithic architecture, database updates are backward compatible in order to ensure that the program deployment can be rolled back. Another reason for compatibility is support for blue and green releases (Blue-Green Deployment). In this deployment, you have both new and old versions of the code, and it is up to the load balancer to decide which version each request points to. They can share a database (which requires the database to be backward compatible), or they can use different data. To put it simply, there are several types of database updates:

Add tables or fields: if the field is null, this operation is backward compatible. Insert a default value if it is a non-null value.

Delete table or field: you can keep the deleted table or field temporarily and delete it after several versions.

Modify the field name: add a new field, copy the data from the old field to the new field, and use the database trigger (or program) to synchronize the old field with the new field (for transition). Then delete the original field after several versions.

Change the table name: if the database supports updatable views, the easiest way is to change the name of the table first, and then create an updatable view that points to the original table. If the database does not support updatable views, the method is similar to changing the field name, and you need to create a new table and synchronize the data.

Modify the field type: it is almost the same as modifying the field name, except that you need to convert the data type when copying the data.

The advantage of backward compatible database updates is that when there is a problem with the deployment of the program, roll back if necessary. Just roll back the program without having to roll back the database. Usually only one version is rolled back when you roll back. All tables or fields that need to be deleted will not be modified during this deployment. Wait until one or more versions and confirm that there are no problems before deleting them. Another benefit is that it does not have an immediate direct impact on shared tables in other microservices. When this microservice is upgraded, other microservices can evaluate the impact of these database updates before deciding whether they need to make corresponding program or database modifications.

Cross-service things:

One of the difficulties of micro-services is how to implement cross-service transaction support. Two-phase commit (Two-Phase Commit) has been proved to be unable to meet the requirements in performance and is now largely unused. The agreed method is called Saga. Its principle is to write a compensation operation (Compensating Transaction) for each operation in the thing, and then perform each compensation operation one by one during the rollback phase. The example is shown in the following figure, where there are three operations in one thing, T1 ~ T2 ~ T2 ~ T3. For each operation, define a compensation operation, C1 Magi C2 Magi C3. When things are executed, T1 is executed first in forward order, and C3 is executed in reverse order when rollback. Each operation in a thing (forward operation and compensation operation) is packaged as a command (Command), and the Saga execution coordinator (Saga Execution Coordinator (SEC)) is responsible for executing all commands. Before execution, all commands are stored in the log in order, and then the Saga execution coordinator fetches the commands from the log and executes them in turn. When an error occurs in an execution, the error is also written to the log, and all executing commands are stopped to start the rollback operation.

Saga relaxes the requirement of Consistency, which can guarantee the final Eventual Consistency, so the data is inconsistent during the execution of things, and this inconsistency will be seen by other processes. In life, in most cases, we don't have that high demand for consistency, and temporary inconsistencies are acceptable. For example, the bank transfer operations, they are not carried out in a database transaction, but in the way of accounting is divided into two actions to ensure the ultimate consistency.

The principle of Saga looks simple, but it is difficult to implement it correctly. Its core problem is the handling of errors. To make it completely clear, you need to write another article. I will only talk about the main points now. The network environment is unreliable, and the executing command may not return a result for a long time, at this time, first, you need to set a timeout. Second, because you don't know that the reason why there is no return value is that the command has been completed but there is something wrong with the network, and it is still sacrificed before it is completed, so you do not know whether to perform the compensation operation. The right thing to do at this point is to retry the original command until it is confirmed that it is complete, and then perform the compensation operation. But there is a requirement for the command that the operation must be Idempotent, that is, it can be executed multiple times, but the end result is the same.

In addition, compensation operations for some operations are relatively easy to generate, such as payment operations, you just need to return the money. But some operations, such as sending an email, have no way to return to the previous state after completion, so you can only send one more email to correct the previous message. Therefore, the compensation operation does not have to return to the original state, but counteracts the effect of the original operation.

Split of microservices:

Most of our original programs are monolithic programs, but now we want to split it into micro-services, what can we do to reduce the impact on existing applications?

Let's use the picture above as an example. It has two programs, one is "Styling app" and the other is "Warehouse app". They share the following database in the diagram, and there are three tables in the library, "core client", "core sku" and "core item".

Suppose we want to split out a microservice called "client-service", which needs to access the "core client" table. In the first step, we split the program from the original code and become a service. The database remains unchanged, and the service still points to the original database. Other programs no longer directly access the table managed by the service, but obtain data through service invocation or by creating another shared table.

The second step is to split the database table of the service so that the microservice has its own database instead of the original shared database. At this point, it becomes a real micro-service.

The above only talked about splitting a micro-service, if there are multiple need to split, then one by one according to the method described above.

In addition, Martin Fowler has a good suggestion in his article "Break Monolith into Microservices". That is, when you split a service from a single program, don't just think about splitting the code. Because the current requirements may be different from the original, the original design may not be very applicable. Moreover, the technology has been updated and the code has to be modified accordingly. A better approach is to rewrite the original function (rather than rewriting the original code), focusing on splitting the business function, rather than splitting the code, and implementing it with new designs and technologies.

Database design is a key point of micro-service design. The basic principle is that each micro-service has its own database, and only the micro-service itself can access this database. Data sharing between microservices can be realized through service invocation, or master and slave tables. When sharing data, find an appropriate way to synchronize. In the micro-service architecture, database modifications have a wide range of implications, and it is necessary to ensure that such modifications are backward compatible. The standard way to implement cross-service things is Saga. When a single program is split into microservices, it can be done step by step to reduce the impact on existing programs.

On the micro-service database design is how to share here, I hope the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.