Shulou(Shulou.com)06/03 Report--

This article introduces the knowledge of "how to build a back-end architecture from 0 to 1". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

The evolution of architecture has gone through four major stages: 1. MVC 2. Service split 3. Micro-service architecture 4. Domain-driven design

1. MVC

At the beginning of the project, there are no more than five back-end colleagues. At this stage, the main work is to implement the prototype of the product, without much consideration of the architecture, and use Django to achieve the function quickly. After the table structure of DB is designed, the functional View is abstracted. Because the product design is not perfect, the back-end needs a lot of reserved design to avoid changes in the whole table structure caused by changes in product logic. The most important thing at this stage of the code is to determine the code specifications and code review rules that are suitable for the team.

In the overall architecture, as shown in the figure above, Nginx is responsible for load balancing, distributing traffic to multiple Django services, Django processing logic, handing over asynchronous tasks to Celery, and then using Redis as cache for large data volumes. At the same time, there is also the need for real-time message notification using Nginx Push Module.

Problems and optimization methods:

1. Poor concurrency performance of Django using uWSGI Master+Worker and gevent Ctrip to support high concurrency

2. The excessive number of Redis connections uses the connection pool that comes with redis-py to achieve connection reuse

3. Excessive number of MySQL connections using djorm-ext- pool (https://github.com/djangonauts/djorm-ext-pool) connection pool multiplexing connection

4. Celery configures gevent to support concurrent tasks

As more and more features are developed, there are more and more app under Django, which brings inconvenience to the release. Every release needs to restart all Django services. If there is a problem with the release, you can only work overtime to solve it. And the amount of code under a single Django project is more and more, which is difficult to maintain.

two。 Service split

With the growth of the back-end team, the requirements assigned to each colleague become more and more detailed. If you continue to develop all the code in a project, the cost of maintenance is too high. In our last architecture, we have divided into high clustering within Django by module and low coupling between app, which facilitates the split of services.

The process of splitting did not encounter too big problems, the initial split is just the separation of the code, the common code is extracted to achieve a common Python library, database, Redis is still shared, with the increase of load, the database also makes multiple instances.

As shown in the figure above, services should avoid calling each other as much as possible. Where interaction is required, http request is used. Calls in private network use hosts to point to private network address.

Problems and optimization methods:

Because Nginx Push Module is not maintained for a long time and the maximum number of persistent connections is not enough, the tormq (https://github.com/zhu327/tormq) service is implemented using Tornado + ZeroMQ to support message notification.

The invocation between services adopts the way of http, and the dependent service host is required to configure hosts to point to the called address, which brings inconvenience in maintenance. And there are no strategies such as retry, error handling, current restriction and so on in the process of calling the chain, resulting in poor service availability.

With the split of the business, it is very troublesome to continue to use Nginx to maintain the configuration, and call errors are often caused by modifying the configuration of Nginx. Each service has a complete authentication process, and authentication depends on the database of the user center. Multiple services need to be reissued when the authentication is modified.

3. Micro-service architecture

First of all, the Kong API Gateway based on OpenResty is introduced in the access layer, and the plug-ins such as authentication and current limit are customized. In the access layer, it undertakes and strips the common authentication and current limiting functions of the application layer. When publishing a new service, calling Kong admin api to register the service address to Kong in the release script and loading api requires the use of plug-ins.

In order to solve the problem of mutual invocation, we maintain a RPC service framework doge based on gevent+msgpack, do service governance with the help of etcd, and realize the functions of current limiting, high availability and load balancing in rpc client.

The most difficult technology selection at this stage, open source API gateways are mostly implemented in Golang and OpenResty (lua), and need to be customized to meet the needs of our business. It took 1 month to learn OpenResty and Golang, and used OpenResty to implement a short URL service shorturl in the business.

The final choice of Kong is based on the convenience of Lua release, Kong out of the box and plug-in development is relatively easy. Performance considerations are not the most important. In order to support more concurrency, LB services provided by the cloud platform are also used to distribute traffic to a cluster of two Kong servers. Automatically synchronize configurations between clusters.

Ele.me maintains a pure Python implementation of thrift protocol framework thriftpy, and provides a lot of supporting tools. If the team is large enough, this set of RPC solution is actually appropriate, but our team is understaffed and the level is uneven. It is difficult to promote this set of expensive learning programs. Finally, we developed a Duboo-like RPC framework doge, and the code mainly referred to weibo's open source motan.

4. Domain-driven design

In this architecture, we try to extract the data service layer from the application service, each data service contains one or more boundary contexts, and the boundary context class has only one aggregate root to expose the methods invoked by RPC. Data services do not depend on application services, which can rely on multiple data services. With the data service layer, applications decouple their dependence on each other, and high-level services only rely on the underlying services.

When I left, domain-driven design was still in the learning design stage and had not yet landed, but I believe that the back-end architecture of the former company will continue to evolve in this direction.

That's all for "how to build a back-end architecture from 0 to 1". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.