Shulou(Shulou.com)06/03 Report--

On June 20, 2019, the third Enterprise Container Innovation Conference (Enterprise Container Innovation Conference, hereinafter referred to as ECIC) hosted by Rancher Labs (hereinafter referred to as Rancher) was held at the Sheraton Hotel in Beijing. This year's ECIC has a large scale, with a total of 17 keynote speeches throughout the day, attracting nearly 1,000 container technology enthusiasts, and more than 10000 viewers watched the event on the live broadcast platform online.

Technical leaders from more than a dozen enterprises, including Rancher, Aliyun, Baidu Yun, Ping an Technology, China Unicom, Flying loan Financial Technology, China Life Insurance, SmartX, Huatai Insurance, Xiamen Airlines, JFrog, New Oriental, Cisco and so on, attended this ECIC, bringing a wonderful sharing of practical experience on corporate container projects and a sharing of corporate containerization experiences for container technology enthusiasts participating in the conference.

At the scene of the conference, Flying loan Financial Technology, as a typical case of database containerization in the financial industry, brought practical experience sharing entitled "Database production containerization and Istio application in the financial field" to container enthusiasts on the spot.

For flying loan financial technology, the difficulty of production containerization and database application lies in how to carry out practical innovation for production containerization and database container application in the financial field, and how to combine R & D and business scenarios to improve the efficiency of resource utilization, product research and development, and operation and maintenance management.

Chen Dingwei, vice president of Flying loan Financial Technology, said: "Financial industry data has more stringent safety and high standards than other industries. In the case of security compliance, we use Flying loan to develop middleware to solve the problems of DB application scenarios in the financial sector, resulting in 10x DB delivery efficiency and the ultimate flexible expansion capacity."

Factual record of speech

Flying loan Financial Technology, founded in 2010, is the overall technical service provider of mobile credit. We take scientific and technological innovation as the driving force for enterprise development, and continue to move forward on the road of scientific and technological innovation.

From 2011 to 2015, Flying loan did the traditional small and micro financial business. In 2015, we decided to carry out the transformation of online Internet. By 2017, our entire company has upgraded its strategy to provide Internet services to customers in the financial industry. So far, Flying loan has provided full-link science and technology services to many financial industry enterprises, such as PICC, Bank of Beijing, China Resources Trust, Tonglian payment and so on.

In 2018, we appeared in time magazine, which was called "Global Financial Technology Best practice" by time magazine. In the same year, we also received the Platinum Award for Product Innovation of the year, the highest honor of the first Global small and Micro Finance Award jointly launched by the World Bank and the G20.

Next, I will introduce to you in detail how Flying loan, as an Internet financial technology enterprise, is combined with containerization, and how to apply containerization in business.

The containerization of the flying loan application is consistent with the enterprise shared before, and it is also based on the containerization application of the whole enterprise. It is worth mentioning that flying loans are in the financial sector, so we are more concerned about safety, fault tolerance and high recovery than companies in other industries. We focus not only on applications, but also on how to recover from disasters quickly.

We deployed the overall architecture using containers, ranging from the familiar DevOps to the DB Mesh section that I'll focus on later. We have divided several platforms, including containerization platform, product research and development platform and data platform. The following is the application security, data security, network security, container security, operation and maintenance security and other parts. Containers are very helpful to us. Now our RD is based on container Kubernetes for application development. In this part, flying loan has reached a leading level in the financial field.

The following picture is the container development roadmap of Flying loan. We started working on containers in 2015 and started production in RD environment in 2016. At that time, we could not fully decide whether Kubernetes was another container technology, so we stayed at the RD stage for the time being. In 2017, as Kubernetes technology becomes more and more mature and stable, we will move the overall direction to Kubernetes. By this year, our production environment has been able to make extensive use of container technology for applications in many directions.

Just now Dr. Liang Sheng, CEO of Rancher, mentioned that Rancher can now manage and deploy multiple K8S clusters and multiple data centers. This is more in line with our business development. We provide financial cloud services based on flying loans, and we have the business needs of multi-tenant cluster management. At present, we can manage K8S multi-cluster in many directions, such as application service, central service, database service and so on. Similarly, we can also achieve multi-tenant network isolation.

From the customer's point of view, before customers cooperate with us or in the process, they may not understand that the business operation of small loans is like this, so banks will put their overall services in our company. Flying loan has become a financial cloud manufacturer. The special thing about flying loan is that we focus on the content related to our business development, and what we provide to our customers is not an overall platform, but an application.

All the contents mentioned just now are closely related to the container. The features of the container include security audit, dynamic storage, high available grayscale release, and so on. We have applied the characteristics of the container to the flying loan production environment and made the best use of it.

The following figure shows the platform components of Flying loan containerization. Whether it is our RD or outsiders, Flying loan will provide them with an app store. If they want to do anything, just click on our management platform, and we will automatically produce a container app to help them deal with it. The parts of our mirror warehouse are together.

In addition to these parts, we also have Prometheus and Jenkins, which are highly related to our research and development. now Flying loan can achieve automatic integration, automatic packaging, automatic release and automatic deployment, which is the result of our research on platform components for more than two years.

Why should the flying loan make DB containerized? Because the application layer of the micro-service part has developed well, but there are still many problems for DB. If the DB goes down, I want to restore the DB quickly so that business production can run normally. How long will it take us? If the DB is very large, the startup time is very long. This is why banks or large financial institutions do not have minicomputers and dare not use open source databases such as MySQL or MangoDB, because it is a big challenge for them to ensure safe and continuous operation.

These are some of the issues I'm going to focus on today. Why MySQL containerization? Is MySQL containerization safe and stable? What is the concrete implementation of containerized MySQL?

We just introduced the container for multi-cluster management of flying loans, and there are some restrictions and requirements. First, it will involve a very complex network structure; second, failures should be switched frequently, which we think is a very important part of the financial industry, because in the event of a failure, the business of the financial industry will basically shut down; third, the capacity should be controlled; and the fourth is to rely on network storage.

There are three reasons why we should do this part. First, we need to achieve standardized and rapid deployment, because after the rapid deployment of the application, if the DB deployment is slow, the overall efficiency is still the same for us, which is in terms of the overall efficiency. The second is the micro-service scenario, our current system is all for service-oriented terminal adjustment, in this scenario, if the data scenario can not be micro-serviced, then what I have done at the upper level is meaningless, we do not want the database to become a deficiency in business flexibility and management; the third is the requirements of MySQL service, automation, networking and intelligence.

The effect of our MySQL containerization is obvious. First, we can achieve efficient auto scaling, capacity expansion, backup, import, export, recovery, snapshot and migration; second, we can achieve performance monitoring and audit of the overall database; third, distributed storage, resources, and multiple copies of data can be synchronized in real time. The part of our application in big data may also be different from that of ordinary companies. Some data in our production environment are separated from big data's real-time data, but we have achieved real-time synchronization. Fourth, computing resources are distributed and multi-node. Technical facilities are highly available; fifth, it has the function of fault self-recovery. If my MySQL goes down, we can recover quickly.

The following figure shows the architecture of our MySQL DB. The application services below correspond to middleware, and all our middleware corresponds to each individual library. In order to implement the DB container, we compress the library with a very large amount of space and limit the capacity of the library, so that it is possible to start it quickly in case of library failure. This part tests our overall business operation, the ability of data table and database, and the ability of reading and writing separation. This part is done through the middleware we developed by ourselves. Without the middleware we developed by ourselves, we would not be able to complete this part of DB Mesh.

The above is basically the network divergence chart of Flying loan DB. The architecture features several parts: high concurrency, low latency, 10000 transactions per second, latency less than 100ms; support for IDC multi-activity; third, support for data routing; fourth, automatic or personalized decision switching; fifth, multiple copies of data.

Up to now, the DB level of Flying loan is PB level. We have about a dozen PB applications, which can be implemented synchronously. If the number of fault containers is greater than 1/2, we can reply automatically. This is why we want to do DB Mesh.

The other part is about our containerized integration of Istio, and on the right is the graphical interface of our production application. You can see that after registering, we can automatically track the health of the library. But there are some minor problems. When the DB is disconnected and then restored, the service is gone and needs to be manually injected again. With regard to this problem, we have studied a lot of Istio documentation, but have not yet overcome this problem. So in the part of DB, we can only inject it at the beginning of production, but when it is dead, we still need to deal with it manually, so there is no way to recover it automatically.

In the part of application and management services, we have achieved full automation, integrating Istio to realize micro-service Service Mesh, and realizing micro-service access, security reinforcement, control and observation. Service tracking, speed limit, circuit breaker, scheduling, load and so on.

The above is the application deployment of Flying loan overall service, from application services to middleware, this is the release map of our overall deployment, so now our RD staff are basically only responsible for development. After development, everything is integrated, released and managed through our platform. After entering the production environment, it will also be handled by our operation and maintenance staff, not by RD. At this point, what we have done is more in line with the requirements of the bank.

Finally, I would like to introduce the achievements brought about by the containerization of flying loans:

The first is to enhance the overall productivity of flying loans. 80% of the basic operation and maintenance of Flying loan are automated; secondly, the delivery capacity has also been improved. We can deliver hundreds of service applications an hour. At present, there are thousands of containers operating on our entire production environment. If we do not carry out the micro-service containerization, the micro-service architecture deployment time will be very long. The last one is that we have hundreds of MySQL instances in the production environment, which is also one of our containerization results.

The second is research and development and expansion, which can be extended according to the pod, physical host node, cabinet and data center level of the container. We have also combined a lot of CMDB content, but we will not elaborate on it here.

The third is the investment of IT cost, which is also a content that our enterprise pays more attention to. Our private cloud was built with CloudStack as a platform, but now we have all replaced it with containers. This saves us about 40% of our resources and 60% of our manpower investment. In the past, in order to deploy an application, we needed to provide a virtual host to deploy on RD, but now the container can be deployed with one click. In addition, the time spent on R & D of the project has also been saved by 40%, because the content such as deploying applications no longer needs to be handled by RD personnel, but is handled automatically by our platform.

The fourth is safe, agile and efficient. We also have a minute-level full backup of this part of amateur data, and our library is small enough that we can back up quickly in a few minutes. Second, in disaster recovery, our business uses one-click recovery in minutes. Data snapshots are second-level, resource utilization increases 10 times, and database delivery capacity increases nearly 100 times. Our entire application has hundreds of MySQL nodes. If the deployment is very slow, we have now done the mirroring, so deployment is very fast.

Finally, the operation and maintenance has become very simple, automated, extreme, flexible container scheduling, grayscale release, pre-release, blue-green deployment, continuous delivery.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.