Shulou(Shulou.com)06/02 Report--

On June 20, 2019, the third Enterprise Container Innovation Conference (Enterprise Container Innovation Conference, hereinafter referred to as ECIC) hosted by Rancher Labs (hereinafter referred to as Rancher) was held at the Sheraton Hotel in Beijing. This year's ECIC has a large scale, with a total of 17 keynote speeches throughout the day, attracting nearly 1000 container technology enthusiasts, and more than 10000 viewers watched the event on the live broadcast platform online.

Technical leaders from more than a dozen enterprises, including Rancher, Aliyun, Baiduyun, Ping an Technology, China Unicom, Fidelity Financial Technology, China Life Insurance, SmartX, Huatai Insurance, Xiamen Airlines, JFrog, New Oriental, Cisco and so on, brought a wonderful sharing of practical experience on enterprise container projects at the conference.

Cdn.xitu.io/2019/7/10/16bd9b3df6e5921c?w=1920&h=1280&f=jpeg&s=336198 ">

The cooperation between Xiamen Airlines and Rancher can be traced back to 2 years ago. In 2017, Xiamen Airlines completed the Xiamen Airlines cloud computing platform project, and built a trinity Xiamen aviation cloud computing platform based on Rancher, IaaS and CMP.

"the development of e-commerce in the aviation industry has given birth to a large number of business requests for access, and the platform needs to have strong stability and automatic elastic contraction, while the original traditional development model and software development model have been unable to meet the existing needs." Zhou Zhao, systems engineer and head of cloud platform of Xiamen Aviation Information Department, shared: "under such circumstances, we found Rancher and worked together to create a supporting platform for Xiamen Airlines'e-commerce strategy through independent research and development and the perfect combination of micro-service architecture and Rancher container platform."

The following is a transcript of the speech by Zhou Zhao, system engineer and head of Cloud platform of Xiamen Aviation Information Department:

Hello, everyone. I am Zhou Zhao, a system engineer of Xiamen Aviation Information Department. The theme I share with you today is the practice of building e-commerce center platform based on micro-service in Xiamen Airlines.

Xiamen Airlines is a domestic medium-sized airline with its main base in Xiamen.

In today's speech, I will share Xiamen Airlines' cloud computing platform with you. At the end of 2014, the Xiamen Airlines cloud computing platform project was put into trial operation as a whole, and the platform consists of three parts. First is the familiar CMP hybrid cloud management platform, then the IaaS cloud computing platform based on Open Stack architecture, and finally our protagonist today, Rancher container cloud platform.

1. Rancher 1.6 + ELK

When we first launched, Rancher was version 1.6. our first containerized application was ELK, when ELK ran on another part of the project, OpenStack's IaaS platform, and used RBD storage to persist the entire ELK. Our ELK is not only used in log analysis, but also widely promoted by ES in some query and search business.

Currently, our ELK is migrated on Rancher's K8S platform, which means the first crab we eat on Rancher and K8S is ELK.

The figure above is the original architecture diagram of ELK on 1.6, which is similar to community best practices, so I won't go into detail here.

two。 Capacitive e-commerce center

This part is our focus-Xiamen Airlines' containerized e-commerce platform.

The e-commerce strategies of China Taiwan and Xiamen Airlines are closely related. As a supporting platform, it can achieve ticket sales as the center, different types of passengers with different forms of travel, including different content of additional services, packaged together for full-process services, to improve the business level of our entire airline.

At present, Xiamen Airlines e-commerce China Taiwan docks all the company's direct sales channels and online OTA channels, that is to say, if you pick up your mobile phone or use your computer to inquire about Xiamen Airlines tickets and buy tickets on official channels or any other ticket purchase channels, you will pass through our e-commerce channel.

The picture above is the architecture diagram of Xiamen Airlines e-commerce in Taiwan. In this diagram, except for the Redis in the red part, the message queue, and the lowest common hardware LB device, all the other components are running on the Rancher 1.6 platform.

The above is a screenshot of the production environment of Xiamen Airlines, including all our services.

This is the first time in the entire e-commerce CCTV to meet a relatively big test, docking Ali Fei Pig, I took a picture on Prometheus at that time, and kept it as a souvenir. There is a processed data, you can take a look at it.

This is the picture taken two days ago when we were preparing for PPT. We can see the growth of our business on the side.

After talking about the achievements of Xiamen Airlines e-commerce platform, I would like to take this opportunity to once again thank Rancher engineers for their strong support to Xiamen Airlines e-commerce platform and container cloud platform.

3. The way for e-commerce to go online in China

Next, we will review some of the experience and experience accumulated by Xiamen e-commerce station in the process of containerization, as well as in the process of testing. The first thing that Rancher brings us to improve is the speed of DevOps update iteration.

We have realized the full-flow CI/CD pipeline in the whole development and test environment, and the whole e-commerce center now has a separate set of Harbor image warehouse.

When I was preparing for PPT, I briefly counted that the recent weekly image growth rate is more than 15G, which is the lowest performance, and sometimes there may be more than 30G image growth per week. A single service has been updated more than 100 times within six months of launch, not including the repair of BUG.

Second, I have specifically counted the utilization of containers in basic resources. If we compare our entire e-commerce console, if we deploy with virtual machines, Rancher will save more than 38% of computing resources, which is close to the 40% data shared by a guest in the morning.

Third, it is well known that containers have greatly improved in terms of flexible expansion and scale-out.

Finally, Xiamen Airlines team developed Publish-helper tools based on Rancher API to achieve near-imperceptibility, that is, grayscale release application updates on the K8S side of the Rancher 1.6 platform. This tool supports version updates for almost all production environments of Xiamen Airlines.

Let's share with you some of the pits we have stepped on. Just now we talked about some of the benefits of containerization, but other than that, we have stepped on three categories: network, storage and key application components.

First of all, the network, at the beginning of our basic platform is not relatively abundant resources. In the beginning, we used a gigabit network, and throughout the platform, we found that many containerized application clusters could not be initialized smoothly.

In the process of troubleshooting, we have a large proportion of network packet loss. Later, we analyzed that the characteristics of old devices, including network cards, which support network multi-session are poor. Later, we updated the overall equipment and switched to a 10-gigabit network with better performance and more characteristics.

But we did a full-link pressure test before the system went online, and found that the single container, especially the LB container in Rancher 1.6, its single container network IO is not high.

The picture above shows the data obtained from our intranet. The 10 Gigabit Nic only has 1.33G in VMWare and only 1G / s on our IaaS platform. This data can not meet our online needs.

After several tuning with Rancher engineers, we raised this data to about 4G per second of storage, which is basically similar to the network of K8S Flannel at that time.

In terms of storage, I'd like to share with you a few examples.

The first example is a pre-launch inspection, our Dockerfile is written by R & D engineers, we will check before the production line to avoid some irregularities. There is a Docker volume in it. When we first checked, we only checked the Docker compose and Rancher compose files to see if there was a definition of Docker volume. We later found that some developers wrote Docker volume instructions in Dockerfile. In conjunction with earlier versions of Docker, it has a feature that the life cycle of Volume is consistent with Container. As a result, if Container disappears, Volume data will be lost. In response to these problems, we later added some checks. During our internal handover, check not only Docker compose but also Dockerfile before launching.

There is also a problem on my side. When I was managing Rancher volume, I deleted all non-active Rancher volume, and someone in the app quickly reported that his data had been lost. It's a good thing it was a test environment. Later, when I was restoring the fault site, I found that it was a feature of Rancher. When Container was rebuilt, Volume would have a Detached status. The volume I deleted happens to be in the state of Detached. In fact, the volume in active status cannot be deleted, and the status of Detached can be deleted. As we summarized later, when managing Rancher Volume, we only focus on volumes with red inactive status, and other volumes are generally not cleaned unless there is a special need.

There are also more critical components that require data persistence, including Redis and message queues mentioned by our e-commerce station at the beginning. We also have full capacity of these components in the test environment, but when we go online, consider its data stability and the critical role of these components to our platform, or put it in the virtual machine. But we have also been testing key components of data persistence.

Before Redis and Oracle, we also did containerization tests for Cassandra and PG, and now we always have these two components running in the test environment. In addition, we have also done some extreme containerization tests. We have made the Oracle 12C containerization of a single node, which is also a test environment for running.

4. Rancher 2.x + Hybrid Cloud + Live data Center

We plan to focus on Rancher to implement the architecture design of hybrid cloud and multi-live data centers, which is the focus of our cooperation with Rancher this year.

Why does Xiamen Airlines want to be a hybrid cloud and multi-active data center? When Xiamen Airlines e-commerce Zhongtai comes online, the growth of the whole e-commerce Zhongtai is very rapid. Internally, in addition to Rancher, I am also responsible for the technology platform, including server virtualization and so on. Then I found that as soon as my server was in place, Rancher would immediately generate demand and eat most of the resources. Limited by our traditional management model, it is basically difficult for our data center to meet the needs of rapid expansion.

Xiamen Airlines e-commerce in Taiwan because of the rapid growth of business, docking systems are also more, its query and search pressure is very huge. In the later stage, in the process of the evolution of our e-commerce, its structure has been changing. We need more flexible and diverse service selections to meet the needs of our different business scenarios, and these issues are what the public cloud is best at, so we have the needs of the hybrid cloud.

With regard to the multi-live data center, Xiamen Airlines carried out the disaster recovery design of two places and three centers five years ago. In May 2017, we realized the one-click switch of the company's core internal system, flight operation control system. Then, in 2017 and 2018, we built disaster recovery in two places and three centers, including all the other core systems of the company, including those with three levels of evaluation. Now, we are going to evolve into a live data center.

We have summed up two principles, one is standardization, the other is service. We believe that based on standardization and service-oriented cloud applications, not only the upper layer can be applied to the bottom layer, but also can be run anywhere. My upper layer can take advantage of any bottom layer, and the bottom layer is transparent to the upper layer.

The evolution of this architecture is relatively clear, now only the standardization and service of the infrastructure layer, that is, what we call IaaS, gradually evolve to the upper layer, turning our data layer and our middleware layer into a standardized and service-oriented architecture. Finally, it evolves towards a comprehensive micro-service architecture.

This is my personal point of view, K8S can carry everything. At the same time, we also began to pay attention to the ecology of K8S. Based on the experience of Rancher 1.6, we focused on Istio, Heptio, Calico and other components in the K8S ecology. We have done some practice and research on these components.

This is technology sharing on our intranet, including Calico routing reflectors, large-scale K8S cluster maintenance, and Heptio-based backup and recovery.

I would like to focus on Calico. With the reform and evolution of containerization and micro-service in IT enterprises, it has now entered the deep water zone. A heavyweight network component like Calico is necessary.

With the deepening of the reform, there are more and more large-scale K8S clusters, and Flannel is a relatively stupid network component, which can not meet the needs of enterprises for container network. We really need an enterprise-class container network plug-in.

Aiming at several unified and standardized features of Rancher 2.2, one is K8S standard container orchestration, and Rancher 2.2's unified management of public cloud infrastructure services, public cloud hosting K8S services, and multi-cluster application management.

Through these features of Rancher 2.2, we want to form a bridge and link between Xiamen Airlines' hybrid cloud and multi-live data center, connecting our business in each cloud and each data center.

Before the launch of Rancher2.2, I brought a few more questions. First of all, there are K8S cluster backup and persistent volume backup management, as well as K8S security and security domain, multi-tenant isolation and other security issues. And the just mentioned network control, network isolation, and the last one is stateful application cluster management.

If K8S can solve these problems perfectly, it may not be far from the goal of carrying everything.

The above is my speech today, thank you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.