Shulou(Shulou.com)06/02 Report--

Author | Zhang Lei, Senior Technical expert of Aliyun Intelligence

Draw the key points

Twitter's infrastructure has shifted from Mesos to Kubernetes Ali Cloud Container platform team is about to open source Kubernetes advanced job management collection

At 7 p.m. western time on May 2, Twitter held a technical conference and Meetup at its headquarters in San Francisco. At the meeting, David McLaughlin, head of products and technology of Twitter Computing platform (Twitter Computing Platform), officially announced that the infrastructure of Twitter will shift from Mesos to Kubernetes.

The Mesos project was released in 2009, and Twitter was one of the early supporters and users of the Mesos project. As one of the most successful social media giants in the world, Twitter has attracted a lot of attention for its huge production cluster size (10,000 nodes). In 2011, Twitter began to develop the Aurora project based on the Mesos project to manage both its internal online and offline businesses, gradually becoming a spokesman for the Mesos community.

After nearly 10 years of continuous investment in the Mesos project, why did Twitter suddenly announce a comprehensive shift to the Kubernetes system? What is the architecture and design behind this eye-catching decision that underpins the 360-degree "gorgeous turn" of Twitter infrastructure?

Twitter Infrastructure faces New challenges in the Cloud era

Twitter was founded in 2006. Today, at least 500 million tweets are generated every day around the world. Under the challenge of rapid growth and huge amounts of data over the past decade, the evolution of Twitter infrastructure has always been a benchmark for technology companies around the world. Among them, excellent old scheduling frameworks like Mesos and choreography engines like Aurora inspired by Google Borg configuration management play an important role.

In fact, in Internet-level technology scenarios, self-building infrastructure relying on top engineers and mature technologies has always been the first choice for front-line non-cloud Internet vendors. It is in this process that relatively mature and low-level Mesos projects gain a large number of large-scale production environment deployment cases.

However, with the popularity of cloud computing and the rapid rise of containerized infrastructure projects with "cloud" as the core, this traditional Internet infrastructure selection method has gradually exposed many unprecedented problems. At this press conference, David gave a concise summary of these issues, citing the current challenges facing Twitter as an example:

1. The diversification and specialization of storage systems lead to a sharp increase in the complexity of traditional infrastructure.

Compared with the single assumption of the traditional technology architecture on the storage system (such as a set of Ceph dominating the world), the software architecture in the cloud era has brought an explosive growth in users' storage choices. Take Aliyun as an example, it can provide more than 10 types of storage services for users on the public cloud, among which there are countless subdivision schemes. With the continuous expansion and development of the infrastructure and software scale of Internet companies, the storage requirements of Internet software itself become more detailed and professional.

For example, in the "atypical" storage demand of Twitter,Local Persistent Volume, it has gradually become a mainstream solution in the process of balancing performance and cost. As the proponent of CSI (Container Storage Inerface), Kubernetes community not only has the most perfect Local PV mechanism, but also can completely smooth out the docking problems of dozens of other different storage services for users by virtue of standard interfaces and PV and PVC systems. This is undoubtedly very important in the increasingly complex and cloud-oriented development trend of Internet software architecture.

2. Mesos and Aurora systems are always drifting away from the "original cloud".

An important technological development trend in the cloud era is that the life cycle of software will gradually move closer to the form of "born on the cloud and grow on the cloud". This means that the core infrastructure project, which supports the software, is bound to evolve in the direction of "giving full play to the maximum value of the cloud".

Unfortunately, the Mesos and Aurora projects may have been released earlier and have never been able to turn the "cloud" into a "first-class citizen" in the entire infrastructure system. In contrast, the Kubernetes system has been advocating concepts such as "declarative API", "container design pattern" and "controller model" since its release, all of which are designed to help users develop, deliver and operate software in a "scalable, replicable and highly automated" way on the cloud. Today, these top-level architecture designs and a variety of best practices have been named "cloud native" by the majority of developers. This is also the biggest difference between the Kubernetes project and other competitors.

3. The cost of traditional multi-cloud and multi-cluster management remains high and rises rapidly in the foreseeable future.

In the traditional Internet architecture, self-built data center and infrastructure system is the first assumption of the whole software system. On the other hand, the role of "cloud" is more like a resource "spare tire" to cope with peak traffic in case of sudden traffic.

Under the guiding ideology of taking "cloud" as the auxiliary role, multi-cloud and multi-cluster can hardly become the top priority of the whole architecture. This makes multi-cloud and multi-cluster capabilities become the responsibility of the underlying resource docking layer, while losing direct relevance to the more important application development, delivery and operation and maintenance systems. This scheme can work in the short term, but the cost of long-term maintenance and iteration can easily be exceeded because of the ever-changing shape and high-speed iteration of the upper application itself.

In addition, the other extreme of this design is to move the overall infrastructure towards the "live" technological abyss: this is actually far beyond the technological capabilities of more than 90% of Internet companies. With the popularity of native cloud systems, "there are countless Kubernetes clusters on each cloud" has gradually become a new normal that application infrastructure can rely on.

This provides a new breakthrough idea for multi-cloud and multi-cluster management: as long as the software chooses Kubernetes-oriented architecture, design and implementation, then "multi-cloud, multi-cluster" will naturally become the default capability of application infrastructure. Under the trend that more and more Twitter business needs to be delivered in a multi-cloud and multi-cluster environment, Kubernetes, which fundamentally helps applications deliver quickly to multi-cloud, has become another important reason why Twitter chooses to change its technology system.

As a fast-growing and iterative Internet company, enterprises under the background of high pressure and fast pace often have no time to consider the standardization and compatibility of infrastructure architecture, which is also one of the main problems faced by Twitter. Therefore, in the process of this transformation, "Kubernetes Native" has become a key word that has been repeatedly emphasized.

"Kubernetes Native" Technology path of large-scale production Environment

As a fast-growing and iterative Internet company, enterprises under the background of high pressure and fast pace often have no time to consider the standardization and compatibility of infrastructure architecture, which is also one of the main problems faced by Twitter. Therefore, in the process of this transformation, "Kubernetes Native" has become a key word that has been repeatedly emphasized. At the press conference, Twitter announced a number of evaluation criteria for choosing the direction of Kubernetes Native.

1. Good open source technology and open source ecology

two。 All public clouds around the world offer Kubernetes services, so you don't have to worry about vendor locking.

3. Native management semantics for stateful business (Stateful Application)

4. The project itself iterates rapidly and has strong innovative ability and advanced nature.

5. Standard storage interface to help Twitter seamlessly migrate existing storage services

In the end, Twitter summed up the results of the assessment in one sentence: "We believe that using the Kubernetes project as the core dependence on the evolution of Twitter's infrastructure will be the right choice."

On this evolution path, Twitter has also announced a number of specific technical initiatives, such as:

1. Develop a stateful application management controller (TwitterSet) dedicated to Twitter

two。 Develop a node controller (NodeController) that meets the Twitter scenario

3. Customize Service Discovery components to interface with Twitter's own traffic management services

4. Write a job management controller compatible with Aurora semantics to migrate business on existing Aurora

5. Develop richer application release strategies and instance stability support

6. Transform the DSL of Aurora to dock Kubernetes and integrate the existing CI/CD system.

"the sheer size of Twitter's infrastructure has always been no secret, but at least today, size is no longer our primary concern, and we can see community leaders like Alibaba pushing larger K8s clusters into the production environment," David said.

David McLaughlin announced that the entire migration plan will continue from now until 2020. At that time, the entire Twitter technology stack will run on the containerized infrastructure based on Kubernetes, and present a multi-cluster combination form of "internal K8s cluster + public cloud K8s service".

Finally, David summarizes the future of Twitter and emphasizes that in 2020, Twitter's own software stack will be developed and delivered in a multi-cluster form "part of which runs on its own K8s cluster and the other part runs on the public cloud".

Obviously, Twitter has got a relatively clear and far-sighted answer to thinking about "how to maximize the value of its infrastructure capabilities through the cloud, and then let the company focus on its more valuable core business." More importantly, this choice is likely to lead to a real win-win situation between the company and Twitter engineers who are able to embrace Kubernetes.

Large-scale cloud native technology supported by world-class Internet companies

It is not difficult to see that the technical essence behind Twitter's move towards Kubernetes Native this time is to maximize the use of the core concepts and scalability of the Kubernetes project itself to achieve a balance between large-scale customized needs and community standards.

This is also a key approach that Alibaba is advocating in the community. Since 2018, Alibaba, together with a large number of top Internet companies such as Google and Facebook,Twitter,LinkedIn,Uber,Netflix,Pinterest, has launched a monthly Web-Scale Meetup in Silicon Valley in the United States to build confidence for more community "watchers" in Internet scenarios by sharing their actual landing practices.

At this conference, Twitter also invited engineers from Aliyun container platform team, such as Li Xiang, Zhang Lei and he Jian, as keynote speakers. Also invited to the press conference was Jago Macleod, the engineering and technical manager of the Kubernetes team of Google.

Ali Cloud Container platform team is about to open source Kubernetes advanced job management collection

At the press conference, the Aliyun Container platform team revealed the Kubernetes Advanced Job Management Collection (Kubernetes Workloads Advanced), which will be honed internally by open source next month. Kubernetes Advanced Job Management Collection will make full use of Kubernetes's "declarative API" and "controller model" to provide users with the ability to "upgrade in place" containerized applications that "rely on survival" in the Internet scenario, as well as more refined business publishing strategies. World-class teams such as Twitter, Pinterest and Netflix will join this innovative "Cloud Native Job Management" project.

In addition, the continuous evolution of Kubernetes in scale and performance improvement is another technical factor that can eventually turn Twitter from a "bystander" to a "practitioner". In this regard, Jago Macleod, the engineering and technical manager of the Google Kubernetes project, specifically introduced the ongoing research and cooperation between Google and Alibaba in this field.

In a recent attempt, engineers from both sides are working together to add "Bookmark" to a large number of WATCH operations in K8s, which will allow the creators of these WATCH operations to trace only a few historical changes other than "bookmarks" after a restart. Under certain circumstances, the performance of K8s APIServer will be improved by more than 40 times.

Kubernetes, the application-centered "highway"

In addition to technical and architectural evolution considerations, there is also a crucial non-technical factor in Twitter's "flamboyant turnaround" to Kubernetes.

The rapid growth of Twitter Company has given birth to its benchmarking basic software team, but it also reflects a problem that must be paid attention to: with the rapid development of Internet business, the company's basic software team soon begins to exceed its proper scale boundary, while the corresponding input-output ratio is stagnant.

So, as David mentioned at the beginning, the basic software development and architecture ideas based on "In-house" in Internet enterprises in the past are subtly changing with the popularity of cloud computing and cloud native ideas. With platform-level project standards like Kubernetes, Internet companies have been able to move their infrastructure to the cloud at less cost.

More importantly, due to the existence of the standard layer of Kubernetes, this "migration" itself will not form a deep-rooted vendor lock-in relationship like Netflix and AWS, but will give full play to the value of "cloud" and multi-cluster management capabilities while retaining most of the benefits of "self-research". The advantages brought by this change will become particularly prominent after the "AWS engineer" in an Internet company has become a "K8s engineer".

It is not difficult to see that the Kubernetes project is application-centric, connecting "cloud", "application developer" and "basic software team". This "highway" ability to communicate, connect and deliver is also an important reference for fast-iterative Internet companies like Twitter to think about the future evolution of their infrastructure. This transformation also makes it a reality that Twitter, a business organization with rapid growth, has always maintained a basic software team of dozens of people.

Write at the end

From the earliest Mesos "spokesperson" to today's overall shift to "Kubernetes Native", Twitter's move once again confirms the assertion that "Kubernetes has become the de facto standard for container layout." More importantly, Twitter's comprehensive embrace of cloud native technology is expected to provide a classic learning model for the industry's large-scale production-level cloud native technology.

Alibaba has invested a lot of technology in cloud native ecology since last year, and is gradually becoming an important leader in the landing of large-scale cloud native technology in the eyes of many world-class Internet companies, such as Facebook, Twitter, LinkedIn, Uber, Netflix, Pinterest and so on.

With the further popularity of cloud computing, the traditional Internet infrastructure has exposed many unprecedented problems, as well as the rapid rise of containerized infrastructure projects with "cloud" as the core, such as Kubernetes, which are prompting more and more world-class enterprises to think about how to embrace open source ecology and open technology standards with the help of "cloud" and cloud native technology. Prepare for a cloud-oriented digital future with strong iterative capabilities.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.