Shulou(Shulou.com)06/02 Report--

PouchContainer is Alibaba's open source rich container technology. GA was officially released in September 2018 and has fully reached the production level. The latest version continues the previous pace, continues to make continuous enhancements in Cloud Native (Kubernetes) ecological support and container security isolation, and begins to incubate the plug-in mechanism of PouchContainer, making it more friendly and convenient for ecological users to expand container functions through self-developed plug-ins.

With the support of the community, PouchContainer has been actively contributed by more than 2700 commit,100 community developers, including domestic first-tier Internet companies and container star startups.

Students who want to know more about PouchContainer can click this article for context: PouchContainer Rich Container Technology

Why should the internal and external versions be the same?

In general, in the process of open source, the first thing is to extract the parts that can be open source and release an "open source version", while the parts related to the internal infrastructure are left in the internal version. In subsequent development, the open source version evolves as the community evolves, and the internal version iterates with internal requirements. If the boundaries of the internal and external versions are not clearly defined, and the code synchronization is incomplete and untimely, the internal and external versions will go further and further and become two unrelated projects.

What motivates us to start doing the same thing between internal and external versions is the following three considerations:

Reduce maintenance costs: this is also the starting point of synchronous work. Reduce the workload of multi-version maintenance, but also facilitate team communication.

Give full play to the advantages of the community: absorbing the power of the community is an important consideration of open source, so that the power of the community can really play its value internally. The community will have some needs that we don't take into account, a stricter review that gathers community strength, a strong constraint CI that integrates with github, and so on.

Reduce the risk of functional synchronization: merging external code is very easy to conflict with different sources of internal and external versions. There are many code modules, and unexpected changes will be introduced in the process of conflict resolution, which is difficult to find in the process of merging.

This article will explain in detail how PouchContainer achieves internal and external version consistency.

Difference carding and boundary definition

PouchContainer has been open source for nearly a year, and until the differences between internal and external versions are sorted out, no one can tell the difference between the two versions. So I use Beyond Compare4 and other software, file-level comparison, one by one to find out the different internal and external code, and then sort out the differences between internal and external functional levels. Fortunately, the difference between inside and outside is smaller than I thought. There are mainly the following types of differences:

Interface compatibility with old internal systems

Backdoor interface for internal system

Logic for interfacing with internal infrastructure, such as storage, network

Bugfix and feature are not synchronized with each other.

The first three types of differences are customized content for PouchContainer within the group. Of course, there is some temporary logic that has to be added at the moment. In the follow-up development, it will be transformed and offline. The fourth category of difference is mainly due to developers' lack of awareness of version synchronization: bugfix is not synchronized to the community after an internal emergency fix. The bugfix and feature of community developers are not synchronized into the internal code in time.

After the differences are sorted out, it is necessary to clarify the functional boundaries of the internal and external. If you can open source, open source, and the unique functions will be retained. Smooth out non-functional inconsistencies.

Consistency transformation

The essence of version synchronization is to improve the scalability of the software, allowing the sharing of a set of core code to be customized for different business scenarios. So the problem we face is not only the consistency of internal and external versions, but also the consistency of three, four and multiple versions. This is also the basis on which PouchContainer enables other business scenarios. It is also a required course for internal use of open source projects, so the second step we do is to improve the scalability of PouchContainer through the plug-in mechanism. At present, we support API, container, daemon and volume,cri plug-ins. See the document for details.

The plug-in is designed to improve the scalability of the software, but does not allow the plug-in to change the original workflow. PouchContainer provides hook for key steps of container, daemon, volume, and cri. Take the daemon plug-in as an example, which provides hook for the start-stop interface. Plug-in implementers can run other programs, such as dfget, before daemon starts. Do some cleanup before daemon stops.

/ / DaemonPlugin defines places where a plugin will be triggered in pouchd lifecycle

Type DaemonPlugin interface {

/ / PreStartHook is invoked by pouch daemon before real start, in this hook user could start dfget proxy or other

/ / standalone process plugins

PreStartHook () error

/ / PreStopHook is invoked by pouch daemon before daemon process exit, not a promise if daemon is killed, in this

/ / hook user could stop the process or plugin started by PreStartHook

PreStopHook () error

}

The API plug-in allows plug-in implementers to extend, delete, and modify API by passing the routing table to the plug-in. This gives the interface a lot of flexibility.

Import "github.com/alibaba/pouch/apis/server/types"

/ / APIPlugin provide the ability to extend PouchContainer HTTP API and change how handler behave.

Type APIPlugin interface {

/ / The default handler of each API would be passed in while starting HTTP server.

/ / UpdateHandler could register extra HTTP API to PouchContainer server

/ / change the behavior of the default handler.

UpdateHandler ([] * types.HandlerSpec) [] * types.HandlerSpec

}

Through the plug-in transformation, most of the internal PouchContainer customization logic is implemented in the plug-in. The plug-in has a separate file directory that rarely conflicts when the code is merged. Then commit the internal plug-in logic and other differences one by one to the open source branch. To achieve the homology of internal and external versions.

Stability guarantee

The open source version of PouchContainer represents the general function. If the new features iterated by external developers on the general version are not available within the group, how to ensure that after the external functions are synchronized to the internal version, the existing internal functions will not be affected?

First of all, the internal version is covered by a complete set of tests, and the internal tests include tests for internal scenarios on the basis of open source tests. After passing the internal test, we believe that the version meets the requirements of the internal scenario, and the open source version does not affect the internal stability. If the test fails, there are two options, one is to re-evaluate the open source common functionality for code defects, and the other is to patch in the internal warehouse. In order to ensure the stability of the open source code after synchronization to the interior.

Establish a new order

Let's take a look at what the workflow looked like before that. Developers will mention code in the internal warehouse and open source warehouse respectively. Urgent requirements will first mention merge request in the internal warehouse, and less important requirements will mention PR in the community first. Some people regularly put the open source branch merge into the internal warehouse. There are several problems. First, there is an internal testing process, which may not be as convenient as travisCI or circleCI to interface with github, and some designed tests will not even run in the internal warehouse. Second, before plug-in, there are two different implementations of some functions inside and outside, which almost always conflict with each manual merge code. It is easy to introduce unexpected changes in the process of conflict resolution, and continue to conflict next time.

Git flow as shown in the picture

After completing the consistency transformation, we set up a set of rules to ensure that version separation will not happen again.

As a non-private enhancement in principle, you should first enter the internal version through the synchronization mechanism after the community submits the PR,merge.

If time is urgent, bugfix will submit it on the build first. The subsequent commiter is responsible for cherry-pick it to the community. If the community review finds that it needs to continue to modify, it will modify and mention another commit to ensure that the commit does not conflict with the internal warehouse.

Code synchronization. The robot periodically submits the merge request to synchronize the open source internally.

In merge, it is guaranteed to be fast-forward, so that the internal and external commit is one-to-one, reducing conflicts.

Summary

Open source can help the project absorb external nutrients and accelerate the evolution of the project. In the process of consistent transformation, help developers clarify the boundaries of internal and external versions, create the same core code, improve the customizable ability of the core code, and better serve different scenarios.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.