Shulou(Shulou.com)06/02 Report--

Reveal the secret of Tencent worker bee: enterprise code management collaboration solution

The agile development of the Internet is inseparable from an efficient code management system. As the basic link of the R & D process, code management not only plays the role of upstream and downstream R & D links such as serial requirements management, continuous integration and continuous delivery, but also carries the construction of engineer culture such as pursuing code quality and encouraging code reuse. With nearly 30,000 R & D personnel, Tencent has a long product line and a wide variety of business. Different team sizes, technology stacks and R & D models all put forward different requirements for R & D collaboration, which also leads to uneven code base size and R & D process. At the same time, compiling system, publishing system and so on need to check out all the code. the higher the degree of automation, the greater the pressure to access the code base. Providing secure and stable code services, managing code repositories of different sizes, and supporting various types of R & D processes are the three major challenges facing code management. Based on the situation of the industry and the needs of its own development, Tencent chose to incubate its own Git system, worker Bee, based on Git.

First of all, it is necessary to solve the problem of server code base storage expansion, because a single storage node can not meet the TB-level growth of storage, there are two schemes that can be considered: custom data slicing and general distributed file storage. The advantage of distributed storage is that the underlying storage structure is shielded from the application layer, and the architecture is relatively simple, but for IO-intensive code management applications, it is too dependent on the IO performance of the distributed file system, and the portability is not strong. On the contrary, custom data sharding can freely control the sharding strategy and flexibly balance the resource load. In addition, on the underlying storage of each shard, it can also be combined with distributed storage to further expand data backup. Worker bee chooses the scheme of data slicing, takes the warehouse path as the routing rule, and implements cross-slicing operation in the application layer. Hundreds of thousands of warehouses are distributed in different clusters, which can realize cluster dynamic expansion and seamless migration between clusters.

After solving the problem of storage expansion, the increase in access volume has gradually exposed the performance bottleneck of the single machine, and the read and write of the code base are concentrated on one host, which will lead to the shortage of computing and memory resources. By analyzing the source, a large number of read requests come from the compilation and publishing system. In view of this scenario of more reads and less writes, the worker bee implements the read-write separation mode of one master and more slaves at the code library level, and the write requests are distributed to the host. Read requests are distributed to the slave according to the current load. The data synchronization between master and slave adopts Git native operation to ensure the atomicity and data consistency of the operation to the maximum extent. At the same time, as real-time hot standby data and remote cold backup, a complete code base data disaster recovery system is established to ensure data security. Figure 1 shows the complete code base back-end storage architecture.

Fig. 1 data fragmentation and read-write separation

How to manage super-large libraries has always been a difficult problem for code management tools. Git was originally designed to manage code files for text classes, but there will inevitably be dependent libraries and resource files in the project, especially Tencent Games business, which contains a large number of pictures, audio and video files, which makes this problem more prominent in Tencent. Worker Bee introduced the open source extension scheme Git LFS, which specializes in managing large binaries. As shown in figure 2, by storing these files outside the Git repository and keeping only the text pointers of the files in the Git repository, you can greatly reduce the size of the Git repository itself and speed up the cloning repository. At present, the single large game warehouse managed by the worker bee exceeds 2.5T, and the problem of the upper limit of the single library has been solved.

Figure 2 large file storage

In terms of the overall architecture, the worker bee has adopted the popular micro-service architecture in the industry. In figure 3, the protocol proxy service provides independent access links for HTTP, SSH and LFS protocols. The data service encapsulates database access, and the routing service addresses the data nodes of the back-end code base for each request. Business services are split according to the functions provided by the platform, such as code browsing, code statistics, code review, code search, and so on. In addition, the unified registry and configuration center provide global functions such as service discovery, service routing, exception breaker and service configuration. All microservices are designed to be stateless and can be easily scaled horizontally. With the ability of containerized deployment, the number of instances can be adjusted at any time to cope with high concurrency scenarios.

Figure 3 Micro-service architecture

If the code tools do not communicate with the upstream and downstream R & D processes, the role of code tools in improving R & D efficiency is very limited. One of the advantages of worker bees is its rich open ability to support third-party system integration access. Webhook push mechanism, which is convenient for the third party to subscribe to the code base to submit events, is widely used to automatically trigger the compilation and construction of continuous integration systems after submitting code. Commit Check interception mechanism is used to automatically pipelined code specification, defect detection, unit testing and other code checks before code integration, and strictly control the quality of integrated code by setting quality red lines. Worker Bee also provides a wealth of API in accordance with the restful standard, improves the private token and OAuth authorization mechanism, provides a safe and effective standardized access mode for third parties, and expands the application scenario of worker bee.

Within Tencent, worker Bee has become popular in six major business groups, serving thousands of business lines, including Wechat and QQ, with nearly 200000 code bases, 10 million daily visits and millions of calls to API every day, effectively improving the overall R & D efficiency of the company. Under the strategic goal of open source collaboration within the company, worker bee is also imperceptibly changing the way the company works. At present, more than half of worker bee projects have achieved completely open source internally, and discussion with Issue is becoming an effective way of communication for cross-team cooperation.

At the end of September this year, the project "Tencent worker Bee-Git-based R & D Engineering platform" stood out in the selection of the Chinese computer Society and won the 2019 CCF Science and Technology Award. It is reported that the "CCF Science and Technology Award" is awarded to outstanding achievements that have important discoveries, inventions, and original innovations in computer science, technology, or engineering, and have certain international influence in related fields. This award is a great affirmation of the worker bee. In the future, the worker bee will devote itself to exploring many aspects, such as the degree of code reuse, the experience of R & D integration, and the data measurement of the R & D process, and continue to dig deeply in the field of code management. to provide greater value for the company and the industry.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.