Shulou(Shulou.com)06/02 Report--

This article mainly explains "what are the knowledge points of Android compilation". Interested friends may wish to have a look. The method introduced in this paper is simple, fast and practical. Next let the editor to take you to learn "what are the knowledge points of Android compilation"!

Basic knowledge of Android compilation

No matter the compilation optimization of Wechat or the Tinker project, they all involve a lot of compilation-related knowledge, so I have a lot of research and experience in Android compilation. The compilation and construction process of Android mainly includes three parts: code, resources and Native Library. The whole process can refer to the construction flow chart of the official document.

Gradle is the official compilation tool of Android, and it is also an open source project on GitHub. As you can see from Gradle's update log, the current project is updated so frequently that there is a new version almost every month or two. For Gradle, I feel the most painful is the writing of Gradle Plugin, mainly because Gradle does not have perfect documentation in this area, so generally can only rely on the source code or breakpoint debugging methods. Recently, my company is going to use Gradle to build a channel packaging tool, and I have a deep understanding of the packaging and construction process of the project.

But compilation is so important that each company has a different situation and must forcibly build its own set of "wheels". Projects that have been open source include Facebook's Buck and Google's Bazel.

Why do you want to build your own wheels? There are mainly the following reasons:

Unified compilation tool. Facebook and Google both have dedicated teams responsible for compiling, and they want all internal projects to use the same set of build tools, including Android, Java, iOS, Go, C++, and so on. All projects will benefit from the unified optimization of compilation tools.

Code organization and management architecture. A very special feature of Facebook and Google code management is that all the projects of the entire company are placed in the same warehouse. So the entire warehouse is very large, so they will not use Git. Currently, Google uses Piper,Facebook, which is modified based on HG, and is also a distributed file system.

The ultimate pursuit of performance. It is true that Buck and Bazel perform better than Gradle, including their various compilation optimizations. But they are more or less customized, and they don't support external dependencies such as Maven and JCenter very well.

"programmers hate writing documents, and others don't write documents", so their documentation is relatively small, and it will be very painful if they want to do secondary custom development. If you want to switch compilation tools to Buck and Bazel, you need to make a lot of determination, and you need to consider collaboration with other upstream and downstream projects. Of course, even if we do not use them directly, their internal optimization ideas are also very worthy of our study and reference.

Gradle, Buck, and Bazel are all aimed at faster compilation and more powerful code optimization, so let's take a look at what they have done.

Compilation speed

Think back to our Android development career and how much time and life we wasted compiling it. As I said earlier, compilation speed is very important to team efficiency.

With regard to compilation speed, we are probably most concerned about the speed of compiling Debug packages, especially the speed of incremental compilation (incremental build), and we want to be able to debug more quickly. As shown in the following figure, we go through two steps of compilation and installation for each code verification.

Here, we look at the compilation speed of Android in terms of compile time and installation time.

Compile time. Compile the Java or Kotlin code to a ".class" file, and then compile it to a Dex file through dx. For incremental compilation, we want to compile as little code and resources as possible, ideally only the parts that change. But this is not feasible in most cases because of the dependencies between the code. At this time, we can only settle for second best and want to compile fewer modules. Android Plugin 3.0 and later versions use Implementation instead of Compile precisely to optimize dependencies

Installation time. We have to go through signature verification first, and after the verification is successful, there will be a lot of file copying work, such as APK files, Library files, Dex files and so on. After that, we also need to compile the Odex file, which can be very time-consuming, especially in Android 5.0 and 6.0. For incremental compilation, the best optimization is to apply the new code directly without having to reinstall the new APK.

For incremental compilation, let me first talk about Gradle's official schema, Instant Run. Prior to Android Plugin 2.3, it used the Multidex implementation. After Android Plugin 2.3, it uses the new Split APK mechanism in Android 5.0.

As shown in the following figure, the resources and Manifest are placed in Base APK, the code in Base APK is only the Instant Run framework, and the code of the application itself is in Split APK.

There are three modes of Instant Run, if it is heat exchange and temperature exchange, we do not need to reinstall the new Split APK, the difference between them is whether to restart Activity. For cold swapping, we need to reinstall the changed Split APK through adb install-multiple-r-t, and the application needs to be restarted.

Although in either mode, we do not need to reinstall Base APK. This makes Instant Run look good, but in large projects, its performance is still very poor, mainly because:

Multi-process problem. "The app was restarted since it uses multiple processes", if there are multiple processes in the application, neither heat exchange nor temperature exchange will take effect. Because most applications are multi-process, the speed of Instant Run is greatly reduced.

Problem with Split APK installation. Although the installation of Split APK will not generate Odex files, there will still be a copy of the signature checksum file (the ping-pong mechanism of the APK installation). This time takes a few seconds to dozens of seconds, which is unacceptable.

The problem follows the Javac. Prior to Gradle 4.6, if Annotation Processor was used in the project. Sorry, this modification and the modules it depends on require full javac, and the process is very slow and may take dozens of seconds. This problem will not be solved until Gradle 4.7. for a discussion of the causes of this problem, you can refer to this Issue.

You can also take a look at this Issue: "full rebuild if a class contains a constant". Suppose the modified class contains a variable of "public static final". It is also embarrassing that this modification and the modules it depends on require full javac. Why is that? Because constant pooling compiles values directly into other classes, Gradle doesn't know which classes might use this constant.

Ask the Gradle staff that the solution they offer is the following:

/ / the original constant definition: public static final int MAGIC = 23 return / replace the constant definition with the method: public static int magic () {constant 23;}

For large-scale projects, this is certainly not feasible. As I wrote in Issue, whether we really change to this constant or not, Gradle will have a mindless full amount of javac, which is definitely wrong. In fact, we can compare this code change to see if there is a real change in the value of a constant.

But students who may have used Ali's Freeline or Mogujie's super-fast compiler will wonder why their solutions don't encounter problems with Annotation and constants.

In fact, their solutions are faster than Instant Run in most cases at the expense of correctness. In other words, in pursuit of faster speed, they directly ignore the compilation products that Annotation and constant changes may lead to errors. As an official solution, Instant Run gives priority to ensuring 100% correctness.

Of course, Google people also found all kinds of problems with Instant Run. After Android Studio 3.5, devices after Android 8.0 will use the new scheme "Apply Changes" instead of Instant Run. So far, I haven't found any more information about this plan, but I think the Split APK mechanism should be abandoned.

For a long time, I have an ideal compilation scheme in mind. The Base APK installed in this solution is still only a shell APK, and the real business code is put into the ClassesN.dex of Assets. Its architecture is shown below.

No installation is required. The method similar to Tinker hotfix is still used, and only the modified and dependent classes can be inserted into the front of pathclassloader each time. Students who are not familiar with it can refer to the Qzone scheme in "the Evolution of Wechat Android Hot Patch practice".

Oatmeal . To solve the Odex time-consuming problem of ClassesN.dex in Assets when running for the first time, we can use cool techs: Oatmeal in ReDex as discussed in "installation package optimization". It can generate a fully interpreted Odex file in less than 100ms.

Close JIT. We turn off the JIT optimization of the virtual machine by adding android:vmSafeMode= "true" to AndroidManifest, so that there are no problems encountered by Tinker in Android N mixed compilation.

I also have a few suggestions for optimizing compilation speed:

Replace the compilation machine. For powerful companies, it is easiest to directly replace Mac or other more powerful devices as compilers.

Build Cache . You can detach most projects that change infrequently and use remote Cache mode to retain the compiled cache

Upgrade Gradle and SDK Build Tools. We should upgrade the latest compilation tool chain in time and enjoy the latest optimization results of Google.

Use Buck. Whether it's the exopackage of Buck or the incremental compilation of code, Buck is more efficient. But as I said earlier, if a large-scale project wants to switch to Buck, there are actually many concerns. Wechat connected to Buck in early 2014, but due to problems in collaboration with other projects, it switched back to Gradle in 2015.

By contrast, Hot Reload second compilation may be more attractive in Flutter, which is currently the hottest.

Of course, in recent versions of Android Studio, Google has also made a number of other optimizations, such as using AAPT2 instead of AAPT to compile Android resources. AAPT2 implements the incremental compilation of resources, which divides the compilation of resources into two steps: Compile and Link. The former resource files are compiled in binary format in Flat format, while the latter merges all the files and then packages them.

In addition to the introduction of D8 and R8 by AAPT2,Google, here are some test data provided by Google, as shown in the figure below.

So what are D8 and R8? Apart from the optimization of compilation speed, what other functions do they have? You can refer to the following introduction: Android D8 and R8

Code optimization

For Debug package compilation, we are more concerned with speed. But for the Release package, code optimization is more important because we care more about the performance of the application.

Next, I'll talk about ProGuard, D8, R8, and ReDex, four code optimization tools that we might use.

ProGuard

During the 12-minute compilation of the Wechat Release package, it takes 8 minutes to ProGuard alone. Although ProGuard is really slow, it is used in almost every project. After joining ProGuard, the process of building the application is as follows:

ProGuard mainly has three major functions: confusion, tailoring and optimization. Its whole processing flow is as follows:

The optimization includes more than 30 inline, modifiers, merge classes and methods, and so on. You can refer to the official documentation for specific introduction and usage.

D8

Android Studio 3.0 launched D8 and officially became the default tool in 3.1. Its purpose is to compile the ".class" file into a Dex file, replacing the previous dx tool.

In addition to the faster compilation speed of D8, another optimization is to reduce the size of the generated Dex. According to Google's test results, there will be about 3% to 5% optimization.

R8

R8, introduced in Android Studio 3.1, is more ambitious, with the goal of replacing ProGuard and D8. We can directly use R8 to turn ".class" files into Dex.

At the same time, R8 also supports confusion, tailoring and optimization in ProGuard. As R8 is still in the experimental stage, there are not many introduction materials on the Internet. You can refer to the following materials:

ProGuard vs. R8:

ProGuard and R8: a comparison of optimizers .

Jake Wharton God's blog recently has a lot of R8-related articles: https://jakewharton.com/blog/.

The ultimate goal of R8 is the same as D8, one is to speed up compilation, the other is more powerful code optimization.

ReDex

If R8 is the tool for ProGuard that you want to replace in the future, then ReDex, which is used internally by Facebook, has already done so. Many projects within Facebook have been switched to ReDex and no longer use ProGuard. Unlike ProGuard, it directly inputs Dex, not ".class" files, that is, it is optimized directly for the end product, WYSIWYG.

In the previous article, I have mentioned the ReDex project more than once, because its features are so powerful, please refer to the previous article in the column, "package size Optimization (1): how to reduce the installation package size?" ".

Interdex: class rearrangement and file rearrangement, Dex subpackage optimization

Oatmeal: directly generated Odex file

StripDebugInfo: removes Debug information from Dex.

In addition, ReDex methods such as Type Erasure and removing Aceess from the code are also very good features, which are helpful in terms of package size and application speed, so I encourage you to study and practice their usage and effect.

But the ReDex documentation is not updated for thousands of years, and it is mixed with some Facebook internal customized logic, so it is really very inconvenient to use. At present, I mainly study its source code directly, refer to its principle, and then directly implement it alone.

In fact, there are a lot of useful things in Buck, but there is still nothing mentioned in the document, so you still need to "read the source code".

Library Merge and Relinker

Multilingual split

Subcontract support

ReDex support

Continuous delivery

Gradle, Buck and Bazel all represent compilation in a narrow sense. I think broad compilation should include processes such as packaging and building, Code Review, code engineering management, code scanning, and so on, that is, continuous integration, which is often mentioned in the industry recently.

At present, the most commonly used continuous integration workers are Jenkins, GitLab CI, Travis CI and so on. GitHub also provides its own continuous integration services. Every big company has its own continuous integration solution, such as Tencent's RDM, Ali's Ferris wheel, Dianping's MCI and so on.

Let me briefly talk about some of my experiences and views on continuous integration:

Custom code checks. Each company has its own coding specification, and the purpose of code review is to prevent non-conforming code from being submitted to the remote warehouse. For example, Wechat defines a set of code specifications and writes special plug-ins to detect them. For example, log specification, can not directly use new Thread, new Handler, etc., and violators will be punished. Custom code detection can be implemented entirely by itself or by extending Findbugs plug-ins. For example, Meituan implemented the Android vulnerability scanning tool Code Arbiter using Findbugs.

Third-party code check. The most commonly used code scanning tools in the industry are paid Coverity and Facebook open source Infer, such as null pointers, multithreading problems, resource leaks and many other problems can be scanned. In addition to increasing the testing process, my greatest experience is the need to increase personnel training at the same time. I have encountered many developers in order to solve the scanned problems, null pointers are directly null, multithreaded locks are added directly, which may eventually lead to more serious problems.

Code Review . For Code Review, integrating GitLab, Phabricator, or Gerrit is a good choice. We must pay attention to Code Review, which is also an opportunity to show others our "great" code. And we should be the first Code Reviewer, and before giving Review to others, we should first look at the code from the perspective of a third party. In this way, we can not only respect other people's time, but also establish a good technology brand for ourselves by passing the test of our own.

There are many processes involved in continuous integration, and you need to take into account the current situation of your team. If you just blindly increase the process, sometimes it may be counterproductive.

At this point, I believe you have a deeper understanding of "what are the knowledge points of Android compilation?" you might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.