Shulou(Shulou.com)06/01 Report--

How to carry out the custom tool chain analysis in Bazel, I believe that many inexperienced people are at a loss about this. Therefore, this paper summarizes the causes and solutions of the problem. Through this article, I hope you can solve this problem.

1 preface

The editor talks about two ways of customizing the tool chain for Bazel, Platform and Non-Platform. The reason for the existence of these two ways is the history of Bazel.

For example, C++-related rules use-cpu and-- crosstool_top to set a build target CPU and C++ tool chain, so that you can choose different tool chains to build C++ projects. However, none of these can correctly express the characteristics of "platform". Using this approach inevitably leads to clumsy and inaccurate building APIs. As a result, there is little mention of the Java tool chain, and the Java tool chain develops its own independent interface-java_toolchain. Therefore, the implementation of custom tool chain in non-platform approach (Non-Platform) does not have a unified APIs to regulate cross-platform construction in different languages. The goal of Bazel is to stand out in large, mixed-language, multi-platform projects. This requires more principled support for these concepts, including clear APIs, these API bindings rather than decentralized languages and projects. This is what the new platform (platform) and tool chain (toolchain) APIs implements.

If you do not understand the difference between Platform and Non-Platform, you may not understand what has been said above. Here is the popular difference between the two. For example, we compile C++ and Java mixed related projects, this project needs to be run on multiple platforms, so it involves tool chains under multiple platforms, while the tool chains of C++ and Java are different, non-platform approach. For Cellular tools, we need to specify the tool chain collection through-- crosstool_top,-- cpu to specify a specific device tool chain. For Java, you need to build Java-related content through-- java_toolchain,-- host_java_toolchain,-- javabase, and-- host_javabase. For such a hybrid project of C++ and Java, you need to specify so many inputs to compile the project completely.

If you use the platform approach, it will be easy. First of all, it is very simple to understand the concept of a platform. A platform is a collection of constraint values (constraint_value). For example, a platform can be determined by two constraint types, OS and CPU, or a platform can be determined by OS, CPU and GLibc_Version. We can bind C++-related platform constraints to the platform, and Java-related platform constraints to the platform, so that we can unify the mixed-language project into one platform, that is, once a platform is determined, you only need to execute the following command on the command line to compile the mixed-language project:

$bazel build /: my_mixed_project-- platforms==//:myplatform

At present, the construction of the platform in Bazel is not perfect. These APIs are not enough for all projects to use the platform. Bazel must also phase out the old APIs. This is not an easy task, because all languages, tool chains, dependencies, and select () of the project must support the new APIs. This requires an orderly migration sequence to keep the project working properly. Bazel's C++-related rules already support the platform, while Android-related rules do not. Your C++ project may not care about Android, but others may. Therefore, it is not safe to enable all C++ platform construction methods around the world. The construction methods of the platform are fully supported:

C/C++RustGoJava

The goal of future Bazel is to achieve $bazel build / /: all, which means that any project and target platform can be built on a single command line.

2 Non-Platform mode

Through the introduction of the previous chapter, the Non-Platform approach is built independently according to the nature of each project, such as C++-related crosstool_top and-- cpu. Java related-- java_toolchain,-- host_java_toolchain,-- javabase and-- host_javabase. In this section, we only implement the Non-Platform construction of C++ (of course, the complete platform construction method is not perfect, such as Apple and Android do not support platform construction).

In Bazel's official documentation, there is a tutorial on how to configure a C++ tool chain, see https://docs.bazel.build/versions/master/tutorial/cc-toolchain-config.html, mainly related to the built-in rules are: cc_common, cc_toolchain, cc_toolchain_suite.

Of course, some engineering optimizations can be done here:

The rules for generating CcToolchainConfigInfo can optimize its input configuration so that all major C++ compilers attrs = {can be configured by writing a toolchain configuration rule.

"include_paths": attr.string_list (doc = "The compiler include directories.")

"compiler_root": attr.string (doc = "The compiler root directory.")

Host_os: attr.string (default = "linux", doc = "The cross toolchain prefix.")

"toolchain_identifier": attr.string ()

"target_os": attr.string (default = "linux")

"target_arch": attr.string (default = "x86-64")

Cc_compiler: attr.string (default = "gcc", doc = "The compiler type.")

"extra_features": attr.string_list ()

}

Optimize multi-tool configuration generation so that all tool chain collections can be automatically generated from only one tool chain configuration file, so that the command line can switch to a tool chain def generate_toolchain_suite () via-- cpu:

Toolchains = {}

Native.filegroup (name = "empty")

For (platform, toolchain_info) in TOOLCHAIN_SUPPORT_MATRIX.items ():

Host_os = toolchain_ info [TOOLCHAIN _ HOST_OS]

Target_os = toolchain_ info [TOOLCHAIN _ TARGET_OS]

Target_arch = toolchain_ info [TOOLCHAIN _ TARGET_ARCH]

Compiler_root = toolchain_ info [TOOLCHAIN _ COMPILER_ROOT]

Include_paths = toolchain_ info [TOOLCHAIN _ INCLUDE_PATHS]

Toolchain_identifier = toolchain_ info [TOOLCHAIN _ IDENTIFIER]

Cc_compiler = toolchain_ info [TOOLCHAIN _ CC_COMPILER]

Base_name = "{platform} _ {target_os} _ {target_arch} _ {cc_compiler} _ {toolchain_identifier}" .format (

Platform = platform

Target_os = target_os

Target_arch = target_arch

Cc_compiler = cc_compiler

Toolchain_identifier = toolchain_identifier

)

Configuration_name = "% s_cc_toolchain_config"% base_name

Cc_name = "% s_cc_toolchain"% base_name

Toolchain_name = "% s_cc"% base_name

My_cc_toolchain_config (

Name = configuration_name

Include_paths = include_paths

Compiler_root = compiler_root

Host_os = host_os

Toolchain_identifier = toolchain_identifier

Target_os = target_os

Target_arch = target_arch

Cc_compiler = cc_compiler

Extra_features = []

)

Cc_toolchain (

Name = cc_name

Toolchain_identifier = toolchain_name

Toolchain_config = ":% s"% configuration_name

All_files = ": empty"

Compiler_files = ": empty"

Dwp_files = ": empty"

Linker_files = ": empty"

Objcopy_files = ": empty"

Strip_files = ": empty"

Supports_param_files = 0

)

If platform in toolchains.keys ():

Print ("s already exist!" platform)

Fail ("s already exist!" platform)

Else:

Toolchains [platform] = cc_name

Print ("toolchains =", toolchains)

Cc_toolchain_suite (

Name = "compiler_suite"

Toolchains = toolchains

)

TOOLCHAIN_SUPPORT_MATRIX = {

"hisi": {

TOOLCHAIN_HOST_OS: "linux"

TOOLCHAIN_TARGET_OS: "linux"

TOOLCHAIN_TARGET_ARCH: "armv7"

TOOLCHAIN_COMPILER_ROOT: ""

TOOLCHAIN_INCLUDE_PATHS: []

TOOLCHAIN_IDENTIFIER: ""

TOOLCHAIN_CC_COMPILER: "gcc"

}

"ubuntu_gcc": {

TOOLCHAIN_HOST_OS: "linux"

TOOLCHAIN_TARGET_OS: "linux"

TOOLCHAIN_TARGET_ARCH: "x86-64"

TOOLCHAIN_COMPILER_ROOT: "/ usr/bin/"

TOOLCHAIN_INCLUDE_PATHS: [

"/ usr/include"

"/ usr/lib/gcc"

"/ usr/local/include"

]

TOOLCHAIN_IDENTIFIER: ""

TOOLCHAIN_CC_COMPILER: "gcc"

}

"ubuntu_clang": {

TOOLCHAIN_HOST_OS: "linux"

TOOLCHAIN_TARGET_OS: "linux"

TOOLCHAIN_TARGET_ARCH: "x86-64"

TOOLCHAIN_COMPILER_ROOT: ""

TOOLCHAIN_INCLUDE_PATHS: []

TOOLCHAIN_IDENTIFIER: ""

TOOLCHAIN_CC_COMPILER: "clang"

}

"ubuntu_arm_linux_gnueabihf": {

TOOLCHAIN_HOST_OS: "linux"

TOOLCHAIN_TARGET_OS: "linux"

TOOLCHAIN_TARGET_ARCH: "aarch74"

TOOLCHAIN_COMPILER_ROOT: "/ usr/bin/"

TOOLCHAIN_INCLUDE_PATHS: [

"/ usr/arm-linux-gnueabihf/include/"

"/ usr/lib/gcc-cross/arm-linux-gnueabihf/7/include"

]

TOOLCHAIN_IDENTIFIER: "arm-linux-gnueabihf-"

TOOLCHAIN_CC_COMPILER: "gcc"

}

}

Example of profile design content:

Finally-- crosstool_top=//toolchains/cpp: {name of cc_toolchain_suite},-- cpu= {name of cc_toolchain}, cross-compilation can be achieved. The whole implementation content will not be posted here. To simplify the $bazel build command, you can write default configuration items to the .bazelrc file:

Build:compiler_config-crosstool_top=//toolchains/cpp:compiler_suite

Build:compiler_config-cpu=ubuntu_gcc

Build:compiler_config-host_crosstool_top=@bazel_tools//tools/cpp:toolchain

3 Platform Mode 3.1 platform 3.1.1 Overview

Bazel can build and test code on a variety of hardware, operating systems, and system configurations, using many different versions of build tools, such as linkers and compilers. To help manage this complexity, Bazel proposed the concepts of constraints and platforms. Constraints are dimensions that may be different from the build or production environment, such as the CPU architecture, the presence or absence of GPU, or the version of the compiler installed by the system. As described in Chapter 1, a platform is a specified selection set of these constraints that represent specific resources available in some environments.

Modeling the environment as a platform helps Bazel automatically select the appropriate tool chain for build operations. The platform can also be used in conjunction with config_setting rules to write configurable properties.

Bazel believes that the platform can play three roles:

Host (host): the platform Execution (execution) on which the Bazel itself runs: the platform on which the build tool performs build operations to produce intermediate and final output, and the execution platform settings are generally fixed. Target (target): the platform on which the final output resides and executes, such as the target platform output that may be cross-compiled on the execution platform, the target platform is variable. "

Note: here the Host platform is just an illustration of the role the platform plays, and has nothing to do with the actual writing of Bazel rules. In toolchain rules, there are only constraint settings for the execution platform and the target platform.

Bazel supports the following build scenarios for the platform:

Single platform build (default): the host, execution, and target platform are the same. For example, build the Linux executable on Ubuntu running on Intel x64 CPU. Cross-compile build: the host and execution platform are the same, but the target platform is different. For example, develop an iOS application running on MacBook Pro on macOS. Multi-platform build: the host, execution, and target platforms are different. 3.1.2 define constraints and platforms

The possible choice space for the platform is defined by using the constraint_setting and constraint_value rules in the build file. Constraint_setting creates a new dimension, so to speak, a collection of constraint values, and constraint_value creates a new value for a given dimension (constraint_setting); together, they effectively define enumerations and their possible values. Simply put, constraint_setting and constraint_value are a single-key, multi-valued map. For example, the following build file snippet introduces a constraint with two possible values for the glibc version of the system.

Constraint_setting (name = "glibc_version")

Constraint_value (

Name = "glibc_2_25"

Constraint_setting = ": glibc_version"

)

Constraint_value (

Name = "glibc_2_26"

Constraint_setting = ": glibc_version"

)

Constraints and their values can be defined between different packages in the workspace. They are referenced by tags and subject to the usual visibility controls. If visibility allows, you can extend existing constraint settings by defining your own values.

The platform rule `platform` [1] introduces a new platform with specific constraint values. A platform called linux_x86 has been created below, describing any environment that runs the Linux operating system on the x86 operating 64 architecture with glibc version 2.25.

Platform (

Name = "linux_x86"

Constraint_values = [

"@ platforms//os:linux"

"@ platforms//cpu:x86_64"

"glibc_2_25"

]

)

Note that it is wrong for a platform to set multiple values for the same constraint, for example, glibc_2_25 and glibc_2_26 cannot be set at the same time, because they are both glibc_version constraints.

3.1.3 generic constraints and platforms

To maintain ecosystem consistency, the Bazel team maintained a repository containing constraint definitions for the most popular CPU architectures and operating systems. These are all located in https://github.com/bazelbuild/platforms. Of course, you can also customize it yourself.

Bazel comes with the following special platform definition: @ local_config_platform//:host. The value of the host platform is automatically detected: indicates the platform of the system on which Bazel is running.

3.1.4 specify platform build

You can use the following command line flag to specify the host and target platform for the build:

-- host_platform: defaults to @ bazel_tools//platforms:host_platform--platforms: defaults to @ bazel_tools//platforms:target_platform and does not specify-- platforms. By default, it represents a platform for local build machines, that is, it is automatically generated by @ local_config_platform//:host. 3.2 tool chain

In the "preface" chapter, you can know that the platform can build mixed-language projects, and if you build for each language, you need to configure the tool chain and implement the platform constraint settings of the tool chain. In this way, you can combine the platform with the tool chain, which is similar to dependency injection.

A tool chain is a goal defined using the toolchain [2] rule that associates a tool chain implementation with a tool chain type. The tool chain type is a target defined using the tooclhain_type () rule (its utility can also be replaced by a string constant). A tool chain implementation is a goal that represents the actual tool chain by listing files that are part of the tool chain (for example, compilers and standard libraries) and the code required to use the tool chain. The tool chain implementation must return ToolchainInfo Provider (Provider can be thought of as the return value of a function). ToolchainInfo stores the configuration information related to the tool chain, and there is no requirement for what to store, that is, you can define any information you want to store.

Anyone who defines a tool chain needs to declare a toolchain_type target, a string identity that marks the category of the tool chain to avoid potential conflicts in workspaces loaded with multiple language rules. For example, Bazel officially provides a CPP logo: @ bazel_tools//tools/cpp:toolchain_type, while rules_go provides @ io_bazel_rules_go//go:toolchain to distinguish between tool chain categories.

For the tool chain implementation, which is the rule of the tool chain, which is called the tool chain implementation, it is consistent with Non-Platform 's tool chain goal. Of course, you can also use any rule that returns ToolchainInfo, not just cc_toolchain, for example, you can create a ToolchainInfo through platform_common.ToolchainInfo, and then create your own tool chain to implement the rules.

HELLOSDK = provider (

Fields = {

"os": "The host OS the SDK was built for."

"arch": "The host architecture the SDK was built for."

"root_file": "A file in the SDK root directory"

"libs": ("List of pre-compiled .a files for the standard library" +

"built for the execution platform."

"headers": ("List of .h files from pkg/include that may be included" +

"in assembly sources."

"srcs": ("List of source files for importable packages in the" +

"standard library. Internal, vendored, and tool packages" +

"may not be included."

"package_list": ("A file containing a list of importable packages" +

"in the standard library."

"hello": "The hello binary file"

}

)

Def _ hello_toolchain_impl (ctx):

Return [platform_common.ToolchainInfo (

Sdk = ctx.attr.sdk

Cflags = ctx.attr.cflags

)]

Hello_toolchain = rule (

_ hello_toolchain_impl

Attrs = {

"sdk": attr.label (

Mandatory = True

Providers = [HELLOSDK]

Cfg = "exec"

Doc = "The SDK this toolchain is based on sdk"

),

"cflags": attr.string_list ()

}

Doc = "Defines a hello toolchain based on SDK"

Provides = [platform_common.ToolchainInfo]

)

Tool chain implementation rules can be thought of as object-oriented classes, we can New a lot of examples, here New comes out of many different platform architectures or different versions of the tool chain. After the tool chain instance is created, you can bind the tool chain type, target platform, and run platform constraints through native.toolchain.

Users register the tool chain they want to use by calling the `register_ toolchains` [3] function in the WORKSPACE file or passing the-- extra_toolchains flag on the command line.

Finally, when Bazel starts building, it checks the constraints of the execution and target platform. Then select an appropriate set of tool chains that are compatible with these constraints. Bazel will provide the ToolchainInfo objects of these toolchains to the rules that request them.

If you want to know how Bazel chooses or rejects a registered tool chain, you can debug it with the-- toolchain_resolution_debug flag.

3.3 Construction of Platform + Toolchain implementation platform

Bazel's C++ rule uses the platform to select the tool chain. You need to set-- incompatible_enable_cc_toolchain_resolution. If it is not set, it will not work even if it is displayed on the command line plus-- platforms.

Similarly, Platform + Toolchain implements platform construction, and an example is provided in the official documentation, see: https://docs.bazel.build/versions/master/toolchains.html. The general steps are summarized here:

Create ToolchainInfo to create xx_toolchain. For example, C++ already has a built-in cc_toolchain, so there is no need for the first step and this step, that is, you do not need to implement the rule manually. You only need to configure cc_toolchain to use the native.toolchain associated tool chain, and set target_compatible_with, platform binding and tool chain type. Here, associated platform constraints also need to be created. To register all declared tool chains in the WORKSPACE file, you can specify registration with register_toolchains () or the command line-- extra_toolchains= through-- platforms= can be built through the platform.

Here again, in the same way as Non-Platform, we can reuse the configuration and cc_toolchain configuration parts of the tool chain for clocked configurations. It can be optimized in engineering:

Platform constraints def generate_constraint_set_platform () can be generated automatically according to the tool chain configuration:

Available = get_available_unique_platform_idetifier ()

Native.constraint_setting (

Name = "platform"

Visibility = ["/ / visibility:public"]

)

For item in available:

Native.constraint_value (

Name = item

Constraint_setting = ": platform"

Visibility = ["/ / visibility:public"]

)

Tool chains can be declared automatically based on the tool chain configuration, just as Non-Platform can declare native.toolchain in batches (

Name = toolchain_name

Exec_compatible_with = [

"@ platforms//cpu:%s" bazel_exec_platform_info ["cpu"]

"@ platforms//os:%s" bazel_exec_platform_info ["os"]

]

Target_compatible_with = [

"/ / platforms:%s" platform

]

Toolchain = "/ / toolchains/cpp:%s"% cc_name

Toolchain_type = "@ bazel_tools//tools/cpp:toolchain_type"

)

Configuration files can configure multiple tool chains on the same platform or tool chains on different platforms

-- incompatible_enable_cc_toolchain_resolution startup platform setting we can also put it in the .bazelrc global build configuration file, thus eliminating the need for command line typing:

Build-incompatible_enable_cc_toolchain_resolution after reading the above, have you mastered how to analyze the custom tool chain in Bazel? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.