Shulou(Shulou.com)06/02 Report--

Author | Xiao Changjun (Dome Valley) Aliyun Intelligent Business Group Technical expert

Introduction: with the evolution of the cloud native system, how to ensure the stability of the system is greatly challenged. Chaos engineering injects faults into the system through the idea of anti-fragility, finds system problems in advance, and improves the fault tolerance of the system. The ChaosBlade tool can perform chaos experiments through declarative configuration, which is simple and efficient. This article will focus on ChaosBlade and cloud native related experimental scenarios.

ChaosBlade introduction

ChaosBlade is an open source chaotic experimental execution tool developed by Alibaba, which follows the chaotic experimental model. It has the characteristics of high scene richness, easy to use, and can easily expand the experimental scene. Soon after open source, it was added to CNCF Landspace and became a mainstream chaos tool.

Experimental scene

The supported experimental scenarios are as follows:

Basic resource scenarios: CPU load, memory usage, disk IO load, disk usage, network delay, network packet loss, network masking, domain name inaccessible, shell script tampering, killing process, process Hang, machine restart, etc.

Application service scenario: supports experimental scenarios in Java application and C++ application. Java has rich scene components, such as support for Dubbo, RocketMQ, HttpClient, Servlet, Druid, etc., and supports writing Java or Groovy scripts to implement complex experimental scenarios.

Container service scenarios: experimental scenarios that support Kubernetes and Docker services, including node, pod and container resources, such as Pod network delay, packet loss, etc. Chaotic experimental model

All the above experimental scenes follow the chaotic experimental model, which is divided into four layers, including:

Target: experimental target. Refers to the components of the experiment, such as containers, application frameworks (Dubbo, Redis), etc.; Scope: the scope of the implementation of the experiment. Refers to the specific machine or cluster that triggers the experiment; Matcher: experimental rule matcher. According to the configured Target, define the relevant experimental matching rules, which can be configured. Since each Target may have its own special matching conditions, such as Dubbo in RPC domain, it can be matched according to the services provided by service providers and services invoked by service consumers, and Redis in cache domain can be matched according to set and get operations. Action: refers to the specific scenarios of experimental simulation. Different Target results in different implementation scenarios, such as disks, which can be drilled full and disk IO read and write high. If it is an application, we can abstract the experimental scenarios such as delay, exception, return specified value (error code, large object, etc.), parameter tampering, repeated invocation and so on.

For example, an IP is an application on 10.0.0.1 machine, and the delay of calling com.example.HelloService [@ 1.0.0] () Dubbo service is 3s. Based on this model, it can be described as an experiment on Dubbo component (Target). The scope of the experiment is 10.0.0.1 host (Scope), and the delay of calling com.example.HelloService [@ 1.0.0] () (Matcher) service is 3s (Action). The corresponding chaosblade command is:

Blade create dubbo delay-time 3000-service com.example.HelloService-version 1.0.0

Therefore, this model is very simple and clear expression of the experimental scene, easy to understand. The cloud native experiment scenario below is also based on this model definition.

Implementation Scheme of Experimental scenario for Cloud Native

According to the above experimental model, the chaotic experimental scene is defined as a resource in Kubernetes and managed by a self-defined controller, which can be executed through Yaml configuration or direct execution of blade commands.

ChaosBlade Operator defines a resource controller and deploys a chaosblade-tool pod on each node to perform chaos experiments in the form of daemonset. Different experimental scenarios are implemented in different ways. For example, in the Node experimental scenario, the chaosblade-tool deployed above can be executed internally, while in the experimental scenario in Container, the controller copies the chaosblade package to the target Container for execution.

How to install the necessary components

To install ChaosBlade Operator, download chaosblade-operator-0.0.1.tgz from the address and install it using the following command:

Helm install-namespace kube-system-name chaosblade-operator chaosblade-operator-0.0.1.tgz

Installed in the kube-system command space. After ChaosBlade Operator starts, chaosblade-tool Pod and a chaosblade-operator Pod are deployed on each node. You can view the installation results with the following command:

Kubectl get pod-n kube-system-o wide | grep chaosblade

Perform an experiment

There are two ways to execute:

One is to configure yaml to execute using kubectl, and the other is to directly use the blade command in the chaosblade package.

The following example is to specify a node to do the CPU load 80% experiment.

Yaml configuration mode

ApiVersion: chaosblade.io/v1alpha1kind: ChaosBlademetadata: name: cpu-loadspec: experiments:-scope: node target: cpu action: fullload desc: "increase node cpu load by names" matchers:-name: names value:-"cn-hangzhou.192.168.0.205"-name: cpu-percent value:-"80"

As shown above, after configuring the file, save it as chaosblade_cpu_load.yaml, and use the following command to execute the experiment scene:

Kubectl apply-f chaosblade_cpu_load.yaml

You can view the execution status of each lab with the following command:

Kubectl get blade cpu-load-o json

See more lab scenario configuration examples.

Blade command execution mode

Download the chaosblade toolkit and unzip it. For the same example above, use the blade command to execute as follows:

Blade create k8s node-cpu fullload-- names cn-hangzhou.192.168.0.205-- cpu-percent 80-- kubeconfig ~ / .kube/config

Execution using the blade command returns the execution result of the experiment.

Modify the experiment

The yaml configuration file supports dynamic modification of the scene. For example, if you adjust the above cpu load to 60%, you only need to change the value of the above value from 80 to 60, for example:

ApiVersion: chaosblade.io/v1alpha1kind: ChaosBlademetadata: name: cpu-loadspec: experiments:-scope: node target: cpu action: load desc: "cpu load" flags:-name: cpu-percent value: "60"-name: ip value: 192.168.0.34

Then use the kubeclt apply-f chaosblade_cpu_load.yaml command to perform the update.

Stop the experiment

You can stop the experiment in three ways:

Stop according to the name of the experimental resource

For example, in the above cpu-load scenario, you can execute the following command to stop the experiment:

Kubectl delete chaosblade cpu-load

Stop through yaml configuration file

Specify the yaml file created above to delete. The command is as follows:

Kubectl delete-f chaosblade_cpu_load.yaml

Stop through the blade command

This method is limited to labs created with blade, and stop using the following command:

Blade destroy

Is the result returned by executing the blade create command. If you forget, you can use the blade status-- type create command to query.

Uninstall chaosblade operator

Perform helm del-- purge chaosblade-operator uninstall, which will stop all experiments and delete all created resources.

Summary

Based on the chaotic experimental model, ChaosBlade amicably combines Kubernetes resource control. The deployment is simple and the use is simple, and the experiment is controllable. In addition, ChaosBlade implements many domain scenario executors based on the experimental model, which can easily expand the experimental scene, which can be detailed in the list of projects in the appendix.

Community co-construction

Since ChaosBlade was opened up, more than 30 contributors have joined and many enterprises have often thanked you. At the same time, more and more people are welcome to participate, so that ChaosBlade becomes more powerful, covers more scenes, and becomes a stable and universal chaos engineering tool for all enterprises.

Contributions can take the form of mentioning bug, submitting code, writing documentation, supplementing unit tests, participating in problem discussions, and so on. ChaosBlade believes that in the open source world, any help is a contribution.

Appendix

The list of projects is as follows:

ChaosBlade CLI (call entry) ChaosBlade experimental model defines basic resource scene executor Docker scene executor Kubernetes scene executor Java application scene executor C++ application scene executor

"Alibaba Yun × × icloudnative × × erverless, containers, Service Mesh and other technical fields, focusing on cloud native popular technology trends, cloud native large-scale landing practice, to do the best understanding of cloud native development × ×

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.