Shulou(Shulou.com)06/03 Report--

At present, K8s mainly supports CPU and memory resources. In order to support the scheduling and allocation of other hardware types of resources that users need to allocate as needed, K8s implements a device plug-in framework (device plugin framework) for resource integration of other hardware types. For example, machine learning now uses resources such as GPU. Today, let's take a look at its internal key implementation.

1. Basic concept

1.1Integration Mode 1.1.1 DaemonSet and Services

When we want to integrate local hardware resources, we can run a GRPC service on the current node through DaemonSet, through which local hardware resources can be reported and allocated.

1.1.2 Service Registration Design

When you need to communicate with kubelet to provide hardware services, you need to register first. Registration is achieved through the original underlying socket file and through the inotify mechanism of the Linux file system.

1.2 plug-in service awareness

1.2.1 Watcher

Watcher is mainly responsible for sensing the services registered on the current node. When a new plug-in service to be registered is found, a corresponding event will be generated and registered to the current kubelet.

1.2.2 expected state and actual state

The status here mainly refers to whether registration is required, because kubelet communicates with the corresponding plug-in service through the network. When there is a problem with the network or the corresponding plug-in service fails, it may cause the service registration to fail, but the socket of the corresponding service still exists, that is, the corresponding plug-in service still exists.

At this point, there will be two states: the expected state and the actual state. Because the socket exists, the expected state of the service actually needs to register the plug-in service, but in fact, for some reason, the plug-in service has not completed the registration, and the actual state will be adjusted continuously through the expected state to achieve consistency.

1.2.3 Coordinator

The coordinator is the core to complete the operation between the above two states. By calling the callback function of the corresponding plug-in, it actually calls the corresponding grpc interface to achieve the consistency between the expected state and the actual state.

1.2.4 plug-in controller

For each type of plug-in, there will be a corresponding controller, that is, to register and unregister the corresponding devices and to complete the underlying resource allocation (Allocate) and ListWatch operations.

two。 Plug-in service discovery

2.1 Core data structure type Watcher struct {/ / path of socket registered by plug-in service path string fs utilfs.Filesystem / / inotify monitors plug-in service socket changes fsWatcher * fsnotify.Watcher stopped chan struct {} / / Storage expected state desiredStateOfWorld cache.DesiredStateOfWorld} 2.2 initialization

Initialization is actually creating the corresponding directory.

Func (w * Watcher) init () error {klog.V (4). Infof ("Ensuring Plugin directory at% s", w.path) if err: = w.fs.MkdirAll (w.path, 0755) Err! = nil {return fmt.Errorf ("error (re-) creating root% s:% v", w.path, err)} return nil} 2.3 plug-in Service Discovery Core go func (fsWatcher * fsnotify.Watcher) {defer close (w.stopped) for {select {case event: =

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.