Shulou(Shulou.com)06/01 Report--

In this issue, the editor will bring you about how to achieve face occlusion detection in video images based on TensorFlow. The article is rich in content and analyzes and describes for you from a professional point of view. I hope you can get something after reading this article.

When making a video call, we often need to do some real-time analysis of the picture, such as identifying people, cars, animals and so on. In this section, we will use the face monitoring module of Shixin Rubik's Cube to detect faces obscured by hands, as shown in the following figure:

Preliminary knowledge

The client side of Shixin Rubik's Cube uses TensorFlow as the machine learning engine and the server uses DJL. In this section, our sample program mainly demonstrates the face detection on the client side, and uses the FaceMonitor module to automatically load the model and predict the image in real time. The whole process is automatically completed by FaceMonitor. Developers do not need to know how to use TensorFlow for machine learning and use it out of the box.

Program interface

First of all, we need to choose an available camera and use the camera to capture real-time portraits. To tell us whether our face is obscured by our hands, we design an indicator light on the far right side of the toolbar.

Then we display a real-time portrait in the center of the page. Finally, in order to facilitate the observation of the data of the system, we design a log display container at the bottom of the screen. The interface structure is shown in the following figure:

Preparatory work

We use the MediaDeviceTool Media device tool to enumerate the available camera devices.

JavaScript Code:

MediaDeviceTool.enumDevices (function (devices) {var html = []; devices.forEach (function (desc) {if (desc.isVideo ()) {videoDevices [desc.getDeviceId ()] = desc; var c = [', desc.getLabel (),'']; html.push (c.join ('));}}) If (html.length > 0) {deviceSelect [XSS _ clean] = html.join ('');} else {deviceSelect [XSS _ clean] ='no available devices detected';}})

For the FaceMonitor module, the event we need to listen for is the Touched event:

JavaScript Code:

/ / get FaceMonitor module const monitor = cube.getModule ('FaceMonitor'); / / listen for Touched event monitor.on (FaceMonitorEvent.Touched, onTouched)

In the Touched event, we change the color of the status light to indicate whether the face is covered by the hand.

Set video display elements

Before starting the program, we need to set the video tag element and its container element on the interface to FaceMonitor so that FaceMonitor automatically starts face detection when the camera image is loaded into the video tag.

JavaScript Code:

Monitor.setup (videoContainer, cameraVideo); startup program

As with the previous process of starting Cube, we start the Shixin Rubik's cube by calling the start () method, and start the FaceMonitor module after successful startup.

JavaScript Code:

Monitor.start ()

We use MediaDeviceTool.getUserMedia () to start the camera, and after successfully starting the camera, we use MediaDeviceTool.bindVideoStream () to pass the camera's stream data to the video tag.

JavaScript Code:

MediaDeviceTool.getUserMedia ({video: {width: {exact: 640}, deviceId: videoDevice.getDeviceId (), groupId: videoDevice.getGroupId ()}}, function (stream) {videoStream = stream; MediaDeviceTool.bindVideoStream (cameraVideo, stream);}, function (error) {appendLog ('failed to acquire video data' + error);})

After starting up, you can see the shot of the camera in the video area, and you can try to use your hand to block the color change of the indicator light on your face. As in the first image in this example, the Touched event is triggered when a face is detected to be obscured by a hand.

In the whole process, you just need to set the parameters and start the module, and then FaceMonitor will automatically load the model and perform real-time screen detection. The test result is told to the application by event, and it can be completed in a few lines of code!

working principle

As mentioned earlier, we use TensorFlow to load the model of BodyPix to identify various parts of the human body.

For video call scenarios, FaceMonitor adjusts some parameters to speed up human body recognition, such as letting BodyPix identify only one person instead of multiple people to reduce the amount of computation to obtain a better real-time processing frame rate, and so on.

After the face is detected, we judge whether the hand covers the face by calculating the overlapping pixels of the nose, left face, right face and hand. Because it is a real-time calculation, the calculated value of each time does not have enough information to support the result. Therefore, we accumulate the recognition data after each sampling of FaceMonitor, and compare the cumulative score with our expected threshold score to determine whether it is effective or not.

Handle camera and microphone call permissions

The first time you run the program, the browser will prompt you whether to allow the program to call your camera and microphone, which requires you to choose allow so that the program can get the video stream and audio stream smoothly.

The media device management window of the Chrome browser is on the right side of the browser address bar:

The media device management window of the Firefox browser is on the left side of the address bar:

The above is how to achieve face occlusion detection in video images based on TensorFlow. If you happen to have similar doubts, please refer to the above analysis to understand. If you want to know more about it, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.