Shulou(Shulou.com)06/02 Report--

Transparent matting problem as a kind of matting problem, its method and model construction are different from the usual model. Transparent matting needs to take the influence of ambient light and refractive index into the calculation, but the general refractive light map is very difficult to obtain, so the model of transparent matting has been difficult to establish or achieve satisfactory results in the past. The Dama visual algorithm team obtained it through two-branch decoder (Object Mask). Opacity prediction), color correction module, to achieve high-precision transparent matting of the image.

Transparent matting vs non-transparent matting

The matting problem of an object can be defined as a linear combination of foreground color F, background color B and Alpha matte for a given image I.

For transparent objects, the color shown in the eyes of the observer is a mixture of its foreground color, background color, and ambient light through the self-reflection and refraction of the foreground object, so its formula is more complicated:

Φ represents the influence of ambient light, which is the double integral of the product of all rays E (w) and reflectivity equation R at all points. The solution is very complex [28], so it is very difficult to realize accurate transparent matting. Therefore, the goal of the existing transparent matting research is also to achieve the visual feeling of "real" matting, not the pursuit of completely real matting results.

Existing research

In the case of providing both the original image and the corresponding trimap, SOTA's Matting algorithm can process semi-transparent objects (such as the effect of GCA-Matting in the following picture), but it is difficult to obtain tripmap in the actual image processing, which limits the use of this kind of algorithm in business.

TOM-Net regards transparent matting as the estimation problem of folded jet. The network supports the input of a single image through a three-branch codec network to predict the Object Mask,attenuative mask,flow mask (refraction flow graph) of the image respectively, and can be further synthesized in the new background through the refraction flow information. The limitation of this method is that it assumes that all objects must be colorless transparent objects, and the refraction flow graph needs to be used as label in the training process, but the refraction flow graph is very difficult to obtain in the real world, so the training number of this method can only rely on graphics synthesis, which is not consistent with the distribution of the real transparent image (the semantic rationality of the image is in doubt, for example, the glass is suspended in front of the mountain). After our test on the actual data, the performance of this method in the actual image is not ideal.

Segmenting Transparent Objects in the Wild proposes a real-world transparent object segmentation network based on semantic branch and edge branch structure, enhances the segmentation accuracy of transparent objects through boundary attention module (Boundary Attention Modeule), and publishes the largest number of transparent object segmentation tagging data set Trans10K. However, the proposed algorithm and the published data set are dealt with to the semantic segmentation level, and there is no further processing on the transparency of the object.

Problem simplification

Considering that the transparent matting problem itself is difficult to solve, and the data construction is also very difficult, in the actual application scene, in order to ensure the generalization ability and matting effect of the algorithm at the same time, we simplify the problem. We assume that the transparent part of the object to be processed is colorless, and the background color distribution of the environment is relatively uniform. Under such conditions, the color of the background self-illumination or reflected light can be considered as a globally consistent color, and there will be no superposition of multiple colors, and the estimation of Φ is only related to the background color. In particular, if the color of the background is predicted, the background noise can be suppressed and removed by introducing it into Φ as a priori.

Model design

Our model is inputted as a single image, and its deep features are extracted through the encoder network. The decoder is designed into two branches. The first branch of the decoder adopts the weight of the decoder matting on the non-transparent object. This branch pays attention to the segmentation and extraction at the semantic level, and strives to obtain the image region of the object completely and accurately, namely Object Mask.

The second branch focuses on the prediction of image object opacity (Opacity). Under the assumption of uniform background, the branch predicts the similarity between each pixel of the image and the background, and the high similarity indicates the high transparency of the medium (such as air, glass). Because Branch 2 has no semantic constraints during training, it is easy to have the influence of noise in the non-subject area, so the fusion of the two can constrain the transparent information within the scope of the subject. With the realization of the fusion module, Opacity and ObjectMask can be fused at the image level, and they can also be spliced in depth dimensions to predict through the further network.

Finally, for scenes where the background color is known to be a priori (such as the known green screen), we can introduce a color correction module to remove the noise from the background. For scenes with unknown background color but low saturation, matting results are still available.

Results and application

When the background color is known a priori, the background noise can be removed through the color correction module (left to right: live shot, Opacity, direct matting result, color deviation correction result)

For scenes with unknown background color but low saturation, matting results are still available.

More results

At present, in the vehicle segmentation algorithm, we have realized the idea based on transparent matting to improve the effect of semi-transparent window area, so that the vehicle after matting can be more natural and harmonious with the new background. At present, vehicle segmentation has been launched on Aliyun Visual Intelligent Open platform. You are welcome to try it.

Summary and prospect

In the face of more diverse objects in real scenes, the current transparent matting algorithms still have the following shortcomings, which need to be further explored and solved:

1. The amount of real transparent object image data is seriously insufficient and difficult to label; 2. The prediction of subject opacity map (Opacity) is easily affected by noise in the image. 3. After obtaining the Opacity diagram, how to remove the background color under the condition of unknown background prior and high background color saturation (such as the cup in the following picture is pan-blue as a whole)

Later, we will consider further extracting the features of the background, introducing the a priori knowledge of the background into the estimation of transparency, increasing the RGB offset output information, and trying to correct the color of the foreground object.

Article source: https://developer.aliyun.com/article/766602?groupCode=aliyunmit

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.