Shulou(Shulou.com)11/24 Report--

Use AI as an assistant to make the machine's night vision as clear as during the day.

Today, a new way to subvert existing thermal imaging technology is on the cover of Nature.

It comes from Purdue University and Michigan State University in the United States and graduated as a doctor from Zhejiang University.

By overcoming the problem of "ghosting" in traditional solutions, this method has a great advantage in benchmarking. It can not only see the texture and depth of the environment as clearly as during the day, but also perceive a variety of physical information other than RGB and thermal vision, which is quite good for machine perception, especially in the self-driving industry.

On the other hand, the author believes that this achievement can directly accelerate the fourth Industrial Revolution.

How can you see that? Let's take a look at the paper.

Night vision is as clear as during the day. At present, the more cutting-edge machine perception method is to use ubiquitous thermal signals to reproduce environmental information.

But it has a very obvious disadvantage, that is, it will produce "ghosting effect".

Specifically, the effect refers to a phenomenon (limited to night vision) in which three physical properties, namely temperature (T, physical state), emissivity (e, material fingerprint) and texture (X, surface geometry) are mixed in the photon flow due to the continuous emission of thermal radiation from the object and environment.

This phenomenon is mainly caused by the lack of texture of the environment / object, as shown in the following image:

Only when the light bulb is turned off can we see the geometric texture on the light bulb. Once the light emits, it disappears completely, and the blackbody radiation cannot be "turned off", so it means that the thermal image we get is always lack of texture. Can't see a completely real dark world.

Here, the author proposes a method called HADAR (heat-assisted detection and ranging), which takes the thermal photon flow as the input, records the hyperspectral imaging thermal cube, and solves the challenge of ghosting by TeX decomposition.

The authors say that TeX decomposition uses machine learning to vividly recover textures from cluttered thermal signals (such as the color part of the image below) and enables artificial intelligence algorithms to reach the limits of information theory, but so far, traditional RGB or thermal vision methods are difficult to do.

Its implementation is shown in the following figure:

According to the author, the physical inspiration of its structure comes from three aspects.

First of all, the TeX decomposition of thermal cubes depends on spatial patterns and spectral thermal characteristics, which inspires them to use spectral and pyramid (spatial) attention layers in the UNet model.

Secondly, because of the degeneracy of TeX, the following mathematical structure must be specified to ensure the uniqueness of the inverse mapping (α and β represent the index of the object, v is the wavenumber), so it is necessary to learn the thermal illumination coefficient V rather than the texture X. In other words, TeX-Net cannot be trained end-to-end.

Finally, the material library M and its dimensions are the key to the whole network.

In addition, the author also proposes a non-machine learning method, namely TeX-SGD, to generate TeX-vison as a supplement.

In the test, we can see that the HADAR method brings ultra-high precision.

As shown in the following figure, the first line shows that the ranging method based on the original thermal image has poor accuracy due to ghosting, while the second line shows that the texture and enhancement in HADAR are about 100 times more accurate than thermal ranging.

In the following scene (black cars, people and Einstein cardboard), we can see:

Vision-driven object detection in optical imaging (a) mistakenly identifies two people and a car, while the laser radar point cloud (c) not only identifies two people but also loses the car, only the HADAR method can bring a comprehensive understanding and accurately frame one person and one car.

Finally, this group of maps fully proves that the overall visual ability of HADAR at night is better than that of the most advanced thermal ranging method (GCNDepth), and its RGB stereo vision is basically at the same level as that tested during the day, that is, HADAR sees the texture and depth of the environment in the dark, just like during the day.

The author introduces Fanglin Bao, a researcher at Purdue University. He received his bachelor's degree in physics from Zhejiang University in June 2011 and his doctorate in optics in June 2016.

Fanglin Bao's previous research focused on the Casimir effect (quantum mechanics) in inhomogeneous systems, but now it extends to tensor networks, neural networks and their applications in quantum physics.

The authors are Zubin Jacob, a professor of electrical and computer engineering at Purdue University, and Vishnu Boddeti, an assistant professor of computer science and engineering at Michigan State University (who is recruiting students with a "strong mathematical background").

Paper address:

Https://www.nature.com/articles/s41586-023-06174-6

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.