Shulou(Shulou.com)11/24 Report--

Michelangelo was reborn in the 16th century! The Nvidia team proposed Neuralangelo to reconstruct 3D large-scale scenes from 2D videos, reproducing the 5.5m statue of David perfectly.

Today, Nvidia recreates the 16th century Michelangelo "Neuralangelo".

Look, Neuralangelo "duplicated" the 3D version of the famous statue David, the details and texture of the marble are lifelike.

You know, the statue of David in the Florence Academy of Fine Arts is only 3.96 meters tall, with a pedestal of 5.5 meters.

It can even reconstruct the internal and external structure of a building, with roof tiles, glass panes, and all kinds of details.

All this is the magic of "nerve Neuralangelo". A new AI model proposed by researchers from Invida and Johns Hopkins University uses neural networks to reconstruct 3D objects.

The latest research has been hired by CVPR 2023.

Paper address: https://research.nvidia.com/ labs / dir / neuralangelo / paper.pdf, in particular, Neuralangelo can reconstruct "high-fidelity large-scale scenes" from mobile phone videos and UAV videos.

Isn't that the future, where you can easily turn a city or even a video from outer space into an immersive world and put it into a game to experience?

Netizens exclaimed, Invida hacked the "matrix" world!

Some people even say that Apple's XR technology, coupled with Neuralangelo, can create a "new worlds".

Effect demonstration

The headquarters of Nvidia

A shabby truck

When reconstructing 3D scene, it is often difficult to accurately capture repetitive texture patterns, uniform colors and strong color changes in the previous AI model of Ignatius statues.

To this end, the team proposed a new method-Neuralangelo, which combines the representation ability of multi-resolution 3D hash mesh with neural surface rendering.

Last year, Nvidia researchers created a new tool, 3D MoMa, that makes it easy to turn photos into 3D objects.

NeuralAngelo builds on this concept, allowing you to import larger, more detailed spaces and objects. What makes it special is that it can accurately capture repeated texture patterns, homogeneous colors and strong color changes.

By using "real-time neurographics primitives", the core of NVIDIA Instant NeRF technology, Neuralangelo can capture more subtle details.

The team's approach depends on two key elements:

(1) the numerical gradient used to calculate the higher derivative as a smoothing operation

(2) the optimization from coarse to fine is carried out on the hash grid with different levels of detail.

Even without auxiliary depth, Neuralangelo can effectively restore dense 3D surface structures from multi-view images, and its fidelity is significantly higher than previous methods, making it possible to reconstruct detailed large-scale scenes from RGB video capture.

The NeuralAngeloNeuralAngelo model is built on multi-resolution hash coding and volume rendering based on SDF.

Step 1: using numerical gradients to calculate higher-order derivatives can optimize beyond local elements by using numerical gradients of step sizes that match the spatial resolution of the hash grid. Compared with the analytical gradient, the numerical gradient plays the role of smoothing operation on SDF.

Step 2: gradually refine the level of detail by gradually reducing the step size of the numerical gradient and enabling a higher resolution hash mesh, the optimization effect can better restore a large area of smooth surface and fine geometric details. This learning process can gradually improve the level of detail.

Step 3: optimize NeuralAngelo using three optimization goals:

RGB synthesis loss

Loss of RGB reconstruction between the input image and the composite image.

Eikonal loss

The bottom SDF is regularized so that its surface normal is unit regular.

Curvature loss

Regularize the underlying SDF so that the mean curvature will not increase arbitrarily.

Now that NeuroLongoro is built, how does it work?

It can be said that Neuralangelo restores the whole process of Michelangelo's portrayal of David:

First of all, the model selects several frames of the object / scene from the 2D video and "sees" its depth, size, and shape. This process is like a sculpture artist's composition from multiple angles at the beginning.

The model then creates a rough 3D scene representation, just as the artist begins to chisel the shape of the main body.

Finally, the model optimizes the rendering to improve the clarity of the details, just as the artist imitates the texture of the fabric or humanoid by carefully decorating it.

In the qualitative comparison of DPU benchmarks, Neuralangelo produces more accurate and fidelity surfaces.

The following are the quantitative results of Neuralangelo in DTU data sets, and the model achieves good reconstruction accuracy and image synthesis quality.

In the qualitative comparison of different optimization schemes from rough to fine, there are artifacts on the rough surface when the analytical gradients AG and AG+P are used.

When using digital gradient (NG), a better rough surface can be reconstructed and the details can be smoothed.

Nvidia's solution (NG+P) produces smooth surfaces and fine details.

The end result is a 3D object or large-scale scene that can be used in virtual reality applications, digital twins or robot development.

Nvidia says Neuralangelo's ability to transform the texture of complex materials, including the roughness of roof tiles and the smoothness of marble, from 2D video to 3D objects significantly exceeds previous methods.

Ming-Yu Liu, senior director of research at Nvidia and author of the paper, imagined the significance of the research:

The 3D reconstruction capabilities provided by Neuralangelo will bring great benefits to creators and help them rebuild the real world in the digital world. This tool will eventually enable developers to import fine objects-whether small statues or large buildings-into video games or industrial digital twins.

Creative professionals can import these 3D objects into design applications for further editing for use in areas such as art, video game development, robotics and industrial digital twins.

The author introduces Zhaoshuo Li (Li Zhaoshuo)

Li Zhao Shuo is currently a doctoral student in computer science at Johns Hopkins University, mentored by Professor Mathias Unberath and Professor Russell H Taylor.

He has a strong interest in computer vision, computer graphics and deep learning, focusing on reproducing motion and structure from images.

In addition, he has a lot of hobbies, such as cameraman, mental health promoter, pet dog lover, surfer, skydiver, snowboarder …

Chen-Hsuan Lin

Chen-Hsuan Lin is a research scientist at NVIDIA Research who works in computer vision, computer graphics and artificial intelligence.

He received his doctorate in robotics from Carnegie Mellon University and the Nvidia Graduate Scholarship. Previously, he worked as an intern at Facebook AI Research and Adobe Research.

Lin is interested in solving the problems of 3D reconstruction, view compositing and 3D content production. Its research aims to move towards real 3D spatial intelligence by learning from Internet-scale visual data and giving artificial intelligence systems the ability of 3D perception and imagination at the human level.

Netizens commented that Nvidia scientist Jim Fan said

To let you know the speed of artificial intelligence in 3D modeling: the field goes from left (original NeRF reconstructed grid) to right (Nvidia's Neuralangelo) within 3 years.

Transferring reality to high-fidelity simulations is no longer a dream.

The new Neuralangelo model is a beast, and Nvidia decided to eliminate us, R.I.P. Photogrammetric software.

It's like Michelangelo in the digital world.

Some netizens say they want to know how much it costs to use it.

We can use drones in the factory, then send video to this model, make a digital twin, and use it to optimize our process.

For the significance of this technology, netizens think that it will have a great impact on the game industry.

Reference:

Https://blogs.nvidia.com/blog/2023/06/01/neuralangelo-ai-research-3d-reconstruction/

This article comes from the official account of Wechat: Xin Zhiyuan (ID:AI_era)

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.