Shulou(Shulou.com)11/24 Report--

Speaking of the cutting-edge research progress in the field of VR (virtual reality), Meta, the most popular VR equipment supplier in the world, is definitely leading the market.

Mark Zuckerberg, CEO of Meta, recently revealed the latest layout of Meta in the meta-universe in an interview. Meta will launch a new VR headset, Meta Quest Pro, in October.

It is reported that the biggest highlight of the head is that it has facial and eye movement tracking functions, which allows virtual digital avatars to make eye contact. Brad Lynch, a hardware analyst, believes that Meta Quest Pro will use an improved XR2 processor, which will have 30 per cent year-on-year performance improvement over existing XR2 processors. Analyst Guo Mingyi predicts that Quest Pro will use a monocular 2160 × 2160 resolution panel.

In the interview, Zuckerberg also talked about meta-social. He believes that social media carriers, from text, images to video, are evolving towards greater information capacity, and the next carrier that can carry more information than video is in the field of VR.

Earlier, Zuckerberg personally recommended his social platform Horizon Worlds and posted a VR selfie. Although the photos were ridiculed by the whole network, Zuckerberg brought a wave of popularity to Horizon Worlds.

▲ Zuckerberg's VR selfie

Recently, Meta has acted frequently in the field of meta-universe. In June this year, Reality Labs, Meta's VR division, launched four VR head display prototypes, which showcased four key technological evolution directions of future VR devices: high resolution, high dynamic range (HDR), light and thin shape, and zoom technology optimization to reduce vertigo. At the SIGGRAPH conference held in August, Douglas Lanman, the head of display system development of Meta, gave a speech, introducing us to the top ten challenges of breaking the bottleneck of VR technology.

At the SIGGRAPH 2022 conference, Sam Sprigg, a reporter from foreign media Auganix, specially experienced the VR prototype Starburst, which featured HDR functions. The Oculus Quest 2 is the pinnacle of Meta's VR all-in-one machine so far, with more than 10 million units shipped last year, making it the best-selling VR product in history, while the peak brightness of the Starburst is 200 times that of the Quest 2.

So, what exactly is the layout of Meta's meta-universe? How does the Meta prototype feel? Where has Meta invested tens of billions of dollars in R & D? With these questions in mind, let's take a look at the latest developments of Meta in the field of VR.

01. In order to explore four major research areas, Meta released four prototypes in June this year, and Meta Reality Labs unveiled four VR head display prototypes in different technology areas.

Butterscotch explores resolution technology, Starburst explores HDR technology, Holocake 2 explores the light and thin shape of VR head, and Half Dome series explores zoom technology. These four prototypes represent the latest progress made by Meta in the research and development of four VR head displays.

Earlier, Mark Zuckerberg, CEO of Meta, proposed the concept of visual Turing test (Visual Turing test). To put it simply, the assessment standard of the visual Turing test is whether the virtual image can be confused with the real image, so that the human eye can not distinguish the virtual image from the real image. Zuckerberg said that the visual Turing test has two key points, namely photorealistic and telepresence.

Comparison of head display Resolution of different VR in ▲

One of the ways to make the image more realistic is to improve the resolution of the image as much as possible. To this end, Meta developed the prototype Butterscotch, which is characterized by a "retinal level" resolution.

Butterscotch's PPD reaches 55, which is very close to the "retinal resolution" level of 60 pixels per degree angle, which is 2.5 times that of Quest 2. Yang Zhao, an optical scientist in the DSR team of Meta Reality Labs research, said that in Butterscotch's custom lens module, there are two high-precision glass lenses with higher surface quality but heavier surface. In addition, there is a special lens that diffracts light. After the combination of glass lens and diffractive lens, a clear image can be presented.

▲ Starburst prototype

The Starburst prototype is characterized by high HDR (high dynamic range imaging) function, and the maximum brightness of its backlight can reach 20000nit, which is 200times higher than that of the existing VR head display Quest 2. The Meta Reality Labs team developed Starburst to restore complete indoor lighting in a virtual environment and to explore the ideal brightness that allows users to achieve the best immersion experience.

Starburst uses COB LED (commonly used for LED on headlights) to replace the original LCD screen backlight, which greatly improves the backlight of the LCD, but also generates more heat. As a result, the Starburst is also equipped with fans and 3D printing cooling chassis. But the equipment is bulky as a whole, and the power consumption is serious. Of the four technologies, this technology is the furthest from going on the market.

▲ Holocake 2 prototype

The Holocake 2 prototype is light and thin, which can improve the wearing comfort of the VR head display. Holocake 2 is a PC VR device, although it is light and thin, but it requires additional computing and battery assembly space, using a split design.

History of ▲ Half Dome

Half Dome is a VR head display prototype based on Oculus Rift, which is characterized by the use of variable focus lens. Since the third generation, the Half Dome lens has been upgraded from a mechanical system to an electronic zoom system, reducing mechanical noise and vibration. Half Dome 3 has 32 switchable focus planes. According to experiencers, when playing games such as "Lucky's Tale" through Half Dome's dynamic zoom, the focus of the image in the field of vision can change according to the distance the handle moves.

02. The peak brightness of Starburst is 20000nit, 200 times that of Quest 2. At SIGGRAPH 2022, the top computer graphics conference held this month, Sam Sprigg, a reporter from foreign media Auganix, specially experienced the VR prototype Starburst, which focuses on HDR functions. The Oculus Quest 2 is the pinnacle of Meta's VR all-in-one machine so far, with more than 10 million units shipped last year, making it the best-selling VR product in history, while the peak brightness of the Starburst is 200 times that of the Quest 2.

At the exhibition site, two Starburst were hung on a frame for participants to experience. The Starburst weighs about 2.5kg, which is a bit too big for headsets. "it's too heavy," Sam Sprigg said. "you can't wear it on your head, you can only hold it in front of your face." Also, the Starburst needs a lot of power to support its high-brightness display, so it is fixed to the frame.

▲ two Starburst are hung on a shelf

The brightness of the Starburst is measured by "nit". According to Sam Sprigg, the indoor environment is brighter than 10000nit, while the peak brightness of Starburst is 20000nit. Meta has said that the peak brightness of Quest 2 is about 100nit. the peak brightness of Quest 2 is 200 times that of Quest 2.

The peak of 20000nit can also explain the "clunkiness" of Starburst to some extent. High brightness can also generate high heat, and Starburst is equipped with a large aluminum radiator to dissipate heat. Nathan Matsuda, a research scientist at Meta's display system research team (Display Systems Research,DSR), says aluminum radiators account for at least 50 per cent of the weight of the equipment. In addition, there is a large fan at the top of the equipment that absorbs heat from the radiator.

▲ Starburst consists of 36 components

However, Starburst has a lower resolution, about 2560x1620 pixels, which is slightly lower than Quest 2. But it's also acceptable. Because Starburst mainly shows HDR solutions, as for high resolution, that is the direction that Butterscotch should explore.

Starburst uses both color and monochrome LCD (liquid crystal display). This is because a single LCD has poor contrast in low light, while two color LCD absorbs most of the light, reducing the display, Matsuda said. Therefore, the best display effect can be obtained by using color and monochrome LCD in series.

In order to focus on exploring HDR technology, Starburst made a compromise on the refresh rate and vision of the screen. However, Sam Sprigg said the compromise did not affect Starburst's performance in terms of HDR and brightness.

▲ Sam Sprigg "wears" Starburst

The first part of the Starburst demonstration is to set up a light source in the virtual space and then place some spheres to display the reflected light source. Sam Sprigg said that in the 10000nit display range, this is the same as the Quest 2 experience, nothing special.

However, when the brightness rises to 10000nit, he can feel more obvious brightness. When showing the 10000nit light, the demonstration switches to the ocean sunset view. The sun passed through the clouds, the light hit the waves of the ocean, and the water sparkled. Then the light rises to its peak 20000nit, and there is a virtual sun in the picture. Sam Sprigg said he felt as if he saw the real sun because the light was so bright. Finally, as the sun went down, the waves gave off a faint reflection. The demonstration was impressive.

▲ placement sphere test light source

Sam Sprigg said that staring directly at the light source of the 20000nit in Starburst might be similar to staring at a 60-watt light bulb. However, the light source in VR is only a few inches from his eyes, which is very close. He believes that this brightness display can bring users a more realistic and immersive experience.

However, Sam Sprigg also pointed out that power, heat and shape are important obstacles that limit the development of HDR technology in VR head display. He believes that it may take a long time for us to wait until we are able to export 20000nit mature products on the market.

03. There are many difficulties, VR head display faces ten major technical challenges. Although Meta has made some technical progress in VR head display, it still encountered many technical bottlenecks that can not be solved at present. In his speech at the SIGGRAPH conference, Douglas Lanman, head of Meta display system development, provided us with some reference directions-there are ten challenges that need to be solved to take the VR experience to new heights!

Whoever can take the lead in overcoming these challenges is expected to take the lead in the increasingly fierce meta-universe hardware entry race.

1. Resolution: Meta develops Butterscotch prototype to improve resolution

In order to improve users' sense of immersion and make the virtual world look more realistic and clear, VR head display must have higher resolution. At present, the resolution of VR head display is still very far from the level of human vision.

Meta says its initial goal is to achieve a monocular 8K resolution and 60PPD (pixel density). But at present, Quest 2 only achieves monocular 2K resolution and 20PPD, which is still a long way from this goal.

In order to improve the resolution of VR head display, Meta has developed a Butterscotch prototype that focuses on exploring resolution. In this regard, the development and production of high-resolution displays is not difficult, the difficulty lies in how to provide the corresponding computing power for high-resolution displays. Lanman points out that two technologies, Foveated Rendering and Cloud streaming, may help improve computing power, but they themselves are difficult to develop.

▲ above is the VR effect of ordinary blurring, and below is the blurring effect of viewpoint rendering technology.

2. Field of view angle: the view angle of VR head is twice as small as that of human eyes.

The average horizontal field-of-view angle of human is 200C, while that of most commercial VR heads is 100C at present. This means that the field of view of the VR head is narrower than what we see in the real world. In addition, there is room for improvement in the vertical field of view angle of VR head display.

The wider the field of view, the more pixels the VR head display presents. This requires higher power and more waste heat (waste heat). In addition, when the line of sight is widened, the image at the edge of the line of sight is easily distorted. Therefore, a wider line of sight needs better lens technical support. In addition, the researchers also need to ensure that the lens and display are constructed in a way that does not increase the size of the VR head display.

3. Ergonomics (Ergonomics): the comfort of VR is not enough.

At present, most VR headers are slightly "bulky". For example, the Quest 2 weighs more than 500g and its device is 8cm thick (protruding from the face). Such a device will obviously not bring a good wearing experience for users. The ideal VR device should be lighter and thinner, allowing users to wear it comfortably for a long time.

Pancake lenses and holographic lenses may help to reduce the head display volume, and Meta's Holocake 2 prototype is currently exploring this direction, but it has not yet been developed to the level of mass production.

▲ "sunglasses" is a prototype of Holocake 1, transparent is the 2016 Oculus Rift

4. Vision correction display screen: a low-cost vision correction display screen has not been found yet

A better VR head display should have a visual correction function so that users do not have to wear regular glasses under the VR head display. This problem can be solved through special lens accessories, but this is not an ideal solution. The best solution is that the head display has its own visual correction function, so that the head display designer no longer needs to consider problems such as adaptive glasses.

But the challenge is, how can we find a solution that is cheap to manufacture without adding extra head weight?

▲ lens accessories with vision correction already exist, but they are not an ideal solution.

5. Zoom: Meta prototype Half Dome can realize human eye zoom.

In the natural environment, in order to see the picture at different distances, the human eye can zoom naturally. But in the VR head display, because the distance between the human eye and the screen has not changed, so the human eye can not zoom, nor can see the difference between the near and far scene. There is the problem of visual convergence adjustment conflict (vergence-accommodation-conflict). So sometimes users will feel eye fatigue, and even headaches and vomiting.

To solve this problem, Meta researchers have developed a series of Half Dome prototypes that support progressive vision (progressive vision). Half Dome can simulate different focus planes and show different degrees of blur, thus helping human eyes to adjust the focal length in the virtual world.

The pupil of ▲ changes with the distance of the object.

6. Eye tracking: it is difficult to adapt to everyone's eye conditions.

Eye movement tracking is a key technology in virtual reality. It is the basis of many other important VR technologies, such as zoom, distortion correction and so on. Through this technology, users can make eye contact with other users in the virtual world and have a more real social experience.

But the problem is that everyone has different pupil shapes and the growth of eyelids and eyelashes is different, so the current eye tracking technology does not work well for all users. As a result, the Lanman team will extensively collect more user data to upgrade eye tracking technology and try to adapt it to more people.

The shape of ▲ pupil varies from person to person, which is a challenge for eye movement tracking technology.

7. Distortion correction: Meta develops a distortion simulator to speed up the iteration of the algorithm.

The movement of the pupil causes the distortion of the image, which in turn reduces the user's sense of immersion, especially when used with zoom technology. In order to develop the correction algorithm, researchers have to adjust the test on the physical head display, but it may take weeks or even months to produce the head display, which greatly delays the iterative time of the algorithm.

To solve this problem, the Meta research team developed a distortion simulator (distortion simulator). With this simulator, researchers can test the correction algorithm without producing test head displays and special lenses.

▲ red light is not corrected, blue light is the result of correction.

8. High dynamic range (HDR): the peak brightness of Starburst prototype is 20000nit.

Physical objects and environments are much brighter than VR headers. To solve this problem, Meta created a Starburst prototype. The peak brightness of Starburst is 20000nit, which is 200 times that of the existing Quest 2.

Starburst can more realistically simulate the lighting conditions of enclosed spaces and nights, making the virtual environment look more realistic. At present, however, Starburst is heavy and energy-intensive.

Meta believes that HDR contributes more to visual realism than resolution and zoom, but this technology is the furthest from practical application.

▲ Starburst prototype close-up

9. Visual fidelity (Visual realism): perspective (Passthrough) technology is not yet mature

A better VR header should also allow VR users to interact with the real world. VR header display can record real environment information and display it as a video image in virtual reality. This technology, called "Passthrough", has been implemented in commercial VR headers, but its quality is poor. For example, Quest 2 provides a black-and-white perspective mode, while Meta's upcoming high-end VR head display "Quest Pro" aims to pursue higher resolution and color perspective.

However, the VR head display has not yet achieved a perfect reconstruction of the physical environment. The perspective captured by perspective technology deviates from the eye in space, and users may feel uncomfortable when using it for a long time. Therefore, Meta is studying artificial intelligence-assisted line-of-sight synthesis, which can generate the correct point of view in real time and has high visual fidelity.

In addition, Meta is working on a "reverse perspective" (Reverse Passthrough) feature that allows real-world users to see VR users' glasses and faces and make eye contact. But at present, the technology is visually uncoordinated and unnatural, and is far from ready for the market.

On the left of the ▲ is an Oculus Rift, which blocks the view of VR users, and on the right is the early reverse "perspective" prototype.

10. Facial reconstruction: Meta carries out research on "virtual avatar" project

Meta hopes that in the future, people will meet in a virtual environment as real as in real life. To this end, Meta is launching a research project called virtual avatar (Codec Avatars), which aims to create 3D digital avatars for people in virtual environments.

At present, VR head display can read the facial expressions of VR users in real time and transfer them to virtual reality. It is reported that the upcoming Quest Pro may be the first Meta device to provide facial tracking (face-tracking).

▲ Quest Pro can recognize facial expressions and transmit them to VR devices

04. Conclusion: Meta's VR technology has made a breakthrough, but it is still necessary to continue to overcome the technical problems. Lanman has put forward ten major challenges facing VR on SIGGRAPH 2022, from which we can see the progress made by Meta and the unsolved problems. Meta has released four prototypes designed to solve the problems of VR head display in resolution, zoom, light and thin shape, and HDR. These four prototypes have their own strengths and have made initial technical progress in their respective fields. In addition, Meta has carried out the virtual avatar project to help the facial reconstruction function of VR head display, and developed a distortion simulator to accelerate algorithm iteration.

These prototypes show the highest level that Meta can achieve in the field of VR individual technology, but these technologies are far from enough to create a mature VR head display. At present, there are still some technical problems that Meta can not overcome temporarily. For example, the angle of view of VR head is narrower than that of human eyes, the comfort of VR head display is not enough, VR head display is not equipped with vision correction function, VR head display eye movement tracking technology can not adapt to everyone, VR head display "perspective" technology is not yet mature, and so on.

Looking at the current exploration of Meta on the head of VR, we can find that Meta has come a long way in the exploration of VR. But Meta is still a long way from the end of the road. At present, all the technologies related to VR are under active development, but these technologies are far from mature and there is still a long way to go before the commercial listing. From this point of view, the explosion of VR is still a long way off.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.