Shulou(Shulou.com)11/24 Report--

The amazing Vision Pro secretly installed a "brain-computer interface"?

The emergence of Apple's Vision Pro caused an uproar all over the world overnight.

As Cook put it, this will usher in the era of "spatial computing", which shows that Vision Pro means a lot to everyone.

Wearing a Vision Pro, all you have to do is blink, pinch your fingers, or even move your mouth, and the science fiction of "player number one" becomes possible.

But the science fiction of Apple's Vision Pro is more than that.

After the headline was released, Sterling Crispin, a former Apple researcher, reviewed his work on Twitter and revealed a lot of cool techs about Vision Pro.

The most amazing thing is that Vision Pro actually hides a "brain-computer interface" technology.

There is no need for chip implantation, it can predict your brain behavior, proper "mind reading".

The gray box may be a secret project that Apple is working on. In addition, Apple head Show can also detect the mental state based on the data of an individual's body and brain.

The seemingly short tweet actually brings unlimited enjoyment to many people on Apple's Vision Pro.

Vision Pro hides a "brain-computer interface"? In 2018, Sterling Crispin joined Apple's technology development team as a researcher on Neurotechnology prototypes.

Three years ago, he began to participate in the research and development of Vision Pro.

As he said on Twitter, he spent 10% of his life contributing to the development of Vision Pro, the longest effort I've ever made.

Most of Crispin's work at Apple is bound by confidentiality agreements, designing a wide range of themes and methods.

But in fact, some of these jobs have been patented by Apple, which can be mentioned.

"there are a lot of tricks involved in making specific predictions possible, which are detailed in the few patents I mentioned," Crispin said. "

The coolest ability is to predict that users will click on something before they actually click.

The main principle behind this technology is that everyone's pupils react before doing everything, in part because you expect something to happen after you click.

Therefore, Apple can use algorithms to monitor your eye behavior.

And redesign the UI in real time to create more of this expected pupillary response, thereby creating biofeedback for the individual's brain.

This is actually the primary "brain-computer interface" realized through the eyes.

"it's a tough job, and it's something I'm proud of," Crispin said. Will undergo invasive brain surgery at any time. "

It can be seen that Apple Touxian uses a large number of neuroscience technologies to develop some of these key functions, with different skills to determine what a person is going to do.

In addition, Crispin says a large part of its work involves measuring the mental state of users through their body and brain data in an immersive experience.

After all, people immersed in the virtual world really need a health check function.

When you are in a mixed reality or virtual reality experience, the AI model tries to predict whether you are curious, thinking about, scared, focused, remembering past experiences, or other cognitive states.

These can be inferred from measurements such as eye tracking, brain electrical activity, heart beat and rhythm, muscle activity, blood density in the brain, blood pressure, skin conductance and so on.

In addition to the techniques mentioned above, speculating about cognitive states includes ways that you may not be aware of.

For example, flash a visual effect or sound to you quickly, and then measure your reaction to it.

There is also a patent on how to use machine learning and signals from the body and brain to predict your concentration, relaxation, or learning.

The header then automatically updates the virtual environment to enhance these states.

So you can imagine an adaptive immersive environment that can help everyone learn, work, or relax by changing what you see and hear in the background.

These and technologies introduced by Sterling Crispin may be able to be seen in future iterative upgrades of VisionPro.

Some of the patents behind the list of patents underpin the effect of the magic demonstration at the press conference is Apple's own technology accumulation over the years of quietly developing VR devices.

More than 5000 VR patents are stunning products and cornerstones and moats at the press conference.

Now let's take a look at the patented technology involved in the demonstration.

Hands + mind control almost all VR devices on the market are equipped with handles.

But Apple boldly relied entirely on gesture tracking and eye tracking to achieve silky control in its first VR product.

The technology behind it was patented in 2023.

According to the patent diagram, FIG.1 is the frame diagram of the 3D operating environment, showing the overall framework of the user's UI implementation.

A FIG.4 is a frame diagram of a hand tracking unit of a head display computer system that captures the user's gesture input.

FIG.5 is the frame diagram of the eye tracking unit of the head display computer system in which the user's eye tracking data is input.

This patent illustration shows the flow chart of eye reflection assisted eye tracking.

In addition to providing the basis for hand and eye tracking technology, the patent also describes how users can interact more effectively in a 3D environment.

The powerful gesture tracking experience also stems from Apple's acquisition of another patent in May this year.

The patent describes how the camera in the head display uses depth mapping to distinguish between different gestures.

Figure note: FIG.7E / F / H / I illustrates the selective movement of virtual objects or viewpoint dimensional environments relative to the 3D environment (and all virtual objects in the 3D environment) based on the grip strength maintained by the user during a series of user inputs.

One click to generate a 3D image do you remember the 3D image that was ridiculed by the whole network in Horizon World?

Apple's plan may partly explain why Apple can detonate the whole network when it is also a VR device, but Meta makes its stock "fall endlessly".

The technology patents behind it were also released last week.

The patent describes future Apple Watch and XR headers that will allow users to set up their own 3D avatars using cameras on the device.

In a diagram of the patent, Apple shows that the future Apple Watch will allow users to take their own Face ID images using the sensor on the back of the Apple Watch.

The sensor technology can also be used in XR environments.

Infinite canvas the Vision Pro demonstrated at Apple's press conference can use the surrounding space as an infinite canvas.

Users can not only use any APP at will, but also adjust them to any size they like.

Apple applied for a patent related to this feature in 2018, a technology that allows future XR headers to provide users with the ability to edit documents in 3D.

The following figure is shown in the patent diagram of Apple.

Another related patent is the technology obtained in 2021 that allows users wearing a XR head display device (HMD) to view the display (Mac, MacBook, or TV) and make virtual extensions as needed.

In fact, the acquisition of AR startup Mira shows that Apple's exploration in the field of AR / VR has been premeditated for a long time.

Back in 2010, Apple bought Polar Rose, a Swedish facial recognition company.

A day after the launch of the Vision Pro, it was rumored that Apple had recently acquired Mira. This is a start-up that builds lightweight AR headsets and provides services to the U.S. Military.

Apple has hired at least 11 Mira employees as part of the acquisition, according to an Ins post by Mira CEO Ben Taft. Excited about Mira's next chapter at Apple. A 7-year journey from dormitory to acquisition.

At present, the exact amount of the acquisition is unknown, and it is not clear what role Mira plays in Apple's augmented reality roadmap.

Jony Ive, Apple's former design director, once served as an adviser to the start-up, according to two former employees.

Founded by Ben Taft in 2016 and headquartered in Los Angeles, Mira has been producing AR headsets for customers around the world, including Nintendo's theme park in Universal Studios, Japan.

The start-up has so far raised about $17 million, including investors from Bluebear Capital, Happiness Capital and Sequoia Capital.

And many of the technologies that Apple has demonstrated in the Vision Pro are also the result of successful acquisitions.

Some patented technologies for gestures and spatial positioning, for example, were acquired through the acquisition of Montreal-based VRvana.

The first sign of the "iPhone moment" is coming, and it will take another five years for Apple to launch the Vision Pro, which indicates that its next era is coming.

The headline marks the first time Apple has created a new computing platform that can replace most of the existing major products.

Although iPhone eats away at iPod,iPad and threatens Mac. However, no Apple product category has been able to beat the current three major market segments at the same time.

Bloomberg reports that it will take at least five years for Vision Pro to become a substitute for iPhone. It may take more time to become a killer product.

Today, the Vision Pro sells for $3499, which is expensive, and it can't connect to a mobile phone or even make a phone call.

And the design is still too bulky, and must carry an external battery pack, only 2 hours of power, can not be used for a long time.

But over the next few years, Apple will be able to solve these problems and launch an "omnipotent" device.

On the face of it, Vision Pro looks like a "high-priced toy" tailored for the rich, or those who have unlimited demand for new technology.

However, Apple is also at pains to show on WWDC that this is everyone's device, from office to entertainment, video, travel and so on.

However, this is not impossible.

Apple has demonstrated its potential in iterative updates to iPhone, MacBook and other products.

Furthermore, the development value of Vision Pro and its more advanced follow-up products will become clearer.

In fact, another point is that Vision Pro is not as easy to use as iPad or iPhone. And everyone wearing the head may also have the uncomfortable feeling of immersion experience.

However, with the passage of time, people will get used to this new "space computer", just like wearing a pair of glasses.

Through Vision Pro, Apple shows that it is working towards the future of reducing the size of computing devices to the smallest and lightest.

But it could be a long and rugged road.

However, for at least two hours on Monday, for millions of fans, the head finally witnessed the arrival of the Vision Pro.

Maybe one day, we may wear Vision Pro and watch WWDC 2030.

Reference:

Https://twitter.com/sterlingcrispin/status/1665792422914453506

Https://www.bloomberg.com/news/newsletters/2023-06-06/apple-vision-pro-a-post-mac-iphone-and-ipad-future-with-new-visionos-and-apps

Https://www.patentlyapple.com/2023/06/today-apple-introduced-the-apple-vision-pro-a-revolutionary-spatial-computer-headset-thats-out-of-this-world-backed-by.html

This article comes from the official account of Wechat: Xin Zhiyuan (ID:AI_era)

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.