Shulou(Shulou.com)11/24 Report--

CTOnews.com November 16, Google recently issued a press release introducing a small artificial intelligence model, Mirasol, which can answer questions about video and set a new record.

At present, the AI model is very difficult to deal with different data streams. If we want AI to understand video, we need to integrate the information of different modes such as video, audio and text, which greatly increases the difficulty.

Researchers at Google and Google Deepmind have proposed a new approach to extend multimodal understanding to the field of long videos.

With the Mirasol AI model, the team tried to address two key challenges:

You need to synchronize video and audio with high-frequency sampling, but process the title and video description asynchronously.

Video and audio will generate a lot of data, which will strain the capacity of the model.

Google uses combiners and autoregressive transformer models in Mirasol.

The model component processes time-synchronized video and audio signals, and then splits the video into separate segments.

The converter processes each fragment, learns the connections between each fragment, and then uses another converter to process the context text, and the two components exchange information about their respective inputs.

A novel conversion module called Combiner extracts a common representation from each fragment and compresses the data by dimensionality reduction. Each segment contains 4 to 64 frames. The model currently has 3 billion parameters and can handle video of 128 to 512 frames.

In the test, Mirasol3B reached a new benchmark in video problem analysis, was significantly smaller and could handle longer videos. Using a variant of the combiner that contains memory, the team can further reduce the computing power required by 18%.

CTOnews.com is enclosing the official press release of Mirasol, which can be read in depth by interested users.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.