Shulou(Shulou.com)06/01 Report--

Today, I will talk to you about which three open source tools AI relies on to restore the old video in Beijing 100 years ago, which many people may not know very well. in order to make you understand better, the editor has summarized the following content for you. I hope you can get something according to this article.

Recently, an image of the old Beijing city taken 100 years ago was forwarded by CCTV and went viral on Weibo. It is understood that the video is from the game creation cabin of bilibili's UP master Otani. Author Otani uses AI technology to render a black-and-white video of Beijing filmed in 1920 into a high-definition and smooth color video, coupled with BGM with local characteristics, restoring the folk customs of the old Beijing city a hundred years ago.

According to Otani, this black-and-white video, which was shot a hundred years ago, has a lot of noise and low frame number and resolution because of its age. So he uses three open source AI tools to process the video, which are:

DAIN, make up the frame of the video to improve the fluency of the video

ESRGAN, super-resolution video processing to improve video definition

DeOldify, color the video

Let's take a look at these three commonly used video repair AI tools, interested friends can also learn to use, their own to deal with old videos.

Video frame completion tool DAIN

The number of frames (FPS) has a great impact on the fluency of a video. At normal playback speed, a video below 30 FPS will feel stuttered, but if it is higher than 60 FPS, the naked eye can't tell the difference. The higher the number of frames, the higher the fluency of the video, especially in slow motion, the difference is especially obvious:

The image above comes from an example of SUPER SLOMO, another AI framing tool. After slowing down the car tail flick of the original 30 FPS by 8 times, the actual FPS is less than 4 FPS, and the stutter is visible to the naked eye. Through the AI frame filling technology, the slow video can be maintained at 240FPS, which shows the fluency.

Generally speaking, the core requirement of frame complement is to insert intermediate frames between two consecutive frames so as to improve the FPS of the video. The problem that the frame complement tool needs to solve is how to automatically generate intermediate frames through the AI algorithm. The full name of DAIN is Depth-Aware Video Frame Interpolation, which is a video frame interpolation tool based on depth perception. The DAIN team proposed a depth-aware frame interpolation model and developed a depth-aware stream projection layer to generate intermediate frames.

Test environment

Ubuntu (Ubuntu = 16.04.5 LTS)

Python:Anaconda3 = 4.1.1 & Python= 3.6.8

Cuda and Cudnn:Cuda = 9.0 & Cudnn = 7.0,

Pythorch: customized depth-aware flow projection and other layers require ATen API in Pythorch = 1.0.0

GCC: GCC=4.9.1 and nvcc=9.0 compilers are required to compile PyTorch 1.0.0 extension files (.c / .cu)

GPU:NVIDIA GPU (the author uses Titan X (Pascal) calculation = 6.1, which supports compute_50/52/60/61 devices)

Installation and use

Download the database:

$git clone https://github.com/baowenbo/DAIN.git

Before you build the Pytorch extension, make sure that you have pytorch > = 1.0.0:

$python-c "import torch; print (torch.__version__)"

Generate PyTorch extensions:

$cd DAIN$ cd my_package $. / build.sh

Generate the Correlation package required for PWCNet:

$cd. / PWCNet/correlation_package_pytorch2_0 $. / build.sh

Test pre-training model:

Create the model weight directory and the Middlebury dataset directory:

$cd DAIN$ mkdir model_weights$ mkdir MiddleBurySet

Download the pre-training model

$cd model_weights$ wget http://vllab1.ucmerced.edu/~wenbobao/DAIN/best.pth

And Middlebury datasets:

$cd.. / MiddleBurySet$ wget http://vision.middlebury.edu/flow/data/comp/zip/other-color-allframes.zip$ unzip other-color-allframes.zip$ wget http://vision.middlebury.edu/flow/data/comp/zip/other-gt-interp.zip$ unzip other-gt-interp.zip$ cd..

Pre-installation:

$cd PWCNet/correlation_package_pytorch2_0 $sh build.sh$ cd. / my_package$ sh build.sh$ cd.

Download result

Download the interpolation results in the following ways:

$wget http://vllab1.ucmerced.edu/~wenbobao/DAIN/UCF101_DAIN.zip$ wget http://vllab1.ucmerced.edu/~wenbobao/DAIN/Vimeo90K_interp_DAIN.zip$ wget http://vllab1.ucmerced.edu/~wenbobao/DAIN/Middlebury_eval_DAIN.zip$ wget http://vllab1.ucmerced.edu/~wenbobao/DAIN/Middlebury_other_DAIN.zip

Slow motion generation:

This model is fully capable of producing slow-motion effects with small modifications to the network architecture. Run the following code by specifying time_step = 0.25 to generate an x4 slow-motion effect:

CUDA_VISIBLE_DEVICES=0 python demo_MiddleBury_slowmotion.py-- netName DAIN_slowmotion-- time_step 0.25

Or set the time_step to 0 or 0.125 as follows

CUDA_VISIBLE_DEVICES=0 python demo_MiddleBury_slowmotion.py-- netName DAIN_slowmotion-- time_step 0.125 $CUDA_VISIBLE_DEVICES=0 python demo_MiddleBury_slowmotion.py-- netName DAIN_slowmotion-- time_step

Generate x8 and x10 slow motion, respectively. Or it's okay if you want to shoot something fun with x100 slow motion.

$CUDA_VISIBLE_DEVICES=0 python demo_MiddleBury_slowmotion.py-- netName DAIN_slowmotion-- time_step

You can create gif animations in the following ways:

$cd MiddleBurySet/other-result-author/ [random number] / Beanbags$ convert-delay 1 * .png-loop 0 Beanbags.gif / / 1*10ms delay

Have a good time!

Details of DAIN: click to view

Download address of DAIN: click download

Resolution improvement tool ESRGAN

We know that resolution will have a direct impact on the clarity of the image. A small low-resolution picture, if we directly zoom in, many details will become "mosaics", which I believe we all have a lot of experience. Therefore, if the ordinary interpolation algorithm is directly used to enlarge the image, the edge of the object in the image will become blurred, and the super-resolution algorithm is to solve this problem.

ESRGAN is an enhanced super-resolution generation countermeasure network, which can generate real texture during the image super-resolution through AI to improve the image resolution. The effect is shown below:

Test environment

Python 3

PyTorch > = 1.0 (if installed with CUDA, CUDA version > = 7.5. More details)

Python Suite: pip install numpy opencv-python

Installation and use

1. Clone this github repository.

Git clone https://github.com/xinntao/ESRGANcd ESRGAN

two。 Place the low-resolution image in the. / LR folder. (there are two sample images-baboons and comics).

3. Download the pre-trained model from Google cloud hard drive or Baidu cloud hard disk. Place the model at. / models.

4. The author provides two models with high perceptual quality and high PSNR performance. Run the test: provide the ESRGAN model and the RRDB_PSNR model, and you can configure test.py.

Python test.py

5. The result is in the. / results folder.

Details of ESRGAN: click to view

Download address of ESRGAN: click download

Black-and-white image coloring tool DeOldify

DeOldify is a deep learning project for shading and restoring old images and videos. It uses NoGAN, a new and efficient image-to-image GAN training method, with better detail processing and more realistic rendering:

NoGAN is a new GAN training model developed by the author, which is used to solve some key problems in the previous DeOldify model. NoGAN training combines the advantages of GAN training (good-looking colors) while eliminating unpleasant side effects (such as flickering objects in videos). The video is generated from isolated images without adding any time modeling. The process performs the GAN portion of 30-60 minutes of "NoGAN" training, using 1 to 3 per cent of image network (imagenet) data at a time. Then, as with still image shading, each frame is "DeOldify" before the video is rebuilt, and the rendering is very consistent even in the moving scene:

Currently, DeOldify has three models to choose from, each of which has key strengths and weaknesses, so it has different use cases. The three models are: Artistic (artistic), Stable (stable) and Video (video only).

Test environment

Linux

Fast.AI = 1.0.51 (and its dependencies). If you use any later version, you will see the grid artifact in the rendering and the tensorboard will malfunction.

PyTorch = 1.0.1

Jupyter Lab conda installation

-c conda forge jupyterlab

Tensorboard (that is, install Tensorflow) and TensorboardX. Not necessary, but FastAI now provides native support for TF, which is easy to use.

Prereqs:conda install-c anaconda tensorflow gpupip install tensorboardX

ImageNet is a good data set for training.

GPU: it's not very demanding. (individual coloring is fine with ordinary set display, but a better graphics card is recommended for large-scale training.)

Installation and use

Open the command line and navigate to the root folder where you want to install, and enter the following command:

Git clone https://github.com/jantic/DeOldify.git DeOldifycd DeOldifyconda env create-f environment.yml

Then start using these commands to run:

Source activate deoldifyjupyter lab

Start running in Jupyter Lab through the URL provided in the console.

DeOldify details: click to learn

DeOldify download address: click download

Friends who want to turn black-and-white old videos into color high-definition videos can take action now.

After watching the above content, do you have any further understanding of which three source workers rely on AI to restore the old video in Beijing 100 years ago? If you want to know more knowledge or related content, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.