Shulou(Shulou.com)06/03 Report--

Boston

introduced

Neural Style Transfer is one of the most amazing applications of AI in creative environments. In this project, we will see how to convert art painting styles into selected images to create amazing effects. Leon A. Gatys et al. conceived of the concept of neural style transfer in their 2015 paper "Neural Algorithms for Artistic Style." Since then, many researchers have applied and refined the method, added loss elements, tried different optimizers and experimented with different neural networks for this purpose. Nevertheless, this original paper is still the best source for understanding this concept, and VGG16 and VGG19 networks are the most commonly used models in this regard. This choice is unusual considering that both performed well in recent networks, and it achieved the highest performance in style migration.

https://arxiv.org/abs/1508.06576

You can check out this GitHub repository for the full code.

https://github.com/maurock/neural_transfer_style

How does it work?

The goal of this technique is to apply the style of the image (we call it a "stylized image") to the target image, thereby preserving the content of the latter. Let's define these two terms:

Style is the texture and visual pattern in an image. One example is the artist's brush strokes.

Content is the macro structure of an image. People, buildings, objects are examples of image content.

This shows a good effect:

Do you want to see more results? You can browse at the end of the article.

Let's look at the advanced steps:

Select the image you want to style

·Select sample style reference images. Usually, this is a painting with a unique and recognizable style.

Initialize the pre-trained deep neural network and obtain a feature representation of the middle layer. This step is done to enable representation of content images and genre images. In content images, the best option is to obtain the feature representations of the highest layers because they contain information about the macrostructure of the image. For stylistic reference images, feature representations are obtained from multiple layers of different scales.

Define a loss function to minimize the sum of content loss, style loss, and variation loss. Each iteration, the optimizer generates an image. Content loss is the difference (L2 normalization) between generated images and content images, while style loss is between generated images and styles. We will see later how these variables are mathematically defined.

Reconsider loss minimization

Image processing and image inverse processing

First, we need to format images for use on the web. The convolutional neural network (CNN) we will use is a pre-trained VGG19 session. When we process images into compatible arrays, we also need to deprocess the resulting images, switching from BGR to RGB format. Let's build two helper functions to do this:

content loss

Content loss preserves the content of the primary input image as style. Since the higher layers of the convolutional neural network contain information about the macrostructure of the image, we calculate the content loss as the difference between the output of the highest layer of the input image and the same layer of the generated image (L2 normalization).

Content loss is defined as:

content loss

In the equation, F is a feature representation of the content image (what the network outputs when we run the input image), and P is one of the images generated at a particular hidden layer l.

This is what was implemented:

style loss

Understanding style loss is not as simple as content loss. The goal is to preserve the style of the image (i.e. visual patterns as strokes) in the newly generated image. In the former case, we compare the raw output of the middle layer. Here, we compare the differences between the style reference image and the Gram matrix of a specific layer of the generated image. The Gram matrix is defined as the inner product between vectorized eigenmaps of a given layer. The meaning of the matrix is to capture the correlation between the features of the layers. Calculating the loss of multiple layers allows similar features to be preserved that are correlated within different layers between the style image and the generated image.

The style loss for a single layer is calculated as follows:

Style loss per layer

In the equation, A is the Gram matrix of the style image and G is the Gram matrix of the generated image, both of which are associated with a given layer. N and M are the width and height of the style image.

Style loss is calculated for each individual layer first, and then applied to each layer considered for style modeling. Let's make it happen:

change damage

Finally, the final component of the loss is the variation loss. This element was not included in the original paper and is not a necessary condition for project success. In fact, experience has shown that adding this element produces better results because it smoothes color variations between pixels.

Let's include this:

overall loss

Finally, taking into account all these contributions, the total loss is calculated. First, we need to extract the output of the specific layer we selected. For this purpose, we define dictionaries as:

We then calculate the cost of code that calls the previous function. Each component is multiplied by a specific weight, which we can adjust to produce a stronger or lighter effect:

Set up the neural network

The VGG19 network takes as input a batch of three images: an input content image, a stylistic reference image, and a symbol tensor containing the generated image. The first two are constant variables, defined as Variable using the package keras.backend. The third variable is defined as a placeholder because it changes over time as the optimizer updates the results.

Once the variables are initialized, we add them to the tensor, which will be provided later in the network.

Once done, we need to define losses, gradients, and outputs. The original paper used the algorithm L-BFGS as the optimizer. One limitation of this algorithm is that it requires the loss and gradient to be converted separately. Since it would be very inefficient to calculate them independently, we will implement an Evaluator class that calculates loss and gradient values once, but returns them separately. We started implementing:

the last step

Finally, everything is ready! The final step is to iterate the optimizer several times until the desired loss or desired result is reached. We will save the results along the iterations to check if the algorithm works as expected. If the results are unsatisfactory, we can use weights to improve the resulting image.

To view the entire code, see the GitHub link provided at the beginning of the page.

results

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.