Shulou(Shulou.com)06/01 Report--

This article introduces how to achieve real-time rendering based on strokes in Unity. The content is very detailed. Interested friends can use it for reference. I hope it will be helpful to you.

What is stroke-based rendering

Stroke-based rendering (stroke-based rendering, hereinafter referred to as SBR), simply speaking, is to generate strokes from the image, and then redraw the image with the stroke. This process is actually the process of imitating the painter's actual painting, that is, the process of observation-> understanding-> expression. When painting, there is a sentence called meaning in the pen, that is, this meaning. You can take a look at this review of SBR: luthuli.cs.uiuc.edu/~da (luthuli.cs.uiuc.edu/~daf/courses/ComputerGraphics/01210867.pdf).

Why to implement SBR in Unity

When I was a graduate student, I studied non-realistic rendering (Non-Photorealistic Rendering, hereinafter referred to as NPR), and found that there are essentially two methods, one is based on experience, that is, there is no theoretical basis, according to the experimental results to design rendering methods. It seems that there is a big guy in graphics who said: it looks right, it is right. Probably talking about the research method based on experience. There is also a research method based on physical simulation, such as studying the actual color mixing of pigments and the flow of water, and so on.

I find that the experience-based method is somewhat based on feeling, and it is difficult to design algorithms or judge the results. The research method based on physical simulation is too bulky, only to reproduce the tools and pigments in the computer, and how to paint depends on the artist to draw by hand. So I was wondering how to make the computer really think and express like an artist.

I happen to have been exposed to a little bit of computer vision before, so I wonder if there is a vision-based research method for computers to understand images and then draw like people, that is, rendering, instead of using a bunch of filters.

At that time, opencv was used to segment the image, and then the saliency of the image was calculated, and then the brush was generated according to the salience. Finally, the brush was rendered in parallel with cuda, and such an offline version of SBR was made. The results were good, but the study was dropped after work.

Recently, I began to learn Unity's Compute Shader, and found that it was very similar to cuda, so I thought of SBR and felt that SBR could be realized in post-processing with Unity, so I made some attempts.

Specific implementation method

All the processing is written in the render of post processing stack, and the process is roughly as follows:

After Gaussian smoothing, the gradient tensor is called smoothing gradient tensor, and the solution is as follows:

IGa is the convolution operation of Gaussian filter, px is the gradient in x direction calculated by sobel operator, and py is the gradient in y direction calculated by sobel operator. As for why the second eigenvector obtained by SVD decomposition is used to calculate the rotation matrix, it is because the first eigenvector (the principal eigenvector) indicates the direction of the greatest change, and the second eigenvector is orthogonal to it, so it can be used to indicate the flow direction of the stroke.

Render the result:

The original frame on the left and the rendered effect of SBR on the right:

Real-time rendering based on strokes

In addition, the interval and size of strokes can be adjusted for different effects:

The interval and size of strokes can be adjusted.

Different stroke textures can have different rendering effects:

Code

(github.com/alpacasking/SBR)

Time is limited, now only the most basic process has been written, in fact, there are still many things worth doing, such as:

Detect the foreground and background according to the depth texture to generate strokes of different densities and sizes

Analyze and understand the image and generate strokes by using the relevant knowledge of CV

Render with different stroke textures at the same time

Let the brush strokes connect smoothly between each frame.

Further research can be done for oil painting, watercolor style, etc.

It will be added slowly if you have time, and you can change it by yourself if you are interested.

On how to achieve real-time stroke-based rendering in Unity to share here, I hope the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.