Shulou(Shulou.com)06/02 Report--

How to use Python code to do a face-changing program, I believe that many inexperienced people do not know what to do, so this paper summarizes the causes of the problem and solutions, through this article I hope you can solve this problem.

Brief introduction

In this article, I will show you how to write a short (200-line) Python script to automatically replace the face of one image with the face of another.

This process is divided into four steps:

Detect facial markers.

Rotate, zoom, pan, and second picture to match the * steps.

Adjust the color balance of the second picture to match the image.

Mix the characteristics of the second image into the * image.

1. Using dlib to extract facial markers

The script uses dlib's Python binding to extract facial markers:

Dlib implements the algorithm in the paper "using regression Tree for one millisecond face alignment" by Vahid Kazemi and Josephine Sullivan. The algorithm itself is very complex, but the dlib interface is very simple to use:

PREDICTOR_PATH = "/ home/matt/dlib-18.16/shape_predictor_68_face_landmarks.dat" detector = dlib.get_frontal_face_detector () predictor = dlib.shape_predictor (PREDICTOR_PATH) def get_landmarks (im): rects = detector (im, 1) if len (rects) > 1: raise TooManyFaces if len (rects) = 0: raise NoFaces return numpy.matrix ([p.x) .y] for p in predictor (im, rects [0]) .parts ())

The get_landmarks () function converts an image into an numpy array and returns a 68 × 2 element matrix. Each feature point of the input image corresponds to an x _ ray coordinate of each row.

The feature extractor (predictor) requires a rough bounding box as the algorithm input, which is provided by a traditional face detector (detector) that can return a rectangular list, each of which corresponds to a face in the image.

two。 Using Procrustes analysis to adjust the face

Now we have two marker matrices, and each row has a set of coordinates corresponding to a specific facial feature (for example, the coordinates of line 30 correspond to the nose). We now need to figure out how to rotate, translate, and scale * * vectors so that they fit the point of the second vector as much as possible. One idea is that a second image can be overwritten on * images with the same transformation.

To mathematize this problem, look for TMagne s and R, and make the following expression:

The result is minimum, where R is a 2 × 2 orthogonal matrix, s is a scalar, T is a two-dimensional vector, and pi and qi are the rows of the marked matrix above.

Facts have proved that this kind of problem can be solved by "conventional Procrustes analysis":

Def transformation_from_points (points1, points2): points1 = points1.astype (numpy.float64) points2 = points2.astype (numpy.float64) C1 = numpy.mean (points1, axis=0) c2 = numpy.mean (points2, axis=0) points1-= C1 points2-= c2s1 = numpy.std (points1) S2 = numpy.std (points2) points1 / = S1 points2 / = S2 U, S Vt = numpy.linalg.svd (points1.T * points2) R = (U * Vt) .T return numpy.vstack ([numpy.hstack ((S2 / S1) * R, c2.T-(S2 / S1) * R * c1.T), numpy.matrix ([0.,0.1,1.]))

The code implements these steps:

1. Converts the input matrix to a floating point number. This is the basis for subsequent operations.

two。 Subtract its centroid from each set of points. Once a method of scaling and rotation is found for the point set, these two centroids C1 and c2 can be used to find a complete solution.

3. Similarly, each set of points is divided by its standard deviation. This eliminates the problem of component scaling deviation.

4. The rotational part is calculated using singular value decomposition. You can see details on solving the orthogonal Procrustes problem on Wikipedia.

5. The affine transformation matrix is used to return the complete transformation.

The result can be inserted into the cv2.warpAffine function of OpenCV to map image two to image one:

Def warp_im (im, M, dshape): output_im = numpy.zeros (dshape, dtype=im.dtype) cv2.warpAffine (im, M [: 2], (dshape [1], dshape [0]), dst=output_im, borderMode=cv2.BORDER_TRANSPARENT, flags=cv2.WARP_INVERSE_MAP) return output_im

The alignment results are as follows:

3. Correct the color of the second image

If we try to cover facial features directly, we will soon see this problem:

The problem is that the different skin color and light between the two images cause the edges of the coverage area to be discontinuous. We try to correct:

COLOUR_CORRECT_BLUR_FRAC = 0.6 LEFT_EYE_POINTS = list (range (42,48)) RIGHT_EYE_POINTS = list (range (36,42)) def correct_colours (im1, im2, landmarks1): blur_amount = COLOUR_CORRECT_BLUR_FRAC * numpy.linalg.norm (numpy.mean (Landmarks1 [left _ EYE_POINTS]) Axis=0)-numpy.mean (Landmarks1 [right _ EYE_POINTS], axis=0) blur_amount = int (blur_amount) if blur_amount% 2 = 0: blur_amount + = 1 im1_blur = cv2.GaussianBlur (im1, (blur_amount, blur_amount), 0) im2_blur = cv2.GaussianBlur (im2, (blur_amount, blur_amount) 0) # Avoid divide-by-zero errors. Im2_blur + = 128* (im2_blur 0) * 1.0im = cv2.GaussianBlur (im, (FEATHER_AMOUNT, FEATHER_AMOUNT), 0) return im mask = get_face_mask (im2, landmarks2) warped_mask = warp_im (mask, M, im1.shape) combined_mask = numpy.max ([get_face_mask (im1, landmarks1), warped_mask], axis=0)

We break down the above process:

Get_face_mask () is defined as generating a mask for an image and a marker matrix, which draws two white convex polygons: the area around the eyes and the area around the nose and mouth. Then it expands from 11 pixels to the outside of the edge of the mask, which can help hide any discontinuous areas.

Such a mask is generated for both images, and using the same conversion as in step 2, the mask of image 2 can be converted into the coordinate space of image 1.

Then, with an element-wise*** value, the two masks are combined into one. The purpose of combining these two masks is to ensure that image 1 is masked, revealing the characteristics of image 2.

* use the mask to get the final image:

Output_im = im1 * (1.0-combined_mask) + warped_corrected_im2 * combined_mask

Complete code (link): import cv2 import dlib import numpy import sys PREDICTOR_PATH = "/ home/matt/dlib-18.16/shape_predictor_68_face_landmarks.dat" SCALE_FACTOR = 1 FEATHER_AMOUNT = 11 FACE_POINTS = list (range (17,68)) MOUTH_POINTS = list (range (48,61)) RIGHT_BROW_POINTS = list (range (17,22)) LEFT_BROW_POINTS = list (range (22,27) RIGHT_EYE_POINTS = list (range (36)) 42) LEFT_EYE_POINTS = list (range (42,48)) NOSE_POINTS = list (range (27,35)) JAW_POINTS = list (range (0,17)) # Points used to line up the images. ALIGN_POINTS = (LEFT_BROW_POINTS + RIGHT_EYE_POINTS + LEFT_EYE_POINTS + RIGHT_BROW_POINTS + NOSE_POINTS + MOUTH_POINTS) # Points from the second image to overlay on the first. The convex hull of each # element will be overlaid. OVERLAY_POINTS = [LEFT_EYE_POINTS + RIGHT_EYE_POINTS + LEFT_BROW_POINTS + RIGHT_BROW_POINTS, NOSE_POINTS + MOUTH_POINTS,] # Amount of blur to use during colour correction, as a fraction of the # pupillary distance. COLOUR_CORRECT_BLUR_FRAC = 0.6 detector = dlib.get_frontal_face_detector () predictor = dlib.shape_predictor (PREDICTOR_PATH) class TooManyFaces (Exception): pass class NoFaces (Exception): pass def get_landmarks (im): rects = detector (im, 1) if len (rects) > 1: raise TooManyFaces if len (rects) = 0: raise NoFaces return numpy.matrix ([p.x) .y] for p in predictor (im, rects [0]). Parts ()] def annotate_landmarks (im, landmarks): im = im.copy () for idx, point in enumerate (landmarks): pos = (point [0,0], point [0,1]) cv2.putText (im, str (idx), pos, fontFace=cv2.FONT_HERSHEY_SCRIPT_SIMPLEX, fontScale=0.4 Color= (0,0,255) cv2.circle (im, pos, 3, color= (0255,255)) return im def draw_convex_hull (im, points, color): points = cv2.convexHull (points) cv2.fillConvexPoly (im, points, color=color) def get_face_mask (im, landmarks): im = numpy.zeros (im.shape [: 2]) Dtype=numpy.float64) for group in OVERLAY_POINTS: draw_convex_hull (im, landmarks [group], color=1) im = numpy.array ([im, im, im]). Transpose ((1,2,0) im = (cv2.GaussianBlur (im, (FEATHER_AMOUNT, FEATHER_AMOUNT), 0) > 0) * 1.0im = cv2.GaussianBlur (im) (FEATHER_AMOUNT, FEATHER_AMOUNT), 0) return im def transformation_from_points (points1, points2): "Return an affine transformation [scurr | T] such that: sum | | scurvy is minimized p1Magi + T-p2Magi | | ^ 2 Magi. "# Solve the procrustes problem by subtracting centroids, scaling by the # standard deviation, and then using the SVD to calculate the rotation. See # the following for more details: # https://en.wikipedia.org/wiki/Orthogonal_Procrustes_problem points1 = points1.astype (numpy.float64) points2 = points2.astype (numpy.float64) C1 = numpy.mean (points1, axis=0) c2 = numpy.mean (points2, axis=0) points1-= C1 points2-= c2 S1 = numpy.std (points1) S2 = numpy.std (points2) points1 / = S1 points2 / = S2 U S, Vt = numpy.linalg.svd (points1.T * points2) # The R we seek is in fact the transpose of the one given by U * Vt. This # is because the above formulation assumes the matrix goes on the right # (with row vectors) where as our solution requires the matrix to be on the # left (with column vectors) R = (U * Vt) .T return numpy.vstack ([numpy.hstack (S2 / S1) * R, c2.T-(S2 / S1) * R * c1.T)), numpy.matrix ([0.,0.1,1.])) Def read_im_and_landmarks (fname): im = cv2.imread (fname, cv2.IMREAD_COLOR) im = cv2.resize (im, (im.shape [1] * SCALE_FACTOR, im.shape [0] * SCALE_FACTOR)) s = get_landmarks (im) return im, s def warp_im (im, M, dshape): output_im = numpy.zeros (dshape, dtype=im.dtype) cv2.warpAffine (im M [: 2], (dshape [1], dshape [0]), dst=output_im, borderMode=cv2.BORDER_TRANSPARENT, flags=cv2.WARP_INVERSE_MAP) return output_im def correct_colours (im1, im2) Landmarks1): blur_amount = COLOUR_CORRECT_BLUR_FRAC * numpy.linalg.norm (numpy.mean (Landmarks1 [left _ EYE_POINTS], axis=0)-numpy.mean (landmarks1 [right _ EYE_POINTS], axis=0)) blur_amount = int (blur_amount) if blur_amount% 2 = 0: blur_amount + = 1 im1_blur = cv2.GaussianBlur (im1) (blur_amount, blur_amount), 0) im2_blur = cv2.GaussianBlur (im2, (blur_amount, blur_amount), 0) # Avoid divide-by-zero errors Im2_blur + = 128* (im2_blur

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.